Incremental clustering method and system, distributed incremental clustering method and system

Abstract

The present invention discloses an incremental clustering method and system, and also discloses a distributed incremental clustering method and system, which relate to the field of data processing, wherein the incremental clustering method comprises the following steps: acquiring data to be clustered and clustered data; performing incremental clustering on the clustered data based on the data to be clustered to obtain a plurality of first clustering clusters, each of which contains at least one target feature; detecting abnormal features in each of the first clustering clusters, and updating the corresponding first clustering clusters based on the obtained abnormal detection results to obtain clustering results; in the incremental clustering process, there is a clustering cluster that is contaminated by abnormal features, so that the clustering cluster contains a large number of features that do not belong to the clustering cluster. The present invention detects abnormal features in each of the first clustering clusters, and updates the corresponding first clustering clusters based on the obtained abnormal detection results to ensure the accuracy of the obtained clustering results.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to an incremental clustering technology. Background Art

[0002] The commonly used clustering method nowadays is full clustering, that is, each time clustering, all the features participating in the clustering are used as the features to be clustered, the similarity between the features to be clustered is recalculated, and the features to be clustered are divided into several clusters based on the obtained similarity; this scheme has high accuracy, but cannot use the results of the previous clustering. When the number of features participating in the clustering is large, the amount of calculation is large and the clustering speed is slow;

[0003] To address the low clustering efficiency of full clustering, an incremental clustering method is proposed today. That is, when new features are added, the clusters obtained from the previous clustering are updated based on the newly added features to improve clustering efficiency. However, the accuracy of incremental clustering is low. Summary of the invention

[0004] In view of the shortcoming of low clustering accuracy of existing incremental clustering technologies, the present invention provides an incremental clustering technology for detecting abnormal features and optimizing clustering results according to the obtained abnormality detection results to improve the accuracy of clustering results.

[0005] In order to solve the above technical problems, the present invention is solved by the following technical solutions:

[0006] An incremental clustering method comprises the following steps:

[0007] Obtain the data to be clustered and the clustered data;

[0008] Performing incremental clustering on the clustered data based on the data to be clustered to obtain a plurality of first clustering clusters;

[0009] Detect abnormal features in each first cluster, and update the corresponding first cluster based on the obtained abnormal detection result to obtain a clustering result, wherein the updating includes removing the abnormal features in the first cluster and splitting the first cluster.

[0010] The abnormal feature in the present invention is a feature that is similar to multiple types of features. Since the abnormal feature is similar to features of different types, in the clustering process, features similar to the abnormal feature will be continuously summarized into the same clustering cluster, resulting in an avalanche of clustering errors, which seriously affects the accuracy of clustering.

[0011] The existing technology solves the above problems as follows:

[0012] Limit the total number of features in a cluster. When there are too many features in a cluster, avoid adding normal features that are similar to abnormal features to the cluster where the abnormal features are located to avoid the continued accumulation of clustering errors.

[0013] Raise the similarity threshold. When the similarity of two features exceeds the preset similarity threshold, the two features are judged to be similar. Raising the similarity threshold can reduce the number of normal features that are similar to abnormal features. However, similar features may be divided into different clusters, which will also affect the accuracy of clustering. For example, the facial features of the same person may be incorrectly clustered into several clusters. When face recognition is performed based on the clustering results, only some clusters can be recalled based on the face image, which affects the final face recognition result.

[0014] The present invention identifies abnormal features in each first clustering cluster, and updates the corresponding first clustering cluster based on the obtained abnormal features to ensure the accuracy of the obtained clustering results.

[0015] As an implementable method, the step of detecting abnormal features in each first cluster is:

[0016] Each first cluster contains at least one target feature, that is, all the features participating in this clustering are used as target features, and the target features as core features in the first cluster are detected to obtain the first features;

[0017] Extracting neighborhood features of each first feature, and counting the number of neighborhood features and the number of neighborhood core features corresponding to the first feature, wherein the neighborhood feature refers to a target feature similar to the first feature, and the number of neighborhood core features refers to the number of neighborhood features belonging to the core feature;

[0018] That is, traverse the neighborhood features of the first feature, take the neighborhood features belonging to the core feature as the neighborhood core features of the first feature, count the number of neighborhood features corresponding to the first feature to obtain the corresponding neighborhood feature number, and count the number of neighborhood core features corresponding to the first feature to obtain the corresponding neighborhood core feature number;

[0019] Based on the neighborhood features, the number of neighborhood features and the number of neighborhood core features of the first feature, it is determined whether the first feature is an abnormal feature, and a corresponding abnormality detection result is obtained.

[0020] As an implementable method, the specific steps of judging whether the first feature is an abnormal feature based on the neighborhood feature, the neighborhood feature number and the neighborhood core feature number of the first feature are:

[0021] Based on the neighborhood feature number and the neighborhood core feature number, performing abnormality judgment on the corresponding first feature according to a preset judgment rule, and when the first feature is judged to be abnormal, using the first feature as the second feature;

[0022] Those skilled in the art can set the judgment rules according to actual needs, and the present invention does not specifically limit them. For example, in the present invention, when N>aN c When the corresponding first feature is judged to be abnormal, N represents the number of neighborhood features of the first feature, N c represents the number of neighborhood core features of the first feature, a is the weight coefficient, and a>1.

[0023] Calculate the discrete value corresponding to each second feature to obtain the discrete value to be determined, wherein the discrete value is used to indicate the degree of discreteness between each neighborhood feature;

[0024] In the present invention, the variance is calculated based on all neighborhood features of the second feature, and the obtained variance is used as the discrete value of the second feature.

[0025] When the discrete value to be determined is greater than the corresponding discrete threshold, the second feature is determined to be an abnormal feature, wherein the discrete value is used to indicate the degree of discreteness between each neighborhood feature.

[0026] Those skilled in the art can set the discrete threshold according to actual needs.

[0027] The present invention combines the distribution of the neighborhood core features corresponding to the first feature and the discreteness of the neighborhood features corresponding to the first feature to jointly determine whether the first feature is an abnormal feature, which can effectively avoid misjudging normal features as abnormal features.

[0028] Furthermore, the discrete threshold is obtained by:

[0029] When it is determined that the first feature is normal, taking the first feature as the third feature;

[0030] Summarize the neighborhood features corresponding to all third features in the first cluster, and calculate the discrete value corresponding to the first cluster based on the obtained summary result to obtain the basic discrete value;

[0031] The discrete threshold is determined based on the base discrete value.

[0032] Those skilled in the art can set the configuration rules of discrete thresholds according to actual needs, and the present invention does not limit them in detail. In the present invention, g = bO, where O g represents the discrete threshold, O represents the corresponding basic discrete value, b is the weight coefficient and b≥1.

[0033] Due to the imbalance in the number of various features, when the discrete threshold is set to a fixed value, the discrete threshold cannot accurately indicate the normal discrete degree corresponding to each first cluster. Therefore, the present invention characterizes the basic discrete value corresponding to the first cluster by the discrete degree of the neighborhood features of all third features, and then determines the discrete threshold according to the obtained basic discrete value, thereby further improving the recognition accuracy of abnormal features.

[0034] As an implementable method:

[0035] The data to be clustered includes several features to be clustered;

[0036] The clustered data includes a plurality of clusters, the clusters are recorded as second clusters, each second cluster includes at least one clustered feature, and the clustered data also includes neighborhood features of each clustered feature;

[0037] Both the features to be clustered and the features that have been clustered are taken as target features, and the neighborhood features of each target feature are determined based on the similarity between the target features;

[0038] Incremental clustering is performed based on the second clusters and neighborhood features of each target feature to obtain corresponding first clusters.

[0039] Further:

[0040] Calculate the similarity between the features to be clustered and each target feature respectively;

[0041] When the similarity is greater than a preset similarity threshold, the feature to be clustered is determined to be similar to the corresponding target feature, the target feature similar to the feature to be clustered is used as the neighborhood feature of the feature to be clustered, and the feature to be clustered is used as the neighborhood feature of the target feature similar to it.

[0042] Furthermore, before performing incremental clustering based on the second cluster and the neighborhood features of each target feature, a judgment step is also performed, specifically:

[0043] It is determined whether each feature to be clustered is a core feature. When there is a feature to be clustered that is a core feature, incremental clustering is performed based on the second cluster clusters and the neighborhood features of each target feature.

[0044] When there is no feature to be clustered as a core feature, incremental clustering is not performed, and the feature to be clustered can participate in clustering as the feature to be clustered in the next clustering.

[0045] When it is determined not to perform incremental clustering, technicians in this field can retain the updates of the data to be clustered and the clustered data in this clustering according to actual needs, that is, update the neighborhood features of the clustered data, and retain the neighborhood features of each feature to be clustered, and use the feature to be clustered as the first feature to be clustered in the next incremental clustering. When performing the next clustering, there is no need to repeatedly calculate the similarity between the feature to be clustered and the target feature, and only the similarity of the newly added features needs to be calculated.

[0046] Further:

[0047] The feature to be clustered is the first feature to be clustered or the second feature to be clustered. The first feature to be clustered is the feature to be clustered that participated in the last incremental clustering and has a neighborhood feature set. The second feature to be clustered is a newly added feature and has no neighborhood feature set.

[0048] Calculating the similarity between the second feature to be clustered and each target feature to obtain a neighborhood feature corresponding to each target feature;

[0049] Based on the obtained similarity, a similarity judgment is made between the second feature to be clustered and each first feature to be clustered, a similarity judgment is made between the second feature to be clustered and other second features to be clustered, and a similarity judgment is made between the second feature to be clustered and the clustered features; since two similar features are each other's neighborhood features, the neighborhood feature corresponding to each target feature can be obtained by only calculating the similarity between the second feature to be clustered and each target feature.

[0050] Further:

[0051] The features to be clustered or the already clustered features to be judged as core features are used as features to be detected;

[0052] The number of neighborhood features of the feature to be detected is counted, and when the number of neighborhood features exceeds a preset quantity threshold, the feature to be detected is determined to be a core feature.

[0053] Those skilled in the art may set the quantity threshold by themselves, and the present invention does not make any specific limitation thereto.

[0054] The present invention also proposes an incremental clustering system, comprising:

[0055] A data acquisition module, used to acquire data to be clustered and clustered data;

[0056] An incremental clustering module, configured to perform incremental clustering on the clustered data based on the data to be clustered to obtain a plurality of first clustering clusters, each of which contains at least one target feature;

[0057] An abnormality processing module is used to detect abnormal features in each first cluster, and update the corresponding first cluster based on the obtained abnormality detection result to obtain a clustering result;

[0058] The exception handling module includes:

[0059] A core detection unit, configured to detect a target feature as a core feature in the first cluster to obtain a first feature;

[0060] A data statistics unit, used to extract neighborhood features of each first feature, and count the number of neighborhood features and the number of neighborhood core features corresponding to the first feature, wherein the neighborhood feature refers to a target feature similar to the first feature, and the number of neighborhood core features refers to the number of neighborhood features belonging to the core feature;

[0061] The anomaly detection unit is used to determine whether the first feature is an abnormal feature based on the neighborhood feature, the neighborhood feature number and the neighborhood core feature number of the first feature, and obtain a corresponding anomaly detection result.

[0062] The present invention also proposes a distributed incremental clustering method, comprising the following steps:

[0063] Acquire data to be clustered and clustered data, wherein the data to be clustered includes a plurality of features to be clustered, the clustered data includes a plurality of second clustering clusters, each second clustering cluster includes at least one clustered feature, and the clustered data also includes neighborhood features of each clustered feature, and use the features to be clustered and the clustered features as target features;

[0064] The features to be clustered are grouped to obtain several feature groups, and the neighborhood features of the features to be clustered in each feature group are distributedly calculated to obtain the calculation results corresponding to each feature group, wherein the neighborhood features of the features to be clustered are target features similar to the features to be clustered;

[0065] Performing incremental clustering based on the calculation results, the clustering clusters of the features to be clustered and the features that have been clustered, and the neighborhood features, to obtain a plurality of first clustering clusters;

[0066] The abnormal features in each first cluster are detected, and the corresponding first cluster is updated based on the obtained abnormal detection result to obtain a clustering result.

[0067] Currently, clustering can only be performed on a single machine and cannot be performed across multiple devices. The present invention distributes computing tasks to each node participating in the computing, thereby achieving cross-terminal clustering, which can greatly improve clustering performance.

[0068] The present invention also proposes a distributed incremental clustering system, including a database, a control node and a plurality of computing nodes:

[0069] The database is used to store data to be clustered and clustered data, the data to be clustered includes a plurality of features to be clustered, the clustered data includes a plurality of second clustering clusters, each second clustering cluster includes at least one clustered feature, and the clustered data also includes neighborhood features of each clustered feature;

[0070] The control node includes a processing module, a calculation module, a clustering module and an update module;

[0071] The processing module is used to obtain the data to be clustered and the clustered data from the database, and use the obtained features to be clustered and the clustered features as target features; and is also used to group the features to be clustered to obtain a plurality of feature groups, and distribute the obtained feature groups to the computing module and each computing node;

[0072] The calculation module is used to receive the feature group sent by the processing module, calculate the neighborhood features of each feature to be clustered in the feature group, obtain corresponding calculation results, and send the calculation results to the clustering module, wherein the neighborhood features of the feature to be clustered are target features similar to the feature to be clustered;

[0073] The clustering module is used to perform incremental clustering based on the calculation results, the clustering clusters of the features to be clustered and the features that have been clustered, and the neighborhood features to obtain a plurality of first clustering clusters;

[0074] The updating module is used to detect abnormal features in each first clustering cluster, and update the corresponding first clustering cluster based on the obtained abnormal detection result to obtain a clustering result;

[0075] The computing node is used to receive the feature group sent by the processing module, calculate the neighborhood features of each feature to be clustered in the feature group, obtain corresponding calculation results, and send the calculation results to the clustering module.

[0076] The present invention has significant technical effects due to the adoption of the above technical solution:

[0077] The present invention adopts an incremental clustering algorithm, which greatly reduces the amount of calculation by utilizing historical clustering results. When the features to be clustered are newly added, real-time online clustering can be realized. In addition, the present invention identifies the abnormal features in each first clustering cluster, updates the corresponding first clustering cluster based on the obtained abnormal features, and can effectively improve the accuracy of the clustering results while ensuring the clustering efficiency.

[0078] The present invention distributes the calculation of the neighborhood features of the features to be clustered in each feature group, thereby realizing cross-terminal clustering, which can greatly improve the clustering performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0080] Figure 1 It is a schematic diagram of the workflow of an incremental clustering method of the present invention;

[0081] Figure 2 It is a schematic diagram of module connection of an incremental clustering system of the present invention;

[0082] Figure 3 yes Figure 2 Module connection diagram of the exception handling module 30;

[0083] Figure 4 yes Figure 2 Schematic diagram of module connections of the mid-increment clustering module 20;

[0084] Figure 5 yes Figure 2 Another schematic diagram of module connection of the incremental clustering module 20;

[0085] Figure 6 It is a schematic diagram of module connection of a distributed incremental clustering system of the present invention. DETAILED DESCRIPTION

[0086] The present invention is further described in detail below in conjunction with embodiments. The following embodiments are for explanation of the present invention but the present invention is not limited to the following embodiments.

[0087] Embodiment 1: An incremental clustering method for incrementally clustering facial features, such as Figure 1 As shown, the following steps are included:

[0088] S100, obtaining data to be clustered and clustered data;

[0089] S110, data to be clustered:

[0090] The data to be clustered includes several features to be clustered;

[0091] In this embodiment, after monitoring that new facial features are added to the database, incremental clustering is triggered to achieve online real-time clustering. In the subsequent face recognition process, the newly added facial features can be effectively used. In the incremental clustering process, the newly added facial features are used as non-clustered features;

[0092] Extract features to be clustered from unclustered features;

[0093] The facial features corresponding to the facial images due to the obstruction of the face or unclear light do not participate in the clustering. In this embodiment, such features that do not participate in the clustering are eliminated from the non-clustered features to obtain the corresponding features to be clustered. This is the prior art and will not be introduced in detail in this embodiment.

[0094] S120, clustered data:

[0095] The clustered data includes a plurality of second clusters, each of which includes at least one clustered feature, and each clustered feature also has a corresponding neighborhood feature set, the neighborhood feature set is used to place neighborhood features of the clustered features, the neighborhood features of the clustered features are facial features similar to the clustered features, and the neighborhood features described in this step belong to the clustered features;

[0096] S200, performing incremental clustering on the clustered data based on the data to be clustered to obtain a plurality of first clusters;

[0097] Each first cluster contains at least one target feature;

[0098] The target feature is the feature to be clustered or the clustered feature obtained in step S100 , that is, the target feature is the feature that will participate in clustering.

[0099] The specific steps are as follows:

[0100] S210, determining a neighborhood feature of each target feature based on the similarity between the target features;

[0101] Two target features whose similarity is greater than a preset similarity threshold are neighboring features of each other;

[0102] Since the clustered data obtained in step S100 includes the neighborhood features of each clustered feature, it is only necessary to calculate the similarity between the feature to be clustered and each target feature to determine other features to be clustered similar to the feature to be clustered and clustered features similar to the feature to be clustered.

[0103] Methods for obtaining neighborhood features of each target feature include:

[0104] S211, similarity calculation:

[0105] Calculate the similarity between the features to be clustered and each target feature respectively;

[0106] Those skilled in the art can calculate the similarity between the feature to be clustered and each target feature based on the existing published similarity calculation formula, such as using the cosine similarity or Euclidean distance between the feature to be clustered and the corresponding target feature as the similarity between the feature to be clustered and the target feature.

[0107] S212, neighborhood feature judgment:

[0108] When the similarity is greater than a preset similarity threshold, the feature to be clustered is determined to be similar to the corresponding target feature, the target feature similar to the feature to be clustered is used as the neighborhood feature of the feature to be clustered, and the feature to be clustered is used as the neighborhood feature of the target feature similar to it.

[0109] Construct the corresponding neighborhood feature set for each feature to be clustered in advance;

[0110] When the similarity between the feature to be clustered and a target feature is greater than a preset similarity threshold, the corresponding target feature is added to the neighborhood feature set of the feature to be clustered, and the feature to be clustered is added to the neighborhood feature set of the corresponding target feature;

[0111] After all the features to be clustered have completed the above similarity calculation and neighborhood feature judgment steps, the neighborhood feature corresponding to each target feature can be determined based on the neighborhood feature set corresponding to each target feature.

[0112] S213, determine the core features;

[0113] In this embodiment, after obtaining the neighborhood features of each target feature, the core features are also determined based on the neighborhood features. The specific determination method is:

[0114] Each target feature is used as a feature to be detected;

[0115] The number of neighborhood features of the feature to be detected is counted. When the number of neighborhood features exceeds a preset quantity threshold, the feature to be detected is determined to be a core feature, and a first core feature set and a second core feature set are obtained. The first core feature set includes several features to be clustered as core features, and the second core feature set includes several clustered features as core features.

[0116] S220, performing incremental clustering based on the second clusters and the neighborhood features of each target feature to obtain corresponding first clusters, the specific steps are as follows:

[0117] S221, establishing a union-find set based on the second cluster and the neighborhood feature set of each target feature to obtain a first union-find set;

[0118] In this embodiment, for each clustered feature, a feature is randomly selected from the corresponding second cluster and the neighborhood feature set as the root node, all face features in the second cluster and the neighborhood feature set are associated with the root node, a union-find set is established, and a second union-find set is obtained;

[0119] The features to be clustered are extracted from the first core feature set, and the second union-find set is updated based on the obtained features to be clustered and the neighborhood features of the obtained features to be clustered to obtain the first union-find set.

[0120] S222, traversing the first union-find set, clustering target features with the same root node into the same cluster, and obtaining a plurality of first clusters, each of which contains at least one target feature.

[0121] S300, detecting abnormal features in each first cluster, and updating the corresponding first cluster based on the obtained abnormal detection result to obtain a clustering result;

[0122] Since the method for performing anomaly detection on each first cluster is the same, this embodiment will introduce the steps of performing anomaly detection on one first cluster in detail, which are:

[0123] S310, detecting a target feature as a core feature in the first cluster to obtain a first feature;

[0124] Based on the first core feature set and the second core feature set, core features in the first cluster are extracted, and the obtained core features are used as first features.

[0125] S320, extracting neighborhood features of each first feature, and counting the number of neighborhood features and the number of neighborhood core features corresponding to the first feature;

[0126] A neighborhood feature set corresponding to the first feature is obtained, the total number of features in the neighborhood feature set is counted to obtain a corresponding neighborhood feature number, and the number of core features in the neighborhood feature set is counted to obtain a corresponding neighborhood core feature number.

[0127] S330: judging whether the first feature is an abnormal feature based on the neighborhood feature, the number of neighborhood features, and the number of neighborhood core features of the first feature, and obtaining a corresponding abnormality detection result, the specific steps are as follows:

[0128] S331, performing abnormality determination on each first feature;

[0129] Those skilled in the art can set the judgment rules according to actual needs. In this embodiment, the judgment rules are:

[0130] When N>2N c When the corresponding first feature is judged to be abnormal, N represents the number of neighborhood features of the first feature, N c The number of core features of the neighborhood of the first feature is represented, and the first feature is used as the second feature;

[0131] When N≤2N c When the corresponding first feature is determined to be normal, the first feature is used as the third feature;

[0132] S332, discrete value calculation;

[0133] Calculate the discrete value corresponding to each second feature to obtain the discrete value to be determined, wherein the discrete value is used to indicate the degree of discreteness between each neighborhood feature;

[0134] That is, a neighborhood feature set of the second feature is obtained, variance is calculated based on all neighborhood features in the neighborhood feature set, and the obtained variance is used as the discrete value of the second feature.

[0135] S333, abnormal verification;

[0136] When the discrete value to be determined is greater than the corresponding discrete threshold, the corresponding second feature is determined to be an abnormal feature;

[0137] The calculation formula of the discrete threshold in this embodiment is: g =1.3O, where O g represents the discrete threshold, and O represents the corresponding basic discrete value;

[0138] In this embodiment, the neighborhood features corresponding to all the third features in the first cluster are summarized, the variance is calculated based on all the obtained neighborhood features, and the obtained variance is used as the basic discrete value O.

[0139] S334, optimizing clustering results;

[0140] Eliminate the nodes corresponding to the obtained abnormal features from the first union-find set, traverse the first union-find set from which all abnormal nodes are eliminated, and obtain corresponding clustering results, wherein the clustering results include a plurality of third clustering clusters;

[0141] The clustering results can be used to recall similar faces during the face recognition process. For example, a face to be recognized is obtained, and similar faces corresponding to the face to be recognized are obtained through vector search to obtain a first face, and third clusters corresponding to each first face are extracted. Based on the obtained third clusters, all faces corresponding to each third cluster are obtained to obtain a second face; the obtained second face is deduplicated, and the obtained second face is output as a similar face of the face to be recognized; that is, only any face photo under the obtained cluster needs to be recognized to recall the entire cluster. Therefore, the improvement of the accuracy of the clustering results in this embodiment can also improve the recall rate in the face recognition process.

[0142] In this embodiment, the clustered data is updated based on the obtained clustering results, and the neighborhood feature set of each third clustering cluster and each facial feature in each clustering cluster is saved. The third clustering cluster is used as the second clustering cluster for the next incremental clustering, and the facial features are used as the clustered features for the next incremental clustering.

[0143] When clustering facial features, there are facial features corresponding to relatively ambiguous facial images, i.e., abnormal features; such abnormal features will be similar to facial features of different people, so that in the clustering process, the same cluster contains facial features of multiple people, resulting in inaccurate clustering results;

[0144] As more facial features are added, more and more facial features that do not belong to the cluster will be clustered into the cluster, which will seriously affect the clustering accuracy and the accuracy of subsequent face recall based on the clustering results.

[0145] This embodiment can effectively identify and eliminate abnormal features by comparing the discreteness of the neighborhood features of the second feature (determined to be abnormal) and the third feature (determined to be normal) under the first cluster cluster, avoid erroneous clustering, and improve clustering accuracy while ensuring clustering efficiency.

[0146] Embodiment 2, on the basis of Embodiment 1, adds an execution judgment step, and the rest is the same as Embodiment 1, that is, before performing incremental clustering based on the second clustering cluster and the neighborhood features of each target feature in step S220, adds an execution judgment step, and the execution judgment step is specifically as follows:

[0147] When the first core feature set obtained in step S213 is empty, it is determined to end clustering, otherwise it is determined to execute step S220, that is, when there are features to be clustered as core features, incremental clustering is performed based on the second cluster clusters and the neighborhood features of each target feature.

[0148] Those skilled in the art can configure the data processing method according to actual needs when determining to end clustering;

[0149] Method 1: No processing is performed, that is, the database storing the data to be clustered and the clustered data is not updated.

[0150] Method 2: retain the neighborhood features of each target feature obtained by this clustering; that is, add the neighborhood feature set of the feature to be clustered to the database, and update the neighborhood feature set of each clustered feature in the database.

[0151] When the above method 2 is adopted, the obtained data to be clustered includes a number of features to be clustered, some of which have corresponding neighborhood features, and the neighborhood features of the features to be clustered and the features already clustered may belong to the features already clustered or to the features to be clustered;

[0152] Since this method retains the similarity calculation result in the previous clustering, in the current clustering, only the newly added features to be clustered need to be calculated for similarity. In this embodiment, the features to be clustered are the first features to be clustered or the second features to be clustered. The first features to be clustered are the features to be clustered that participated in the previous incremental clustering and have the corresponding neighborhood feature set. The second features to be clustered are newly added features.

[0153] In the similarity calculation of step S211, it is only necessary to calculate the similarity between the second feature to be clustered and each target feature, that is, only the similarity calculation is performed on the newly added features. Based on the obtained similarity, the second feature to be clustered and each first feature to be clustered can be judged to be similar, the second feature to be clustered and other second features to be clustered can be judged to be similar, and the second feature to be clustered and the clustered features can be judged to be similar.

[0154] Embodiment 3, a distributed incremental clustering method, the similarity calculation method in step S211 of embodiment 1 or embodiment 2 is changed to distributed calculation, and the rest is the same as embodiment 1 or embodiment 2. The method of distributed calculation of similarity specifically includes the following steps:

[0155] The features to be clustered are grouped to obtain several feature groups, and the neighborhood features of the features to be clustered in each feature group are distributedly calculated to obtain the calculation results corresponding to each feature group;

[0156] When the features to be clustered are divided into the first features to be clustered (with corresponding neighborhood feature sets) and the second features to be clustered (without corresponding neighborhood feature sets), the second features to be clustered are grouped, otherwise the features to be clustered are grouped directly;

[0157] The number of feature groups is determined by the number of nodes participating in the calculation. A feature group is issued to each node participating in the calculation. Each node performs similarity calculation on the features to be clustered in the received feature group to obtain the corresponding calculation results.

[0158] Those skilled in the art can set the calculation results according to actual needs. For example, the calculation results can be set as each feature to be clustered and its neighborhood feature set, and can also include the features to be clustered belonging to the core feature and its neighborhood feature set, that is, after obtaining the neighborhood features of the corresponding features to be clustered, it is determined whether the features to be clustered are core features based on the obtained neighborhood features, and the neighborhood feature set of the features to be clustered and the core feature judgment result are used as the base number to calculate the result. In this embodiment, the calculation results include the features to be clustered of the core features, and the calculation results reported by each node are summarized and merged to obtain the corresponding first core feature set;

[0159] Since two similar features are each other's neighborhood features, after obtaining the neighborhood feature set of the feature to be clustered, the feature to be clustered can be added to the neighborhood feature set of its neighborhood features based on the neighborhood feature set.

[0160] Referring to step S220 in Example 1 or Example 2, incremental clustering is performed based on the calculation results, the clusters of the features to be clustered and the features that have been clustered, and the neighborhood features to obtain a plurality of first clusters;

[0161] Referring to step S300 in embodiment 1 or embodiment 2, abnormal features in each first cluster are detected, and the corresponding first cluster is updated based on the obtained abnormal detection result to obtain a clustering result.

[0162] Embodiment 4: An incremental clustering system, such as Figure 2 As shown, including:

[0163] A data acquisition module 10 is used to acquire data to be clustered and clustered data;

[0164] An incremental clustering module 20, configured to perform incremental clustering on the clustered data based on the data to be clustered, to obtain a plurality of first clusters, each of which contains at least one target feature;

[0165] The abnormality processing module 30 is used to detect abnormal features in each first cluster, and update the corresponding first cluster based on the obtained abnormality detection result to obtain a clustering result;

[0166] Reference Figure 3 , further, the exception handling module 30 includes:

[0167] A core detection unit 31, configured to detect a target feature as a core feature in the first cluster to obtain a first feature;

[0168] A data statistics unit 32 is used to extract neighborhood features of each first feature, and count the number of neighborhood features and the number of neighborhood core features corresponding to the first feature, wherein the neighborhood feature refers to a target feature similar to the first feature, and the number of neighborhood core features refers to the number of neighborhood features belonging to the core feature;

[0169] The anomaly detection unit 33 is used to determine whether the first feature is an abnormal feature based on the neighborhood feature, the neighborhood feature number and the neighborhood core feature number of the first feature, and obtain a corresponding anomaly detection result.

[0170] Further, the abnormality detection unit 33 includes a judgment unit and a verification unit;

[0171] The judgment unit is used to judge the abnormality of the corresponding first feature according to a preset judgment rule based on the neighborhood feature number and the neighborhood core feature number, and when the first feature is judged to be abnormal, use the first feature as the second feature;

[0172] The verification unit is used to calculate the discrete value corresponding to each second feature to obtain the discrete value to be determined, wherein the discrete value is used to indicate the degree of discreteness between each neighborhood feature; and also determine whether the second feature is an abnormal feature based on the discrete value to be determined.

[0173] Reference Figure 4 , further, the incremental clustering module 20 includes:

[0174] A neighborhood determination unit 21, configured to determine a neighborhood feature of each target feature based on the similarity between target features;

[0175] The incremental clustering unit 22 is used to perform incremental clustering based on the second clusters and the neighborhood features of each target feature to obtain corresponding first clusters.

[0176] Further, the neighborhood determination unit 21 includes:

[0177] A similarity calculation unit, used to calculate the similarity between the feature to be clustered and each target feature;

[0178] The neighborhood judgment unit determines that the feature to be clustered is similar to the corresponding target feature when the similarity is greater than a preset similarity threshold, takes the target feature similar to the feature to be clustered as the neighborhood feature of the feature to be clustered, and takes the feature to be clustered as the neighborhood feature of the target feature similar to it.

[0179] This embodiment is a device embodiment corresponding to Embodiment 1. Since it is basically similar to Embodiment 1, the description is relatively simple, and the relevant parts can refer to the partial description of Embodiment 1.

[0180] Example 5, reference Figure 5 , an execution judgment unit 23 is added to the incremental clustering module 20 of Example 4, and the rest is the same as Example 4;

[0181] The execution judgment unit 23 is used to judge whether each feature to be clustered is a core feature. When there is a feature to be clustered that is a core feature, the incremental clustering unit 22 is instructed to perform incremental clustering based on the second cluster cluster and the neighborhood features of each target feature.

[0182] This embodiment is a device embodiment corresponding to Embodiment 2. Since it is basically similar to Embodiment 2, the description is relatively simple, and the relevant parts can be referred to the partial description of Embodiment 2.

[0183] Embodiment 6: A distributed incremental clustering system, comprising a database and a plurality of nodes;

[0184] The database is used to store data to be clustered and clustered data, the data to be clustered includes a plurality of features to be clustered, the clustered data includes a plurality of second clustering clusters, each second clustering cluster includes at least one clustered feature, and the clustered data also includes neighborhood features of each clustered feature;

[0185] Reference Figure 6 , the nodes are divided into a control node and a plurality of computing nodes 200;

[0186] In this embodiment, the distributed incremental clustering system has 5 nodes involved in the calculation. During initialization, the 5 nodes elect a control node and use the remaining nodes as the calculation nodes 200, thereby obtaining a control node and four calculation nodes 200;

[0187] The control node executes the incremental clustering method disclosed in Example 1 or Example 2, and when performing similarity calculation, each computing node 200 is also allowed to participate in the calculation.

[0188] like Figure 6 As shown, the control node includes a processing module 110, a calculation module 120, a clustering module 130 and an update module 140;

[0189] The processing module 110 is used to obtain the data to be clustered and the clustered data from the database, and use the obtained features to be clustered and the clustered features as target features; it is also used to group the features to be clustered to obtain a plurality of feature groups, and distribute the obtained feature groups to the computing module 120 and each computing node 200. In this embodiment, the features to be clustered are divided into 5 parts and distributed to 5 nodes;

[0190] The calculation module 120 is used to receive the feature group sent by the processing module 110, calculate the neighborhood features of each feature to be clustered in the feature group, obtain corresponding calculation results, and send the calculation results to the clustering module 130, wherein the neighborhood features of the feature to be clustered are target features similar to the feature to be clustered;

[0191] The clustering module 130 is used to perform incremental clustering based on the calculation results, the clustering clusters of the features to be clustered and the clustered features, and the neighborhood features to obtain a plurality of first clustering clusters. In this embodiment, the control node asynchronously collects and merges the calculation results returned by each computing node 200, and performs incremental clustering with reference to the method disclosed in Example 1 or Example 2 to obtain the corresponding first clustering clusters.

[0192] The updating module 140 is used to detect abnormal features in each first cluster, and update the corresponding first cluster based on the obtained abnormal detection result to obtain a clustering result;

[0193] In this embodiment, each computing node 200 is used to receive the feature group sent by the processing module 110, calculate the neighborhood features of each feature to be clustered in the feature group, obtain corresponding calculation results, and send the calculation results to the clustering module 130.

[0194] This embodiment is a device embodiment corresponding to Embodiment 3. Since it is basically similar to Embodiment 3, the description is relatively simple, and the relevant parts can be referred to the partial description of Embodiment 3.

[0195] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0196] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0197] The present invention is described with reference to the flowcharts and / or block diagrams of the method, terminal device (system), and computer program product according to the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0198] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0199] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0200] It should be noted that:

[0201] The "one embodiment" or "embodiment" mentioned in the specification means that the specific features, structures or characteristics described in conjunction with the embodiment are included in at least one embodiment of the present invention. Therefore, the phrases "one embodiment" or "embodiment" appearing in various places throughout the specification do not necessarily refer to the same embodiment.

[0202] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0203] In addition, it should be noted that the shapes and names of the parts and components of the specific embodiments described in this specification may be different. All equivalent or simple changes made according to the structure, features and principles described in the patent concept of the present invention are included in the protection scope of the patent of the present invention. Technicians in the technical field to which the present invention belongs can make various modifications or supplements to the specific embodiments described or replace them in a similar manner, as long as they do not deviate from the structure of the present invention or exceed the scope defined by the claims, they should all fall within the protection scope of the present invention.

Claims

1. An incremental clustering method, characterized in that: The following steps are involved: Obtaining face feature data to be clustered and face feature data that has been clustered; Performing incremental clustering on the clustered facial feature data based on the facial feature data to be clustered to obtain a plurality of first clusters; Detecting abnormal features in each first cluster, and updating the corresponding first cluster based on the obtained abnormal detection result to obtain a clustering result; Among them, the step of detecting abnormal features in each first cluster is: each first cluster contains at least one target feature, and the target feature as the core feature in the first cluster is detected to obtain the first feature; the neighborhood feature of each first feature is extracted, and the number of neighborhood features and the number of neighborhood core features corresponding to the first feature are counted, the neighborhood feature refers to the target feature similar to the first feature, and the number of neighborhood core features refers to the number of neighborhood features belonging to the core feature; based on the neighborhood feature, the number of neighborhood features and the number of neighborhood core features of the first feature, it is judged whether the first feature is an abnormal feature to obtain the corresponding abnormality detection result.

2. The incremental clustering method according to claim 1, characterized in that: The specific steps of judging whether the first feature is an abnormal feature based on the neighborhood feature, the neighborhood feature number and the neighborhood core feature number of the first feature are: Based on the neighborhood feature number and the neighborhood core feature number, performing abnormality judgment on the corresponding first feature according to a preset judgment rule, and when the first feature is judged to be abnormal, using the first feature as the second feature; Calculate the discrete value corresponding to each second feature to obtain the discrete value to be determined, wherein the discrete value is used to indicate the degree of discreteness between each neighborhood feature; When the discrete value to be determined is greater than the corresponding discrete threshold, the second feature is determined to be an abnormal feature.

3. The incremental clustering method according to claim 2, characterized in that: The method for obtaining the discrete threshold is: When it is determined that the first feature is normal, taking the first feature as the third feature; Summarize the neighborhood features corresponding to all third features in the first cluster, and calculate the discrete value corresponding to the first cluster based on the obtained summary result to obtain the basic discrete value; The discrete threshold is determined based on the base discrete value.

4. The incremental clustering method according to any one of claims 1 to 3, characterized in that: The face feature data to be clustered includes a number of face features to be clustered; The clustered facial feature data includes a plurality of second clusters, each of which includes at least one clustered facial feature, and the clustered facial feature data also includes neighborhood features of each clustered facial feature; The face features to be clustered and the face features that have been clustered are taken as target features, and the neighborhood features of each target feature are determined based on the similarity between the target features; Incremental clustering is performed based on the second clusters and neighborhood features of each target feature to obtain corresponding first clusters.

5. The incremental clustering method according to claim 4, characterized in that: Before performing incremental clustering based on the second cluster and the neighborhood features of each target feature, a judgment step is also performed, specifically: Determine whether the features to be clustered on each face are core features; When there are human face features to be clustered as core features, it is determined that clustering is to be performed, and incremental clustering is performed based on the second cluster clusters and neighborhood features of each target feature.

6. The incremental clustering method according to claim 5, characterized in that: The face feature to be clustered is the first feature to be clustered or the second feature to be clustered, and the second feature to be clustered is a newly added feature; When the execution judgment step of the last incremental clustering determines that clustering is not to be performed, the neighborhood features of the face features to be clustered corresponding to the last incremental clustering are saved, and the face features to be clustered corresponding to the last incremental clustering are used as the first features to be clustered of the current incremental clustering; The similarity between the second feature to be clustered and each target feature is calculated to obtain a neighborhood feature corresponding to each target feature.

7. An incremental clustering system, characterized in that: The incremental clustering system executes the incremental clustering method according to any one of claims 1 to 6, comprising: A data acquisition module, used to acquire the face feature data to be clustered and the face feature data that has been clustered; An incremental clustering module, configured to perform incremental clustering on the clustered facial feature data based on the facial feature data to be clustered, to obtain a plurality of first clusters; The abnormality processing module is used to detect abnormal features in each first clustering cluster, and update the corresponding first clustering cluster based on the obtained abnormality detection result to obtain a clustering result.

8. A distributed incremental clustering method, characterized in that: The distributed incremental clustering method performs distributed incremental clustering based on the incremental clustering method according to any one of claims 1 to 6, comprising the following steps: Acquire face feature data to be clustered and face feature data that has been clustered, wherein the face feature data to be clustered includes a plurality of face features to be clustered, the face feature data that has been clustered includes a plurality of second clustering clusters, each second clustering cluster includes at least one face feature that has been clustered, and the face feature data that has been clustered also includes neighborhood features of each face feature that has been clustered, and use the face features to be clustered and the face features that have been clustered as target features; The face features to be clustered are grouped to obtain a number of feature groups, and the neighborhood features of the face features to be clustered in each feature group are distributedly calculated to obtain the calculation results corresponding to each feature group, wherein the neighborhood features of the face features to be clustered are target features similar to the face features to be clustered; Performing incremental clustering based on the calculation results, the features to be clustered of each face, the clusters of the features that have been clustered of each face, and the neighborhood features to obtain a plurality of first clusters; The abnormal features in each first cluster are detected, and the corresponding first cluster is updated based on the obtained abnormal detection result to obtain a clustering result.

9. A distributed incremental clustering system, characterized in that: The distributed incremental clustering system executes the distributed incremental clustering method according to claim 8, comprising a database, a control node and a plurality of computing nodes: The database is used to store face feature data to be clustered and clustered face feature data, the face feature data to be clustered includes a plurality of face features to be clustered, the clustered face feature data includes a plurality of second clustering clusters, each second clustering cluster includes at least one clustered face feature, and the clustered face feature data also includes neighborhood features of each clustered face feature; The control node includes a processing module, a calculation module, a clustering module and an update module; The processing module is used to obtain the face feature data to be clustered and the face feature data already clustered from the database, and use the obtained face features to be clustered and the face features already clustered as target features; It is also used to group the features to be clustered on the face, obtain a number of feature groups, and distribute the obtained feature groups to the computing module and each computing node; The calculation module is used to receive the feature group sent by the processing module, calculate the neighborhood features of each face feature to be clustered in the feature group, obtain corresponding calculation results, and send the calculation results to the clustering module, wherein the neighborhood features of the face feature to be clustered are target features similar to the face feature to be clustered; The clustering module is used to perform incremental clustering based on the calculation results, the clustering clusters of the features to be clustered and the features of the clustered faces, and the neighborhood features, to obtain a plurality of first clustering clusters; The updating module is used to detect abnormal features in each first clustering cluster, and update the corresponding first clustering cluster based on the obtained abnormal detection result to obtain a clustering result; The computing node is used to receive the feature group sent by the processing module, calculate the neighborhood features of each face feature to be clustered in the feature group, obtain corresponding calculation results, and send the calculation results to the clustering module.

Images