An online face clustering method based on image search
By employing an online face clustering method based on image search, and utilizing convolutional neural networks to extract features and perform class center retrieval and correction, the accuracy and efficiency issues of online real-time face recognition and data labeling in existing technologies are resolved, achieving efficient data classification and labeling processing.
Patent Information
- Application Number
- CN202010955736.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-09-11
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2040-09-11
AI Technical Summary
Existing technologies cannot provide accurate and efficient online real-time face recognition and data labeling methods. In particular, the computational load is huge and difficult to meet engineering requirements under large-scale data. Furthermore, existing clustering algorithms lack the ability to construct complex cluster structures, which limits the improvement of noisy clustering and performance indicators.
An online face clustering method based on image search is adopted. It uses a trained convolutional neural network to extract face features, and achieves fast and accurate data archiving and classification by retrieving and weighting class center features, combining quantized feature class centers and online clustering correction.
It realizes automated real-time data clustering of the face recognition system, which improves recognition efficiency and system resource utilization, reduces computational cost, and enhances the robustness and accuracy of the recognition system.
Smart Images

Figure CN112633051B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the fields of computer vision and image processing, and in particular to an online face clustering method based on image search. Background Art
[0002] In recent years, the research and application of face recognition technology has reached a high level in academia, video surveillance, public security, access control and other fields. How to better enable public security services for such a large face database and how to use these information resources to improve the accuracy of face recognition to a new level are two very challenging tasks. Therefore, effectively utilizing unlabeled face data becomes an attractive option, namely, unsupervised face clustering algorithm.
[0003] Face clustering analysis aims to cluster large face images into several clusters based on the similarity between faces, which is a natural idea to use unlabeled data as "pseudo-classes" so that these data can be used for supervised learning like labeled data. Commonly used methods include K-means clustering, spectral clustering, hierarchical clustering and other methods to group unlabeled face images. These methods have their own advantages but also many shortcomings. For example, the K-means method requires a preset number of clusters, that is, it is necessary to assume the number of possible categories of the data, but the number of class centers in the actual accumulated data set is completely uncertain. For example, spectral clustering requires that the sizes of different categories of clusters are relatively balanced, but this is not the case in reality. Therefore, the current clustering scheme lacks the ability to have complex clustering structures, resulting in noisy clustering, especially when applied to large-scale images collected from real scenes. This problem seriously limits the improvement of performance indicators.
[0004] In addition, another serious problem of clustering algorithms is the huge amount of calculation, especially when clustering large-scale data. The current common clustering algorithms are time-consuming and cannot meet the actual engineering needs. In addition, in some scenarios, image data needs to be clustered and stored in real time, so it is necessary to combine image retrieval methods to realize an online real-time clustering algorithm. Summary of the invention
[0005] The purpose of the present invention is to solve the problem that the prior art cannot provide an accurate and efficient online real-time portrait data labeling and one-person-one-file construction method for face recognition, and to provide an online face clustering method based on image search.
[0006] To achieve the above object, the present invention is implemented according to the following technical solutions:
[0007] An online face clustering method based on image search comprises the following steps:
[0008] S1, obtaining archived portrait images and new portrait images that have been clustered and grouped into multiple categories;
[0009] S2, using the trained convolutional neural network model to extract facial features from the archived portrait images and the new portrait images, and then inputting the extracted facial features into the database;
[0010] S3, performing class center feature retrieval on the face feature vector in the database;
[0011] S4. Determine the category to which the face image belongs based on the result of the class center feature retrieval and archive it in the database.
[0012] Furthermore, the extraction of facial feature vectors of archived portrait images and new portrait images by using the trained convolutional neural network model in S2 specifically includes:
[0013] S21, capturing facial images from the archived portrait images and the new portrait images;
[0014] S22, performing facial key point detection on the facial image and performing facial correction;
[0015] S23. Input the corrected face image into the trained convolutional neural network model, and output a multi-dimensional face feature vector.
[0016] Furthermore, the S3 specifically includes:
[0017] S31. Calculate the class center feature using feature weighting method: Let the face feature set in each class be F n ={f1, f2, ... f n}, then the calculation method of the class center feature of the corresponding category is shown in formula (1):
[0018]
[0019] Where: f i represents the i-th face feature in a category; w i Represents the weight of the i-th facial feature;
[0020] S32, using the quantitative feature class center to calculate the above class center feature:
[0021] Assume that the weight of each facial feature is w = {w1, w2, ..., w n}, the face feature set of this category is f = {f1, f2, ..., f n}, quantize the feature value to: 1→1, 0→-1; then calculate the weight, the i-th dimension center feature is:
[0022] f cen (i)=w1·f1(i)+w2·f2(i)+…+w n ·f n (i) = w1-w2+…+w n (2);
[0023] Finally, the quantified feature class center feature is obtained as
[0024] S33, calculate the pairwise similarity of the facial features in each category, construct a similarity matrix based on the similarity scores, and after obtaining the similarity score matrix, calculate the average similarity between the facial features of the new face image and other facial features in the category, and the calculation formula is as shown in formula (3):
[0025]
[0026] Among them, S i,j Represents the similarity between the i-th and j-th features;
[0027] S34, sorting the calculation results by similarity, and selecting the category with the highest similarity as the candidate category.
[0028] Furthermore, the S4 specifically includes:
[0029] S41, filing the new face image based on the retrieval similarity, if the similarity with the candidate category is higher than the preset threshold, the new face image is classified into the candidate category group; if the similarity with the candidate category is lower than the preset threshold, a new category group is created and the new face image is classified into the category, and the face feature of the new face image is used as the central feature of the category, and its feature weight is reset to 1.0;
[0030] S42: If the new face image is classified into an existing category, update the feature weight, feature center and other information of the category.
[0031] Furthermore, the specific steps of updating the feature weight, feature center and other information of the category in S42 are:
[0032] Assume that there are n face features in the current category, and the new face feature is the n+1th feature, and the weight w i and the category center feature f c To update:
[0033] The face feature set of the current category is recorded as F n ={f1, f2, ... f n}, and the sum of their average similarities is as well as The current feature weight is but
[0034] First, calculate the similarity sequence between the new facial features and other facial features, as well as the corresponding average similarity:
[0035]
[0036]
[0037] After obtaining the similarity set between the new face features and the existing face features, the existing average similarity is updated:
[0038]
[0039] After updating the existing average similarity, calculate the sum of the new average similarities In order to calculate the subsequent weights:
[0040]
[0041] After obtaining the new similarity sum, calculate the new feature weight:
[0042]
[0043] because Then we have:
[0044]
[0045] The weight of the new feature is:
[0046]
[0047] Substitute the weight of the new feature into formula (1) to obtain the new class center feature of the category.
[0048] Furthermore, the method further includes S5, online clustering correction; the online clustering correction specifically includes:
[0049] S51. In the existing clustering results, use a higher similarity threshold to find the most similar connected node pair set;
[0050] S52, re-aggregating similar nodes into one category according to the connection relationship of the connected nodes, and disconnecting those that do not meet the threshold, thereby splitting different categories in the error results;
[0051] S53. Because subclusters are established by connecting node pairs, a larger cluster will be split into multiple small clusters. Therefore, after the split, the adjacent node set overlap rate is used to re-merge the data so that the data of the same category is as complete as possible.
[0052] The principle of the present invention is as follows: the present invention first uses CNN (convolutional neural network) to express the image as corresponding features, and then quickly selects the category to which the new data belongs through feature retrieval (i.e., image retrieval); in order to improve the image retrieval speed to meet engineering needs, the present invention uses the class center to represent the entire category for the storage retrieval of new images, thereby greatly improving the retrieval speed.
[0053] Compared with the prior art, the present invention has the following beneficial effects:
[0054] 1. The present invention provides an automated real-time data clustering function for the face recognition system, which can classify face data while recognizing faces, automatically complete the online establishment of one person per file, and label the data;
[0055] 2. The class center retrieval method and the use of the quantitative class center of the present invention can effectively improve the efficiency of the recognition system and the system resource occupancy;
[0056] 3. The weighted class center calculation scheme of the present invention effectively improves the accuracy of the center feature and reduces the interference of noise, so that the recognition system has stronger robustness;
[0057] 4. The online weight and center update algorithm of the present invention can make the most of the existing calculation information without recalculation, thereby greatly reducing the calculation cost of the clustering algorithm. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] Figure 1 It is a flow chart of the present invention.
[0059] Figure 2 This is a schematic diagram of the facial feature weight calculation principle of the present invention.
[0060] Figure 3 This is a flowchart of the online clustering correction of the present invention. DETAILED DESCRIPTION
[0061] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments. The specific embodiments described herein are only used to explain the present invention and are not used to limit the invention.
[0062] like Figure 1 As shown, an online face clustering method based on image search in this embodiment includes the following steps:
[0063] Step 1: Obtain archived portrait images and new portrait images that have been clustered and grouped into multiple categories.
[0064] Step 2: Use the trained convolutional neural network model to extract facial features from archived portrait images and new portrait images, and then input the extracted facial features into the database; specifically:
[0065] 1) Capture the face images from the archived portrait images and new portrait images;
[0066] 2) Detect facial key points of face images and perform face correction;
[0067] 3) Input the corrected face image into the trained convolutional neural network model and output a multi-dimensional face feature vector.
[0068] Step 3: Perform class center feature retrieval on the face feature vector in the database:
[0069] Considering the diversity and differences of large-scale images collected from actual application scenarios, the clustering grouping results may result in a person being assigned to multiple category IDs, or different face images in one category ID. When there are multiple face images in one category ID, if the method of directly calculating the Euclidean center is used, these noise images will cause the calculated center to deviate from the true center, and cannot well represent the group of face data, resulting in the continuous accumulation and amplification of errors in subsequent classification tasks. Therefore, to solve this problem, this embodiment uses a feature weighting method to calculate the class center feature: Let the face feature set in each category be F n ={f1, f2, ... f n}, then the calculation method of the class center feature of the corresponding category is shown in formula (1):
[0070]
[0071] Where: f i represents the i-th face feature in a category; w i Represents the weight of the i-th facial feature;
[0072] In order to improve the speed of retrieval and similarity calculation and reduce system resource usage, the quantitative feature class center is used to calculate the above class center features:
[0073] Assume that the weight of each facial feature is w = {w1, w2, ..., w n}, the face feature set of this category is f = {f1, f2, ..., f n}, quantize the feature value to: 1→1, 0→-1; then calculate the weight, the i-th dimension center feature is:
[0074] f cen (i)=w1·f1(i)+w2·f2(i)+…+w n·f n (i) = w1-w2+…+w n (2);
[0075] Finally, the quantified feature class center feature is obtained as
[0076] Based on the above formula (1), we need to i Calculate its corresponding weight w i In order to consider the correlation between images in the ID, a face calculation method based on feature similarity is adopted here. The principle is as follows Figure 2 As shown. Calculate the pairwise similarity of facial features in each category and construct a similarity matrix based on their similarity scores ( Figure 2 After obtaining the similarity score matrix, the average similarity between the facial features of the new face image and other facial features in the category is calculated (i.e., the mean of each row in the figure). The calculation formula is as follows:
[0077]
[0078] Among them, S i,j Represents the similarity between the i-th and j-th features;
[0079] Then the calculation results are sorted by similarity, and the category with the highest similarity is selected as the candidate category.
[0080] Step 4: Determine the category of the face image based on the result of the class center feature retrieval and archive it in the database:
[0081] a) Based on the retrieval similarity, the new face image is archived. If the similarity with the candidate category is higher than the preset threshold, the new face image is classified into the candidate category group; if the similarity with the candidate category is lower than the preset threshold, a new category group is created and the new face image is classified into the category. The facial features of the new face image are used as the central features of the category, and its feature weight is reset to 1.0;
[0082] b) If the new face image is classified into an existing category, the feature weight, feature center and other information of the category to which it belongs are updated. The calculation method of the ID center feature and feature weight has been explained above. When a new face image is added to the ID in the future, each weight under the ID needs to be updated to obtain the new center feature. The final weight update result is as follows:
[0083]
[0084]
[0085] From the above formula, we can see that the parameters needed for actual weight update are the weights of each feature in the current state. And the sum of average similarities of all features The detailed formula is derived as follows:
[0086] Assume that there are n face features in the current category, and the new face feature is the n+1th feature, and the weight w i and the category center feature f c To update:
[0087] Basic variable description:
[0088] 1.f i represents the i-th feature in the class;
[0089] 2.S i,j Represents the similarity between the i-th and j-th features;
[0090] 3.S mi represents the mean similarity between the i-th face and other faces in the class, that is,
[0091] 4.S sum Represents the sum of all average similarities within the ID, that is,
[0092] 5.w i represents the weight of the i-th facial feature,
[0093] 6.f c represents the class center feature, that is
[0094] Assuming that there are n features in the current ID and the newly collected face is the n+1th feature, the weight w needs to be i and the ID center feature f c The update is specifically derived as follows. The ultimate goal is to update the central feature vector: S mi →w i →f c :
[0095] Let the feature set of the current ID be F n ={f1, f2, ... f n}, and the sum of their average similarities is as well as The current feature weight is but
[0096] First, calculate the similarity sequence between the newly added features and other facial features, as well as the corresponding average similarity:
[0097]
[0098]
[0099] After obtaining the similarity set between the new features and the existing features, the existing average similarity needs to be updated:
[0100]
[0101]
[0102] After updating the existing average similarity, calculate the sum of the new average similarities In order to calculate the subsequent weights:
[0103]
[0104] After obtaining the new similarity sum, the new feature weight can be calculated:
[0105]
[0106] Also because Then we have:
[0107]
[0108] The weight of the new feature is:
[0109]
[0110] In summary, in actual use, the variables that need to be saved are the weight of each feature in the ID and the sum of the average similarity, a total of n+1 parameters;
[0111] Substitute the weight of the new feature into formula (1) to obtain the new class center feature of the category.
[0112] Based on the above clustering principle, online face clustering can be realized. However, since clustering is based on the calculation of class centers, the direct online clustering solution cannot handle the problem of erroneous samples generated by clustering. Moreover, if there is no real-time error correction solution, errors will continue to accumulate, resulting in serious performance degradation in large data volumes and long-term clustering. Therefore, the present invention adds an erroneous sample separation mechanism based on the above clustering, splitting the clustered erroneous samples to ensure the accuracy of the class center.
[0113] The core strategy of this module is that when the amount of sample data of a certain class exceeds a certain level, a simplified clustering algorithm is used to associate and divide the ID, divide the samples of the ID into different small clusters, and use each small cluster as a new ID to recalculate the class center and the corresponding weight. The process is as follows: Figure 3 As shown. Step 1, in the existing clustering results, use a higher similarity threshold to find the most similar connected node pair set; Step 2, re-aggregate similar nodes as one category based on the connectivity of the connected nodes, and disconnect those that do not meet the threshold, so as to split different categories in the erroneous results; Step 3, because sub-clusters are established by connecting node pairs, the larger cluster will be split into multiple small clusters, so after the split, it is necessary to re-merge the adjacent node set overlap rate to make the data belonging to one category as complete as possible; In this way, a clustered result can be quickly corrected.
[0114] In summary, the present invention first uses CNN (convolutional neural network) to express the image as corresponding features, and then quickly selects the category of the new data through feature retrieval (i.e., image retrieval); in order to improve the image retrieval speed to meet engineering needs, the present invention uses the class center to represent the entire category for the storage retrieval of new images, thereby greatly improving the retrieval speed.
[0115] The technical solution of the present invention is not limited to the above-mentioned specific embodiments. All technical variations made according to the technical solution of the present invention fall within the protection scope of the present invention.
Claims
1. An online face clustering method based on image search, characterized in that: The following steps are involved: S1, obtaining archived portrait images and new portrait images that have been clustered and grouped into multiple categories; S2. Extracting facial features from the archived portrait images and the new portrait images using the trained convolutional neural network model, and then inputting the extracted facial features into the database; extracting facial feature vectors from the archived portrait images and the new portrait images using the trained convolutional neural network model specifically includes: S21, capturing facial images from the archived portrait images and the new portrait images; S22, performing facial key point detection on the facial image and performing facial correction; S23, inputting the corrected face image into the trained convolutional neural network model, and outputting a multi-dimensional face feature vector; S3, performing class center feature retrieval on the face feature vector in the database; S31. Calculate the class center feature using feature weighting method: Let the face feature set in each class be F n ={f1, f2, ... f n }, then the calculation method of the class center feature of the corresponding category is shown in formula (1): Where: f i represents the i-th face feature in a category; w i Represents the weight of the i-th facial feature; S32, using the quantitative feature class center to calculate the above class center feature: Assume that the weight of each facial feature is w = {w1, w2, ..., w n }, the face feature set of this category is f = {f1, f2, ..., f n }, quantize the feature value to: 1→1, 0→-1; then calculate the weight, the i-th dimension center feature is: f cen (i)=w1·f1(i)+w2·f2(i)+…+w n ·f n (i)=w1-w2+…+w n (2); Finally, the quantified feature class center feature is obtained as S33, calculate the pairwise similarity of the facial features in each category, construct a similarity matrix based on the similarity scores, and after obtaining the similarity score matrix, calculate the average similarity between the facial features of the new face image and other facial features in the category, and the calculation formula is as shown in formula (3): Among them, S i,j Represents the similarity between the i-th and j-th features; S34, sorting the calculation results by similarity, and selecting the category with the highest similarity as the candidate category; S4. Determine the category to which the face image belongs based on the result of the class center feature retrieval and archive it in the database.
2. The online face clustering method based on image search according to claim 1, characterized in that: The S4 specifically includes: S41, filing the new face image based on the retrieval similarity, if the similarity with the candidate category is higher than a preset threshold, the new face image is classified into the candidate category; if the similarity with the candidate category is lower than the preset threshold, a new category is created and the new face image is classified into the category, and the face feature of the new face image is used as the central feature of the category, and its feature weight is reset to 1.0; S42: If the new face image is classified into an existing category, update the feature weight, feature center and other information of the category.
3. The online face clustering method based on image search according to claim 2, characterized in that: The specific steps of updating the feature weight, feature center and other information of the category in S42 are: Assume that there are n face features in the current category, and the new face feature is the n+1th feature, with a weight of w i and the category center feature f c To update: The face feature set of the current category is recorded as F n ={f1, f2, ... f n }, and the sum of their average similarities is as well as The current feature weight is but First, calculate the similarity sequence between the new facial features and other facial features, as well as the corresponding average similarity: After obtaining the similarity set between the new face features and the existing face features, the existing average similarity is updated: After updating the existing average similarity, calculate the sum of the new average similarities In order to calculate the subsequent weights: After obtaining the new similarity sum, calculate the new feature weight: because Then we have: The weight of the new feature is: Substitute the weight of the new feature into formula (1) to obtain the new class center feature of the category.
4. The online face clustering method based on image search according to claim 3, characterized in that: It also includes S5, online clustering correction; The online clustering correction specifically includes: S51, in the existing clustering results, using a higher preset threshold to find the most similar connected node pair set; S52, re-aggregating similar nodes into one category according to the connection relationship of the connected nodes, and disconnecting those that do not meet the threshold, thereby splitting different categories in the error results; S53. Because subclusters are established by connecting node pairs, a large cluster will be split into multiple small clusters. Therefore, after the split, the adjacent node set overlap rate is used to re-merge the data so that the data of the same category is as complete as possible.
Citation Information
Patent Citations
Mass image retrieval system based on cluster compactness
CN103605765A
Face retrieval method based on multitask convolution nerve network
CN106815566A
A fast BGP face retrieval method based on clustering
CN109388727A
An index-based rapid face retrieval system application
CN109710792A
Archiving method and device
CN109815370A
