A face clustering optimization method based on graph convolution

Abstract

The present invention discloses a face clustering optimization method based on graph convolution. The method comprises the following steps: face feature F, face feature similarity and face image quality score are used; a face feature similarity graph is constructed using the face feature similarity and the face image quality score to obtain graph convolution network input data; each graph sample #imgabs2# containing a point attribute matrix #imgabs0# and an adjacency matrix #imgabs1# is reconstructed with a graph convolution network, and the reconstructed graph convolution network outputs a result of whether each face node in each graph sample #imgabs3# belongs to the face category; a judgment is made based on the output face category result. If a face does not belong to this category, the face is removed from this category. At the same time, faces with a high similarity to this category are verified. If the face to be verified belongs to this category, the face is added to this category. The purpose of face clustering optimization and purification is achieved.

Description

Technical Field

[0001] The present invention relates to the field of pattern recognition and computer vision technology, and in particular to a face clustering optimization method based on graph convolution. Background Art

[0002] With the widespread deployment of video surveillance and the explosive growth of live video streaming, an increasing amount of video data needs to be analyzed and processed. One efficient approach is to classify the subject of the video (people) for further analysis and processing. While the advent of deep learning techniques has driven rapid development in face recognition technology, achieving recognition accuracy that reaches or exceeds human levels, the challenge of recognizing faces in videos with mismatched scenes remains unresolved due to numerous factors, such as pose, lighting, and blur. Current deep face recognition algorithms often require massive amounts of annotated face data. Unsupervised face clustering algorithms are an effective way to provide these large amounts of face annotations. Face clustering algorithms classify large amounts of unannotated face data by person, grouping facial images of the same person into a single category or cluster. This plays a crucial role in the effective analysis and processing of video data. However, current face clustering algorithms often suffer from low clustering accuracy and the accumulation of clustering errors due to the influence of multi-class noise in face data. To address this issue, we propose a face clustering optimization method based on graph convolution. Summary of the Invention

[0003] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0004] The present invention provides a face clustering optimization method based on graph convolution, which includes the following steps:

[0005] Step 1: Dataset preparation and preprocessing; use the pre-trained face recognition model to extract the face recognition data to be trained, obtain the face feature F and the face feature similarity, for each face feature F is expressed as a p-dimensional face feature vector, then F∈R p ; Then use the face image quality determination method to determine the face image of the training face recognition data, the face image quality score class value Q, and obtain the face image quality score;

[0006] Step 2: Use facial feature similarity and facial image quality score to construct a facial feature similarity graph to obtain graph convolutional network input data; the input graph sample T is constructed by using the graph convolutional network G = {V, E, A} using facial feature similarity and facial image quality score. i , and get each graph sample The node attribute matrix in and the adjacency matrix

[0007] Step 3: Get the matrix containing point attributes and the adjacency matrix Each sample of Reconstruct with the graph convolutional network and output each graph sample through the reconstructed graph convolutional network Whether each face node in the result belongs to the face category;

[0008] Step 4: Make a judgment based on the output face category results. If a face does not belong to this category, remove it from this category. At the same time, verify the faces with a high similarity to this category. If the face to be verified belongs to this category, add it to this category; thus achieving the purpose of face clustering optimization and purification.

[0009] As a preferred technical solution of the present invention, in the step 2, the facial feature similarity graph is constructed using the facial feature similarity and the facial image quality score to obtain the graph convolution network input data; the input graph sample T is constructed by using the facial feature similarity and the facial image quality score through the graph convolution network G = {V, E, A}. i , and get each graph sample The node attribute matrix in and the adjacency matrix The method is,

[0010] The graph convolution network G = {V, E, A} represents a graph convolution network consisting of several nodes V, several edges E and an adjacency matrix A, where each node has a d-dimensional attribute vector. When a graph has N nodes, the node attribute matrix X∈R of the graph convolution network input is N×D and the adjacency matrix A∈R N×N , let v i ∈V,e ij =(v i , v j )∈E as an edge between node vi and node Vj, then the NxN adjacency matrix A, where A ij =w ij >0 if e ij ∈E,A ij =w ij =0 if

[0011] Among them, D represents the face recognition training data set, and Represents a face training image sample, where i represents the i-th face ID in the training face ID, and j represents the j-th face image under the i-th face ID. Similarly, Represents the facial feature vector of a face image sample, using Represents the face image quality score of the face image sample, for the graph convolutional network G = {V, E, A} an input sample T i Randomly sample k face samples from a certain face ID As a positive sample, k is less than the total number of face samples N under the ID, generally k is greater than 3, and the H set is selected. The best quality face sample is As a high-quality node sample, l face samples are randomly selected from the face ID samples that do not belong to i as abnormal node samples and Usually l is less than or equal to 5, where The positive sample class label is 0, The class label of the sample marked as abnormal is 1, and the similarity distance between the face vectors is used between the samples, and the adjacency relationship between the nodes is established to form a graph convolution network with an input graph sample;

[0012] Where X∈R N×D It is represented as the node attribute moment of the graph convolutional network G = {V, E, A} input, where the number of node attribute vectors is the sum of the number of positive samples and abnormal samples. The attribute data vector of the face node is established. The node attribute data vector is calculated by the similarity between nodes. Define M ij is the similarity between different face image samples, then M ij =φ(F i ,F j ); where face node attributes include: similarity with high-quality nodes, best similarity between a node and other nodes, suboptimal similarity between a node and other nodes, and worst similarity between a node and other nodes; node attribute data is constructed based on face node attributes;

[0013] In summary, the sampling graph convolutional network inputs a graph sample set K, where each graph sample Node attribute matrix and the adjacency matrix It is composed as the input data of the graph convolutional network.

[0014] As a preferred technical solution of the present invention, the point attribute matrix and the adjacency matrix Each sample of Reconstruct with the graph convolutional network and output each graph sample through the reconstructed graph convolutional network The method for determining whether each face node in the face belongs to the face category is to use a graph convolutional neural network to aggregate feature information from its neighboring nodes to encapsulate the hidden representation of each node; specifically, after feature aggregation, a nonlinear transformation is applied to the result output, and the attribute state of the hidden layer node is updated using the following formula:

[0015]

[0016] in represents the degree matrix, X (l) Represents the node features at layer l, X (l+1) Represents the node features at the l+1 layer, W (l) Represents the weight of the first layer, and obtains the hidden state of each node's graph perception. For each node, its hidden state contains information from neighboring nodes. In the last layer of the graph convolution network, the output of each node state is a binary classification vector to indicate whether the node belongs to this category. According to this formula, a 2-3 layer graph convolution network is constructed, and then the reconstructed graph convolution network is used to classify each graph sample. Each face node in the is the face category result for recognition and outputs the recognition category result.

[0017] As a preferred technical solution of the present invention, it also includes step 5, training the graph convolution network, which is to construct the input samples of the graph convolution network according to the steps shown in step 2. In the process of constructing the generated data samples, difficult samples are constructed at a ratio of 70%, that is, in the process of selecting positive samples, samples with lower similarity to the high-quality samples of this batch are preferred. When selecting abnormal samples, sample data with higher similarity to the positive samples are preferred. The input sample data of the graph convolution network is divided into training data, verification data and test data at a ratio of 3:1:1. The graph convolution network constructed in step 4 is trained on the training set, and the accuracy is verified on the verification set. When the accuracy on the verification set no longer increases, the training is stopped and the results are tested on the test set. The graph convolution network model training weight parameter file is saved recently.

[0018] The beneficial effects of the present invention are:

[0019] This face clustering optimization method based on graph convolution extracts the face recognition data to be trained by using the pre-trained face recognition model to obtain the face features F and the face feature similarity. For each face feature F, it is expressed as a p-dimensional face feature vector, then F∈R p ; Then, the face image quality judgment method is used to judge the face image of the training face recognition data, the face image quality score class value Q, and the face image quality score are obtained; the face feature similarity and the face image quality score are used to construct a face feature similarity graph to obtain the graph convolution network input data; the input graph sample T is constructed by using the face feature similarity and the face image quality score through the graph convolution network G = {V, E, A} i , and get each graph sample The node attribute matrix in and the adjacency matrix The matrix containing the point attributes will be obtained and the adjacency matrix Each sample of Reconstruct with the graph convolutional network and output each graph sample through the reconstructed graph convolutional network Whether each face node in the face belongs to the face category result; judge according to the output face category result, if a face does not belong to this category, then remove the face from this category, and at the same time, verify the faces with a high similarity to the category, if the face to be verified belongs to the category, then add the face to this category; achieve the purpose of face clustering optimization and purification; the present invention is based on the graph convolutional neural network and proposes to use the similarity relationship between nodes to build a graph convolutional network input. The graph network model determines the category to which the node belongs based on the similarity relationship and similarity of the faces of the same type, rather than using a unified fixed threshold method for classification. This method is more flexible and robust, and its results are better than most clustering algorithms based on unified rules, greatly improving the accuracy of face clustering results. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0021] Figure 1 This is a flow chart of a face clustering optimization method based on graph convolution of the present invention;

[0022] Figure 2 Schematic diagram of the structure of an input graph sample formed by establishing adjacency relationships between nodes based on similarity distances obtained between face vectors in the present invention;

[0023] Figure 3 Schematic diagram of the node attribute data structure of the input graph sample constructed by step 2 of the present invention;

[0024] Figure 4 It is a structural diagram of the graph convolutional network reconstructed by the present invention. DETAILED DESCRIPTION

[0025] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0026] Example: Figure 1-4 As shown, the present invention provides a face clustering optimization method based on graph convolution, which includes the following steps:

[0027] Step 1: Dataset preparation and preprocessing; use the pre-trained face recognition model to extract the face recognition data to be trained, obtain the face feature F and the face feature similarity, for each face feature F is expressed as a p-dimensional face feature vector, then F∈R p ; Then use the face image quality determination method to determine the face image of the training face recognition data, the face image quality score class value Q, and obtain the face image quality score;

[0028] Step 2: Use facial feature similarity and facial image quality score to construct a facial feature similarity graph to obtain graph convolutional network input data; the input graph sample T is constructed by using the graph convolutional network G = {V, E, A} using facial feature similarity and facial image quality score. i , and get each graph sample The node attribute matrix in and the adjacency matrix

[0029] Step 3: Get the matrix containing point attributes and the adjacency matrix Each sample of Reconstruct with the graph convolutional network and output each graph sample through the reconstructed graph convolutional network Whether each face node in the result belongs to the face category;

[0030] Step 4: Make a judgment based on the output face category result. If a face does not belong to this category, the face is removed from this category. At the same time, faces with a high similarity to the category are verified. If the face to be verified belongs to this category, the face is added to this category; the purpose of face clustering optimization and purification is achieved. Based on the traditional face feature clustering results, the present invention uses each face image data as a graph node to be identified, uses face feature similarity and face image quality score to construct a face feature similarity graph, inputs the feature similarity graph into the graph convolutional neural network, and outputs the binary classification category of each face node, that is, whether the face belongs to this class or this cluster. If a face does not belong to this class, the face is removed from this class. At the same time, faces with a high similarity to the category can also be verified. If the face to be verified belongs to this category, the face is added to this category, thereby achieving the effect of face clustering optimization and purification.

[0031] In the step 2, the facial feature similarity graph is constructed using the facial feature similarity and the facial image quality score to obtain the graph convolution network input data; the input graph sample T is constructed by using the graph convolution network G = {V, E, A} using the facial feature similarity and the facial image quality score i , and get each graph sample The node attribute matrix in and the adjacency matrix The method is,

[0032] The graph convolution network G = {V, E, A} represents a graph convolution network consisting of several nodes V, several edges E and an adjacency matrix A, where each node has a d-dimensional attribute vector. When a graph has N nodes, the node attribute matrix X∈R of the graph convolution network input is N×D and the adjacency matrix A∈R N×N , let v i ∈V,e ij =(v i , v j )∈E as an edge between node vi and node Vj, then the NxN adjacency matrix A, where A ij =w ij >0 if e ij ∈E,A ij =w ij =0 if

[0033] Among them, D represents the face recognition training data set, and Represents a face training image sample, where i represents the i-th face ID in the training face ID, and j represents the j-th face image under the i-th face ID. Similarly, Represents the facial feature vector of a face image sample, using Represents the face image quality score of the face image sample, for the graph convolutional network G = {V, E, A} an input sample T i Randomly sample k face samples from a certain face ID As a positive sample, k is less than the total number of face samples N under the ID, generally k is greater than 3, and the H set is selected. The best quality face sample is As a high-quality node sample, l face samples are randomly selected from the face ID samples that do not belong to i as abnormal node samples and Usually l is less than or equal to 5, where The positive sample class label is 0, The class label of the sample marked as abnormal is 1, and the similarity distance between the face vectors is used between the samples, and the adjacency relationship between the nodes is established to form a graph convolution network with an input graph sample;

[0034] Where X∈R N×D It is represented as the node attribute moment of the graph convolutional network G = {V, E, A} input, where the number of node attribute vectors is the sum of the number of positive samples and abnormal samples. The attribute data vector of the face node is established. The node attribute data vector is calculated by the similarity between nodes. Define Mij is the similarity between different face image samples, then M ij =φ(F i ,F j ); The attributes of face nodes include: similarity with high-quality nodes, the best similarity between nodes and other nodes, the second-best similarity between nodes and other nodes, and the worst similarity between nodes and other nodes; construct node attribute data based on face node attributes; construct node attribute data based on the above rules as follows Figure 3 shown.

[0035] In summary, the sampling graph convolutional network inputs a graph sample set K, where each graph sample Node attribute matrix and the adjacency matrix It is composed as the input data of the graph convolutional network.

[0036] The above method establishes the node adjacency relationship based on the similarity distance between face vectors between samples to form a graph convolution network G = {V, E, A}. The rules of an input graph sample are as follows: a. All nodes establish adjacent edges with the optimal node; b. All nodes establish adjacent edge relationships with the two nodes with the highest similarity except the high-quality nodes. The graph adjacency matrix A can be generated based on the adjacent edge relationship. Specifically, Figure 2 shown.

[0037] The above will get a matrix containing point attributes and the adjacency matrix Each sample of Reconstruct with the graph convolutional network and output each graph sample through the reconstructed graph convolutional network The method for determining whether each face node in the face belongs to the face category is to use a graph convolutional neural network to aggregate feature information from its neighboring nodes to encapsulate the hidden representation of each node; specifically, after feature aggregation, a nonlinear transformation is applied to the result output, and the attribute state of the hidden layer node is updated using the following formula:

[0038]

[0039] in represents the degree matrix, X (l) Represents the node features at layer l, X (l+1) Represents the node features at the l+1 layer, W (l) Represents the weight of the first layer, and obtains the hidden state of each node's graph perception. For each node, its hidden state contains information from neighboring nodes. In the last layer of the graph convolution network, the output of each node state is a binary classification vector to indicate whether the node belongs to this category. According to this formula, a 2-3 layer graph convolution network is constructed, and then the reconstructed graph convolution network is used to classify each graph sample. Each face node in the image is a face category result for recognition, and the recognition category result is output. The reconstructed graph convolutional network is as follows: Figure 4 shown.

[0040] The present invention is implemented in the following process:

[0041] 1) Perform facial feature extraction and quality score extraction on the unlabeled face image data according to the method described in step 1, and save the results;

[0042] 2) Based on the similarity between facial features obtained in process 1), the traditional unsupervised clustering algorithm is used to perform rough clustering of face categories to obtain preliminary rough clustering results;

[0043] 3) Based on the preliminary coarse clustering results obtained in process 2), a graph convolutional neural network is constructed according to the clustering method in step 3 and the weight file trained in step 4 is loaded. According to the method described in step 2, the coarse clustering results are used as the ID category to construct the graph convolutional neural network input data. Here, the node with the best quality is directly selected as the high-quality node within the coarse clustering category. The first batch of data can only screen abnormal samples within the class. The second batch of data can select the topN nodes of the quality ranking of the filtered class data as the node data within the class. Some isolated nodes with high similarity to the high-quality samples in the class are selected to jointly construct the input samples to detect whether the high-similarity isolated nodes belong to the class;

[0044] 4) Based on the input data constructed in the above process, clustering and purification work is performed. First, based on the first batch of data constructed, the data is input into the graph network model prepared in process 3) to infer the node category, and the data inferred as abnormal nodes are removed from the category. Then, the second batch of data can be input into the graph network model for inference, and the data determined to be positive samples are added to the category.

[0045] 5) Process 4) completes the purification of the rough clustering results, and the results can be repeatedly optimized for multiple clustering processes.

[0046] Finally, it should be noted that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions described in the aforementioned embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A face clustering optimization method based on graph convolution, characterized by: It includes the following steps: Step 1: Dataset preparation and preprocessing: Use the pre-trained face recognition model to extract the face recognition data to be trained, obtain the face features F and the face feature similarity. For each face feature F, it is expressed as a p-dimensional face feature vector, then F∈R p ; Then, the face image quality determination method is used to determine the face image of the training face recognition data to obtain the face image quality score; Step 2: Use facial feature similarity and facial image quality score to construct a facial feature similarity graph to obtain graph convolution network input data; construct input graph sample T using graph convolution network G = {V, E, A} using facial feature similarity and facial image quality score i , and get each graph sample The node attribute matrix in and the adjacency matrix Step 3: Get the matrix containing point attributes and the adjacency matrix Each sample of Reconstruct with the graph convolutional network and output each graph sample through the reconstructed graph convolutional network Whether each face node in the result belongs to the face category; Step 4: Determine the face category based on the output. If a face does not belong to this category, remove it from this category. At the same time, verify faces with a high degree of similarity to this category. If the face to be verified belongs to this category, add it to this category. The purpose of face clustering optimization and purification is achieved.

2. The face clustering optimization method based on graph convolution according to claim 1, characterized in that: In the step 2, the facial feature similarity graph is constructed using the facial feature similarity and the facial image quality score, and the graph convolution network input data is obtained. The input graph sample T is constructed by the graph convolution network G = {V, E, A} using the facial feature similarity and the facial image quality score. i , and get each graph sample The node attribute matrix in and the adjacency matrix The method is, The graph convolution network G = {V, E, A} represents a graph convolution network consisting of several nodes V, several edges E and an adjacency matrix A, where each node has a d-dimensional attribute vector. When a graph has N nodes, the node attribute matrix X∈R of the graph convolution network input is N×D and the adjacency matrix A∈R N×N , let v i ∈V,e ij =(v i , v j )∈E as an edge between node Vi and node Vj, then the NxN adjacency matrix A, where A ij =w ij >0 if e ij ∈E,A ij =w ij =0 if Among them, D represents the face recognition training data set, and Represents a face training image sample, where i represents the i-th face ID in the training face ID, and j represents the j-th face image under the i-th face ID. Similarly, Represents the facial feature vector of a face image sample, using Represents the face image quality score of the face image sample, for the graph convolutional network G = {V, E, A} an input sample T i Randomly sample k face samples from a certain face ID As a positive sample, k is less than the total number of face samples N under the ID, k is greater than 3, and the H set is selected The best quality face sample is As a high-quality node sample, l face samples are randomly selected from the face ID samples that do not belong to i as abnormal node samples and Take l less than or equal to 5, where The positive sample class label is 0, The class label of the sample marked as abnormal is 1, and the similarity distance between the face vectors is used between the samples, and the adjacency relationship between the nodes is established to form a graph convolution network with an input graph sample; Where X∈R N×D Represented as the node attribute matrix of the graph convolutional network G = {V, E, A} input, where the number of node attribute vectors is the sum of the number of positive samples and abnormal samples; establish the attribute data vector of the face node, and the node attribute data vector is calculated by the similarity between nodes, and define M ij is the similarity between different face image samples, then M ij =φ(F i , F j ); where face node attributes include: similarity with high-quality nodes, best similarity between a node and other nodes, suboptimal similarity between a node and other nodes, and worst similarity between a node and other nodes; node attribute data is constructed based on face node attributes; In summary, the sampling graph convolutional network inputs a graph sample set K, where each graph sample Node attribute matrix and the adjacency matrix It is composed as the input data of the graph convolutional network.

3. The face clustering optimization method based on graph convolution according to claim 2, characterized in that: The above will get a matrix containing point attributes and the adjacency matrix Each sample of Reconstruct with the graph convolutional network and output each graph sample through the reconstructed graph convolutional network The method to determine whether each face node belongs to the face category is to use the graph convolutional network to aggregate the feature information from its neighboring nodes to encapsulate the hidden representation of each node. Specifically, after feature aggregation, a nonlinear transformation is applied to the result output, and the attribute state of the hidden layer node is updated using the following formula: in represents the degree matrix, X (l) Represents the node features at layer l, X (l+1) Represents the node features at the l+1 layer, W (l) Represents the weight of the first layer, and obtains the hidden state of each node's graph perception. For each node, its hidden state contains information from neighboring nodes. In the last layer of the graph convolution network, the output of each node state is a binary classification vector to indicate whether the node belongs to this category. According to this formula, a 2-3 layer graph convolution network is constructed, and then the reconstructed graph convolution network is used to classify each graph sample. Each face node in the is the face category result for recognition and outputs the recognition category result.

4. The face clustering optimization method based on graph convolution according to claim 1, characterized in that: It also includes step 5, training the graph convolution network, which is to construct the input samples of the graph convolution network according to the steps shown in step 2. In the process of constructing the generated data samples, difficult samples are constructed at a ratio of 70%, that is, in the process of selecting positive samples, samples with lower similarity to the high-quality samples of this batch are selected. When selecting abnormal samples, sample data with higher similarity to the positive samples are selected. The input sample data of the graph convolution network is divided into training data, verification data and test data at a ratio of 3:1:

1. The graph convolution network constructed in step 4 is trained on the training set, and the accuracy is verified on the verification set. When the accuracy on the verification set no longer increases, the training is stopped and the results are tested on the test set. The graph convolution network model training weight parameter file is saved recently.

Images