A face clustering update method and device

Abstract

This invention belongs to the field of image processing technology and provides a method and apparatus for updating face clustering. The method includes: receiving an image to be processed; when facial features are detected in the image to be processed, capturing a face image; inputting the captured face image into a feature extraction model, outputting a current face feature vector and a face confidence score corresponding to the face image; calculating the similarity between the current face feature vector and representative face feature vectors in a pre-established face feature database; when the calculated similarity is less than a first similarity threshold, re-establishing a new face identifier and updating the face feature database; when the calculated similarity is greater than or equal to the first similarity threshold, determining whether the output face confidence score is greater than a confidence threshold, and deciding whether to update the face feature database. This invention can accurately and dynamically update the face feature database, ensuring the multi-angle and diversity of face features in the database.

Description

Technical Field

[0001] This invention relates to the field of generative large model technology, and in particular to a face clustering update method and apparatus. Background Technology

[0002] As the demand for photo data storage space increases, people are uploading more and more photos to cloud storage. With the growing number of images in cloud storage albums, smart album functions are often added to improve user search convenience. Smart albums typically use features including face clustering, which uses artificial intelligence algorithms to distinguish photos of different people and group photos of the same person together. Furthermore, due to the large volume of user data, an incremental face clustering strategy should be implemented, allowing for real-time updates of face clustering results with minimal overhead as users add new photos. In addition, the face clustering method should also be able to handle newly added individuals.

[0003] Existing methods have the following problems: whenever a new character appears, the learning rate needs to be dynamically adjusted and retrained, increasing the model training complexity; in multi-user scenarios, if all characters are added, there are too many categories, requiring too many models to be trained based on the number of user categories, necessitating multiple iterations until the stopping condition is met, which increases the overall computational complexity; existing methods require specific conditions to trigger clustering, which cannot guarantee the real-time performance of the clustering algorithm; global clustering calculations are required every time a new image is added, but problems such as scene limitations, single perspective, and lack of consideration for diversity still exist.

[0004] Therefore, it is necessary to provide a new face clustering update method to solve the above problems. Summary of the Invention

[0005] This invention aims to provide a face clustering update method and apparatus to solve the technical problems of existing technologies that require specific conditions to be met to trigger clustering, which cannot guarantee the real-time performance of the clustering algorithm; and that global clustering calculation is required every time a new image is added, but there are still technical problems such as scene limitation, single perspective, and failure to consider diversity. The technical problems to be solved by this invention are achieved through the following technical solutions.

[0006] In a first aspect, the present invention proposes a face clustering update method, comprising: receiving an image to be processed; when a face feature is detected in the image to be processed, cropping a face image; inputting the cropped face image into a feature extraction model, and outputting a current face feature vector and a face confidence score corresponding to the face image; calculating the similarity between the output current face feature vector and a representative face feature vector in a pre-established face feature database; when the calculated similarity is less than a first similarity threshold, re-establishing a new face identifier to update the face feature database; when the calculated similarity is greater than or equal to the first similarity threshold, further determining whether the output face confidence score is greater than a confidence threshold to determine whether to update the face feature database.

[0007] According to an optional implementation, the method further includes the following steps: retrieving a representative face feature vector with the highest similarity to the output current face feature vector from a pre-established face feature database, and determining whether the highest similarity is greater than a first similarity threshold, wherein, when the highest similarity is greater than or equal to the first similarity threshold, the face corresponding to the current face feature vector is clustered into the existing person cluster corresponding to the representative face feature vector; when the highest similarity is less than the first similarity threshold, a new person cluster is established based on the face corresponding to the current face feature vector to update the face feature database.

[0008] According to an optional implementation, the method further includes the following steps: when determining to establish the new person cluster, performing multi-view diversity judgment on the current face feature vector, combining the current face feature vector with the representative face feature vector that has the highest similarity to the current face feature vector, and using it as model input to the diversity discrimination model to obtain a difference evaluation value.

[0009] According to an optional implementation, the method further includes the following steps: using a convolutional neural network algorithm to establish a diversity discrimination model; inputting the model obtained by the concatenation process into M[v1,v2], and inputting it into the diversity discrimination model to obtain the difference evaluation value σ(M):

[0010] ,

[0011] Where σ(M) represents the facial difference between the current facial feature vector and the representative facial feature vector with the highest similarity to the current facial feature vector, and M represents the vector matrix obtained after concatenating the current facial feature vector and the representative facial feature vector with the highest similarity to the current facial feature vector, i.e., the input of the neural network of the diversity discrimination model; e -CNN(M)denoted as e raised to the power of -CNN(M), its function is to convert CNN(M) into a non-linear range. When CNN(M) is positive, the closer the value of -CNN(M) is to 0, the closer σ(M) is to 1; when CNN(M) is negative, the closer the value of -CNN(M) is to 1, the closer σ(M) is to 0, where CNN(M) represents the output of the neural network of the diversity discrimination model.

[0012] According to an optional implementation, the method further includes the following steps: based on the obtained difference evaluation value, determine whether to establish a new person cluster; if the obtained difference evaluation value is greater than a set threshold, determine to establish a new person cluster; if the obtained difference evaluation value is less than or equal to the set threshold, further determine the quality of the face image corresponding to the current face feature vector, so as to determine whether to replace the existing face image corresponding to the representative face feature vector that has the highest similarity to the current face feature vector.

[0013] According to an optional implementation, the method further includes the following step: calculating a confidence threshold, specifically:

[0014] The detected face region is processed using a set of convolutional and fully connected layers. The Lth convolutional layer outputs h. L Represented as:

[0015] h L =ReLU(W L * h L-1 +b L ),

[0016] Among them, W L and b L These are the weights and biases of the Lth convolutional layer, respectively; * indicates the convolution operation; ReLU is the activation function, h L-1 It is the output of the previous layer;

[0017] The output f of the k-th fully connected layer k Represented as:

[0018] f k =ReLU(W k * f k-1 +b k ),

[0019] Among them, W k and b k These are the weights and biases of the k-th fully connected layer, and ReLU is the activation function, f. k-1 It is the output of the previous layer;

[0020] The output of the fully connected layer is passed to the single-neuron output layer. The output of a single neuron output layer is z. The sigmoid activation function is used to convert the output into a probability between 0 and 1, and the confidence score σ(z) is calculated.

[0021] z = W0* f k + b0,

[0022] ,

[0023] Where z represents the output of a single neuron in the output layer; W0 and b0 are the weights and biases of the output layer, respectively; f k This represents the output of the k-th layer of the fully connected layer. The purpose of representing e to the power of -z is to convert z into a non-linear range; when z is positive, The closer the value of z is to 0, the closer σ(z) is to 1; when z is negative, The closer the value is to 1, the closer σ(z) is to 0.

[0024] According to an optional implementation, the method further includes the following steps: for determining the first similarity threshold, the specific determination process includes the following steps: selecting a candidate threshold; evaluating the candidate threshold; selecting the optimal threshold; testing the candidate threshold to calculate the first similarity threshold.

[0025] According to an optional implementation, when the calculated similarity is greater than or equal to the first similarity threshold, further determining whether the output face confidence is greater than the confidence threshold to determine whether to update the face feature database includes the following steps: when it is further determined that the output face confidence is greater than the confidence threshold, determining to replace the existing face image corresponding to the representative face feature vector with the highest similarity to the current face feature vector with the face image corresponding to the current face feature vector, so as to update the face feature database.

[0026] According to an optional implementation, receiving the image to be processed and, when facial features are detected in the image to be processed, extracting the facial image includes the following steps: inputting the image to be processed into a deep learning model to obtain the position coordinate information of all faces in the image to be processed and the facial confidence score corresponding to each face; and extracting the facial image based on the position coordinate information of each face and the facial confidence score corresponding to each face.

[0027] Secondly, the present invention proposes a face clustering update device for implementing the face clustering update method described in the first aspect. The face clustering update device includes: a receiving and processing module for receiving an image to be processed and, when a face feature is detected in the image to be processed, capturing a face image; an output module for inputting the captured face image into a feature extraction model and outputting a current face feature vector and a face confidence score corresponding to the face image; a similarity calculation module for calculating the similarity between the output current face feature vector and a representative face feature vector in a pre-established face feature database; a first update processing module for re-establishing a new face identifier to update the face feature database when the calculated similarity is less than a first similarity threshold; and a second update processing module for further determining whether the output face confidence score is greater than a confidence threshold when the calculated similarity is greater than or equal to the first similarity threshold, to determine whether to update the face feature database.

[0028] Thirdly, the present invention provides an electronic device comprising: one or more processors; a storage device for storing one or more programs; wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in the first aspect of the present invention.

[0029] Fourthly, the present invention provides a computer-readable medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the method described in the first aspect of the present invention.

[0030] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0031] 1. This invention utilizes a feature extraction model built upon a sophisticated convolutional neural network structure to effectively extract the current facial feature vector from detected face images. This transforms the original image data of the image to be processed into feature vectors that accurately describe facial attributes. The output current facial feature vector is then compared with representative facial feature vectors in a pre-established facial feature database to calculate similarity. This comparison is then compared with a first similarity threshold, enabling precise and dynamic updates to the facial feature database. This ensures that the facial features in the database possess sufficient multi-angle and diversity, while avoiding the storage of overly similar features. Furthermore, it improves the robustness and accuracy of the facial recognition method.

[0032] 2. The present invention uses a diversity discrimination model established by a convolutional network structure to learn and capture the complex similarities and differences between faces at different angles, so as to accurately judge the complex similarities and differences between the current face image and the existing face images in the face feature database, thereby further promoting the face clustering task to achieve higher accuracy and robustness. Attached Figure Description

[0033] Figure 1 This is a flowchart illustrating an example of the face clustering and updating method of the present invention;

[0034] Figure 2 This is a schematic diagram illustrating a specific example of the image to be processed and the captured face image in the face clustering update method of the present invention;

[0035] Figure 3 This is a partial flowchart of an example of the face clustering and updating method of the present invention;

[0036] Figure 4 This is a schematic diagram of a frame of an example of the face clustering and updating device of the present invention;

[0037] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention;

[0038] Figure 6 This is a schematic diagram of a computer-readable medium embodiment according to the present invention. Detailed Implementation

[0039] Example

[0040] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0041] In view of the above problems, this invention proposes a face clustering update method, which is an incremental face clustering update method based on multiple perspectives. Compared with traditional batch clustering methods, the method of this invention can use an incremental strategy for face clustering, effectively eliminating the need for any global clustering calculations. To further enhance the robustness and diversity of the clustering results, a representative face feature vector selection strategy based on multiple perspectives is designed. The core of this strategy is to construct a representative face feature vector library containing a wide range of perspectives and update the representative face feature vector library. When selecting feature vectors, not only the shooting angle is considered, but also the diversity of multiple dimensions and aspects such as facial expressions, skin color, and hair color is fully considered. Therefore, it can ensure that the selected feature vectors fully capture the multi-angle characteristics of the face, thereby improving the accuracy and reliability of face recognition and classification.

[0042] Figure 1 This is a flowchart illustrating an example of the face clustering and updating method of the present invention.

[0043] The following reference Figure 1 , Figure 2 and Figure 3 The present invention will be described in detail below.

[0044] like Figure 1 As shown, in step S101, the image to be processed is received, and when facial features are detected in the image to be processed, the facial image is captured.

[0045] Specifically, the image to be processed is input into a deep learning model to obtain the position coordinates of all faces in the image and the face confidence score corresponding to each face; based on the position coordinates of each face and the face confidence score corresponding to each face, the face image is extracted.

[0046] In one specific implementation, a first deep learning model (RetinaFace) is used to detect the locations of all faces in the image to be processed, and the image to be processed (e.g., Figure 2 The image on the left is input into the deep learning model (RetinaFace). The first model (P-Net) proposes candidate facial regions, the second model (R-Net) filters bounding boxes, and the third model (O-Net) outputs the bounding boxes, returning, for example, the positional coordinates of all faces (in the case of multiple faces), such as the coordinates of the top-left and bottom-right corners of each face. It also obtains face confidence scores, which characterize the sharpness of the face. For example, the horizontal and vertical coordinates of the top-left and bottom-right corners of each face. And the facial confidence score corresponding to each face. .

[0047] The confidence threshold for face recognition is calculated using the following process: A set of convolutional and fully connected layers is applied to the detected face portion for calculation. The output of the Lth convolutional layer is represented as follows:

[0048] h L =ReLU(W L * h L-1 +b L ),

[0049] Among them, W L and b L These are the weights and biases of the Lth convolutional layer, respectively; * indicates the convolution operation; ReLU is the activation function, h L-1 It is the output of the previous layer.

[0050] The output f of the k-th fully connected layer k Represented as:

[0051] f k =ReLU(W k * f k-1 +b k ),

[0052] Among them, Wk and b k These are the weights and biases of the k-th fully connected layer, where ReLU is the activation function, f. k-1 It is the output of the previous layer.

[0053] Finally, the output of the fully connected layer is passed to the single-neuron output layer. The output of the single-neuron output layer is z. The Sigmoid activation function is used to convert the output into a probability between 0 and 1, and the confidence score σ(z) is calculated.

[0054] z = W0* f k + b0,

[0055] ,

[0056] Where z represents the output of a single neuron in the output layer; W0 and b0 are the weights and biases of the output layer, respectively; f k This represents the output of the k-th layer of the fully connected layer. The purpose of representing e to the power of -z is to convert z into a non-linear range; when z is positive, The closer the value of z is to 0, the closer σ(z) is to 1; when z is negative, The closer the value is to 1, the closer σ(z) is to 0.

[0057] Furthermore, based on the obtained location coordinates of all faces (in the case of multiple faces) and the face confidence score, face images are extracted from the image to be processed.

[0058] For example, if the image to be processed contains a human face, based on the obtained coordinates of the top-left and bottom-right corners of the face, and if the face confidence score is greater than a first confidence threshold (i.e., the face clarity is greater than the first confidence threshold), a face image is extracted from the image to be processed (e.g., ...). Figure 2 The face image shown on the right side of the middle section.

[0059] It should be noted that the above is only an optional example and should not be construed as a limitation of the present invention.

[0060] Next, in step S102, the captured face image is input into the feature extraction model, and the current face feature vector and face confidence corresponding to the face image are output.

[0061] Specifically, by detecting the coordinate information of the face, the face image is captured from the image to be processed, and the captured input feature extraction model extracts the face feature vector.

[0062] For example, a face feature extraction module integrating a feature extraction model can be used to extract face features from a face image. This involves further inputting the cropped face image into a second deep learning model (ArcFace) to obtain the current face feature vector. And face confidence. Each face feature vector is, for example, a 400-dimensional to 600-dimensional vector, preferably a 512-dimensional vector. Face confidence represents the accuracy of each dimension of the feature vector.

[0063] The feature extraction model established by the sophisticated convolutional neural network structure can effectively extract the current facial feature vector of the detected face image. This feature extraction stage is the key link in the entire face recognition process. Its purpose is to transform the original image data into feature vectors that can accurately describe the facial attributes.

[0064] It should be noted that the above is only an optional example and should not be construed as a limitation of the present invention.

[0065] Next, in step S103, the similarity between the output current face feature vector and the representative face feature vector in the pre-established face feature database is calculated.

[0066] Based on cosine similarity, the similarity between the current face feature vector and each representative face feature vector in the pre-established face feature database is calculated.

[0067] Specifically, the system retrieves the representative face feature vector with the highest similarity to the output current face feature vector from the pre-established face feature database, and determines whether the highest similarity is greater than the first similarity threshold.

[0068] The determination of the first similarity threshold specifically includes the following steps.

[0069] Step S201: Select candidate thresholds.

[0070] A grid search method is used to select candidate thresholds and determine the confidence score range, which in this invention is, for example, [0, 1]. Then, the values ​​at equal intervals within the above range are used as candidate thresholds for evaluation.

[0071] Step S202: Evaluate candidate thresholds.

[0072] Specifically, for each candidate threshold, various performance metrics, such as precision and recall, are calculated using the validation set. Receiver operating characteristic (ROC) curves, precision-recall (PR) curves, and F1 scores are used for analysis to evaluate each candidate threshold.

[0073] For example, the verification set includes images labeled with faces, face location information, etc.

[0074] Step S203: Select the optimal threshold.

[0075] Specifically, based on the calculation of the above performance indicators, the candidate threshold with the highest accuracy and recall rate of over 96% is selected as the optimal threshold.

[0076] Step S204: Test candidate thresholds to calculate the first similarity threshold.

[0077] Specifically, the selected candidate thresholds are evaluated using a separate test set to obtain an unbiased estimate of their performance on unseen data.

[0078] The test set includes image data without labeled faces or face location information.

[0079] For example, a first similarity threshold can be obtained based on a large number of sample verifications (specifically using sample data in the verification dataset). The first similarity can be used as the decomposition point for two feature vectors belonging to two different person clusters, in order to determine whether to update the person clusters, existing face images, etc. in the face feature database.

[0080] It should be noted that face confidence can also be calculated using the same steps as calculating the first similarity threshold. The validation set for calculating face confidence specifically includes image data with or without face labels, or image data labeled with features such as faces, face locations, and the number of faces, but without face labels.

[0081] For example, when the person clustering module performs the retrieval step, it retrieves the representative face feature vector with the highest similarity to the current output face feature vector from the pre-established face feature database, obtains the person ID corresponding to the representative face feature vector, and returns the person ID.

[0082] Specifically, the first similarity threshold can effectively determine whether the current face feature vector and the representative face feature vector with the highest similarity to the current face feature vector belong to the same person. If it is determined that the two feature vectors do not belong to the same person (corresponding to...), the threshold will be lowered. Figure 3 If the highest similarity score is less than the first similarity threshold (indicated by "No"), then the current face feature vector is used as the vector to be clustered, and the current face feature vector is stored in the face feature database as a new person cluster, and a new person ID is generated. If it is determined that the two feature vectors belong to the same person (corresponding to...), then... Figure 3If the highest similarity score is greater than or equal to the first similarity threshold, then the face corresponding to the current face feature vector is added to the existing person cluster corresponding to the representative face feature vector. See [link to details] for more information. Figure 3 .

[0083] Incremental clustering based on feature similarity not only ensures clustering accuracy but also enhances the scalability and flexibility of the facial feature database.

[0084] In one specific implementation, when the highest similarity is greater than or equal to the first similarity threshold, the face corresponding to the current face feature vector is clustered into the existing person cluster corresponding to the representative face feature vector.

[0085] It should be noted that the above is only an optional example and should not be construed as a limitation of the present invention.

[0086] Next, in step S104, when the calculated similarity is less than the first similarity threshold, a new face identifier is re-established to update the face feature database.

[0087] Specifically, when the calculated highest similarity is less than or equal to the first similarity threshold, a new person cluster is established based on the face corresponding to the current face feature vector in order to update the face feature database.

[0088] In a preferred embodiment, when determining to establish a new person cluster, a multi-view diversity judgment is performed on the current face feature vector. Specifically, the current face feature vector and the representative face feature vector with the highest similarity to the current face feature vector are combined and used as model input to input into the diversity discrimination model to obtain a difference evaluation value.

[0089] A diversity discrimination model is established using a convolutional neural network algorithm.

[0090] Specifically, a convolutional neural network algorithm was employed, and multiple face images with significant differences in angle were carefully selected from an existing face image sample database. Images of the same individual (i.e., the same person), taken from the same angle, and with low visual differences were paired and defined as negative samples. Conversely, images of the same person taken from different angles and with high visual differences were paired and defined as positive samples. These positive and negative samples were used to build a training dataset for training the diversity discrimination model. The diversity discrimination model was then trained using this training dataset, resulting in a well-trained diversity discrimination model.

[0091] The process involves combining the current facial feature vector with the representative facial feature vector that has the highest similarity to the current facial feature vector. For example, this can be done by directly concatenating them, by weighted multiplication, or by weighted addition.

[0092] Input the model obtained from the concatenation process into M[v1, v2], input it into the diversity discrimination model, and obtain the difference assessment value σ(M):

[0093] ,

[0094] Where σ(M) represents the facial difference between the current facial feature vector and the representative facial feature vector with the highest similarity to the current facial feature vector, and M represents the vector matrix obtained after concatenating the current facial feature vector and the representative facial feature vector with the highest similarity to the current facial feature vector, i.e., the input of the neural network of the diversity discrimination model; e -CNN(M) denoted as e raised to the power of -CNN(M), its function is to convert CNN(M) into a non-linear range. When CNN(M) is positive, the closer the value of -CNN(M) is to 0, the closer σ(M) is to 1; when CNN(M) is negative, the closer the value of -CNN(M) is to 1, the closer σ(M) is to 0, where CNN(M) represents the output of the neural network of the diversity discrimination model.

[0095] In the training process of the diversity discrimination model, the binary cross-entropy loss function can be used to effectively measure the model's performance.

[0096] The binary cross-entropy loss function Loss is expressed by the following expression.

[0097] ,

[0098] Where Loss represents the binary cross-entropy loss function of the diversity discriminant model; N is the number of samples in the training dataset; y i It is the true label of the i-th sample data, which in this invention is, for example, 0 or 1; It is the predicted value (e.g., the predicted probability) of the i-th sample data.

[0099] Specifically, the sigmoid activation function of the diversity discrimination model outputs the difference evaluation value σ(M) between the current face feature vector and the representative face feature vector with the highest similarity to the current face feature vector.

[0100] By using a diversity discrimination model built with a convolutional network structure, it is possible to learn and capture the complex similarities and differences between faces at different angles, so as to accurately judge the complex similarities and differences between the current face image and the existing face images in the face feature database, thereby effectively promoting the face clustering task to achieve higher accuracy and robustness.

[0101] Next, based on the obtained difference assessment value (i.e. whether the output difference assessment value is greater than the set threshold), it is determined whether to establish a new character cluster.

[0102] When the obtained difference assessment value is greater than the set threshold (i.e., as shown below) Figure 3 The "Yes" option indicates that a new character cluster will be established when the output difference assessment value is greater than the set value.

[0103] When the obtained difference assessment value is less than or equal to a set threshold, the quality of the face image corresponding to the current face feature vector is further judged to determine whether to replace the existing face image corresponding to the representative face feature vector with the highest similarity to the current face feature vector (i.e., the corresponding face image). Figure 3 The "Determine whether to replace the existing face image" shown in the image.

[0104] It should be noted that the threshold values ​​are determined based on factors such as differences in local areas of the image, differences in the mouth, nose, hair, or other specified areas of a face, image clarity, and image quality. The above are merely optional examples and should not be construed as limiting the invention.

[0105] In one specific implementation, if the obtained difference evaluation value is less than or equal to a set threshold, and the quality of the face image corresponding to the current person's feature vector is lower than the current quality (specifically, the current quality of the existing face image corresponding to the representative face feature vector with the highest similarity to the current person's feature vector), then it is not stored in the face feature vector database, i.e., it is deleted. Conversely, if the quality of the face image corresponding to the current person's feature vector is higher than the current quality, then the existing face image corresponding to the representative face feature vector with the highest similarity to the current person's feature vector is replaced with the face image corresponding to the current person's feature vector. If the obtained difference evaluation value is greater than the set threshold, and the quality meets the specified requirements, then it is stored as a new person cluster in the face feature vector database.

[0106] It should be noted that for diversity assessment of multi-view facial feature vectors, if a feature vector is determined to be clustered into an existing person cluster, a convolutional neural network (CNN) model will be used to perform diversity assessment. The input to the CNN includes the current person's feature vector and the feature vector with the highest similarity to it, aiming for 0-1 binary classification. This process primarily quantifies the difference between the two feature vectors. If the difference between the two feature vectors is low, the output is 0, meaning the two features are visually similar, and therefore this facial feature vector will not be stored in the representative facial feature database. Conversely, if the difference between the two feature vectors is large, the output is 1, representing a significant visual difference, and the current person's feature vector will be stored in the representative facial feature database.

[0107] The above methods ensure that the facial features in the facial feature database have sufficient multi-angle and diversity, while avoiding the storage of overly similar features. This improves the robustness and accuracy of facial recognition methods, enables precise and dynamic updates to the facial feature vector database, increases the diversity of representative facial feature vectors, and enhances clustering confidence. Furthermore, by storing only representative facial feature vectors, storage space requirements are effectively reduced.

[0108] Next, in step S105, when the calculated similarity is greater than or equal to the first similarity threshold, it is further determined whether the output face confidence score is greater than the confidence score threshold in order to determine whether to update the face feature database.

[0109] Specifically, when it is further determined that the confidence score of the output face is greater than the confidence score threshold, the existing face image corresponding to the representative face feature vector with the highest similarity to the current face feature vector is replaced with the face image corresponding to the current face feature vector in order to update the face feature database.

[0110] If the output face confidence score is less than or equal to the confidence score threshold, the face image corresponding to the current face feature vector will be deleted, and the face feature database will not be updated.

[0111] It should be noted that since the calculation and determination process of the confidence threshold corresponding to the face confidence threshold in step S105 is roughly the same as the calculation and determination process of the confidence threshold corresponding to the face confidence threshold in step S101, the explanation of the same part is omitted.

[0112] Furthermore, the accompanying drawings are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention and are not intended to be limiting. It is readily understood that the processes shown in the drawings do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0113] Compared with existing technologies, this invention, through a feature extraction model established by a sophisticated convolutional neural network structure, can effectively extract the current facial feature vector of the detected face image, transforming the original image data of the image to be processed into a feature vector that can accurately describe facial attributes. The output current facial feature vector is then compared with representative facial feature vectors in a pre-established facial feature database to calculate similarity. This comparison with a first similarity threshold allows for precise and dynamic updates to the facial feature database, ensuring that the facial features in the database have sufficient multi-angle and diversity while avoiding the storage of overly similar features. Furthermore, it improves the robustness and accuracy of the face recognition method.

[0114] Furthermore, the diversity discrimination model established by using a convolutional network structure can learn and capture the complex similarities and differences between faces at different angles, so as to accurately judge the complex similarities and differences between the current face image and the existing face images in the face feature database, thereby further promoting the face clustering task to achieve higher accuracy and robustness.

[0115] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the embodiments of the apparatus of the present invention, please refer to the embodiments of the method of the present invention.

[0116] Figure 4 This is a schematic diagram of an example of a face clustering and updating device according to the present invention.

[0117] Reference Figure 4 The second aspect of this disclosure provides a face clustering update device 400, which employs the face clustering update method described in the first aspect of this invention. The face clustering update device 400 includes a receiving and processing module 410, an output module 420, a similarity calculation module 430, a first update processing module 440, and a second update processing module 450.

[0118] Specifically, the receiving and processing module 410 receives the image to be processed and, upon detecting facial features in the image, extracts a facial image. The output module 420 inputs the extracted facial image into the feature extraction model and outputs the current facial feature vector and facial confidence score corresponding to the facial image. The similarity calculation module 430 calculates the similarity between the output current facial feature vector and representative facial feature vectors in a pre-established facial feature database. The first update processing module 440, when the calculated similarity is less than a first similarity threshold, re-establishes a new facial identifier to update the facial feature database. The second update processing module 450, when the calculated similarity is greater than or equal to the first similarity threshold, further determines whether the output facial confidence score is greater than a confidence threshold to determine whether to update the facial feature database.

[0119] In one specific implementation, the image to be processed is input into a deep learning model to obtain the positional coordinates of all faces in the image and the face confidence score corresponding to each face. Based on the positional coordinates of each face and the face confidence score corresponding to each face, face images are extracted.

[0120] In an optional embodiment, the system further includes a judgment processing module. This module retrieves a representative face feature vector from a pre-established face feature database that has the highest similarity to the output current face feature vector, and determines whether this highest similarity is greater than a first similarity threshold. If the highest similarity is greater than or equal to the first similarity threshold, the face corresponding to the current face feature vector is clustered into an existing person cluster corresponding to the representative face feature vector. If the highest similarity is less than the first similarity threshold, a new person cluster is established based on the face corresponding to the current face feature vector to update the face feature database.

[0121] In one optional implementation, when determining to establish the new person cluster, the current face feature vector is subjected to multi-view diversity judgment. The current face feature vector and the representative face feature vector with the highest similarity to the current face feature vector are combined and used as model input to the diversity discrimination model to obtain the difference evaluation value.

[0122] Specifically, a convolutional neural network algorithm is used to establish a diversity discrimination model.

[0123] Input the model obtained from the concatenation process into M[v1, v2], input it into the diversity discrimination model, and obtain the difference assessment value σ(M):

[0124] ,

[0125] Where σ(M) represents the facial difference between the current facial feature vector and the representative facial feature vector with the highest similarity to the current facial feature vector, and M represents the vector matrix obtained after concatenating the current facial feature vector and the representative facial feature vector with the highest similarity to the current facial feature vector, i.e., the input of the neural network of the diversity discrimination model; e -CNN(M) denoted as e raised to the power of -CNN(M), its function is to convert CNN(M) into a non-linear range. When CNN(M) is positive, the closer the value of -CNN(M) is to 0, the closer σ(M) is to 1; when CNN(M) is negative, the closer the value of -CNN(M) is to 1, the closer σ(M) is to 0, where CNN(M) represents the output of the neural network of the diversity discrimination model.

[0126] In one optional implementation, based on the obtained difference evaluation value, it is determined whether to establish a new person cluster. If the obtained difference evaluation value is greater than a set threshold, it is determined to establish a new person cluster. If the obtained difference evaluation value is less than or equal to the set threshold, the quality of the face image corresponding to the current face feature vector is further judged to determine whether to replace the existing face image corresponding to the representative face feature vector with the highest similarity to the current face feature vector.

[0127] In one optional implementation, a confidence threshold is calculated by applying a set of convolutional and fully connected layers to the detected face portion, with the Lth layer of the convolutional layer outputting h. L Represented as:

[0128] h L =ReLU(W L * h L-1 +b L ),

[0129] Among them, W L and b L These are the weights and biases of the Lth convolutional layer, respectively; * indicates the convolution operation; ReLU is the activation function, h L-1 It is the output of the previous layer;

[0130] The output f of the k-th fully connected layer k Represented as:

[0131] f k =ReLU(W k * f k-1 +b k ),

[0132] Among them, W k and b k These are the weights and biases of the k-th fully connected layer, and ReLU is the activation function, f.k-1 It is the output of the previous layer;

[0133] The output of the fully connected layer is passed to the single-neuron output layer. The output of a single neuron output layer is z. The sigmoid activation function is used to convert the output into a probability between 0 and 1, and the confidence score σ(z) is calculated.

[0134] z = W0* f k + b0,

[0135] ,

[0136] Where z represents the output of a single neuron in the output layer; W0 and b0 are the weights and biases of the output layer, respectively; f k This represents the output of the k-th layer of the fully connected layer. The purpose of representing e to the power of -z is to convert z into a non-linear range; when z is positive, The closer the value of z is to 0, the closer σ(z) is to 1; when z is negative, The closer the value is to 1, the closer σ(z) is to 0.

[0137] In one optional implementation, the determination of the first similarity threshold specifically includes the following steps: selecting candidate thresholds; evaluating candidate thresholds; selecting the optimal threshold; and testing candidate thresholds to calculate the first similarity threshold.

[0138] In an optional implementation, when the calculated similarity is greater than or equal to a first similarity threshold, further determining whether the output face confidence score is greater than a confidence threshold to determine whether to update the face feature database includes the following steps:

[0139] If the confidence score of the output face is greater than the confidence score threshold, the existing face image corresponding to the representative face feature vector with the highest similarity to the current face feature vector will be replaced with the face image corresponding to the current face feature vector to update the face feature database.

[0140] It should be noted that, in this embodiment, the face clustering update method performed by the face clustering update device is the same as... Figure 1 The face clustering update methods are largely the same, so the explanations of the common parts are omitted.

[0141] Figure 5 This is a schematic diagram of an embodiment of an electronic device according to the present invention.

[0142] like Figure 5As shown, the electronic device is embodied in the form of a general-purpose computing device. There can be one or more processors working collaboratively. This invention also does not preclude distributed processing, meaning that processors can be distributed across different physical devices. The electronic device of this invention is not limited to a single entity, but can also be the sum of multiple physical devices.

[0143] The memory stores a computer-executable program, typically machine-readable code. The computer-readable program can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least some steps of the method.

[0144] The memory includes volatile memory, such as random access memory (RAM) and / or cache memory, and may also be non-volatile memory, such as read-only memory (ROM).

[0145] Optionally, in this embodiment, the electronic device further includes an I / O interface for exchanging data with external devices. The I / O interface can represent one or more of several bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0146] It should be understood that Figure 5 The electronic device shown is merely one example of the present invention, and the electronic device of the present invention may also include elements or components not shown in the above examples. For example, some electronic devices also include display units such as displays, and some electronic devices also include human-computer interaction elements such as buttons and keyboards. Any electronic device capable of executing a computer-readable program in memory to implement the method of the present invention or at least some steps of the method can be considered as an electronic device covered by the present invention.

[0147] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software, or by combining software with necessary hardware. Therefore, as... Figure 6 As shown, the technical solution according to the embodiments of the present invention can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.) or on a network, and includes several commands to cause a computing device (such as a personal computer, server, or network device, etc.) to execute the above-described method according to the embodiments of the present invention.

[0148] The software product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0149] The computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting programs for use by or in connection with a command execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0150] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0151] The aforementioned computer-readable medium carries one or more programs (e.g., computer-executable programs) that, when executed by a device, enable the computer-readable medium to implement the data interaction method of this disclosure.

[0152] Those skilled in the art will understand that the above modules can be distributed in the device as described in the embodiments, or they can be modified accordingly and placed in one or more devices that are unique to this embodiment. The modules in the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0153] Through the description of the above embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions of the embodiments of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, portable hard drive, etc.) or on a network, including several commands to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of the present invention.

[0154] Exemplary embodiments of the present invention have been specifically shown and described above. It should be understood that the present invention is not limited to the detailed structures, arrangements, or implementations described herein; rather, the present invention is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. A face clustering update method, characterized in that, Includes the following steps: Receive the image to be processed, and when facial features are detected in the image to be processed, extract the facial image; The captured face image is input into the feature extraction model, which outputs the current face feature vector and face confidence corresponding to the face image. The similarity between the output current face feature vector and the representative face feature vector in the pre-established face feature database is calculated. When the calculated similarity is less than the first similarity threshold, a new face identifier is re-established to update the face feature database. This includes establishing new person clusters based on the faces corresponding to the current face feature vectors to update the face feature database; when determining to establish the new person clusters, performing multi-view diversity judgment on the current face feature vectors, combining the current face feature vectors with the representative face feature vectors that have the highest similarity to the current face feature vectors, and using this as input to the diversity discrimination model to obtain a difference evaluation value; specifically, this includes using a convolutional neural network algorithm to establish a diversity discrimination model; Input the model obtained from the concatenation process into M[v1, v2], input it into the diversity discrimination model, and obtain the difference assessment value σ(M): Where σ(M) represents the quantified facial difference between the current facial feature vector and the representative facial feature vector with the highest similarity to the current facial feature vector, i.e., the difference evaluation value; where M represents the vector matrix obtained by concatenating the current facial feature vector and the representative facial feature vector with the highest similarity to the current facial feature vector, and also serves as the input to the neural network of the diversity discrimination model; e -CNN(M) σ(M) represents e raised to the power of -CNN(M), which transforms CNN(M) into a non-linear range. When CNN(M) is positive, the closer -CNN(M) is to 0, the closer σ(M) is to 1; when CNN(M) is negative, the closer -CNN(M) is to 1, the closer σ(M) is to 0. Here, CNN(M) represents the output of the neural network of the diversity discrimination model. Based on the obtained difference evaluation value, it is determined whether to establish a new character cluster. If the obtained difference evaluation value is greater than a set threshold, it is determined to establish a new character cluster. When the calculated similarity is greater than or equal to the first similarity threshold, it is further determined whether the output face confidence score is greater than the confidence score threshold in order to determine whether to update the face feature database.

2. The face clustering update method according to claim 1, characterized in that, It also includes the following steps: The system retrieves the representative face feature vector with the highest similarity to the output current face feature vector from a pre-established face feature database, and determines whether the highest similarity is greater than a first similarity threshold. When the highest similarity is greater than or equal to the first similarity threshold, the face corresponding to the current face feature vector is clustered into the existing person cluster corresponding to the representative face feature vector; When the highest similarity is less than the first similarity threshold, a new person cluster is established based on the face corresponding to the current face feature vector to update the face feature database.

3. The face clustering update method according to claim 1, characterized in that, It also includes the following steps: When the obtained difference evaluation value is less than or equal to the set threshold, the quality of the face image corresponding to the current face feature vector is further judged to determine whether to replace the existing face image corresponding to the representative face feature vector with the highest similarity to the current face feature vector.

4. The face clustering update method according to claim 1, characterized in that, It also includes the following steps: The confidence threshold is calculated as follows: The detected face region is processed using a set of convolutional and fully connected layers. The Lth convolutional layer outputs h. L Represented as: h L =ReLU(W L *h L-1 +b L ), Among them, W L and b L These are the weights and biases of the Lth convolutional layer, respectively; * indicates the convolution operation; ReLU is the activation function, h L-1 It is the output of the previous layer; The output f of the k-th fully connected layer k Represented as: f k =ReLU(W k *f k-1 +b k ), Among them, W k and b k These are the weights and biases of the k-th fully connected layer, and ReLU is the activation function, f. k-1 It is the output of the previous layer; The output of the fully connected layer is passed to the single-neuron output layer. The output of a single neuron output layer is z. The sigmoid activation function is used to convert the output into a probability between 0 and 1, and the confidence score σ(z) is calculated. z=W0*f K +b0, Where z represents the output of a single neuron in the output layer; W0 and b0 are the weights and biases of the output layer, respectively; f k This represents the output of the k-th layer of the fully connected layer, e. -z The purpose of e raised to the power of -z is to convert z into a non-linear range; when z is positive, e -z The closer the value of σ(z) is to 0, the closer σ(z) is to 1; when z is negative, e -z The closer the value is to 1, the closer σ(z) is to 0.

5. The face clustering update method according to claim 1, characterized in that, It also includes the following steps: The determination of the first similarity threshold includes the following steps: Select candidate thresholds; Evaluate candidate thresholds; Select the optimal candidate threshold; The optimal candidate threshold selected by the test is used to calculate the first similarity threshold.

6. The face clustering update method according to claim 1 or 2, characterized in that, When the calculated similarity is greater than or equal to the first similarity threshold, the next step is to determine whether the output face confidence score is greater than the confidence threshold in order to decide whether to update the face feature database. This includes the following steps: If the confidence score of the output face is greater than the confidence score threshold, the existing face image corresponding to the representative face feature vector with the highest similarity to the current face feature vector will be replaced with the face image corresponding to the current face feature vector to update the face feature database.

7. The face clustering update method according to claim 1, characterized in that, The process of receiving the image to be processed and, upon detecting facial features in the image, extracting the facial image includes the following steps: The image to be processed is input into a deep learning model to obtain the location coordinates of all faces in the image and the face confidence score corresponding to each face. Based on the location coordinates of each face and the face confidence level corresponding to each face, the face image is extracted.

8. A face clustering and updating device, used to implement the face clustering and updating method as described in claim 1, characterized in that, The face clustering update device includes: The receiving and processing module is used to receive the image to be processed and to extract the face image when facial features are detected in the image to be processed. The output module is used to input the captured face image into the feature extraction model and output the current face feature vector and face confidence corresponding to the face image. The similarity calculation module is used to calculate the similarity between the output current face feature vector and the representative face feature vectors in the pre-established face feature database. The first update processing module re-establishes new face identifiers to update the face feature database when the calculated similarity is less than the first similarity threshold; this includes establishing new person clusters based on the faces corresponding to the current face feature vectors to update the face feature database. The second update processing module, when the calculated similarity is greater than or equal to the first similarity threshold, further determines whether the output face confidence score is greater than the confidence score threshold in order to determine whether to update the face feature database.

Images