Clustering model training method, image clustering method, device and equipment

Abstract

The embodiment of the invention provides a clustering model training method, an image clustering method, an image clustering device and image clustering equipment, and relates to the technical field of deep learning. According to the specific implementation scheme, a training data set is obtained; wherein the training data set comprises a plurality of face images; grouping the face images in the training data set to obtain multiple groups of face images; wherein the number of any group of face images is a first preset number; for each group of face images, processing the group of face images by using a specified clustering model to obtain a first similarity matrix output by the specified clustering model; and for each group of face images, based on the difference between the first similarity matrix corresponding to the group of face images and the similarity labels corresponding to the image pairs in the group of face images obtained by pre-calculation, adjusting the model parameters of the specified clustering model. Visibly, through the scheme, the clustering model can be trained by using the large-scale face image under the condition of limited equipment resources.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a clustering model training method, image clustering method, device and equipment. Background Art

[0002] In scenarios such as smart security, facial images captured by cameras usually need to be clustered according to the people they belong to for the purpose of personnel archiving.

[0003] Currently, when clustering multiple facial images, features are usually extracted from each facial image first to obtain a feature vector corresponding to each facial image; then, the feature vector corresponding to each facial image is input into a clustering model that can calculate the correlation between multiple feature vectors to obtain a clustering result.

[0004] However, the clustering model used in the above clustering method often requires a large amount of memory during training. That is, when the number of facial images is huge, for example, reaching millions or hundreds of millions, the model training needs to input the feature vectors corresponding to all facial images at one time, resulting in the training device requiring a large amount of memory to accommodate the feature vectors corresponding to all facial images at one time.

[0005] Therefore, how to use large-scale facial images to train clustering models when device resources are limited is a technical problem that needs to be solved urgently. Summary of the invention

[0006] The purpose of the embodiments of the present application is to provide a clustering model training method, device and equipment, so as to train the clustering model using a large number of facial images when the equipment resources are limited; in addition, the embodiments of the present application also provide an image clustering method to improve the accuracy of the clustering results. The specific technical solutions are as follows:

[0007] In a first aspect, an embodiment of the present application provides a clustering model training method, the method comprising:

[0008] Obtain a training data set; wherein the training data set includes a plurality of facial images;

[0009] Grouping the facial images in the training data set to obtain a plurality of groups of facial images; wherein the number of facial images in any group is a first preset number;

[0010] For each group of facial images, the specified clustering model is used to process the group of facial images to obtain a first similarity matrix output by the specified clustering model; wherein the first similarity matrix is ​​a square matrix having the first preset number of rows, and each element in the first similarity matrix represents the similarity of an image pair in the group of facial images; the similarity of any image pair represents the similarity between the two images included in the image pair;

[0011] For each group of facial images, the model parameters of the specified clustering model are adjusted based on the difference between the first similarity matrix corresponding to the group of facial images and the similarity labels corresponding to the image pairs in the group of facial images calculated in advance; wherein the similarity labels corresponding to the image pairs in the group of facial images are obtained from a pre-constructed similarity label set; and the similarity labels in the similarity label set are obtained based on normalizing the similarities of each image pair in the training data set.

[0012] Optionally, the designated clustering model is a visual transformation model structure ViT model, and the ViT model has a second preset number of input nodes;

[0013] The method of processing each group of facial images by using a specified clustering model to obtain a first similarity matrix output by the specified clustering model includes:

[0014] For each facial image in the group of facial images, extracting image regions occupied by a second preset number of designated facial features from the facial image to obtain a partial image corresponding to each designated facial feature in each facial image;

[0015] The local image corresponding to the same designated facial features in each facial image is used as input data of an input node and input into the ViT model to obtain a first similarity matrix corresponding to the group of facial images.

[0016] Optionally, the similarity tag set is constructed in the following manner:

[0017] Extracting features from each facial image in the training data set to obtain a feature vector corresponding to each facial image;

[0018] Calculate the similarity between the feature vectors corresponding to each two facial images to obtain the similarity of the image pair formed by the two facial images;

[0019] Based on normalizing the similarity of each image pair, a similarity label of each image pair is generated to obtain a similarity label set.

[0020] Optionally, the generating a similarity label for each image pair based on normalizing the similarity of each image pair includes:

[0021] Normalize the similarity of each image pair to obtain the normalized result corresponding to each image pair;

[0022] For each image pair, if the identities of the persons included in the facial images in the image pair are the same, the normalized result corresponding to the image pair is adjusted using the first penalty factor, and the adjusted result is determined as the similarity label of the image pair;

[0023] For each image pair, if the identities of the persons included in the facial images in the image pair are different, the normalized result corresponding to the image pair is adjusted using the second penalty factor, and the adjusted result is determined as the similarity label of the image pair;

[0024] The first penalty factor increases the adjusted result, and the second penalty factor decreases the adjusted result.

[0025] Optionally, the normalizing the similarities of each image pair to obtain a normalized result corresponding to each image pair includes:

[0026] Based on the maximum and minimum values ​​of the similarities of each image pair, the similarities of each image pair are normalized to the interval (-1, 1) to obtain the normalized results corresponding to each image pair.

[0027] In a second aspect, an embodiment of the present application provides an image clustering method, the method comprising:

[0028] Obtain multiple facial images to be clustered;

[0029] Processing the acquired multiple facial images using a pre-trained designated clustering model to obtain a second similarity matrix; wherein the designated clustering model is trained based on any of the clustering model training methods described above;

[0030] Based on a predetermined designated threshold and the second similarity matrix, a plurality of facial images are clustered to obtain a clustering result.

[0031] Optionally, clustering the plurality of facial images based on a predetermined specified threshold and the second similarity matrix to obtain a clustering result includes:

[0032] The image pairs corresponding to the elements in the second similarity matrix that are not less than the specified threshold are determined as image pairs belonging to the same identity, and a clustering result representing the same identity is obtained.

[0033] Optionally, the method for determining the specified threshold includes:

[0034] Using the specified clustering model, a first preset number of facial images in a pre-acquired test data set are processed to obtain a third similarity matrix;

[0035] For each candidate threshold, determining the image pairs corresponding to the elements in the third similarity matrix that are not less than the candidate threshold as image pairs belonging to the same identity;

[0036] Calculate the accuracy of the image pairs belonging to the same identity determined based on each candidate threshold;

[0037] The candidate threshold with the highest accuracy is determined as the designated threshold.

[0038] Optionally, the candidate threshold is selected from a specified interval according to a preset interval; the specified interval is a value range of labels in a similarity label set used when training the specified clustering model.

[0039] In a third aspect, an embodiment of the present application provides a clustering model training device, the device comprising:

[0040] A first acquisition module is used to acquire a training data set; wherein the training data set includes a plurality of facial images;

[0041] A grouping module, used for grouping the facial images in the training data set to obtain a plurality of groups of facial images; wherein the number of facial images in any group is a first preset number;

[0042] A first processing module is used to process each group of facial images using a specified clustering model to obtain a first similarity matrix output by the specified clustering model; wherein the first similarity matrix is ​​a square matrix having the first preset number of rows, and each element in the first similarity matrix represents the similarity of an image pair in the group of facial images; the similarity of any image pair represents the similarity between the two images included in the image pair;

[0043] The adjustment module is used to adjust the model parameters of the specified clustering model for each group of facial images based on the difference between the first similarity matrix corresponding to the group of facial images and the similarity labels corresponding to the image pairs in the group of facial images calculated in advance; wherein the similarity labels corresponding to the image pairs in the group of facial images are obtained from a pre-constructed similarity label set; and the similarity labels in the similarity label set are obtained based on normalizing the similarities of each image pair in the training data set.

[0044] Optionally, the designated clustering model is a visual transformer ViT model, and the ViT model has a second preset number of input nodes;

[0045] The first processing module comprises:

[0046] an extraction submodule, for extracting, for each facial image in the group of facial images, image regions occupied by a second preset number of designated facial features from the facial image, to obtain a local image corresponding to each designated facial feature in each facial image;

[0047] The input submodule is used to input the local image corresponding to the same designated facial features in each facial image as input data of an input node, input it into the ViT model, and obtain the first similarity matrix corresponding to the group of facial images.

[0048] Optionally, the similarity tag set is constructed in the following manner:

[0049] Extracting features from each facial image in the training data set to obtain a feature vector corresponding to each facial image;

[0050] Calculate the similarity between the feature vectors corresponding to each two facial images to obtain the similarity of the image pair formed by the two facial images;

[0051] Based on normalizing the similarity of each image pair, a similarity label of each image pair is generated to obtain a similarity label set.

[0052] Optionally, the generating a similarity label for each image pair based on normalizing the similarity of each image pair includes:

[0053] Normalize the similarity of each image pair to obtain the normalized result corresponding to each image pair;

[0054] For each image pair, if the identities of the persons included in the facial images in the image pair are the same, the normalized result corresponding to the image pair is adjusted using the first penalty factor, and the adjusted result is determined as the similarity label of the image pair;

[0055] For each image pair, if the identities of the persons included in the facial images in the image pair are different, the normalized result corresponding to the image pair is adjusted using the second penalty factor, and the adjusted result is determined as the similarity label of the image pair;

[0056] The first penalty factor increases the adjusted result, and the second penalty factor decreases the adjusted result.

[0057] Optionally, normalizing the similarity of each image pair to obtain a normalized result corresponding to each image pair includes:

[0058] Based on the maximum and minimum values ​​of the similarities of each image pair, the similarities of each image pair are normalized to the interval (-1, 1) to obtain the normalized results corresponding to each image pair.

[0059] In a fourth aspect, an embodiment of the present application provides an image clustering device, the device comprising:

[0060] A second acquisition module is used to acquire multiple facial images to be clustered;

[0061] A second processing module is used to process the acquired multiple facial images using a pre-trained designated clustering model to obtain a second similarity matrix; wherein the designated clustering model is trained based on any of the clustering model training methods described above;

[0062] The clustering module is used to cluster the multiple facial images based on a predetermined specified threshold and the second similarity matrix to obtain a clustering result.

[0063] Optionally, the clustering module is specifically used to:

[0064] The image pairs corresponding to the elements in the second similarity matrix that are not less than the specified threshold are determined as image pairs belonging to the same identity, and a clustering result representing the same identity is obtained.

[0065] Optionally, the method for determining the specified threshold includes:

[0066] Inputting a first preset number of facial images in a pre-acquired test data set into the specified clustering model to obtain a third similarity matrix;

[0067] For each candidate threshold, determining the image pairs corresponding to the elements in the third similarity matrix that are not less than the candidate threshold as image pairs belonging to the same identity;

[0068] Calculate the accuracy of the image pairs belonging to the same identity determined based on each candidate threshold;

[0069] The candidate threshold with the highest accuracy is determined as the designated threshold.

[0070] Optionally, the candidate threshold is selected from a specified interval according to a preset interval; the specified interval is a value range of labels in a similarity label set used when training the specified clustering model.

[0071] In a fifth aspect, an embodiment of the present application provides an electronic device, including:

[0072] Memory, used to store computer programs;

[0073] The processor is used to implement any of the above-mentioned clustering model training methods or image clustering methods when executing the program stored in the memory.

[0074] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the training method of the clustering model described in any one of the above items, or the image clustering method, is implemented.

[0075] In a seventh aspect, an embodiment of the present application provides a computer program product, which includes executable instructions. When the executable instructions are executed on a computer, the computer executes any one of the clustering model training methods described above, or the image clustering method.

[0076] Beneficial effects of the embodiments of the present application:

[0077] The solution provided by the embodiment of the present application is to group the facial images in the training data set, and for each group of facial images, use the specified clustering model to process the group of facial images, obtain the first similarity matrix output by the specified clustering model, and adjust the model parameters of the specified clustering model based on the difference between the first similarity matrix corresponding to the group of facial images and the similarity labels corresponding to the image pairs in the group of facial images calculated in advance. In this way, the training data set can be trained in batches using the specified clustering model, so that the device used to train the specified clustering model only needs to put one group of facial images into the memory for training at a time, which significantly reduces the memory usage, so that the model training can be carried out in an environment with limited device resources. In addition, since the similarity label set is obtained based on the normalization of the similarity of each image pair in the training data set, the similarity labels corresponding to the image pairs in each group of facial images have a unified range, so that the distribution of the similarity labels corresponding to each group of facial images is more uniform, thereby avoiding the problem of model performance degradation caused by the uneven distribution of the similarity labels corresponding to each group of facial images. It can be seen that through this solution, a large-scale facial image can be used to train the clustering model under limited device resources. Therefore, it is possible to adapt the training of ultra-large-scale training data using limited device resources.

[0078] In addition, the image clustering method provided in the embodiment of the present application is not limited by device resources when training the specified clustering model according to the above-mentioned clustering model training method. Therefore, it is possible to use a large-scale training data set to train a specified clustering model with higher accuracy. The specified clustering model trained by the clustering model training method can be used to process multiple facial images to be clustered, thereby improving the accuracy of the clustering results.

[0079] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0081] Figure 1 A flowchart of a clustering model training method provided in an embodiment of the present application;

[0082] Figure 2 A flowchart of a method for constructing a similarity tag set provided in an embodiment of the present application;

[0083] Figure 3 A flowchart of an image clustering method provided in an embodiment of the present application;

[0084] Figure 4 A flowchart of a specific example of the image clustering method provided in an embodiment of the present application;

[0085] Figure 5 A schematic diagram of the structure of a clustering model training device provided in an embodiment of the present application;

[0086] Figure 6 A schematic diagram of the structure of an image clustering device provided in an embodiment of the present application;

[0087] Figure 7 A block diagram of an electronic device for implementing the method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0088] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0089] First, the professional terms involved in the embodiments of the present application are introduced.

[0090] Face clustering: a technique for classifying large amounts of unlabeled face images based on specific facial feature information;

[0091] ViT (Vision Transformer) model: a Transformer structure that uses a convolutional kernel sliding pane algorithm to extract image features;

[0092] Transformer structure: a deep learning neural network structure for processing sequential inputs.

[0093] Next, a clustering model training method provided in an embodiment of the present application is introduced.

[0094] The training method of a clustering model provided in the embodiment of the present application can be applied to various electronic devices, such as cloud servers, server clusters, and other devices with data processing capabilities. In addition, it is understood that the data storage method provided in the embodiment of the present application can be implemented by software, hardware, or a combination of software and hardware.

[0095] Among them, a clustering model training method provided in an embodiment of the present application may include:

[0096] Obtain a training data set; wherein the training data set includes multiple facial images;

[0097] Grouping the facial images in the training data set to obtain a plurality of groups of facial images; wherein the number of facial images in any group is a first preset number;

[0098] For each group of facial images, the group of facial images is processed using a specified clustering model to obtain a first similarity matrix output by the specified clustering model; wherein the first similarity matrix is ​​a square matrix having a first preset number of rows, and each element in the first similarity matrix represents the similarity of an image pair in the group of facial images; the similarity of any image pair represents the similarity between the two images included in the image pair;

[0099] For each group of facial images, model parameters of a specified clustering model are adjusted based on a difference between a first similarity matrix corresponding to the group of facial images and similarity labels corresponding to image pairs in the group of facial images calculated in advance; wherein the similarity labels corresponding to the image pairs in the group of facial images are obtained from a pre-constructed similarity label set; and the similarity labels in the similarity label set are obtained based on normalizing the similarities of each image pair in a training data set.

[0100] The solution provided by the embodiment of the present application is to group the facial images in the training data set, and for each group of facial images, use the specified clustering model to process the group of facial images, obtain the first similarity matrix output by the specified clustering model, and adjust the model parameters of the specified clustering model based on the difference between the first similarity matrix corresponding to the group of facial images and the similarity labels corresponding to the image pairs in the group of facial images calculated in advance. In this way, the training data set can be trained in batches using the specified clustering model, so that the device used to train the specified clustering model only needs to put one group of facial images into the memory for training at a time, which significantly reduces the memory usage, so that the model training can be carried out in an environment with limited device resources. In addition, since the similarity label set is obtained based on the normalization of the similarity of each image pair in the training data set, the similarity labels corresponding to the image pairs in each group of facial images have a unified range, so that the distribution of the similarity labels corresponding to each group of facial images is more uniform, thereby avoiding the problem of model performance degradation caused by the uneven distribution of the similarity labels corresponding to each group of facial images. It can be seen that through this solution, a large-scale facial image can be used to train the clustering model under limited device resources. Therefore, it is possible to adapt the training of ultra-large-scale training data using limited device resources.

[0101] The following is an introduction to the training method of the clustering model provided in the embodiment of the present application in conjunction with the accompanying drawings.

[0102] like Figure 1 As shown, the clustering model training method provided in the embodiment of the present application includes steps S101-S104:

[0103] S101, obtaining a training data set; wherein the training data set includes a plurality of facial images;

[0104] In this embodiment, the multiple facial images included in the training data set can be facial images obtained from an open source facial image library on a website, or facial images collected by cameras in security scenarios, which are all reasonable. For example, in a security scenario, facial images can be collected by cameras at traffic intersections or cameras in access control systems, and the facial images collected by the cameras within a week or a month are used as training data sets. In addition, it is worth mentioning that in order to improve the training effect of the model, the number of facial images included in the training data set can reach millions or hundreds of millions.

[0105] It should be noted that the facial images in this embodiment may come from a public data set, or the acquisition of the facial images is authorized by the corresponding users; and the collection, storage, use, processing, etc. of the facial images comply with the relevant laws and regulations and do not violate public order and good morals.

[0106] S102, grouping the facial images in the training data set to obtain a plurality of groups of facial images; wherein the number of facial images in any group is a first preset number;

[0107] In this embodiment, in order to train the clustering model using a large number of facial images when the device resources are limited, the facial images in the training data set are first grouped, and each group contains a first preset number of facial images. The first preset number can be determined according to the size of the device resources of the device used in the model training. For example, if the number of facial images that can be processed at one time by the device resources of the device used in the model training is 10,000, then the first preset number contained in each group can be any number not exceeding 10,000.

[0108] S103, for each group of facial images, use a specified clustering model to process the group of facial images to obtain a first similarity matrix output by the specified clustering model; wherein the first similarity matrix is ​​a square matrix with a first preset number of rows, and each element in the first similarity matrix represents the similarity of an image pair in the group of facial images; the similarity of any image pair represents the similarity between two images included in the image pair;

[0109] In this embodiment, the designated clustering model may be a neural network model whose output layer is composed of a fully connected layer and a 1-dimensional convolutional layer. The output layer of the model may also include a feature extraction layer composed of a multi-layer convolutional neural network. The 1-dimensional convolutional layer includes a first preset number of convolution kernels, so that the output of the model is a square matrix with a first preset number of rows. Exemplarily, if the first preset number is 100, the 1-dimensional convolutional layer includes 100 convolution kernels, and after a group of facial images are input into the designated clustering model, a 100*100 square matrix is ​​output, which is the first similarity matrix.

[0110] Exemplarily, for each group of facial images, the first similarity matrix can be expressed as:

[0111]

[0112] Among them, S ij It represents the similarity between the i-th facial image and the j-th facial image in the group of images. i, j, and m are all positive integers, and the value range of i and j is [1, m].

[0113] Each element in the first similarity matrix represents the similarity of an image pair in the group of facial images, wherein the elements on the main diagonal represent the similarity of the image pair formed by the facial image and itself. 13 represents the similarity between the first and third facial images in the group of images; S11 Indicates the similarity between the first facial image in the group of images and the image pair itself.

[0114] In addition, it is worth mentioning that using the specified clustering model to process the group of facial images can be either inputting the group of facial images into the specified clustering model to process the group of facial images using the specified clustering model, or inputting the areas belonging to the facial features in the group of facial images into the specified clustering model to process the group of facial images using the specified clustering model, which are both reasonable.

[0115] Optionally, in one implementation, the clustering model is specified as a ViT model, and the ViT model has a second preset number of input nodes;

[0116] Accordingly, in the present implementation, in the above step S103, for each group of facial images, the group of facial images is processed using a specified clustering model to obtain a first similarity matrix output by the specified clustering model, including steps A1-A2:

[0117] A1, for each facial image in the group of facial images, extracting image regions occupied by a second preset number of designated facial features from the facial image to obtain a partial image corresponding to each designated facial feature in each facial image;

[0118] A2, taking the local image corresponding to the same designated facial features in each facial image as the input data of an input node, inputting it into the ViT model, and obtaining the first similarity matrix corresponding to the group of facial images.

[0119] It is understandable that the core idea of ​​the ViT model is to divide the image into patches of fixed size, then treat these patches as sequence data, and use the Transformer structure to calculate the attention mechanism. Through the attention mechanism, the ViT model can capture long-distance dependencies in the image while maintaining sensitivity to local features. In addition, when the training data set reaches a relatively large scale, the accuracy that the ViT model can achieve after training can surpass the convolutional neural network. Therefore, the ViT model is used to replace the convolutional neural network to extract facial features. After using a very large-scale training data set, a feature extraction layer with stronger feature extraction capabilities can be obtained.

[0120] Exemplarily, the second preset number may be 5, 10, etc., and the second preset number is not limited in the embodiment of the present application. In addition, the designated five sense organs may be the five sense organs selected by relevant technical personnel based on experience. For example, in one implementation, the second preset number is 4, and the designated five sense organs include: left eye + left eyebrow, right eye + right eyebrow, nose and mouth.

[0121] Exemplarily, the method of extracting the image area occupied by the second preset number of designated facial features from each facial image may be: according to the preset cropping area, the facial image is cropped to obtain the partial image corresponding to each designated facial feature in the facial image. For example, if the designated facial features include: left eye + left eyebrow, right eye + right eyebrow, nose and mouth, then for a facial image with a height of h and a width of w, the preset cropping area may include:

[0122] The area of ​​[0,h / 2] row and [0,w / 2] column is intercepted as patch1, which represents the image area occupied by the left eye + left eyebrow;

[0123] The area of ​​[0,h / 2] row and [w / 2,w] column is intercepted as patch2, which represents the image area occupied by the right eye + right eyebrow;

[0124] The area of ​​[h / 4,3h / 4] rows and [w / 4,3w / 4] columns is intercepted as patch3, which represents the image area occupied by the nose;

[0125] The area of ​​[h / 2,h] rows and [w / 4,3w / 4] columns is intercepted as patch4, which represents the image area occupied by the mouth.

[0126] It should be noted that the embodiments of the present application do not limit the method of extracting the image area occupied by the second preset number of specified facial features from each facial image. For example, in another implementation method, the positions of facial key areas, including eyebrows, eyes, nose, mouth, etc., can be marked by a facial key point marking method. According to the marked facial key area positions, the image area occupied by the specified facial features can be intercepted, which is reasonable.

[0127] After obtaining the local image corresponding to each designated facial feature in each facial image through step A1, the local image corresponding to the same designated facial feature in each facial image is used as the input data of an input node and input into the ViT model. For example, if the number of a group of facial images is m and the second preset number is 4, after extracting 4 patches corresponding to the designated facial features from each facial image according to the above-mentioned preset cropping area, the m patches 1 are stacked in sequence to form an input matrix of h / 2×w / 2×3m as the input data of input node 1; the m patches 2 are stacked in sequence to form an input matrix of h / 2×w / 2×3m as the input data of input node 2, and so on, to form a complete input of h / 2×w / 2×3m×4. Among them, 3m represents m facial images with three RGB (Red Green Blue) channels.

[0128] In addition, the output layer of the ViT model is designed as a network structure consisting of a fully connected layer and a one-dimensional convolutional layer, wherein the one-dimensional convolutional layer includes a first preset number of convolution kernels so that the output of the ViT model is a square matrix.

[0129] S104, for each group of facial images, based on the difference between a first similarity matrix corresponding to the group of facial images and similarity labels corresponding to image pairs in the group of facial images obtained in advance, adjusting model parameters of the specified clustering model; wherein the similarity labels corresponding to the image pairs in the group of facial images are obtained from a pre-constructed similarity label set; and the similarity labels in the similarity label set are obtained based on normalizing the similarities of each image pair in the training data set.

[0130] In this embodiment, for each group of facial images, the similarity labels corresponding to the image pairs in the group of facial images are pre-calculated, and used as the true value of the similarity of each image pair in the group of facial images during model training, to train the specified clustering model. It can be understood that, since for each group of facial images, each element in the first similarity matrix corresponding to the group of facial images represents the similarity of an image pair in the group of facial images, the model loss value can be calculated based on the difference between the similarity labels corresponding to each image pair and the similarity of the image pairs represented by each element in the first similarity matrix, so as to adjust the model parameters of the specified clustering model according to the model loss value. Exemplarily, the model parameters of the specified clustering model can be adjusted by the back propagation method of minimizing the model loss value, until the model parameters of the specified clustering model are adjusted based on the loss values ​​of each group of facial images, so that the specified clustering model reaches a predetermined convergence state and the training ends. Among them, the predetermined convergence state can be that the model loss value is less than a preset threshold.

[0131] For example, if the training data set contains N (N is a positive integer) facial images, then by calculating the similarity of each image pair formed by two images in the training data set, (N-1)! similarities can be obtained. The (N-1)! similarities are normalized, and the values ​​obtained after the normalization are used as similarity labels to obtain a similarity label set consisting of (N-1)! similarity labels. The similarity labels corresponding to the image pairs in any group of facial images are the similarity labels obtained from the similarity label set and corresponding to each image pair in the group of facial images.

[0132] For example, if the training data set contains 1,000,000 facial images, and every 10,000 facial images are grouped together to obtain 100 groups of facial images, then for the first group of facial images, the similarity labels corresponding to each image pair in the first to 10,000 facial images are obtained from the similarity label set as the similarity labels corresponding to the image pairs in the first group of facial images; for the second group of facial images, the similarity labels corresponding to each image pair in the 10,001st to 20,000th facial images are obtained from the similarity label set as the similarity labels corresponding to the image pairs in the second group of facial images, and so on. It should be noted that, in order to make the layout of the scheme clear, the construction method of the similarity label set is introduced in the following embodiment, which will not be repeated here.

[0133] Exemplarily, if 100 groups of facial images are obtained, the 100 groups of facial images are processed in sequence using the specified clustering model during training. After each group of facial images is processed, the first similarity matrix corresponding to the group of facial images is calculated, and the difference between the similarity labels corresponding to the image pairs in the group of facial images calculated in advance is calculated. Based on the difference, the model parameters of the specified clustering model are adjusted until the model parameters of the specified clustering model are adjusted based on the processing of the 100 groups of facial images, and the training is completed.

[0134] Wherein, for each group of facial images, for each element in the first similarity matrix corresponding to the group of facial images, determine the target label corresponding to the element in the similarity labels corresponding to the image pairs in the group of facial images; calculate the model loss value based on the difference between each element and the corresponding target label; adjust the model parameters of the specified clustering model by back propagation method that minimizes the model loss value. Wherein, the target label corresponding to each element is the same as the image pair corresponding to the element. Exemplarily, the model loss value can be calculated using the MSE (Mean Squared Error) function and the MAE (Mean Absolute Error) function, which are both reasonable.

[0135] It can be understood that by dividing the training data set into multiple groups of facial images, the training data set can be trained in batches using a specified clustering model. In this way, when device resources are limited, a large number of facial images can be used to train the clustering model.

[0136] Furthermore, since the pre-constructed similarity label set is obtained based on normalizing the similarities of each image pair in the training data set, the value of the similarity label corresponding to each image pair in the similarity label set is affected by the similarities corresponding to other image pairs, so that for any group of facial images, the similarity label corresponding to the image pairs in the group of facial images is a label affected by the similarities of each image pair in other groups of facial images, thereby making the similarity labels of each group of facial images associated with each other. In addition, since the similarity label set is obtained based on normalizing the similarities of each image pair in the training data set, the similarity labels corresponding to the image pairs in each group of facial images have a unified range, making the distribution of the similarity labels corresponding to each group of facial images more uniform, thereby avoiding the problem of model performance degradation caused by the uneven distribution of similarity labels corresponding to each group of facial images.

[0137] The solution provided by the embodiment of the present application is to group the facial images in the training data set, and for each group of facial images, use the specified clustering model to process the group of facial images, obtain the first similarity matrix output by the specified clustering model, and adjust the model parameters of the specified clustering model based on the difference between the first similarity matrix corresponding to the group of facial images and the similarity labels corresponding to the image pairs in the group of facial images calculated in advance. In this way, the training data set can be trained in batches using the specified clustering model, so that the device used to train the specified clustering model only needs to put one group of facial images into the memory for training at a time, which significantly reduces the memory usage, so that the model training can be carried out in an environment with limited device resources. In addition, since the similarity label set is obtained based on the normalization of the similarity of each image pair in the training data set, the similarity labels corresponding to the image pairs in each group of facial images have a unified range, so that the distribution of the similarity labels corresponding to each group of facial images is more uniform, thereby avoiding the problem of model performance degradation caused by the uneven distribution of the similarity labels corresponding to each group of facial images. It can be seen that through this solution, a large-scale facial image can be used to train the clustering model under limited device resources. Therefore, it is possible to adapt the training of ultra-large-scale training data using limited device resources.

[0138] Optionally, in another embodiment of the present application, Figure 2 As shown, the method for constructing a similarity tag set includes steps S201-S203:

[0139] S201, extracting features from each facial image in the training data set to obtain a feature vector corresponding to each facial image;

[0140] In this embodiment, a deep neural network for facial recognition can be used to extract features from facial images to obtain a feature vector corresponding to each facial image. Exemplarily, the input of the deep neural network is an h×w×3 image, and the output is an n-dimensional feature vector, for example, the input is a 112×112×3 image, and the output is a 512-dimensional feature vector.

[0141] It should be noted that the embodiment of the present application does not limit the manner in which features are extracted from each facial image. For example, a fully connected layer may be used to extract features from a facial image to obtain a feature vector corresponding to the facial image.

[0142] S202, calculating the similarity between the feature vectors corresponding to each two facial images to obtain the similarity of the image pair formed by the two facial images;

[0143] Exemplarily, after obtaining the feature vector corresponding to each facial image through step S201, the similarity algorithm can be used to calculate the similarity between the feature vectors corresponding to each two facial images. Exemplarily, the similarity algorithm can be a cosine similarity algorithm, which uses a pre-similarity algorithm to calculate the cosine similarity between the feature vectors corresponding to the two facial images, and uses it as the similarity of the image pair formed by the two facial images. Among them, the closer the cosine similarity is to 1, the more similar the two vectors are. It should be noted that the embodiment of the present application does not limit the specific type of the similarity algorithm. For example, the Euclidean distance algorithm and the Harmanton distance algorithm can also be used as similarity algorithms to calculate the similarity between two feature vectors, which are all reasonable. Among them, the smaller the Euclidean distance, the more similar the two vectors are, and the smaller the Harmanton distance, the more similar the two vectors are.

[0144] S203 , based on normalizing the similarities of each image pair, a similarity label of each image pair is generated to obtain a similarity label set.

[0145] It can be understood that if the training data set contains N facial images, by calculating the similarity between the feature vectors corresponding to each two images in the training data set, the similarities of (N-1)! image pairs can be obtained. By normalizing the similarities of the (N-1)! image pairs and generating similarity labels for each image pair, a similarity label set consisting of (N-1)! similarity labels can be obtained.

[0146] Exemplarily, the similarity of each image pair can be normalized using normalization algorithms such as minimum-maximum normalization and interval length normalization to map the similarity of each image pair to a specified interval. It should be noted that the embodiment of the present application does not limit the normalization method.

[0147] Optionally, in one implementation, based on normalizing the similarities of each image pair, generating a similarity label for each image pair includes steps A1-A3:

[0148] A1, normalize the similarity of each image pair to obtain the normalized result corresponding to each image pair;

[0149] Optionally, in one implementation, based on the maximum and minimum values ​​of the similarities of each image pair, the similarities of each image pair are normalized to the interval (-1, 1) to obtain the normalized results corresponding to each image pair.

[0150] For example, for the similarity of each image pair, the similarity of the image pair can be normalized to the interval (-1, 1) using the following formula:

[0151]

[0152] Among them, new_temp_value is the normalized result corresponding to the image pair, origin_temp_value is the similarity of the image pair, upper_bound is the maximum value of the similarities of each image pair, and lower_bound is the minimum value of the similarities of each image pair.

[0153] For example, if the maximum similarity of each image pair is 0.9 and the minimum similarity is 0.3, then the similarity of an image pair is 0.35. After normalization to the interval (-1,1), the normalized result is:

[0154] It can be understood that normalizing the similarity of each image pair to the (-1, 1) interval can make it easier to converge during model training.

[0155] A2, for each image pair, if the identities of the persons contained in the facial images in the image pair are the same, the normalized result corresponding to the image pair is adjusted using the first penalty factor, and the adjusted result is determined as the similarity label of the image pair;

[0156] A3, for each image pair, if the identities of the persons included in the facial images in the image pair are different, the normalized result corresponding to the image pair is adjusted using the second penalty factor, and the adjusted result is determined as the similarity label of the image pair;

[0157] The first penalty factor increases the adjusted result, and the second penalty factor decreases the adjusted result.

[0158] In this implementation, the identity of the person contained in each facial image in the training data set can be pre-labeled. After obtaining the normalized results corresponding to each image pair through step A1, for each image pair, it is possible to identify whether the identities of the persons contained in the two facial images in the image pair are the same based on the labels of the two facial images in the image pair. The labeling method can be manual labeling or machine labeling based on artificial intelligence, which is reasonable.

[0159] It can be understood that since the result after adjusting the normalized result corresponding to each image pair is used as the similarity label of each image pair during model training, in order to improve the accuracy of the trained model, the similarity label needs to be close to the true value. In this implementation, if the identities of the people contained in the facial images in the image pair are the same, the normalized result corresponding to the image pair is increased by the first penalty factor, and if the identities of the people contained in the facial images in the image pair are different, the normalized result corresponding to the image pair is decreased by the second penalty factor. In this way, it can be ensured that the specified clustering model is still aware of the differences and similarities in identity when learning the similarity of facial structure, so that some similar facial images of people with different identities can be distinguished. In addition, by adjusting the normalized results corresponding to the image pair using the penalty factor, the ability of the specified clustering model to handle difficult samples with similar facial features but different identities can be improved.

[0160] Exemplarily, the method of adjusting the normalized result corresponding to the image pair using the first penalty factor may be: adding the first penalty factor to the normalized result corresponding to the image pair, and the value range of the first penalty factor may be (0, 0.5), so that the adjusted result increases; the method of adjusting the normalized result corresponding to the image pair using the second penalty factor may be: adding the second penalty factor to the normalized result corresponding to the image pair, and the value range of the second penalty factor may be (-0.5, 0), so that the adjusted result decreases.

[0161] It should be noted that the embodiment of the present application does not limit the method of adjusting the normalized result corresponding to the image pair using the first penalty factor or the second penalty factor. For example, in another implementation method, the method of adjusting the normalized result corresponding to the image pair using the first penalty factor may be: multiplying the normalized result corresponding to the image pair by the first penalty factor, and the first penalty factor is a value greater than 1; the method of adjusting the normalized result corresponding to the image pair using the second penalty factor may be: multiplying the normalized result corresponding to the image pair by the second penalty factor, and the second penalty factor is a positive number less than 1. Both are reasonable.

[0162] In addition, it is worth mentioning that in another implementation, the similarity of each image pair may be adjusted using the first penalty factor or the second penalty factor, and then the adjusted similarity may be normalized, which is reasonable.

[0163] It can be seen that through this solution, a uniformly distributed similarity label set can be constructed, so that when the similarity labels in the similarity label set are used to train a specified clustering model, the performance of the trained specified clustering model can be improved.

[0164] Corresponding to the above training method embodiment, the present application embodiment also provides an image clustering method, such as Figure 3 As shown, steps S301-S303 are included:

[0165] S301, obtaining a plurality of facial images to be clustered;

[0166] In this embodiment, the multiple images to be clustered may be facial images captured by cameras at traffic intersections or cameras in access control systems in security scenarios.

[0167] S302, using a pre-trained designated clustering model to process the acquired multiple facial images to obtain a second similarity matrix; wherein the designated clustering model is trained based on any of the training methods for the above-mentioned clustering models;

[0168] In this embodiment, the manner of processing the acquired multiple facial images using the specified clustering model can refer to the relevant description of the above step S103, which will not be repeated here.

[0169] S303: Clustering the multiple facial images based on a predetermined specified threshold and a second similarity matrix to obtain a clustering result.

[0170] In this embodiment, the designated threshold is a threshold calculated by using facial images that have not participated in the training to perform model inference after the designated clustering model training is completed, and based on the accuracy of the inferred clustering results.

[0171] Optionally, in one implementation, clustering the multiple facial images based on a predetermined specified threshold and a second similarity matrix to obtain a clustering result includes:

[0172] The image pairs corresponding to the elements in the second similarity matrix that are not less than a specified threshold are determined as image pairs belonging to the same identity, and a clustering result representing the same identity is obtained.

[0173] It can be understood that the second similarity matrix and the first similarity matrix are only model outputs of different stages distinguished by naming, and each element in the second similarity matrix represents the similarity of an image pair among the multiple facial images obtained. By determining the image pairs corresponding to the elements in the second similarity matrix that are not less than the specified threshold as image pairs belonging to the same identity, multiple image pairs of the same identity can be obtained. Among the multiple image pairs of the same identity, the images contained in each image pair containing the same image are determined as clustering results representing the same identity.

[0174] Optionally, in one implementation, a method for determining the specified threshold includes steps B1-B4:

[0175] B1, using a specified clustering model, processing a first preset number of facial images in a pre-acquired test data set to obtain a third similarity matrix;

[0176] In this implementation, the facial images in the test data set are labeled with the identities of the persons included. After the designated clustering model is trained, the designated clustering model is used to process the first preset number of facial images in the test data set. The process of processing using the designated clustering model can refer to the relevant description of step S103 above, and the method for obtaining the test data set can refer to the method for obtaining the training data set in step S101 above, which will not be repeated here.

[0177] B2, for each candidate threshold, determining the image pairs corresponding to the elements in the third similarity matrix that are not less than the candidate threshold as image pairs belonging to the same identity;

[0178] In this implementation, multiple candidate thresholds may be set, and image pairs corresponding to elements in the third similarity matrix that are not less than the candidate thresholds are determined as image pairs belonging to the same identity. Exemplarily, the candidate thresholds may be 0.5, 0.6, 0.7, and so on.

[0179] Exemplarily, in one implementation, the candidate threshold is selected from a specified interval according to a preset interval; the specified interval is a value range of labels in a similarity label set used when training a specified clustering model.

[0180] For example, if the value range of the label in the similarity label set used in the training of the specified clustering model is (-1, 1), all values ​​in the interval can be traversed with 0.05 as the preset interval as the candidate threshold. It should be noted that the preset interval is an interval smaller than the specified interval, and the embodiment of the present application does not limit the value of the preset interval.

[0181] B3, calculating the accuracy of the image pairs belonging to the same identity determined based on each candidate threshold;

[0182] In this implementation, for each candidate threshold, the image pairs corresponding to the elements in the third similarity matrix that are not less than the candidate threshold are determined as image pairs belonging to the same identity. Then, based on the labels corresponding to the identities of the persons contained in each facial image, the accuracy of the image pairs belonging to the same identity determined under the candidate threshold is calculated.

[0183] Exemplarily, for each candidate threshold, the ratio of the number of image pairs with accurate determination results among the image pairs belonging to the same identity determined under the candidate threshold to the number of all image pairs determined to belong to the same identity can be used as the accuracy of the image pairs belonging to the same identity determined under the candidate threshold. For example, if for candidate threshold 1, 8000 out of 10000 image pairs belonging to the same identity are determined to be accurate, the accuracy is 80%.

[0184] B4, determine the candidate threshold with the highest accuracy as the designated threshold.

[0185] It can be understood that the candidate threshold with the highest accuracy indicates that among all the candidate thresholds, the image pairs belonging to the same identity determined under this candidate threshold have the highest accuracy. Therefore, the candidate threshold with the highest accuracy can be determined as the designated threshold to improve the accuracy of the clustering results determined using the designated threshold.

[0186] It can be seen that through this solution, since the training of the specified clustering model according to the above-mentioned clustering model training method is not limited by device resources, it is possible to use a large-scale training data set to train a specified clustering model with higher accuracy, and then use the specified clustering model trained by the clustering model training method to process multiple facial images to be clustered, which can improve the accuracy of the clustering results.

[0187] In order to better understand the present solution, the method provided in the embodiment of the present application is described below with reference to a specific example.

[0188] Objective,Face clustering technology generally uses the feature vector output by the convolutional neural network for face recognition tasks as the input of the clustering algorithm, and implements clustering through a deep neural network model that calculates the correlation between multiple feature vectors. The performance of the face clustering algorithm depends largely on the performance of the individual facial feature extraction.

[0189] On the one hand, existing facial clustering algorithm models often require a lot of memory during training to ensure that the complete training set can be included in the memory / video memory for training at one time. On the other hand, although convolutional neural networks have great advantages in deployment on edge devices, they are easier to converge than ViT models when trained on a certain scale of data set. However, when the training data set reaches a relatively large and massive scale, the capabilities that ViT model training can achieve will appear to be reversed. Therefore, this example uses the ViT model to replace the convolutional neural network to extract facial features, and uses a very large-scale training data set in order to obtain stronger feature extraction capabilities; the training data set is trained in batches to ensure that only a certain amount of memory is required to complete the training of a very large-scale training data set.

[0190] In addition, since the goal of the facial clustering task is actually to efficiently provide massive labeled facial image data for the subsequent supervised training of facial recognition models without consuming extremely large amounts of manpower, the training and deployment of this task do not involve edge devices and are not restricted by extremely small algorithm storage space or tight chip computing power. On the premise of ensuring that the performance of the ViT model can surpass that of convolutional neural networks, the ViT model can be trained and deployed on server clusters and cloud services.

[0191] Before training the ViT model, it is necessary to prepare a similarity label set consisting of similarity labels corresponding to each image pair. The process is as follows:

[0192] First, a deep neural network for facial recognition is trained using the conventional method: the input is an h×w×3 image, and the output is an n-dimensional feature vector (taking an input image of 112×112×3 and an output of 512 dimensions as an example).

[0193] Next, prepare a training data set consisting of ultra-large-scale facial images with real labels (i.e., the identities of the people contained in the facial images have been annotated), with an expected size of at least 100,000,000. Use the aforementioned deep neural network for facial recognition to forward propagate the training data set to output the n-dimensional feature vector of each facial image, and calculate the similarity between every two feature vectors to obtain the similarity of each image pair. That is, if the size of the training data set is N, then (N-1)! similarities are calculated, these similarities are counted and normalized, and the specific steps are as follows:

[0194] Set the normalization range to (-1,1). If the maximum similarity of each image pair is 0.9 (upper_bound) and the minimum similarity is -0.3 (lower_bound), then the original similarity range of each image pair is proportionally expanded to the new normalization range, that is, 0.9 becomes 1, and -0.3 becomes -1. Then for the original similarity origin_temp_value of each image pair, the normalization result new_temp_value corresponding to the image pair is:

[0195]

[0196] Taking the original similarity of 0.35 as an example, the normalized result is:

[0197]

[0198] That is, the normalized result corresponding to this image pair is 0.083.

[0199] Finally, based on the normalized results corresponding to the above-mentioned image pairs, a penalty factor is applied to the normalized results corresponding to each image pair using the real labels of the facial images. The purpose is to ensure that the model is still aware of the identity differences when learning the similarity of facial structures, so that it can distinguish some similar faces of different identities. The specific operation example is: given a hyperparameter α, when the two facial images in any image pair contain people of the same identity, the normalized result corresponding to the image pair is added with α to obtain the similarity label corresponding to the image pair (corresponding to the adjustment of the normalized result corresponding to the image pair using the first penalty factor above); when the two facial images in any image pair contain people of different identities, the normalized result corresponding to the image pair is subtracted from α to obtain the similarity label corresponding to the image pair (corresponding to the adjustment of the normalized result corresponding to the image pair using the second penalty factor above). Wherein, the value range of α is (0,0.5).

[0200] It is worth mentioning that the step of applying the penalty factor to the normalized results corresponding to each image pair can also be placed before the normalization processing of the similarity of each image pair, that is, the penalty factor is first applied to the original similarity value and then the normalization processing is performed. It should be noted that there are many ways to apply the penalty factor, and this application is not limited to this.

[0201] The model training process is as follows:

[0202] First, m is the input batch_size (batch size, corresponding to the first preset number above) of the ViT model, such as 32 or 128, and m h×w×3 facial images (for example, 112×112×3) are selected for each input;

[0203] Next, for each facial image, segment it and extract the left eye + left eyebrow area, right eye + right eyebrow area, nose area and mouth. The interception method can be intercepted by using facial key point annotation or facial segmentation. Take the simpler direct interception method as an example:

[0204] The area of ​​[0,h / 2] row and [0,w / 2] column is intercepted as patch1, which represents the image area occupied by the left eye + left eyebrow;

[0205] The area of ​​[0,h / 2] row and [w / 2,w] column is intercepted as patch2, which represents the image area occupied by the right eye + right eyebrow;

[0206] The area of ​​[h / 4,3h / 4] rows and [w / 4,3w / 4] columns is intercepted as patch3, which represents the image area occupied by the nose;

[0207] The area of ​​[h / 2,h] rows and [w / 4,3w / 4] columns is intercepted as patch4, which represents the image area occupied by the mouth.

[0208] According to the actual situation, the number of patches can be increased, for example, adding: left eye + nose area, mouth and nose area or randomly intercepted area, etc.

[0209] The cropped areas of the m facial images in each batch (a batch corresponding to a group of facial images in the above text) are corresponding to each other to form new inputs in sequence, that is, the patch1 of the m facial images are stacked in sequence to form an input matrix of h / 2×w / 2×3m, which is used as the input data of input node 1, and the patch2 of the m facial images are stacked in sequence to form an input matrix of h / 2×w / 2×3m, which is used as the input data of input node 2, and so on, to form a complete input of h / 2×w / 2×3m×4.

[0210] The output layer of the ViT model is designed as a network structure consisting of a fully connected layer and a 1D convolutional layer, where the 1D convolutional layer includes m convolution kernels, so that the output of the ViT model is an m×m matrix. The value of the (x, y) position in the matrix is: the similarity between the xth facial image and the yth facial image in the current batch of m facial images.

[0211] For each batch, the ViT model is used to process the m facial images in the batch to obtain an m×m matrix (corresponding to the first similarity matrix mentioned above) output by the model. Based on the similarity of the image pairs represented by each element in the matrix and the difference between the similarities of each image pair in the batch in the pre-built similarity label set, the model loss value is calculated, and the model parameters of the ViT model are adjusted by back propagation by minimizing the model loss value. The training ends when the model parameters of the ViT model are adjusted based on each batch in the training data set.

[0212] After training a ViT model according to the above method, a threshold needs to be calculated in the inference stage. For a batch of unlabeled facial images, when the elements in the matrix output by the ViT model are lower than the threshold, the characters in the image pair corresponding to the element are considered to belong to different identities. When it is not lower than the threshold, the characters in the image pair corresponding to the element belong to the same identity. The inference process is as follows: Figure 4 As shown, steps S401-S404 are included:

[0213] S401, cropping the facial image to be clustered; wherein the cropping method is the same as the cropping method in the above-mentioned model training process.

[0214] S402, input the cropped image into the ViT model to obtain a matrix output by the model;

[0215] S403, classifying all images based on a predetermined threshold (corresponding to the specified threshold in the above text) and the matrix output by the model;

[0216] The image pairs corresponding to the elements in the matrix output by the model that are not less than the predetermined threshold are determined as image pairs belonging to the same identity, and multiple image pairs of the same identity can be obtained, thereby achieving classification of all images. Among them, the predetermined threshold is a statistical threshold, and there are many statistical methods, for example: select another large-scale labeled facial image that does not participate in the training as a training data set (the facial image is similar to the actual usage scenario), traverse all values ​​between (-1,1) at an interval of 0.05 as candidate thresholds, and select the candidate threshold when the model performance is optimal as the predetermined threshold.

[0217] S404, summarizing the similarities and differences of the identities of all the images to be clustered to form a clustering result.

[0218] After a plurality of image pairs of the same identity are obtained, the images included in each image pair including the same image among the plurality of image pairs of the same identity are determined as clustering results representing the same identity, thereby forming a clustering result.

[0219] It can be seen that this solution uses the characteristics of the ViT model to implement batch input of training data, solving the problem that large-scale training data sets cannot be input into the model at one time, and can complete the training of ultra-large-scale training data sets under limited equipment resources. And relying on the increase in the size of the training data set, it is possible to train a model with better performance than the convolutional neural network.

[0220] Corresponding to the above-mentioned clustering model training method embodiment, the present application embodiment also provides a clustering model training device, such as Figure 5 As shown, the device comprises:

[0221] A first acquisition module 510 is used to acquire a training data set; wherein the training data set includes a plurality of facial images;

[0222] A grouping module 520, configured to group the facial images in the training data set to obtain a plurality of groups of facial images; wherein the number of facial images in any group is a first preset number;

[0223] A first processing module 530 is used to process each group of facial images using a specified clustering model to obtain a first similarity matrix output by the specified clustering model; wherein the first similarity matrix is ​​a square matrix with a first preset number of rows, and each element in the first similarity matrix represents the similarity of an image pair in the group of facial images; the similarity of any image pair represents the similarity between the two images included in the image pair;

[0224] The adjustment module 540 is used to adjust the model parameters of the specified clustering model for each group of facial images based on the difference between the first similarity matrix corresponding to the group of facial images and the similarity labels corresponding to the image pairs in the group of facial images calculated in advance; wherein the similarity labels corresponding to the image pairs in the group of facial images are obtained from a pre-constructed similarity label set; and the similarity labels in the similarity label set are obtained based on normalizing the similarities of each image pair in the training data set.

[0225] Optionally, the designated clustering model is a visual transformer ViT model, and the ViT model has a second preset number of input nodes;

[0226] The first processing module 530 includes:

[0227] an extraction submodule, for extracting, for each facial image in the group of facial images, image regions occupied by a second preset number of designated facial features from the facial image, to obtain a local image corresponding to each designated facial feature in each facial image;

[0228] The input submodule is used to input the local image corresponding to the same designated facial features in each facial image as input data of an input node, input it into the ViT model, and obtain the first similarity matrix corresponding to the group of facial images.

[0229] Optionally, the similarity tag set is constructed in the following manner:

[0230] Extracting features from each facial image in the training data set to obtain a feature vector corresponding to each facial image;

[0231] Calculate the similarity between the feature vectors corresponding to each two facial images to obtain the similarity of the image pair formed by the two facial images;

[0232] Based on normalizing the similarity of each image pair, a similarity label of each image pair is generated to obtain a similarity label set.

[0233] Optionally, the generating a similarity label for each image pair based on normalizing the similarity of each image pair includes:

[0234] Normalize the similarity of each image pair to obtain the normalized result corresponding to each image pair;

[0235] For each image pair, if the identities of the persons included in the facial images in the image pair are the same, the normalized result corresponding to the image pair is adjusted using the first penalty factor, and the adjusted result is determined as the similarity label of the image pair;

[0236] For each image pair, if the identities of the persons included in the facial images in the image pair are different, the normalized result corresponding to the image pair is adjusted using the second penalty factor, and the adjusted result is determined as the similarity label of the image pair;

[0237] The first penalty factor increases the adjusted result, and the second penalty factor decreases the adjusted result.

[0238] Optionally, normalizing the similarity of each image pair to obtain a normalized result corresponding to each image pair includes:

[0239] Based on the maximum and minimum values ​​of the similarities of each image pair, the similarities of each image pair are normalized to the interval (-1, 1) to obtain the normalized results corresponding to each image pair.

[0240] Corresponding to the above-mentioned embodiment of the image clustering method, the present application embodiment also provides an image clustering device, such as Figure 6 As shown, the device comprises:

[0241] A second acquisition module 610 is used to acquire a plurality of facial images to be clustered;

[0242] A second processing module 620 is used to process the acquired multiple facial images using a pre-trained designated clustering model to obtain a second similarity matrix; wherein the designated clustering model is trained based on any of the clustering model training methods described above;

[0243] The clustering module 630 is used to cluster the multiple facial images based on a predetermined specified threshold and the second similarity matrix to obtain a clustering result.

[0244] Optionally, the clustering module 630 is specifically configured to:

[0245] The image pairs corresponding to the elements in the second similarity matrix that are not less than the specified threshold are determined as image pairs belonging to the same identity, and a clustering result representing the same identity is obtained.

[0246] Optionally, the method for determining the specified threshold includes:

[0247] Inputting a first preset number of facial images in a pre-acquired test data set into the specified clustering model to obtain a third similarity matrix;

[0248] For each candidate threshold, determining the image pairs corresponding to the elements in the third similarity matrix that are not less than the candidate threshold as image pairs belonging to the same identity;

[0249] Calculate the accuracy of the image pairs belonging to the same identity determined based on each candidate threshold;

[0250] The candidate threshold with the highest accuracy is determined as the designated threshold.

[0251] Optionally, the candidate threshold is selected from a specified interval according to a preset interval; the specified interval is a value range of labels in a similarity label set used when training the specified clustering model.

[0252] In the technical solution of the present application, the operations involved in acquiring, storing, using, processing, transmitting, providing and disclosing facial images are all carried out with the user's authorization.

[0253] It should be noted that the facial image in this embodiment is not a facial image of a specific user and cannot reflect the personal information of a specific user.

[0254] It should be noted that the facial images in this embodiment come from a public dataset.

[0255] The present application also provides an electronic device, such as Figure 7 As shown, including:

[0256] Memory 701, used for storing computer programs;

[0257] The processor 702 is used to implement any of the above-mentioned clustering model training methods or image clustering method steps when executing the program stored in the memory 701.

[0258] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 702, the communication interface, and the memory 701 communicate with each other via the communication bus.

[0259] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0260] The communication interface is used for communication between the above electronic device and other devices.

[0261] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0262] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0263] In another embodiment provided in the present application, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, the training method of any of the above-mentioned clustering models, or the steps of the image clustering method, is implemented.

[0264] In another embodiment provided by the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any clustering model training method or image clustering method in the above embodiments.

[0265] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0266] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0267] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0268] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A clustering model training method, characterized in that: The method comprises: Obtain a training data set; wherein the training data set includes a plurality of facial images; Grouping the facial images in the training data set to obtain a plurality of groups of facial images; wherein the number of facial images in any group is a first preset number; For each group of facial images, the specified clustering model is used to process the group of facial images to obtain a first similarity matrix output by the specified clustering model; wherein the first similarity matrix is ​​a square matrix having the first preset number of rows, and each element in the first similarity matrix represents the similarity of an image pair in the group of facial images; the similarity of any image pair represents the similarity between the two images included in the image pair; For each group of facial images, the model parameters of the specified clustering model are adjusted based on the difference between the first similarity matrix corresponding to the group of facial images and the similarity labels corresponding to the image pairs in the group of facial images calculated in advance; wherein the similarity labels corresponding to the image pairs in the group of facial images are obtained from a pre-constructed similarity label set; and the similarity labels in the similarity label set are obtained based on normalizing the similarities of each image pair in the training data set.

2. The method according to claim 1, characterized in that The designated clustering model is a visual transformation model structure ViT model, and the ViT model has a second preset number of input nodes; The method of processing each group of facial images by using a specified clustering model to obtain a first similarity matrix output by the specified clustering model includes: For each facial image in the group of facial images, extracting image regions occupied by a second preset number of designated facial features from the facial image to obtain a partial image corresponding to each designated facial feature in each facial image; The local image corresponding to the same designated facial features in each facial image is used as input data of an input node and input into the ViT model to obtain a first similarity matrix corresponding to the group of facial images.

3. The method according to claim 1, characterized in that The similarity tag set is constructed in the following manner: Extracting features from each facial image in the training data set to obtain a feature vector corresponding to each facial image; Calculate the similarity between the feature vectors corresponding to each two facial images to obtain the similarity of the image pair formed by the two facial images; Based on normalizing the similarity of each image pair, a similarity label of each image pair is generated to obtain a similarity label set.

4. The method according to claim 3, characterized in that The generating of similarity labels of each image pair based on normalizing the similarity of each image pair includes: Normalize the similarity of each image pair to obtain the normalized result corresponding to each image pair; For each image pair, if the identities of the persons included in the facial images in the image pair are the same, the normalized result corresponding to the image pair is adjusted using the first penalty factor, and the adjusted result is determined as the similarity label of the image pair; For each image pair, if the identities of the persons included in the facial images in the image pair are different, the normalized result corresponding to the image pair is adjusted using the second penalty factor, and the adjusted result is determined as the similarity label of the image pair; The first penalty factor increases the adjusted result, and the second penalty factor decreases the adjusted result.

5. The method according to claim 4, characterized in that The normalizing process is performed on the similarity of each image pair to obtain the normalized result corresponding to each image pair, including: Based on the maximum and minimum values ​​of the similarities of each image pair, the similarities of each image pair are normalized to the interval (-1, 1) to obtain the normalized results corresponding to each image pair.

6. An image clustering method, characterized in that: The method comprises: Acquire multiple facial images to be clustered; Processing the acquired multiple facial images using a pre-trained designated clustering model to obtain a second similarity matrix; wherein the designated clustering model is trained based on the method according to any one of claims 1 to 5; Based on a predetermined designated threshold and the second similarity matrix, a plurality of facial images are clustered to obtain a clustering result.

7. The method according to claim 6, characterized in that The clustering of the plurality of facial images based on the predetermined specified threshold and the second similarity matrix to obtain a clustering result includes: The image pairs corresponding to the elements in the second similarity matrix that are not less than the specified threshold are determined as image pairs belonging to the same identity, and a clustering result representing the same identity is obtained.

8. The method according to claim 6 or 7, characterized in that: The method for determining the specified threshold includes: Using the specified clustering model, a first preset number of facial images in a pre-acquired test data set are processed to obtain a third similarity matrix; For each candidate threshold, determining the image pairs corresponding to the elements in the third similarity matrix that are not less than the candidate threshold as image pairs belonging to the same identity; Calculate the accuracy of the image pairs belonging to the same identity determined based on each candidate threshold; The candidate threshold with the highest accuracy is determined as the designated threshold.

9. The method according to claim 8, characterized in that The candidate threshold is selected from a specified interval according to a preset interval; the specified interval is a value range of labels in the similarity label set used when training the specified clustering model.

10. A clustering model training device, characterized in that: The device comprises: A first acquisition module is used to acquire a training data set; wherein the training data set includes a plurality of facial images; A grouping module, used for grouping the facial images in the training data set to obtain a plurality of groups of facial images; wherein the number of facial images in any group is a first preset number; A first processing module is used to process each group of facial images using a specified clustering model to obtain a first similarity matrix output by the specified clustering model; wherein the first similarity matrix is ​​a square matrix having the first preset number of rows, and each element in the first similarity matrix represents the similarity of an image pair in the group of facial images; the similarity of any image pair represents the similarity between the two images included in the image pair; The adjustment module is used to adjust the model parameters of the specified clustering model for each group of facial images based on the difference between the first similarity matrix corresponding to the group of facial images and the similarity labels corresponding to the image pairs in the group of facial images calculated in advance; wherein the similarity labels corresponding to the image pairs in the group of facial images are obtained from a pre-constructed similarity label set; and the similarity labels in the similarity label set are obtained based on normalizing the similarities of each image pair in the training data set.

11. An image clustering device, characterized in that: The device comprises: A second acquisition module is used to acquire multiple facial images to be clustered; A second processing module, used to process the acquired multiple facial images using a pre-trained designated clustering model to obtain a second similarity matrix; wherein the designated clustering model is trained based on the method according to any one of claims 1 to 5; The clustering module is used to cluster the multiple facial images based on a predetermined specified threshold and the second similarity matrix to obtain a clustering result.

12. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-9 when executing a program stored in a memory.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

14. A computer program product, characterized in that The computer program product comprises executable instructions, and when the executable instructions are executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 9.

Images