Cross-modal sketch face recognition method based on deep metric learning

Abstract

The application discloses a cross-modal sketch face recognition method based on deep metric learning, which comprises the following steps: using a dataset expansion stage of StyleGAN, a model pre-training stage based on a bidirectional acceleration ternary inter-modal loss function, a re-training stage based on an inter-modal intra-modal loss of metric learning, and a model effect testing stage; sketch face dataset is expanded by using a generative adversarial neural network, different facial expressions can be generated, a bidirectional acceleration center ternary loss is proposed to reduce the influence of inter-modal differences, and the center loss thought is integrated, so that similar samples are more gathered, and different samples are more dispersed, and the accuracy of the application reaches 91.56%.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, and particularly relates to a cross-modal sketch face recognition method based on deep metric learning. BACKGROUND

[0002] In actual application, since there is a large modal difference between a face photo and a face sketch, a face recognition method based on a sketch cannot directly recognize a probe image (a face sketch or a photo) from a gallery image (a face photo or a sketch), and therefore the effect is poor. There are different information expression modes and generation mechanisms between a drawn sketch face image and a photo face image, resulting in a large texture and grayscale difference between the sketch and the photo. Therefore, there is a very obvious difference between a sketch sketch and a face image on some existing sketch face databases. Therefore, how to reduce the difference caused by different modalities is a very important challenge to improve the accuracy of sketch face recognition. In addition, the existing sketch face dataset has a small sample size, usually in the order of magnitude of thousands of images, which is also a big challenge for deep network training.

[0003] The existing sketch face recognition technology uses transfer learning or a recurrent generative adversarial network, but the existing technology cannot be implemented under a small sample condition and cannot fully eliminate the modal difference, and it is difficult to meet the existing demand. SUMMARY

[0004] The present application is directed to the insufficient research on the modal difference of the existing sketch face recognition, and proposes a cross-modal sketch face recognition method based on deep metric learning in combination with the metric learning technology. The method uses a generative adversarial neural network to expand the sketch face dataset, can generate faces with different expressions, considers the modal difference, proposes a bidirectional acceleration center ternary loss to reduce the influence of the modal difference, and integrates the center loss idea, so that the same samples are more gathered and the different samples are more dispersed. On this basis, the accuracy of the present application reaches 91.56%.

[0005] The present application is implemented through the following technical solutions:

[0006] The present application relates to a cross-modal sketch face recognition method based on deep metric learning, which comprises a dataset expansion stage of StyleGAN, a model pre-training stage based on a bidirectional acceleration ternary modal loss function, a re-training stage based on metric learning inter-modal and intra-modal loss, and a model effect test stage, and specifically comprises:

[0007] S1) Expanding the dataset of the generative adversarial network.

[0008] S2) Establishing a VGG16 deep convolutional neural network for extracting features, which comprises thirteen convolutional layers and three fully connected layers.

[0009] S3) Preprocess the sketch face data set pictures and the expanded pictures.

[0010] S4) Pre-train the neural network of step S2 using the large-scale face data set VGG-Face.

[0011] S5) Migrate the pre-trained model in step S4, that is, keep the parameters of the convolutional layer unchanged, and re-initialize the parameters of the fully connected layer in a uniform manner.

[0012] S6) Establish a bidirectional triple.

[0013] S7) Retrain the pre-trained model obtained in step S4 through a bidirectional accelerated center triple loss to obtain a final training model.

[0014] S8) Test using a test set, extract features from the photo face pictures of the test set to form a test face library, use the network and classifier obtained in step S7 to calculate the photo face matched with the corresponding sketch face, use the top1 method to calculate the final accuracy.

[0015] The present application relates to a system for implementing the above method, comprising: a data set expansion unit, a model pre-training unit, a loss function re-training unit based on the present application, and an effect testing unit, wherein: the data set expansion unit uses an existing StyleGAN generation network model to train the encoder built in the present application, and calculates the direction vector of the smile attribute using an existing smile face data set, changes the encoding of the dlatent domain through the direction vector, inputs the changed encoding into the generator, and obtains augmented data samples. The model pre-training unit builds a VGG16 network, directly reads the model parameters pre-trained by the VGGFace data set in PyTorch, and obtains a pre-trained network model. The loss function re-training unit based on the present application re-trains the previous pre-trained model according to the expanded training set through the constraint of the loss function, and obtains the final feature extraction network. The effect testing unit extracts features from the test set according to the trained feature extraction network, then directly calculates the Euclidean distance for sample matching, and obtains the highest accuracy.

[0016] Technical effects

[0017] Compared with the existing data augmentation method which mostly uses simple picture rotation and occlusion, the present application independently constructs and trains an encoder, proposes a calculation method of encoding direction vector to generate pictures with stronger flexibility and more quantity, constructs a bidirectional triple through a bidirectional accelerated center triple loss, eliminates the difference between modalities, and makes the classes compact and separable. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 This is a structural diagram of the training system of the present invention;

[0019] Figure 2 This is a flowchart illustrating the dataset expansion process of the present invention.

[0020] Figure 3 This is a diagram of the encoder network structure in the dataset extension of this invention;

[0021] Figure 4 This is a diagram of the feature extraction network structure of the present invention;

[0022] Figure 5 This is a schematic diagram illustrating the effect of the loss function of this invention on the distribution of triplet feature vectors; Detailed Implementation

[0023] like Figure 1 As shown in the figure, this embodiment relates to a sketch face recognition method based on deep metric learning, which includes: a dataset expansion stage, a model pre-training stage, a retraining stage based on metric learning inter-modal and intra-modal loss, and a model performance testing stage.

[0024] The method specifically includes:

[0025] S1) as Figure 2 As shown, the dataset for extended generative adversarial networks specifically includes:

[0026] S1.1) Obtain the trained ResNet50-based feature extractor, i.e., the encoder, and obtain the StyleGAN generative network as the image generator, i.e., the generator.

[0027] The generator uses the generative network portion of a StyleGAN model pre-trained on the FFHQ dataset. The generator takes an (18, 512)-dimensional vector as input and outputs an image generated from the input.

[0028] The specific network structure of the encoder is as follows: Figure 3 As shown, ResNet50 is a pre-trained model in PyTorch. Two convolutional layers and three fully connected layers are then added, and the final output feature vector has dimensions of (18, 512).

[0029] The encoder's training data comes from the generator, because the generator is a pre-trained model, and a large number of codes and their corresponding generated images can be easily obtained. The generated images are used as the encoder's input, and the corresponding ground truth codes are used as labels. The loss function is the Euclidean distance between the encoder's output code and the ground truth code.

[0030] The training is to make the loss function converge, input an image into the encoder to get the encoding, and then input the encoding into the generator to get another image, which is not much different from the first one in subjective view.

[0031] S1.2) Use the feature extractor to map the image into the dlatent domain with the encoding vector size of (18, 512) dimensions.

[0032] The dlatent refers to disentangled latent, and the dlatent domain refers to the metric space where the encoding mapped by the encoder is located.

[0033] S1.3) The goal is to achieve dataset expansion by adjusting the smile attribute, and calculate the direction vector of the smile attribute in the dlatent domain.

[0034] The calculation method of the smile attribute direction vector is: Where: dlatent a is the encoding of a randomly selected smiley face from the GENKI smiley face dataset after passing through the encoder, and dlatent b is the encoding of a randomly selected non-smiley face from the GENKI smiley face dataset after passing through the encoder. n is 3000, and the final smile attribute direction vector is obtained.

[0035] S1.4) Add the direction vector to the encoding of the original sample to obtain the encoding of the expanded sample. The greater the weight of the direction vector, the greater the smile degree. The weight of the direction vector is from 0.1 to 1 with a step of 0.1, so that each encoding is expanded to 10 encodings.

[0036] S1.5) Use the image generator to map the (18, 512) dimensional encoding obtained in step S1.4 to a (1024, 1024, 3) dimensional generated image, i.e. a generated image with a length and width of 1024 pixels and a channel number of 3, to achieve dataset expansion.

[0037] S2) Establish a VGG16 deep convolutional neural network for feature extraction as shown in Figure 4 The network includes thirteen convolutional layers and three fully connected layers. This embodiment is built and trained by PyTorch.

[0038] S3) Preprocess the sketch face dataset CUFSF and the expanded images: first convert the color image to grayscale, then use the multi-task convolutional neural network (MTCNN) for face positioning and cropping as a sample set, and divide the dataset into a training set and a test set. The training set is used to train the VGG16 deep convolutional neural network built in step S2, and the test set is used to test the model effect.

[0039] S4) Pre-training the neural network of step S2 using a large-scale face dataset VGG-Face. Here, the VGG16 pre-trained model parameters provided in PyTorch are directly used.

[0040] S5) Migrate the pre-trained model in step S4, that is, keep the parameters of the convolutional layer unchanged, and re-initialize the parameters of the fully connected layer in a uniform manner.

[0041] S6) Establish bidirectional triplets, specifically: from the training set obtained after preprocessing in step S3, use one sketch as the anchor sample, one photo image as the positive sample of the same class as the anchor sample, and one photo image as the negative sample of a different class from the anchor sample to form a forward triplet; then use one photo image as the anchor sample, one sketch as the positive sample, and one sketch as the negative sample to form a reverse triplet.

[0042] Preferably, since the number of negative samples in the bidirectional triplet is much larger than the number of positive samples, in order to improve the training effect, difficult-to-distinguish sample screening is needed, that is, by traversing the same batch of samples to search for the negative sample closest to the anchor image in the metric space. From the picture effect, the difficult-to-distinguish sample is the negative sample that looks most like the anchor sample visually.

[0043] The metric space specifically uses: Euclidean space, and the metric distance uses Euclidean distance.

[0044] S7) Retrain the pre-trained model obtained in step S5 by using bidirectional center accelerated triplet loss to obtain the final trained model.

[0045] The retraining specifically includes: using a loss function including inter-modal loss and intra-modal loss to perform a backpropagation algorithm to train the parameters. To make the training process more robust, a learning rate decay is used, starting with a larger learning rate of 0.0001, and then gradually reducing the learning rate to 0.000001 as the training round increases. The epoch is set to 100, and the batch is set to 16.

[0046] The loss function includes inter-modal loss and intra-modal loss, that is, L t =L Bt +L id .

[0047] The inter-modal loss specifically includes:

[0048] wherein: L s is the loss of the forward triplet, L p is the loss of the reverse triplet, D is the distance metric in the metric space, which is the Euclidean distance here, and N is the number of bidirectional triplets in the same batch, [x]+ = max(0, x), V a is the anchor point picture feature vector, V p is the positive sample picture feature vector, V n is the negative sample picture feature vector, the superscript s represents a sketch face picture, the superscript p represents a photo face picture, β is a hyperparameter, and represents the distance between the expected positive sample and the negative sample. The loss is put into the exponential e to accelerate convergence.

[0049] Then the center loss idea is integrated into the ternary loss to replace V c with V a , replace V a,p with V p . V c is the arithmetic mean of V a and V p . V a,p is the feature vector of the anchor point sample or the positive sample, because the Euclidean distances of the two to V c are the same, so the effect is the same no matter which one is taken. The bidirectional accelerated center ternary loss is realized, and the formula is as follows:

[0050]

[0051] The intra-modal loss is specifically: wherein the probability of being predicted as the i-th sample is the superscript s represents a sketch face picture, the superscript p represents a photo face picture, q i is the ground truth, V i is the feature vector of the i-th sample, V j is the feature vector of the j-th sample.

[0052] S8) Test using the test set, perform feature extraction on the photo face pictures of the test set to form a test face library, use the network and the classifier obtained in step S7 to calculate the photo face matched with the corresponding sketch face, use the top1 method to calculate the final accuracy.

[0053] Through specific experiments, it is realized under Ubuntu 16.04 system based on Python 3.6 and taking PyTorch as a framework. The CUFSF dataset contains 1194 pairs of sketch face images and visible light face photos, 500 pairs of sketch face images-visible light face photo pairs are randomly selected for training, and the remaining 694 pairs of face images are used for testing. The dimension sizes of the full connection layers of VGG16 are 4096, 4096 and 1024 respectively, the deep feature representation V is set to 1024 dimensions, and the random gradient descent algorithm (SGD) is used for optimization through back propagation. At the same time, the learning rate attenuation strategy is used, the learning rate is gradually reduced from the initial 0.0001 to 0.000001, the epoch is 100, the batch is set to 16, beta is set to 1, the evaluation standard is the top1 accuracy, and the experimental data can be obtained as shown in Table 1:

[0054] Table 1

[0055]

[0056]

[0057] Compared with the prior art, the features of the face pictures are often designed manually, and it is difficult to extract common features of different modalities, so the effect is not satisfactory, compared with the present application, the method of VGG is used to directly train in a single mode, without considering the difference between modalities. The data set expansion technology is used in the embodiment to further solve the small sample problem, and the intra-class diversity of the sample is greatly enriched. It can be seen that the accuracy using data augmentation is higher than that without using data augmentation. The bidirectional triplets are used to further eliminate the difference between modalities, and the intra-modal loss and inter-modal loss are combined, that is, the inter-class information in the intra-modal is considered and the inter-modal difference is considered. It can be seen that the accuracy of the embodiment is higher than that of directly using triple loss.

[0058] The application initiatively uses the adversarial generative network to expand the data set, so as to improve the training and testing effect. And fully considering the difference between pictures in the cross-modal recognition problem, the bidirectional accelerated center triple loss is used in the metric space, the concept of bidirectional is used to greatly reduce the influence of the difference between modalities, so as to reduce the distance between the same samples and expand the distance between the different samples, and the effect form is as shown in Figure 5 .

[0059] The above specific embodiments can be adjusted in different ways by those skilled in the art without departing from the principles and purposes of the application, the protection scope of the application is subject to the claims and is not limited by the above specific embodiments, and each implementation scheme in the scope is subject to the constraints of the application.

Claims

1. A cross-modal sketch face recognition method based on deep metric learning, characterized in that, The application relates to a sketch-photo face recognition method based on StyleGAN, and belongs to the technical field of computer vision. The method comprises a dataset expansion stage using StyleGAN, a model pre-training stage based on a bidirectional accelerated triple inter-modal loss function, a re-training stage based on a metric learning inter-modal intra-modal loss, and a model effect testing stage, and specifically comprises the following steps: S1) expanding a dataset of a generative adversarial network; S2) establishing a VGG16 deep convolutional neural network comprising 13 convolutional layers and 3 fully connected layers for extracting features; S3) preprocessing pictures in a sketch face dataset and expanded pictures; S4) pre-training the neural network in step S2 using a large-scale face dataset VGG-Face; S5) migrating the pre-trained model in step S4, that is, keeping the parameters of the convolutional layers unchanged, and reinitializing the parameters of the fully connected layers in a uniform manner; S6) establishing bidirectional triplets; S7) re-training the pre-trained model obtained in step S5 by using a bidirectional accelerated center triplet loss to obtain a final trained model, and specifically, the parameters are trained by using a loss function comprising an inter-modal loss and an intra-modal loss to perform a back propagation algorithm; S8) testing by using a test set, extracting features of photo face pictures in the test set to form a test face library, calculating photo faces matched with corresponding sketch faces by using the network and the classifier obtained in step S7, and calculating a final accuracy by using a top1 method; The loss function includes inter-modal loss and intra-modal loss, i.e., L t = L Bt + L id ; The inter-modal loss is specifically: Wherein: L s is the loss of the forward triplets, L p is the loss of the backward triplets, D is a distance metric in a metric space, here is the Euclidean distance, N is the number of bidirectional triplets in the same batch, [x] + = max(0, x), V a is the anchor picture feature vector, V p is the positive sample picture feature vector, V n is the negative sample picture feature vector, the superscript s represents a sketch face picture, the superscript p represents a photo face picture, and β is a hyperparameter, indicating that the distance between the positive sample and the negative sample is expected to be pulled apart; the loss is put into the exponential e to accelerate convergence; Then the center loss idea is integrated into the triplet loss to replace V c , V a , V a,p , V p ; V c is the arithmetic mean of V a and V p ; V a,p is the feature vector of anchor samples or positive samples, and the bidirectional accelerated center triplet loss is realized, specifically as follows: ; The intra-modal loss is specifically: wherein: the probability predicted as the i-th sample the superscript s represents the sketch face picture, the superscript p represents the photo face picture, q i is the ground truth, V i is the feature vector of the i-th sample, V j is the feature vector of the j-th sample. 2.The deep metric learning based cross-modal sketch face recognition method according to claim 1, characterized in that, The step S1 specifically comprises: S1.1) obtaining a trained feature extractor based on ResNet50, that is, an encoder, and obtaining a generative network of a StyleGAN model as a picture generator, that is, a generator; The generator uses a generative network part in a pre-trained StyleGAN model on an FFHQ dataset; the input of the generator is a (18, 512)-dimensional vector, and the output is a picture generated according to the input; The encoder comprises ResNet50, and two convolutional layers and three fully connected layers are additionally added after the ResNet50; the feature vector dimension of the final output is (18, 512); The training data of the encoder is obtained from the generator, because the generator is a trained model, and a large amount of encoding and corresponding generated images can be easily obtained; the generated images are used as the input of the encoder, and the corresponding real encoding is used as a label; a loss function is an Euclidean distance between the encoding output by the encoder and the real encoding; S1.2) mapping the picture to a dlatent domain with a (18, 512)-dimensional encoding vector size by using the feature extractor; S1.3) the target is to realize dataset expansion by adjusting a smiling attribute, and a direction vector of the smiling attribute in the dlatent domain is calculated; S1.4) adding the direction vector to the encoding of the original sample to obtain the encoding of the expanded sample; the greater the weight of the direction vector, the greater the smiling degree; the weight of the direction vector is from 0.1 to 1 at a step of 0.1, so that each encoding is expanded into 10 encodings. S1.5) using the picture generator to map the (18, 512) dimensional encoding obtained in step S1.4 to a (1024, 1024, 3) dimensional generated picture, i.e., a generated picture with a length and a width of 1024 pixels and a channel number of 3, to realize dataset expansion. 3.The deep metric learning based cross-modal sketch face recognition method according to claim 2, characterized in that, The calculation method of the smile attribute direction vector is: Wherein: dlatent a is the encoding of the smile face randomly selected from the GENKI smile face dataset through the encoder, dlatent b is the encoding of the non-smile face randomly selected from the GENKI smile face dataset through the encoder, n is 3000, and finally the smile attribute direction vector is obtained. 4.The deep metric learning based cross-modal sketch face recognition method of claim 1, characterized in that, The step S3 is specifically: preprocessing the pictures in the sketch face dataset CUFSF and the expanded pictures: first converting the color pictures to grayscale pictures, and then using a multi-task convolutional neural network (MTCNN) to perform face positioning and clipping as a sample set, and dividing the dataset into a training set and a test set, the training set being used to train the VGG16 deep convolutional neural network built in step S2, and the test set being used to test the model effect. 5.The deep metric learning based cross-modal sketch face recognition method of claim 1, characterized in that, The step S6 is specifically: from the training set obtained after the preprocessing in step S3, respectively using one sketch sketch as an anchor sample, one photo image as a positive sample of the same class as the anchor sample, and one photo image as a negative sample of a different class from the anchor sample to form a forward triple; and using one photo image as an anchor sample, one sketch sketch as a positive sample, and one sketch sketch as a negative sample to form a reverse triple. 6.The deep metric learning based cross-modal sketch face recognition method of claim 5, characterized in that, Since the number of negative samples in the bidirectional triple is much larger than the number of positive samples, in order to improve the training effect, difficult-to-distinguish sample screening is needed, that is, by traversing the same batch of samples to search for the negative sample closest to the anchor picture in the metric space, from the picture effect, the difficult-to-distinguish sample is the negative sample that looks most like the anchor sample from the visual point of view. 7.The deep metric learning based cross-modal sketch face recognition method of claim 1, wherein, The step S7, when training the parameters, in order to make the training process more robust, learning rate decay is used, first using a larger learning rate of 0.0001, and then gradually reducing the learning rate to 0.000001 as the training round increases.

8. A system for implementing the method of cross-modal sketch face recognition based on deep metric learning according to any one of claims 1-7, characterized in that, It comprises: The dataset expansion unit uses the existing StyleGAN generator network model to train the encoder, and uses the existing smile face dataset to calculate the direction vector of the smile attribute, changes the encoding of the dlatent domain through the direction vector, inputs the changed encoding into the generator, and obtains the augmented data samples; the model pre-training unit builds a VGG16 network, directly reads the model parameters pre-trained by the VGGFace dataset in PyTorch, and obtains a pre-trained network model; the loss function based on the proposed loss function retraining unit, according to the expanded training set, through the constraint of the loss function, the pre-trained model is retrained again to obtain the final feature extraction network; the effect test unit extracts features from the test set according to the trained feature extraction network, and then directly calculates the Euclidean distance to perform sample matching, and obtains the highest accuracy.

Images