A Sketch Face Recognition Method

Abstract

The present invention discloses a sketch face recognition method, including: extracting face sketch features and face photo features respectively through a face sketch feature extractor and a face photo feature extractor; designing an identity discriminative feature retention module to learn and remove identity non-discriminative features from the face sketch features and face photo features, so as to capture the identity discriminative features of each modality; designing a sketch-photo cross-reconstruction module to reconstruct the global face photo features using the face sketch features and reconstruct the global face sketch features using the face photo features; embedding the reconstructed global face photo features into the face sketch identity discriminative features to obtain joint features containing information of both modalities; embedding the reconstructed global face sketch features into the face photo identity discriminative features to also obtain joint features containing information of both modalities; training the overall network based on the first and second training phases, and recognizing faces based on the trained network.

Description

Technical Field

[0001] The present invention relates to the field of face recognition, and in particular to a method for face recognition from sketches. Background Art

[0002] In recent years, the rapid development of face scanning and imaging technologies has promoted the rapid growth of two-dimensional face image data volume. Accurately identifying the identity of the person contained in a face image has become a research hotspot in the fields of information forensics and security. As an important branch in the field of face recognition, face recognition from sketches aims to retrieve face photos with the same identity information as a given face sketch from a database, and has currently been importantly applied in fields such as intelligent monitoring and identity authentication.

[0003] Intuitively, for any face sketch or face photo, the features of key facial regions such as eyes, nose, and mouth are discriminative for identifying different faces, while the features of other facial regions have a weaker correlation with face recognition. To capture the features of key facial regions, Peng et al. used rectangular frames to locate the key facial regions of face sketches and face photos, and then used a convolutional neural network to extract deep local descriptors from these regions for face recognition. However, due to the complexity of facial structures, the shapes of key facial regions are usually irregular, which will inevitably cause the key facial regions located by rectangular frames to contain information that interferes with recognition, thus limiting the improvement of recognition performance. Therefore, how to effectively learn identity discriminative features from face sketches and face photos still needs to be studied in depth.

[0004] In addition, face photos contain rich color and texture information, while face sketches only contain simple lines, which makes there be a huge cross-modal difference between face sketches and face photos. This cross-modal difference makes it extremely challenging to effectively align face sketches and face photos. To address the above challenges, Huang et al. designed a method of discriminant spectral regression, which first maps face sketch features and face photo features into a common discriminant subspace, and then aligns cross-modal features in this space to achieve face recognition from sketches. However, this method of mapping from a high-dimensional to a low-dimensional feature space will inevitably cause the loss of some key facial information, thereby affecting the performance of face recognition from sketches. Therefore, exploring a reliable feature mapping method to achieve more effective cross-modal feature alignment is crucial for face recognition from sketches. Summary of the Invention

[0005] To effectively extract and align the identity discriminative features of face sketches and face photos for recognition, the present invention proposes a face sketch recognition method. This method respectively mines and removes the identity non-discriminative features from the face sketch features and face photo features to effectively capture the identity discriminative features of each modality; reconstructs the face sketch features into the global face photo features and the face photo features into the global face sketch features through sketch-photo feature cross-reconstruction; embeds the reconstructed global features of a certain modality into the features of another modality to obtain cross-modal aligned face sketch features and face photo features, thereby achieving effective face sketch recognition, as described below:

[0006] A face sketch recognition method, the method comprising:

[0007] Extract face sketch features and face photo features respectively through a face sketch feature extractor and a face photo feature extractor;

[0008] Design an identity discriminative feature retention module for learning and removing the identity non-discriminative features from the face sketch features and face photo features to capture the identity discriminative features of each modality;

[0009] Design a sketch-photo cross-reconstruction module to reconstruct the global face photo features using the face sketch features and reconstruct the global face sketch features using the face photo features;

[0010] Embed the global face photo features reconstructed from the global face sketch features into the face sketch identity discriminative features to obtain a joint feature containing both face sketch modality information and face photo modality information; embed the global face sketch features reconstructed from the global face photo features into the face photo identity discriminative features to obtain a joint feature containing both face photo modality information and face sketch modality information; train the overall network based on the first and second training stages, and recognize faces based on the trained network.

[0011] Among them, the identity discriminative feature retention module is specifically:

[0012] Based on an attention-based write operation, update the memory bank m using the face sketch features s and update the memory bank m using the face photo features p so that m s and m p can respectively store the shared information in the face sketch features and the shared information in the face photo features.

[0013] Among them, the identity discriminative feature retention module is specifically:

[0014] For the i-th memory vector of the memory bank ms 中 of Calculate the similarity between each eigenvector in and extract from the nth eigenvector with the maximum similarity to

[0015]

[0016] Among them, represents cosine similarity calculation, represents the jth eigenvector in

[0017] Use and to update as follows:

[0018]

[0019] Among them, ← represents the update operation, δ represents a constant, d = c / 8, and represent three mapping functions, and aim to map the input feature embeddings to query q s , key k s and value v s ;

[0020] Based on m s , through the memory reading operation, recall the identity non-discriminative features from The reading process is as follows:

[0021]

[0022] Among them, represents the relationship matrix between s and m

[0023] Remove the identity non-discriminative features from to obtain the identity discriminative features of the face sketch The formula is as follows:

[0024]

[0025] Among them, represents element-wise subtraction;

[0026] For the face photo modality, learn the identity non-discriminative features of the face photo and from the face photo features​​​ Removed from it to obtain the identity discriminative features of the face photo

[0027] Among them, the sketch-photo cross-reconstruction module is specifically as follows:

[0028] Introduce two learnable memory banks and To assist in the reconstruction of the global features of the face sketch and the global features of the face photo, where t represents the number of memory vectors in each memory bank, based on u s and u p , Reconstructed into the global features of the face photo As follows:

[0029]

[0030] Among them, Represents The relationship matrix between and u s ;

[0031] For the face photo modality, in the same way as above, the global features of the face photo Reconstructed into the global features of the face sketch

[0032] Among them, the feature embedding is:

[0033]

[0034] Among them, D p Represents the photo feature decoder, D s Represents the sketch feature decoder, σ(·,·) represents channel-level concatenation, ∈(·) represents a mapping function that maps the input feature embedding to a joint feature space containing face sketch modality information and face photo modality information; the photo feature decoder D p And the sketch feature decoder D s Are both built using four deconvolution-convolution blocks and one convolutional layer, and the mapping function ∈(·) consists of one fully connected layer. By measuring the And The Euclidean distance between them is used to achieve sketch face recognition.

[0035] The beneficial effects of the technical solution provided by the present invention are:

[0036] 1. The present invention realizes effective sketch face recognition by mining and aligning the identity discriminative features of face sketches and face photos;

[0037] 2. The present invention designs an identity discriminative feature retention module, which captures the identity discriminative features of each modality by learning and removing identity non-discriminative features from face sketch features and face photo features;

[0038] 3. The present invention designs a sketch-photo cross-reconstruction module and a global feature embedding module, which capture more aligned cross-modal features for sketch face recognition by reconstructing the global features of a certain modality and embedding them into another modality. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a method for sketch face recognition. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0040] To make the objectives, technical solutions and advantages of the present invention clearer, the following further describes in detail the embodiments of the present invention.

[0041] The following illustrates the specific implementation manners of the sketch face recognition method in the embodiments of the present invention through examples.

[0042] I. Construct a feature extraction module

[0043] First, construct a feature extraction module, including: a face sketch feature extractor and a face photo feature extractor. Specifically, the face sketch feature extractor and the face photo feature extractor are respectively used to extract face sketch features and face photo features where L represents the number of convolutional blocks in each feature extractor. and respectively represent the features output by the i-th convolutional block in the face sketch feature extractor and the face photo feature extractor.

[0044] In the embodiments of the present invention, both the face sketch feature extractor and the face photo feature extractor are built using the first four convolutional blocks of the ResNet50 network, that is, L is set to 4. Among them, the ResNet50 network is well-known to those skilled in the art, and the embodiments of the present invention will not elaborate on this.

[0045] II. Design an identity discriminative feature retention module

[0046] In order to adaptively learn and remove identity non-discriminative features from face sketch features and face photo features to capture the identity discriminative features of each modality, an identity discriminative feature retention module is designed.

[0047] Specifically, in order to adaptively learn non-discriminative features from face sketch features and face photo features, two learnable memory banks are first introduced and where r represents the number of memory vectors in each memory bank, r is set to 128, c represents the channel dimension of each memory vector, c is set to 1024, s represents the sketch modality, and s represents the photo modality. Then, through an attention-based write operation, the face sketch features are used to update m s , and the face photo features are used to update m p , so that m s and m p can store the shared information in the face sketch features and the shared information in the face photo features respectively. Taking the face sketch modality as an example, for the nth memory vector in m s calculate the similarity between and each feature vector in , where represents the face sketch features output by the pth convolutional block of the face sketch feature extractor. After that, the Lth feature vector with the maximum similarity to is taken out from , and the specific formula is as follows:

[0048]

[0049] where represents the cosine similarity calculation, represents the jth feature vector in.

[0050] Then, use and to update where represents the face sketch features output by the (L-1)th convolutional block of the face sketch feature extractor, and this process is shown as follows:

[0051]

[0052] where ← represents the update operation, δ represents the constant 0.99, and d = c / 8. and represent three mapping functions. and are designed to map the input feature embeddings into the query q s , the key k s , and the value v s . In the embodiments of the present invention, consists of a single fully-connected layer, and both consist of two fully-connected layers.

[0053] Subsequently, based on m s , through the memory read operation, from​ Medium-recall identity non-discriminative features The reading process is expressed as follows:

[0054]

[0055] Among them, represents the relationship matrix with m s

[0056] Finally, remove the identity non-discriminative features from to obtain the identity discriminative features of the face sketch The formula is as follows: The formula is as follows:

[0057]

[0058] Among them, represents element-wise subtraction.

[0059] For the face photo modality, in the same way as above, the designed identity discriminative feature retention module can learn the identity non-discriminative features of the face photo and remove them from the face photo features to obtain the identity discriminative features of the face photo

[0060] III. Design of the sketch-photo cross-reconstruction module

[0061] Considering the potential similarity between the face sketch features and face photo features of the same identity, a sketch-photo cross-reconstruction module is designed to reconstruct the global features of the face photo using the face sketch features and reconstruct the global features of the face sketch using the face photo features. Specifically, the module first introduces two learnable memory banks and to assist in the reconstruction of the global features of the face sketch and face photo, where t represents the number of memory vectors in each memory bank, and t is set to 4. Taking the face sketch modality as an example, the identity discriminative features of the face sketch are fed into a sketch feature encoder E s to obtain the global features of the face sketch The formula is expressed as follows:

[0062]

[0063] In the embodiment of the present invention, the sketch feature encoder E s is built using the last convolutional block of the ResNet50 network and the subsequent fully connected layer.

[0064] ​After that, based on u s and t p , is reconstructed into the global feature of the face photo The reconstruction process is shown as follows:

[0065]

[0066] Among them, represents the relationship matrix between s and u

[0067] For the face photo modality, in the same way as above, the global feature of the face photo is reconstructed into the global feature of the face sketch

[0068] IV. Design the global feature embedding module

[0069] Design the global feature embedding module. By embedding the global feature of a certain modality reconstructed by the sketch-photo cross-reconstruction module into the feature of another modality, the face sketch feature and the face photo feature are mapped into a common feature space, so as to capture the cross-modal alignment features for face recognition. Specifically, the global feature of the face photo reconstructed from the global feature of the face sketch is embedded into the face sketch identity discriminative feature to obtain the joint feature Similarly, the global feature of the face sketch reconstructed from the global feature of the face photo is embedded into the face photo identity discriminative feature

[0070]

[0071] where D p represents a photo feature decoder, D s represents a sketch feature decoder, σ(·,·) represents channel-level concatenation, ∈(·) represents a mapping function, which aims to map the input feature embedding into a joint feature space containing face sketch modality information and face photo modality information. In the embodiment of the present invention, the photo feature decoder D p and the sketch feature decoder D s are both built with four deconvolution-convolution blocks and a convolutional layer, and the mapping function ∈(·) consists of a single fully connected layer.

[0072] Finally, by measuring and The Euclidean distance between them is used to realize sketch face recognition.

[0073] 5. Training Sketch Face Recognition Network

[0074] The network training proposed in the embodiment of the present invention is divided into two stages.

[0075] 1. First phase of training

[0076] The first stage of training aims to use the orthogonal loss L ort And the triplet loss L tri Jointly train the feature extraction module and the identity-discriminating feature preservation module.

[0077] Specifically, the orthogonal loss L is introduced ort Measuring the identity discriminative features of face sketches Non-identifying characteristics Similarity between people and identity identification features of face photos Non-identifying characteristics The specific formula of orthogonal loss is as follows:

[0078]

[0079] in, and Respectively and The i-th eigenvector in . |·| represents the absolute value calculation. N represents the number of eigenvectors contained in each identity discriminant feature or identity non-discriminant feature, and N is set to 256.

[0080] In addition, the triplet loss L is adopted tri To narrow down the identity identification features of face sketches containing the same identity information and facial photo identity identification features distance between faces, and at the same time increase the identity identification features of face sketches containing different identity information and facial photo identity identification features The triplet loss formula is expressed as follows:

[0081]

[0082] in, Indicates from The identity discriminative features extracted from facial photos containing different identity information. D(·,·) represents the Euclidean distance between two features. Δ represents a threshold constant, which is set to 0.3.

[0083] By combining the above two losses, the total loss L of the first-stage training FEFP is defined as follows:

[0084] L FEFP = L ort + L tri

[0085] 2. Second-stage training

[0086] The second-stage training aims to jointly train the sketch-photo cross-reconstruction module and the global feature embedding module by using the triplet loss L tri , the reconstruction loss L rec and the structure alignment loss Lx a . In this training stage, the parameters of the feature extraction module and the identity discriminative feature retention module trained in the first stage are loaded and fixed.

[0087] Specifically, the triplet loss L tri is introduced to reduce the distance between the joint features and the joint features that contain the same identity information, while increasing the distance between the joint features and the joint features that contain different identity information. The formula of the triplet loss is expressed as follows:

[0088]

[0089] where represents the joint feature obtained from the face photo whose identity information is different from that contained in .

[0090] To enable u s and u p to store the prototype features of the face sketch modality and the face photo modality respectively, the reconstruction loss L rec is introduced to constrain the consistency between the feature reconstructed from based on u s and , and the consistency between the feature reconstructed from based on u p and . The specific formula of the reconstruction loss is expressed as follows:

[0091]

[0092] where represents the feature reconstructed from , represents the feature reconstructed from . MSE(·,·) represents the calculation of the mean square error.

[0093] In addition, the structural alignment loss L sa is adopted to constrain the distribution consistency with so as to encourage u s and u p to be structurally aligned. The specific formula is expressed as follows:

[0094]

[0095] where KL(·||·) represents the calculation of KL divergence, represents the relationship matrix between s and u represents the relationship matrix between p and u

[0096] In summary, the total loss L FA of the second-stage training is defined as follows:

[0097] L FA = L tri + αL rec + βL sa

[0098] where α and β are both loss balance weights. In the embodiment of the present invention, α is set to 1.0 and β is set to 100.0.

[0099] Those skilled in the art can understand that the attached drawings are only schematic diagrams of a preferred embodiment, and the serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0100] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A sketch face recognition method, characterized in that: The method comprises: Construct a feature extraction module to extract face sketch features and face photo features through face sketch feature extractor and face photo feature extractor respectively; Design an identity-discriminative feature-preserving module to learn and remove identity-non-discriminative features from face sketch features and face photo features to capture the identity-discriminative features of each modality; Design a sketch-photo cross-reconstruction module to reconstruct the global features of the face photo based on the identity-discriminating features of the face sketch, and reconstruct the global features of the face sketch based on the identity-discriminating features of the face photo; A global feature embedding module is designed to embed the global features of the face photo reconstructed from the global features of the face sketch into the identity-discriminating features of the face sketch to obtain a joint feature that contains both the modal information of the face sketch and the modal information of the face photo. The global features of the face sketch reconstructed from the global features of the face photo are embedded into the identity-discriminating features of the face photo to obtain a joint feature that contains both the modal information of the face photo and the modal information of the face sketch. Sketch face recognition is achieved by measuring the Euclidean distance between the joint features of the two modalities. The constructed sketch face recognition overall network includes: feature extraction module, identity identification feature retention module, sketch-photo cross-reconstruction module and global feature embedding module; the overall network is trained, and faces are recognized based on the trained network.

2. A sketch face recognition method according to claim 1, characterized in that: The identity identification feature retention module is specifically: Attention-based write operation uses face sketch features to update the memory bank m s , using facial photo features to update the memory library m p , so that m s With m p It can store shared information in face sketch features and shared information in face photo features separately.

3. The sketch face recognition method according to claim 2, characterized in that: The identity identification feature retention module is specifically: For memory bank m s The i-th memory vector in calculate The face sketch feature output by the Lth convolutional block of the face sketch feature extractor The similarity between each feature vector in Take out and The nth eigenvector with the largest similarity : in, Indicates cosine similarity calculation, express The jth eigenvector in ; use The face sketch feature output by the L-1 convolutional block of the face sketch feature extractor renew like Among them, ← represents the update operation, δ represents a constant, d = c / 8, c represents the channel dimension of each memory vector, and Represents three mapping functions, and Aims to map the input feature embeddings to query q s , key k s Sum value v s ; Based on m s , through memory read operation, from Recalling non-discriminative features of identity The reading process is as follows: in, express With m s Relationship matrix; from Remove non-identifying features To obtain the identity-identifying features of the face sketch The formula is as follows: Among them, Θ represents element-wise subtraction; For face photo modality, learn identity-based non-discriminative features of face photos And from the facial features Remove the identity identification features of the face photo 4. The sketch face recognition method according to claim 1, characterized in that: The sketch-photo cross reconstruction module is specifically: Introducing two learnable memory banks and To assist in the reconstruction of global features of face sketches and face photos, where t represents the number of memory vectors in each memory bank, c represents the channel dimension of each memory vector, based on u s and u p , the global features of the face sketch obtained by passing the face sketch identity identification features through the sketch feature encoder Reconstructing the global features of face photos as follows: in, express with u s The relationship matrix between Indicates cosine similarity calculation; For the face photo modality, the global features of the face photo are obtained by passing the face photo identity identification features through the photo feature encoder in the same way as above Reconstructing the global features of face sketch 5. The sketch face recognition method according to claim 4, characterized in that: The global feature embedding module is specifically: in, Represents the identity identification features of face sketches, Represents the identity identification features of face photos, Represents the joint features of the face sketch modality, Represents the joint features of face photo modality, D p Denotes the photo feature decoder, D s Denotes the sketch feature decoder, σ(·,·) denotes the channel-level cascade, ∈(·) denotes the mapping function, which maps the input feature embedding to a joint feature space containing the face sketch modality information and the face photo modality information; the photo feature decoder D p With sketch feature decoder D s Four deconvolution-convolution blocks and one convolution layer are used to build the network. The mapping function ∈(·) consists of a fully connected layer. and The Euclidean distance between them is used to realize sketch face recognition.

Images