Sketch face recognition method

Abstract

The invention discloses a sketch face recognition method. The method comprises the following steps: respectively extracting face sketch features and face photo features through a face sketch feature extractor and a face photo feature extractor; designing an identity identification feature retention module, and learning and removing identity non-identification features from the face sketch features and the face photo features to capture identity identification features of each mode; a sketch-photo cross reconstruction module is designed, the face sketch features are utilized to reconstruct the face photo global features, and the face photo features are utilized to reconstruct the face sketch global features; embedding the reconstructed face photo global feature into the face sketch identity identification feature to obtain a joint feature containing two-mode information; embedding the global features of the reconstructed face sketch into the identity identification features of the face photo, and similarly obtaining a joint feature containing two-modal information; and training the whole network based on the first training stage and the second training stage, and recognizing the face based on the trained network.

Description

Technical Field

[0001] The present invention relates to the field of face recognition, and in particular to a sketch face recognition method. Background Art

[0002] In recent years, the rapid development of face scanning and imaging technology has promoted the rapid growth of two-dimensional face image data. Accurately identifying the identity of the person contained in the face image has become a research hotspot in the field of information forensics and security. As an important branch of face recognition, sketch face recognition aims to retrieve face photos with the same identity information as the face sketch from the database based on a given face sketch. It has been widely used in the fields of intelligent monitoring and identity authentication.

[0003] Intuitively, for any face sketch or face photo, the features of key facial regions such as eyes, nose, and mouth are discriminative for identifying different faces, while the features of other facial regions are less relevant to face recognition. In order to capture the features of key facial regions, Peng et al. used rectangular boxes to locate the key facial regions of face sketches and face photos, and then used convolutional neural networks to extract deep local descriptors from these regions for face recognition. However, due to the complexity of facial structure, the shapes of key facial regions are usually irregular, which will inevitably cause the key facial regions located by rectangular boxes to contain information that interferes with recognition, thereby limiting the improvement of recognition performance. Therefore, how to effectively learn identity discriminative features from face sketches and face photos still needs in-depth research.

[0004] In addition, face photos contain rich color and texture information, while face sketches only contain simple lines, which leads to huge cross-modal differences between face sketches and face photos. This cross-modal difference makes it extremely challenging to effectively align face sketches and face photos. To address the above challenges, Huang et al. designed a discriminative spectral regression method, which first maps face sketch features and face photo features into a common discriminative subspace, and then aligns cross-modal features in this space to achieve sketch face recognition. However, this method of mapping from high-dimensional to low-dimensional feature space inevitably leads to the loss of some key facial information, which in turn affects the performance of sketch face recognition. Therefore, it is crucial to explore a reliable feature mapping method to achieve more effective cross-modal feature alignment for sketch face recognition. Summary of the invention

[0005] In order to effectively extract and align the identity-discriminating features of face sketches and face photos for identification, the present invention proposes a sketch face recognition method, which mines and removes identity-discriminating features from face sketch features and face photo features respectively to effectively capture the identity-discriminating features of each modality; reconstructs face sketch features into face photo global features through sketch-photo feature cross-reconstruction, and reconstructs face photo features into face sketch global features; embeds the reconstructed global features of a certain modality into the features of another modality to obtain cross-modally aligned face sketch features and face photo features, thereby achieving effective sketch face recognition, as described below:

[0006] A sketch face recognition method, the method comprising:

[0007] The face sketch feature extractor and the face photo feature extractor are used to extract the face sketch feature and the face photo feature respectively;

[0008] Design an identity-discriminative feature-preserving module to learn and remove identity-non-discriminative features from face sketch features and face photo features to capture the identity-discriminative features of each modality;

[0009] Design a sketch-photo cross reconstruction module to reconstruct the global features of the face photo using the features of the face sketch, and reconstruct the global features of the face sketch using the features of the face photo;

[0010] The global features of the face photo reconstructed from the global features of the face sketch are embedded in the identity-discriminating features of the face sketch to obtain a joint feature that contains both the modal information of the face sketch and the modal information of the face photo; the global features of the face sketch reconstructed from the global features of the face photo are embedded in the identity-discriminating features of the face photo to obtain a joint feature that contains both the modal information of the face photo and the modal information of the face sketch; the overall network is trained based on the first and second training stages, and the face is recognized based on the trained network.

[0011] The identity identification feature retention module is specifically:

[0012] Attention-based write operation uses face sketch features to update the memory bank m s , using the face photo features to update the memory library m p , so that m s With m p It is possible to store shared information in face sketch features and shared information in face photo features separately.

[0013] The identity identification feature retention module is specifically:

[0014] For memory banks 中 The i-th memory vector calculate and The similarity between each feature vector in Take out and The nth eigenvector with the largest similarity

[0015]

[0016] in, represents the cosine similarity calculation, express The j-th eigenvector in ;

[0017] use and renew as follows:

[0018]

[0019] Among them, ← represents the update operation, δ represents a constant, d = c / 8, and Represents three mapping functions, and Aims to map the input feature embeddings to query q s , key k s Sum value v s ;

[0020] Based on m s , through the memory read operation, from Recalling non-discriminative features of identity The reading process is as follows:

[0021]

[0022] in, express With m s The relationship matrix of

[0023] from Remove non-identifying features from the To obtain the identity-identifying features of the face sketch The formula is as follows:

[0024]

[0025] in, Represents element-wise subtraction;

[0026] For face photo modality, learn identity-based non-discriminative features of face photos And from the facial photo features Remove the identity features of the face photo

[0027] The sketch-photo cross-reconstruction module is specifically:

[0028] Introducing two learnable memory banks and To assist in the reconstruction of global features of face sketches and face photos, where t represents the number of memory vectors in each memory bank, based on u s and u p ,Will Reconstructing the global features of face photos as follows:

[0029]

[0030] in, express with u s The relationship matrix between them;

[0031] For the face photo modality, the global features of the face photo are transformed into Reconstructing the global features of face sketch

[0032] Wherein, the feature embedding is:

[0033]

[0034] Among them, D p represents the photo feature decoder, D s represents the sketch feature decoder, σ(·,·) represents the channel-level cascade, ∈(·) represents the mapping function, which maps the input feature embedding to a joint feature space containing face sketch modality information and face photo modality information; photo feature decoder D p With the sketch feature decoder D s Four deconvolution-convolution blocks and one convolution layer are used to build the mapping function ∈(·) which consists of a fully connected layer. and The Euclidean distance between them is used to realize sketch face recognition.

[0035] The beneficial effects of the technical solution provided by the present invention are:

[0036] 1. The present invention realizes effective sketch face recognition by mining and aligning the identity-identifying features of face sketches and face photos;

[0037] 2. The present invention designs an identity discriminative feature preservation module, which captures the identity discriminative features of each modality by learning and removing identity non-discriminative features from face sketch features and face photo features;

[0038] 3. The present invention designs a sketch-photo cross-reconstruction module and a global feature embedding module, which reconstructs the global features of a certain modality and embeds them into another modality to capture more aligned features across modalities for sketch face recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A sketch face recognition method. DETAILED DESCRIPTION

[0040] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0041] The specific implementation of the sketch face recognition method in the embodiment of the present invention is explained below by using examples.

[0042] 1. Constructing feature extraction module

[0043] First, a feature extraction module is constructed, including: a face sketch feature extractor and a face photo feature extractor. Specifically, the face sketch feature extractor and the face photo feature extractor are used to extract face sketch features. and facial photo features Where L represents the number of convolutional blocks in each feature extractor. and They represent the features output by the i-th convolutional block in the face sketch feature extractor and the face photo feature extractor, respectively.

[0044] In the embodiment of the present invention, both the face sketch feature extractor and the face photo feature extractor are constructed using the first four convolution blocks of the ResNet50 network, that is, L is set to 4. The ResNet50 network is well known to those skilled in the art, and the embodiment of the present invention will not be described in detail.

[0045] 2. Designing an Identity-Retaining Feature Module

[0046] In order to adaptively learn and remove identity non-discriminative features from face sketch features and face photo features to capture the identity discriminative features of each modality, an identity discriminative feature preservation module is designed.

[0047] Specifically, in order to adaptively learn non-discriminative features from face sketch features and face photo features, two learnable memory banks are first introduced and Where r represents the number of memory vectors in each memory bank, r is set to 128, c represents the channel dimension of each memory vector, c is set to 1024, s represents the sketch modality, and s represents the photo modality. Then, through the attention-based write operation, the face sketch features are used to update m s , update m using face photo features p , so that m s With m p It can store the shared information in the face sketch features and the shared information in the face photo features respectively. Taking the face sketch modality as an example, for m s The nth memory vector in calculate and The similarity between each feature vector in , where represents the face sketch feature output by the pth convolutional block of the face sketch feature extractor. Take out and The Lth eigenvector with the largest similarity The specific formula is as follows:

[0048]

[0049] in, represents the cosine similarity calculation, express The j-th eigenvector in .

[0050] Then, use and renew in, Represents the face sketch feature output by the L-1th convolutional block of the face sketch feature extractor. The process is expressed as follows:

[0051]

[0052] Among them, ← represents an update operation, δ represents a constant 0.99, and d = c / 8. and Represents three mapping functions. and Aims to map the input feature embeddings to query q s , key k s Sum value v s In an embodiment of the present invention, It consists of a fully connected layer. and Both are composed of two fully connected layers.

[0053] Then, based on m s , through the memory read operation, from Recalling non-discriminative features of identity The reading process is shown as follows:

[0054]

[0055] in, express With m s The relationship matrix.

[0056] Finally, from Remove non-identifying features from the To obtain the identity-identifying features of the face sketch The formula is as follows:

[0057]

[0058] in, Represents element-wise subtraction.

[0059] For the face photo modality, the designed identity discriminative feature preserving module can learn the identity non-discriminative features of face photos in the same way as above. And from the facial photo features Remove it from the image to obtain the identity identification features of the face photo

[0060] 3. Design Sketch-Photo Cross-Reconstruction Module

[0061] Considering the potential similarity between the features of face sketches and face photos of the same identity, a sketch-photo cross-reconstruction module is designed to reconstruct the global features of face photos using face sketch features, and reconstruct the global features of face sketches using face photo features. Specifically, this module first introduces two learnable memory banks: and To assist in the reconstruction of the global features of the face sketch and the global features of the face photo, where t represents the number of memory vectors in each memory bank, and t is set to 4. Taking the face sketch modality as an example, the identity identification features of the face sketch are Feed a sketch feature encoder E s To obtain the global features of the face sketch The formula is as follows:

[0062]

[0063] In the embodiment of the present invention, the sketch feature encoder E s The last convolutional block of the ResNet50 network and the subsequent fully connected layer are used for construction.

[0064] Afterwards, based on u s and t p ,Will Reconstructing the global features of face photos The reconstruction process is shown as follows:

[0065]

[0066] in, express with u s The relationship matrix between them.

[0067] For the face photo modality, the global features of the face photo are transformed into Reconstructing the global features of face sketch

[0068] 4. Design of global feature embedding module

[0069] A global feature embedding module is designed to embed the global features of one modality reconstructed by the sketch-photo cross reconstruction module into the features of another modality to map the face sketch features and face photo features into a common feature space, thereby capturing cross-modal aligned features for face recognition. Global features of reconstructed face photos Embedding discriminative features into face sketches To obtain the joint features Similarly, the global features of the face photo Global features of reconstructed face sketch Embedding identity discriminative features into face photos To obtain the joint features The feature embedding process is expressed as follows:

[0070]

[0071] Among them, D p represents a photo feature decoder, D s represents a sketch feature decoder, σ(·,·) represents a channel-level cascade, and ∈(·) represents a mapping function, which aims to map the input feature embedding to a joint feature space containing face sketch modality information and face photo modality information. In this embodiment of the present invention, the photo feature decoder D p With the sketch feature decoder D s They are all built with four deconvolution-convolution blocks and one convolution layer, and the mapping function ∈(·) consists of a fully connected layer.

[0072] Finally, by measuring and The Euclidean distance between them is used to realize sketch face recognition.

[0073] 5. Training Sketch Face Recognition Network

[0074] The network training proposed in the embodiment of the present invention is divided into two stages.

[0075] 1. First phase of training

[0076] The first stage of training aims to use the orthogonal loss L ort And the triplet loss L tri Jointly train the feature extraction module and the identity-discriminating feature preservation module.

[0077] Specifically, the orthogonal loss L is introduced ort Measuring the identity discriminative features of face sketches Non-identifying characteristics Similarity between people and identity identification features of face photos Non-identifying characteristics The specific formula of orthogonal loss is as follows:

[0078]

[0079] in, and Respectively and The i-th eigenvector in . |·| represents the absolute value calculation. N represents the number of eigenvectors contained in each identity discriminative feature or identity non-discriminative feature, and N is set to 256.

[0080] In addition, the triplet loss L is adopted tri To reduce the identity discriminative features of face sketches containing the same identity information and facial photo identity identification features The distance between the two faces is increased, and the identity identification features of the face sketches containing different identity information are increased. and facial photo identity identification features The triplet loss formula is expressed as follows:

[0081]

[0082] in, Indicates from The identity discriminative features extracted from the face photos with different identity information. D(·,·) represents the calculation of the Euclidean distance between two features. Δ represents a threshold constant, and Δ is set to 0.3.

[0083] By combining the above two losses, the total loss L in the first stage of training FEFP The definition is as follows:

[0084] L FEFP =L ort +L tri

[0085] 2. Second stage training

[0086] The second stage of training aims to use the triplet loss L tri , reconstruction loss L rec And the structural alignment loss Lx a The sketch-photo cross reconstruction module and the global feature embedding module are jointly trained. In this training phase, the parameters of the feature extraction module and the identity discrimination feature retention module trained in the first phase are loaded and fixed.

[0087] Specifically, the triplet loss L is introduced tri To narrow down the joint features that contain the same identity information and joint features distance between them, and at the same time increase the joint features containing different identity information and joint features The triplet loss formula is expressed as follows:

[0088]

[0089] in, Indicates from The joint features obtained from face photos with different identity information.

[0090] To make u s and u p The prototype features of the face sketch modality and the face photo modality can be stored respectively, and the reconstruction loss L is introduced rec To constrain Based on u s The self-reconstructed features and consistency, and Based on u p The self-reconstructed features and The specific formula of reconstruction loss is as follows:

[0091]

[0092] in, express The features obtained from the reconstruction are express Features obtained from self-reconstruction. MSE(·,·) represents mean square error calculation.

[0093] In addition, the structure alignment loss L is adopted sa To constrain and The distribution consistency of s with u p Aligned in structure, the specific formula is expressed as follows:

[0094]

[0095] Among them, KL(·||·) represents the KL divergence calculation, express with u s The relationship matrix between express with u p The relationship matrix between them.

[0096] In summary, the total loss L of the second stage training FA The definition is as follows:

[0097] L FA =L tri +αL rec +βL sa

[0098] Wherein, α and β are both loss balance weights. In the embodiment of the present invention, α is set to 1.0 and β is set to 100.0.

[0099] Those skilled in the art will appreciate that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0100] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A sketch face recognition method, characterized in that: The method comprises: The face sketch feature extractor and the face photo feature extractor are used to extract the face sketch feature and the face photo feature respectively; Design an identity-discriminative feature-preserving module to learn and remove identity-non-discriminative features from face sketch features and face photo features to capture the identity-discriminative features of each modality; Design a sketch-photo cross reconstruction module to reconstruct the global features of the face photo using the features of the face sketch, and reconstruct the global features of the face sketch using the features of the face photo; The global features of the face photo reconstructed from the global features of the face sketch are embedded in the identity-discriminating features of the face sketch to obtain a joint feature that contains both the modal information of the face sketch and the modal information of the face photo; the global features of the face sketch reconstructed from the global features of the face photo are embedded in the identity-discriminating features of the face photo to obtain a joint feature that contains both the modal information of the face photo and the modal information of the face sketch; the overall network is trained based on the first and second training stages, and the face is recognized based on the trained network.

2. A sketch face recognition method according to claim 1, characterized in that: The identity identification feature retention module is specifically: Attention-based write operation uses face sketch features to update the memory bank m s , using the face photo features to update the memory library m p , so that m s With m p It is possible to store shared information in face sketch features and shared information in face photo features separately.

3. A sketch face recognition method according to claim 2, characterized in that: The identity identification feature retention module is specifically: For memory bank m s The i-th memory vector in calculate and The similarity between each feature vector in Take out and The nth eigenvector with the largest similarity in, represents the cosine similarity calculation, express The j-th eigenvector in ; use and renew as follows: Among them, ← represents the update operation, δ represents a constant, d = c / 8, and Represents three mapping functions, and Aims to map the input feature embeddings to query q s , key k s Sum value v s ; Based on m s , through the memory read operation, from Recalling non-discriminative features of identity The reading process is as follows: in, express With m s The relationship matrix of from Remove non-identifying features from the To obtain the identity-identifying features of the face sketch The formula is as follows: in, Represents element-wise subtraction; For face photo modality, learn identity-based non-discriminative features of face photos And from the facial photo features Remove the identity features of the face photo 4. A sketch face recognition method according to claim 1, characterized in that: The sketch-photo cross reconstruction module is specifically: Introducing two learnable memory banks and To assist in the reconstruction of global features of face sketches and face photos, where t represents the number of memory vectors in each memory bank, based on u s and u p ,Will Reconstructing the global features of face photos as follows: in, express with u s The relationship matrix between them; For the face photo modality, the global features of the face photo are transformed into Reconstructing the global features of face sketch 5. A sketch face recognition method according to claim 4, characterized in that: The feature embedding is: Among them, D p represents the photo feature decoder, D s represents the sketch feature decoder, σ(·,·) represents the channel-level cascade, ∈(·) represents the mapping function, which maps the input feature embedding to a joint feature space containing face sketch modality information and face photo modality information; photo feature decoder D p With the sketch feature decoder D s Four deconvolution-convolution blocks and one convolution layer are used to build the mapping function ∈(·) which consists of a fully connected layer. and The Euclidean distance between them is used to realize sketch face recognition.

Images