Sketch face recognition method and device

Abstract

The invention discloses a sketch face recognition method and device. The method comprises the steps that a target sketch face image and a to-be-matched optical face image library are acquired, and the optical face image library comprises a plurality of optical face images; analyzing each optical face image and the target sketch face image in the optical face image library by using a sketch face synthesis model to obtain a plurality of target pseudo sketch face images; and analyzing the target sketch face image, the plurality of optical face images and the corresponding target pseudo-sketch face images by using a sketch face recognition model so as to determine a target optical face image from the plurality of optical face images, wherein the similarity between the target sketch face image and the target optical face image is not lower than a preset similarity threshold. The technical problem that the recognition accuracy of a related face sketch technology is low is solved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a sketch face recognition method and device. Background Art

[0002] With the rapid development of machine learning and deep learning, face recognition technology has received unprecedented attention and has been widely used in various fields such as video surveillance, access control systems, network applications, etc. Due to the variety of human faces, in addition to the most common optical images, there are also sketch images, stick figures, cartoon images, etc., especially in the field of criminal investigation, sketch face recognition (SFR) has gradually become a research hotspot in the field of face recognition technology.

[0003] The purpose of sketch face recognition is to achieve identity matching between sketch images and optical images, that is, to compare a given sketch image with an optical image with identity information in an optical face database, and to determine the identity of the sketch by identifying the most similar optical image, or to narrow the scope of reconnaissance by identifying some relatively similar optical images. However, since sketch samples are difficult to obtain, the sketch face dataset is small. This small sample problem makes cross-modal recognition between sketch images and optical images difficult to achieve.

[0004] In addition, the algorithms currently used for sketch face recognition can be divided into the following two categories: one is the cross-modal algorithm. The cross-modal algorithm is also called the intra-modal algorithm. It relies on sketch face synthesis to convert the face sketch image (optical image) into an optical image (sketch image) to reduce the difference between the modalities; the other is the common subspace algorithm. The common subspace algorithm is also called the inter-modal algorithm. It divides the image features of two different modalities into a common space and then uses a classifier for recognition. However, the above two methods still use the Convolutional Neural Network (CNN) as the backbone network to construct a model for extracting the features of optical images and sketch images. This model does not have good generalization ability and is only suitable for high-quality data sets. It cannot meet the needs of low-quality sketch image recognition in real scenes.

[0005] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0006] The embodiments of the present application provide a sketch face recognition method and device to at least solve the technical problem of low recognition accuracy of related sketch face technologies.

[0007] According to one aspect of an embodiment of the present application, a sketch face recognition method is provided, comprising: obtaining a target sketch face image and an optical face image library to be matched, wherein the optical face image library includes multiple optical face images; using a pre-trained sketch face synthesis model to analyze each optical face image and the target sketch face image in the optical face image library to obtain multiple target pseudo-sketch face images, wherein the sketch face synthesis model includes a global sketch synthesis module and multiple local sketch refinement modules, and the local sketch refinement module is used to perform local analysis on each facial part in the optical face image and the sketch face image to obtain a local pseudo-sketch image; using a pre-trained sketch face recognition model to analyze the target sketch face image, multiple optical face images and the corresponding target pseudo-sketch face image to determine a target optical face image from the multiple optical face images whose similarity with the target sketch face image is not less than a preset similarity threshold.

[0008] Optionally, the training process of the sketch face synthesis model includes: obtaining a first training set, wherein the first training set includes a first training sample set and a first sample label set, and the first training sample set includes multiple sketch face sample images and multiple optical face sample images, and the first sample label set includes a face label of each sketch face sample image and a face label of each optical face sample image; constructing an initial synthesis model; and iteratively training the initial synthesis model using the first training set to obtain the sketch face synthesis model.

[0009] Optionally, the global sketch synthesis module and each local sketch refinement module include a generator, a discriminator and a block feature extractor, wherein the initial synthesis model is iteratively trained using the first training set to obtain a sketch face synthesis model, including: for each training batch in the iterative training process, randomly selecting a preset number of sketch face sample images and multiple optical face sample images from the first training sample set, and determining face labels corresponding to the selected preset number of sketch face sample images and face labels corresponding to the multiple optical face sample images from the first sample label set to obtain multiple groups of first training samples; for each group of first training samples corresponding to each training batch, the following steps are performed: using the global generator to train the optical face sample images The method comprises the following steps: performing processing to obtain a global pseudo-sketch face sample image, and using multiple local generators to process each face part in the optical face sample image to obtain multiple first local pseudo-sketch sample images, wherein the global generator includes an encoder and a decoder, and the encoder includes at least a residual block; dividing the global pseudo-sketch face sample image, the optical face sample image and the sketch face sample image according to the face parts, respectively, to obtain multiple corresponding second local pseudo-sketch sample images, multiple local optical sample images and multiple local sketch sample images; using a global discriminator to perform discriminant analysis on the global pseudo-sketch face sample image and the sketch face sample image to obtain a corresponding first authenticity discrimination result, and using multiple local discriminators to respectively discriminate the corresponding second local pseudo-sketch face sample images and the sketch face sample images. A first local pseudo-sketch sample image and a local optical sample image, a second local pseudo-sketch sample image and a local optical sample image of the same face part are subjected to discriminant analysis to obtain a corresponding second authenticity discrimination result and a third authenticity discrimination result; a global block feature extractor is used to process the global pseudo-sketch face sample image and the sketch face sample image to obtain a plurality of first image blocks corresponding to the global pseudo-sketch face sample image and a plurality of second image blocks corresponding to the sketch face sample image; and a plurality of local block feature extractors are used to process the plurality of first local pseudo-sketch sample images, the plurality of second local pseudo-sketch sample images and the plurality of local optical sample images respectively to obtain a plurality of third image blocks corresponding to each first local pseudo-sketch sample image and a plurality of a plurality of fourth image blocks corresponding to two local pseudo-sketch sample images and a plurality of fifth image blocks corresponding to each local optical sample image, wherein the first image block, the second image block, the third image block, the fourth image block and the fifth image block are respectively obtained by dividing the feature maps output by each convolution layer in the encoder of the global pseudo-sketch face sample image, the sketch face sample image, each first local pseudo-sketch sample image, each second local pseudo-sketch sample image and each local optical sample image; a global generative adversarial loss function is constructed based on the first authenticity discrimination results of multiple groups of first training samples, the sketch face sample images and the optical face sample images, and a global multi-layer block contrast loss function is constructed based on the first image blocks and the second image blocks corresponding to the multiple groups of first training samples;Based on the second authenticity discrimination results of multiple groups of first training samples, multiple first local pseudo sketch sample images and multiple local optical sample images, a first type of local generative adversarial loss function is constructed, and based on the third authenticity discrimination results of multiple groups of first training samples, multiple local optical sample images and multiple second local pseudo sketch sample images, a second type of local generative adversarial loss function is constructed; based on the third image block and the fifth image block corresponding to multiple groups of first training samples, a first type of multi-layer block contrast loss function is constructed, and based on the fourth image block and the fifth image block corresponding to multiple groups of first training samples, a second type of multi-layer block contrast loss function is constructed; based on the third image block corresponding to the first local pseudo sketch sample image and the fourth image block corresponding to the second local pseudo sketch sample image of the same face part in multiple groups of first training samples, a local consistency loss function is constructed; based on the global generative adversarial loss function, the global multi-layer block contrast loss function, the first type of local generative adversarial loss function, the second type of local generative adversarial loss function, the first type of multi-layer block contrast loss function, the second type of multi-layer block contrast loss function and the local consistency loss function, a first target loss function is constructed; the model parameters of the initial synthesis model are adjusted using the first target loss function to obtain a sketch face synthesis model. ;

[0010] Optionally, a global block feature extractor is used to process a global pseudo-sketch face sample image and a sketch face sample image to obtain multiple first image blocks corresponding to the global pseudo-sketch face sample image and multiple second image blocks corresponding to the sketch face sample image, including: obtaining multiple first feature maps obtained by encoding the global pseudo-sketch face sample image using multiple convolutional layers in the encoder, and using the global block feature extractor to perform block processing on each first feature map to obtain multiple first image blocks corresponding to each first feature map; obtaining multiple second feature maps obtained by encoding the optical face sample image using multiple convolutional layers in the encoder, and using the global block feature extractor to perform block processing on each second feature map to obtain multiple second image blocks corresponding to each second feature map, wherein the second image block at the same position in the feature map output by the same convolutional layer as the first image block is used as a positive block, and the second image block at a different position in the feature map output by the same convolutional layer as the first image block is used as a negative block.

[0011] Optionally, the training process of the sketch face recognition model includes: obtaining a second training set, wherein the second training set includes a second training sample set and a second sample label set, and the second training sample set includes: the first training sample set and multiple global pseudo sketch face sample images obtained by processing the first training sample set using a sketch face synthesis model, and the second sample label set includes: a face label for each sketch face sample image, a face label for each optical face sample image, and a face label for each global pseudo sketch face sample image; constructing an initial recognition model, wherein the initial recognition model includes at least: a convolutional neural network, a serialization module, a multi-layer grouped Transformer network, a generalized average pooling layer, a domain adaptive network, and each layer of the grouped Transformer network includes a multi-head self-attention module, a cross-group multi-head self-attention module, and a feedforward network module; using the second training set to iteratively train the initial recognition model to obtain a sketch face recognition model.

[0012] Optionally, the initial recognition model is iteratively trained using the second training set to obtain a sketch face recognition model, including: for each training batch in the iterative training process, randomly selecting a preset number of optical face sample images, sketch face sample images and global pseudo-sketch face sample images from the second training sample set, and determining the face labels of the selected preset number of optical face sample images, sketch face sample images and global pseudo-sketch face sample images from the second sample label set to obtain multiple groups of second training samples, wherein the face labels of the optical face sample images, sketch face sample images and global pseudo-sketch face sample images in each group of second training samples are the same; and the initial recognition model is trained using each of the multiple groups of second training samples corresponding to the multiple training batches to obtain a sketch face recognition model.

[0013] Optionally, the initial recognition model is trained using multiple groups of second training samples corresponding to multiple training batches to obtain a sketch face recognition model, including: for any target image in each group of second training samples, the following steps are performed: convolution processing is performed on the target image using a convolutional neural network to obtain a low-resolution feature map; the low-resolution feature map is serialized using a serialization module to obtain a visual marker sequence; the visual marker sequence is grouped using a multi-head self-attention network to obtain multiple first groups, and self-attention operations are performed on all serialized features in each first group, and the self-attention modeling results obtained from each group are spliced ​​to obtain grouped multi-head self-attention modeling results corresponding to each first group; and a cross-group multi-head self-attention network is used to perform cross-group grouping on multiple grouped multi-head self-attention modeling results. to obtain multiple second groups, and perform self-attention operation on all serialized features in each second group, and splice the self-attention modeling results obtained in each group to obtain the cross-group multi-head self-attention modeling results corresponding to each second group; use a feedforward network to analyze the multiple cross-group multi-head self-attention modeling results and the visual tag sequence to obtain an image feature vector, and use a generalized average pooling layer to process the image feature vector to obtain a target image feature vector; use a domain adaptive network to determine the similarity between the target image feature vectors of each target image in each group of second training samples; construct a second target loss function based on the similarity between the target image feature vectors of each target image in multiple groups of second training samples, and use the second target loss function to adjust the model parameters of the initial recognition model to obtain a sketch face recognition model.

[0014] Optionally, the feedforward network includes: a first normalization layer, a linear layer, and a second normalization layer, wherein the feedforward network is used to analyze multiple cross-group multi-head self-attention modeling results and a visual marker sequence to obtain an image feature vector, including: performing jump connections on multiple cross-group multi-head self-attention modeling results and a visual marker sequence to obtain a corresponding first jump connection result, and using a first normalization layer to normalize the first jump connection result to obtain a first modeling result; using a linear layer to perform FFN mapping on the first modeling result to obtain an FFN mapping result, and performing jump connections on the FFN mapping result and the first modeling result to obtain a corresponding second jump connection result; using a second normalization layer to normalize the second jump connection result to obtain a second modeling result; and performing jump connections on the second modeling result and a serialized feature to obtain an image feature vector.

[0015] Optionally, a second target loss function is constructed based on the Euclidean distance between the target image feature vectors of each target image in multiple groups of second training samples, including: determining the probability distribution of sketch face sample images in multiple groups of second training samples based on the Euclidean distance; and constructing the second target loss function based on the probability distribution.

[0016] According to another aspect of an embodiment of the present application, a sketch face recognition device is also provided, including: an acquisition module, used to acquire a target sketch face image and an optical face image library to be matched, wherein the optical face image library includes multiple optical face images; a synthesis module, used to analyze each optical face image and the target sketch face image in the optical face image library using a pre-trained sketch face synthesis model to obtain multiple target pseudo-sketch face images, wherein the sketch face synthesis model includes a global sketch synthesis module and multiple local sketch refinement modules, and the local sketch refinement module is used to perform local analysis on each facial part in the optical face image and the sketch face image to obtain a local pseudo-sketch image; a recognition module, used to analyze the target sketch face image, multiple optical face images and the corresponding target pseudo-sketch face image using a pre-trained sketch face recognition model to determine a target optical face image from the multiple optical face images whose similarity with the target sketch face image is not less than a preset similarity threshold.

[0017] In an embodiment of the present application, a global pseudo-sketch face image is synthesized by a global sketch synthesis module in a sketch face synthesis model, and a plurality of local sketch refinement modules are used to refine the local features of the synthesized global pseudo-sketch face image to obtain a pseudo-sketch face image with more refined details. Then, the pseudo-sketch face image, the sketch face sample image and the optical face image are input into a sketch face recognition model to obtain the similarity between the sketch image to be identified and each optical image, and the optical image whose similarity meets the preset conditions can be used as the recognition result of the target sketch image. Among them, the highly complex self-attention mechanism is optimized into two stages of group self-attention and cross-group self-attention through a grouped Transformer network. The group self-attention performs self-attention within the group by marking the group to improve efficiency. The cross-group self-attention realizes cross-group interaction of different group marks through cross-group reorganization to learn global contextual connections, which can capture long-distance features and enhance the key role of identification features in sketch face recognition, thereby effectively improving the accuracy of sketch face recognition, thereby solving the technical problem of low recognition accuracy of related sketch face technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 It is a hardware structure block diagram of an optional computer terminal for implementing the sketch face recognition method according to an embodiment of the present application;

[0020] Figure 2is a flow chart of an optional sketch face recognition method according to an embodiment of the present application;

[0021] Figure 3 is a schematic structural diagram of an optional initial synthesis model according to an embodiment of the present application;

[0022] Figure 4 is a processing flow chart of an optional global multi-layer block comparison module according to an embodiment of the present application;

[0023] Figure 5 is a schematic structural diagram of an optional initial recognition model according to an embodiment of the present application;

[0024] Figure 6 is a schematic diagram of the structure of an optional multi-layer packet Transformer network according to an embodiment of the present application;

[0025] Figure 7 is a schematic structural diagram of an optional sketch face recognition device according to an embodiment of the present application;

[0026] Figure 8 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] In addition, the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. For example, an interface is set up between this system and the relevant user or organization. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or organization.

[0030] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0031] Meta-learning, also known as "learning to learn", is to use previous knowledge and experience to learn new tasks, and has the ability to learn to learn. Meta-learning raises the learning level from data to tasks, and learns from a set of labeled tasks instead of learning from a set of labeled sample data in traditional deep learning, thereby achieving rapid learning of new class classifiers with a certain amount of training data.

[0032] Example 1

[0033] With the development of deep learning technology, methods based on convolutional neural networks (CNN) have become the mainstream of sketch face recognition (SFR). These methods can be mainly divided into intra-modal methods and inter-modal methods, among which:

[0034] Intra-modal methods refer to sketch face recognition algorithms based on CNN in the modality. This type of method converts the cross-modal image to the target modality through a CNN-based modality converter, and then calculates the similarity in the target modality to complete the identity recognition. However, the recognition effect of this type of method is directly related to the quality of the generated image, and when the modality is too large, it is difficult to describe the nonlinear relationship of the modality conversion. When the CNN-based modality converter generates an image of the target modality, the decoded target image is not structurally coherent and lacks detailed textures due to the lack of global connection of the encoded features.

[0035] Inter-modal methods refer to inter-modal sketch face recognition methods based on CNN. This type of method projects the extracted modal features into a common subspace to reduce modal differences. The goal is to learn a classifier that maximizes the inter-class distance while minimizing the intra-class distance to extract modality-invariant features for recognition. However, the difficulty of inter-modal methods based on CNN lies in the extraction of cross-modal consistent discriminant features. Due to the small size of the sketch face dataset, the model is prone to overfitting. At the same time, CNN as a feature extraction network is also limited to capturing local information and cannot establish the interactive relationship of semantic concepts in space, resulting in insufficient recognition performance. It is also affected by the semantic errors introduced in the sketch production process, causing the feature extraction network to ignore key features.

[0036] In order to solve the technical problem of low recognition accuracy of sketch face images in the above method, in an embodiment of the present invention, a sketch face synthesis model constructed by a generative adversarial network based on hierarchical block comparison is used to generate a pseudo sketch face image with finer details, and the sketch face image to be identified, the optical face image library to be matched, and the pseudo sketch face image generated based on the two are used as input features of the sketch face recognition model, so as to identify the target optical face image with the highest similarity to the sketch face image from the optical face image library as the final recognition result.

[0037] According to an embodiment of the present application, a method embodiment of a sketch face recognition method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0038] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a sketch face recognition method. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0039] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0040] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the sketch face recognition method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the sketch face recognition method of the above-mentioned application is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0041] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0042] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0043] In the above operating environment, Figure 2is a flow chart of an optional sketch face recognition method according to an embodiment of the present application, such as Figure 2 As shown, the method can be applied to a server, which can be a single server, a cluster server, or a cloud server, etc., and at least includes steps S202-S206:

[0044] Step S202, obtaining a target sketch face image and an optical face image library to be matched.

[0045] In the technical solution provided in step S202, the optical face image library includes a plurality of optical face images authorized by the user, where the optical face images may be face images taken by a camera of an actual face. The target sketch face image, i.e. the sketch face image to be identified, may be a face image obtained by sketching an actual face.

[0046] Step S204, using the pre-trained sketch face synthesis model to analyze each optical face image and the target sketch face image in the optical face image library to obtain multiple target pseudo sketch face images.

[0047] In the technical solution provided in step S204, it is considered that the traditional generative adversarial network only constitutes a dynamic "game" process through a generator and a discriminator. Therefore, a single generative adversarial loss cannot fully utilize the detail features contained in the image, especially for asymmetric image pairs (that is, the optical face image and the sketch face image input by the network have different labels). The facial details of the pseudo-sketch face image generated by it are not rich enough, resulting in poor accuracy of the subsequent face recognition process. To this end, the sketch face synthesis model used in the embodiment of the present application includes a global sketch synthesis module and multiple local sketch refinement modules, wherein the global sketch synthesis module is used to take an asymmetric image pair as input to finally obtain a global pseudo-sketch face image; and the local sketch refinement module is used to perform local analysis on each facial part in the asymmetric image pair (that is, the optical face image and the sketch face image) to obtain local pseudo-sketch images of multiple facial parts.

[0048] Step S206, using a pre-trained sketch face recognition model to analyze the target sketch face image, multiple optical face images and the corresponding target pseudo sketch face image, so as to determine a target optical face image whose similarity to the target sketch face image is not less than a preset similarity threshold from among the multiple optical face images.

[0049] In the technical solution provided in step S206, the target sketch face image, the optical face image and the target pseudo sketch face image synthesized by using the two images are used as inputs of the sketch face recognition model, and are analyzed, so as to determine the target optical face image whose similarity with the target sketch face image is not less than the preset similarity threshold value among the multiple optical face images. In other words, the target sketch face image, the multiple optical face images and the corresponding target pseudo sketch face image are analyzed by the sketch face recognition model to determine the similarity between the target sketch face image and each optical face image, and the similarities are sorted in order from large to small or from small to large, and the target optical face image with the maximum similarity is determined as the recognition result of the target sketch face image, or multiple target recognition results with similarities greater than the preset similarity threshold value are selected to narrow the reconnaissance range.

[0050] The above method of this embodiment is further introduced below.

[0051] As an optional implementation, in the technical solution provided in the above step S204, the training process of the above sketch face synthesis model may include the following steps S11-S13, wherein:

[0052] Step S11, obtaining a first training set, wherein the first training set includes a first training sample set and a first sample label set, and the first training sample set includes multiple sketch face sample images and multiple optical face sample images, and the first sample label set includes a face label of each sketch face sample image and a face label of each optical face sample image.

[0053] In the technical solution provided in the above step S11, a standard sketch face database such as the UOMSGFS database can be used as the first training set of the model, wherein the UOMSGFS database consists of 600 pairs of sketch optical face images authorized by the user. Multiple pairs of sketch optical face images are randomly obtained from the standard sketch face database as the first training sample set, and the first sample label set is composed of the face labels of the multiple groups of sketch optical face images.

[0054] Step S12, constructing an initial synthesis model.

[0055] Specifically, Figure 3 is a schematic diagram of an optional initial synthesis model according to an embodiment of the present application, such as Figure 3 The sketch face synthesis model includes a global sketch synthesis module ( Figure 3 above) and multiple local sketch refinement modules (to simplify the description, Figure 3The following shows only the structure of a local sketch refinement module, and the structures of the other local sketch refinement models are the same), and the global sketch synthesis module includes a generator G, a discriminator D and a block feature extractor H, and each local sketch refinement module also includes a local generator G k , local discriminator D k and local patch feature extractor H k Among them, the main functions of the generator, discriminator, and block feature extractor are as follows:

[0056] The generator consists of two parts: an encoder and a decoder. The encoder takes the optical image as input to obtain the corresponding feature map, and uses the decoder to decode the feature map to generate the corresponding pseudo sketch image.

[0057] The discriminator takes the pseudo-sketch image and the sketch image as input to discriminate the authenticity of the sketch image and the pseudo-sketch image;

[0058] The block feature extractor uses the feature maps of each convolution layer obtained by encoding the pseudo-sketch image and the sketch image and / or the optical image and the sketch image as input to divide the feature maps of each layer to obtain image blocks, and compare the image blocks of the pseudo-sketch image and the sketch image and / or compare the image blocks of the optical image and the sketch image. It should be noted that when comparing, ensure that the feature map levels corresponding to the compared image blocks and their positions in the feature map are the same.

[0059] Step S13, iteratively training the initial synthesis model using the first training set to obtain a sketch face synthesis model.

[0060] Optionally, the technical solution provided in the above step S13 can be implemented by the following steps:

[0061] Step S131, for each training batch in the iterative training process, randomly select a preset number of sketch face sample images and multiple optical face sample images from the first training sample set, and determine the face labels corresponding to the preset number of sketch face sample images and the face labels corresponding to the multiple optical face sample images from the first sample label set to obtain multiple groups of first training samples.

[0062] Step S132: for each group of first training samples corresponding to each training batch, perform the following steps:

[0063] Step 1: Processing the sketch face sample image using a global generator to obtain a global pseudo sketch face sample image (also referred to as a "pseudo sketch face sample image"), and processing each face part in the sketch face sample image using multiple local generators to obtain multiple first local pseudo sketch sample images, wherein the global generator includes an encoder and a decoder, and the encoder includes at least a residual block;

[0064] Step 2: Divide the global pseudo-sketch face sample image, the optical face sample image and the sketch face sample image according to face parts to obtain corresponding multiple second local pseudo-sketch sample images, multiple local optical sample images and multiple local sketch sample images;

[0065] Step 3: Use a global discriminator to perform discriminant analysis on the global pseudo-sketch face sample image and the sketch face sample image to obtain a corresponding first authenticity discrimination result, and use multiple local discriminators to perform discriminant analysis on the first local pseudo-sketch sample image and the local optical sample image, the second local pseudo-sketch sample image and the local optical sample image of the same face entry part, to obtain a corresponding second authenticity discrimination result and a third authenticity discrimination result;

[0066] Step 4: Use a global block feature extractor to process the global pseudo-sketch face sample image and the sketch face sample image to obtain multiple first image blocks corresponding to the global pseudo-sketch face sample image and multiple second image blocks corresponding to the sketch face sample image, and use multiple local block feature extractors to process the multiple first local pseudo-sketch sample images, the multiple second local pseudo-sketch sample images and the multiple local optical sample images respectively to obtain multiple third image blocks corresponding to each first local pseudo-sketch sample image, multiple fourth image blocks corresponding to each second local pseudo-sketch sample image and multiple fifth image blocks corresponding to each local optical sample image, wherein the first image block, the second image block, the third image block, the fourth image block and the fifth image block are respectively obtained by dividing the feature maps output by each convolutional layer in the encoder for the global pseudo-sketch face sample image, the sketch face sample image, the first local pseudo-sketch sample images, the second local pseudo-sketch sample images and the local optical sample images.

[0067] Step S133, constructing a loss function based on the above-mentioned results according to the following rules: constructing a global generative adversarial loss function based on the first authenticity judgment results, sketch face sample images and optical face sample images of multiple groups of first training samples, and constructing a global multi-layer block contrast loss function based on the first image blocks and second image blocks corresponding to the multiple groups of first training samples; constructing a first type of local generative adversarial loss function based on the second authenticity judgment results, multiple first local pseudo-sketch sample images and multiple local optical sample images of multiple groups of first training samples, and constructing a second type of local generative adversarial loss function based on the third authenticity judgment results, multiple local optical sample images and multiple second local pseudo-sketch sample images of multiple groups of first training samples; constructing a first type of multi-layer block contrast loss function based on the third image blocks and fifth image blocks corresponding to the multiple groups of first training samples, and constructing a second type of multi-layer block contrast loss function based on the fourth image blocks and fifth image blocks corresponding to the multiple groups of first training samples; constructing a local consistency loss function based on each first local pseudo-sketch sample image and each second local pseudo-sketch sample image of the same face part in the multiple groups of first training samples.

[0068] Specifically, for a given first training set D1 train = {P1,P2,…P N ; S1, S2, ... S N}.in Represents an optical face sample image represents the sketch face sample image, and j represents the face label (i.e., the category label). Therefore, in the iterative model training process, if the sample data of each training batch is selected sequentially, the model trained by these sample data can only distinguish the category labels of different images in the same batch well, but cannot distinguish the category labels of images between different batches. Therefore, in order to avoid the above problems, the embodiment of the present application can introduce a meta-learning training strategy that is effective for small samples to avoid model overfitting. Sample sampling using meta-learning method - from the first training set D1 with N pairs of images train M (M<N) pairs of images are randomly selected to form multiple groups of first training samples in each training batch.

[0069] For the first training sample used in each of the above training batches, Figure 3 The global sketch synthesis module and each local sketch refinement model within the shown model can be processed as follows:

[0070] (1) Global Sketch Synthesis Module

[0071] The global sketch synthesis module mainly includes a generator G, a discriminator D and a block feature extractor H. Among them, the generator G includes an encoder E and a decoder. In particular, the encoder adds a residual block in its middle layer to solve the network degradation problem after the number of network layers is deepened, which can improve the network training effect, thereby generating a more realistic and high-quality pseudo-sketch face image; the discriminator D used in the embodiment of the present application is preferably PatchGAN, because the discriminator only focuses on the local structure of the image, so it can effectively learn the high-frequency information and detail features of the image, while reducing the number of parameters in the training process and improving the training efficiency; the block feature extractor H is used to divide the feature map generated by each convolutional layer in the encoder into blocks and generate a block feature stack.

[0072] Specifically, the generator G is used to process the optical face sample images P in each group of first training samples to obtain the corresponding global pseudo sketch face sample images S i ', use the discriminator D to analyze the global pseudo sketch face sample image S i ' and sketch face sample image S i Perform discriminant analysis to obtain the corresponding first authenticity discrimination result. i The generated global pseudo sketch face sample image S i ', sketch face sample image S i The first authenticity discrimination result constructs a global generation adversarial loss function:

[0073] L GAN (G,D,P,S)=E Pi～p log(1-D(G(P i )))-E Si～S log(1-D(S i ))

[0074] In addition, the optical face sample image P input by the generator G i The output global pseudo sketch face sample image is closely related. Therefore, the embodiment of the present application can obtain the optical face sample image P by the following method: i and the optical face sample image P i The corresponding multiple image blocks are used to make the optical face sample image P i and the global pseudo sketch face sample image S i ' to maximize the mutual information between them to enhance the optical face sample image P i and the global pseudo sketch face sample image S i 'The connection between.

[0075] Optionally, multiple first feature maps are obtained by encoding the global pseudo-sketch face sample image using multiple convolutional layers in the encoder, and each first feature map is divided into blocks using a block feature extractor to obtain multiple first image blocks corresponding to each first feature map; multiple second feature maps are obtained by encoding the optical face sample image using multiple convolutional layers in the encoder, and each second feature map is divided into blocks using a block feature extractor to obtain multiple second image blocks corresponding to each second feature map.

[0076] Figure 4 is a processing flow chart of an optional global multi-layer block comparison module according to an embodiment of the present application, such as Figure 4 As shown. For the global pseudo sketch face sample image S i ′, obtain multiple first feature maps encoded by each convolution layer in the encoder E in the generator G, and use the block feature extractor H to divide each first feature map to obtain multiple first image blocks (also called query blocks). For the optical face sample image P, obtain multiple second feature maps encoded by each convolution layer in the encoder E in the generator G, and use the block feature extractor H to divide each second feature map to obtain multiple second image blocks, wherein the second image block at the same position in the feature map output by the same convolution layer as the first image block is used as the positive class block, and the second image block at a different position in the feature map output by the same convolution layer as the first image block is used as the negative class block. In order to ensure that the query block is most associated with the positive class block and least associated with the negative class block, N negative class blocks and one positive class block can be selected at different positions of the input optical face sample image to construct N+1 classifications, and a cross entropy loss function can be constructed based on the classification results of the query block and each block, so that the probability of the positive class block being selected is greater than that of the negative class block. The expression of the cross entropy loss function constructed above can be written as:

[0077]

[0078] Where ν represents the query block, ν + represents the positive class block, ν - represents a negative block, τ represents a scaling factor, and in the embodiment of the present application, τ is preferably set to 0.07.

[0079] Based on the cross entropy loss function defined above, we calculate l and calculate the global pseudo-sketch face sample images S i 'The corresponding first image block and optical face sample image P i The corresponding second image block constructs a global multi-layer block contrast loss function:

[0080]

[0081] In the formula It represents the global face sample four images S i 'The first feature map output by the mth layer of encoder E is then divided into blocks by block feature extractor H to obtain the nth query block; It represents the optical face sample image P i The second feature map output by the mth layer of the encoder E is then divided into blocks by the block feature extraction network H to obtain the nth positive block; It represents the optical face sample image P i The second feature map output by the mth layer of the encoder E is divided into blocks by the block feature extraction network H to obtain blocks other than positive blocks (negative blocks).

[0082] (2) Local sketch refinement module

[0083] The number of local refinement sketch modules can be determined according to the number of facial parts of the image. In the embodiment of the present application, four local refinement sketch modules are used as an example, and each local refinement sketch module is used to analyze the four facial parts of the left eye, right eye, nose, and mouth. Each local refinement sketch module includes a generator, a discriminator, and a block feature extractor. Among them:

[0084] Four generators G k For each group of optical face images P in the first training sample i The left eye P iel , right eye P ier 、Nose P in 、Mouth P im Processing is performed to obtain the corresponding first local pseudo sketch sample image

[0085] S′ iel ,S′ ier ,S′ in ,S′ im ;

[0086] Four discriminators D k The local optical sample image P iel ,P ier ,P in ,P im The generated first partial pseudo sketch face sample image S i ' el ,S i ' er ,S i ' n ,S i ' m With the local optical sample image P iel ,P ier ,P in ,Pim Perform discriminant analysis to obtain corresponding sets of second authenticity discrimination results, and then use the first partial pseudo sketch face sample image S i ' el ,S i ' er ,S i ' n ,S i ' m With the local optical sample image P iel ,P ier ,P in ,P im The second authenticity discrimination result of constructs a local generation adversarial loss function (i.e., the first type of local generation adversarial loss function); at the same time, the four discriminators D k We can also divide the global pseudo sketch face sample image S i The obtained second local pseudo sketch sample image S i ' iel ,S i ' ier ,S i ' in ,S i ' im With the local optical sample image P iel ,P ier ,P in ,P im Perform discriminant analysis to obtain the third authenticity discrimination result, and according to the second local pseudo sketch sample image S i ' iel ,S i ' ier ,S i ' in ,S i ' im With the local optical sample image P iel ,P ier ,P in ,P im The third authenticity discrimination result constructs a global partial generation adversarial loss function (i.e., the second type of local generation adversarial loss function);

[0087] Four block feature extractors H k Can be used with four generators G k Encoder E k A local multi-layer block comparison module is formed, which can transform the first local pseudo sketch sample image S i ' el ,S i ' er ,S i ' n ,S i 'm The corresponding third image block (which is a block of the first image block corresponding to the global pseudo sketch image) and the local optical sample image P iel ,P ier ,P in ,P im The corresponding fifth image block is compared, and a local multi-layer block contrast loss function (i.e., the first type of multi-layer block contrast loss function) is constructed according to the comparison result. At the same time, the local multi-layer block contrast module can also convert the second local pseudo sketch sample image S i ' iel ,S i ' ier ,S i ' in ,S i ' im The corresponding fourth image block and the local optical sample image P iel ,P ier ,P in ,P im The corresponding fifth image block is compared, and a global partial multi-layer block contrast loss function (i.e., the second type of multi-layer block contrast loss function) is constructed according to the comparison result, wherein the processing flow of the local multi-layer block contrast module is the same as that of the global multi-layer block contrast module.

[0088] The specific expressions of the above loss functions are explained below.

[0089] Specifically, the expression of the local generative adversarial loss function (i.e., the first type of local generative adversarial loss function) can be written as:

[0090]

[0091] The expression of the global partial generation adversarial loss function (i.e. the second type of local generation adversarial loss function) can be written as:

[0092]

[0093] In addition, when constructing the first type of multi-layer block contrast loss function, the local multi-layer block contrast module can perform the following process on the first local pseudo sketch sample image S i ' el ,S i ' er ,S i ' n ,S i ' m and the local optical sample image P iel ,P ier ,P in ,P im To process:

[0094] For the first local pseudo sketch sample image S i ' el ,S i ' er ,S i ' n ,S i ' m For example, get the generator G k Encoder E k The convolution layers in the image processing unit encode the multiple third feature maps (which are partial maps of the first feature maps obtained by the global pseudo sketch sample image through multiple convolution layers), and use the block feature extractor H k Each third feature map is divided to obtain a plurality of third image blocks (also called query blocks). iel ,P ier ,P in ,P im For example, get the generator G k Encoder E k The convolution layers in the image processing unit encode the plurality of fifth feature maps (which are partial maps of each second feature map obtained by the optical face sample image through multiple convolution layers), and use the block feature extractor H k Each fifth feature map is divided to obtain a plurality of fifth image blocks, wherein the fifth image block at the same position in the feature map output by the same convolution layer as the third image block is used as a positive class block, and the fifth image block at a different position in the feature map output by the same convolution layer as the third image block is used as a negative class block. Therefore, based on the cross entropy loss function operation l defined above, and according to the first local pseudo sketch sample image S i ' el ,S i ' er ,S i ' n ,S i ' m The corresponding third image block and the local optical sample image P iel ,P ier ,P in ,P im The corresponding sixth image block constructs a local multi-layer block contrast loss function (i.e., the first type of multi-layer block contrast loss function), which can be written as:

[0095]

[0096] In the formula The first partial pseudo sketch image S of the nose part i ' n In encoder E in The feature map output by the mth convolutional layer is then extracted by the local block feature extractor Hin The nth query block obtained after block division; It represents the local optical image P of the nose part in Via encoder E in The feature map output by the mth convolutional layer is then extracted by the local block feature extractor H in The nth query block obtained after block division; It represents the local optical image P of the nose part in Via encoder E in The feature map output by the mth convolutional layer is then extracted by the local block feature extractor H in After segmentation, blocks other than positive blocks (negative blocks) are obtained.

[0097] When constructing the second type of multi-layer block contrast loss function, the local multi-layer block contrast module can perform the following process on the second local pseudo sketch sample image S i ' iel ,S i ' ier ,S i ' in ,S i ' im and the local optical sample image P iel ,P ier ,P in ,P im To process:

[0098] For the second local pseudo sketch sample image S i ' iel ,S i ' ier ,S i ' in ,S i ' im For example, get the generator G k Encoder E k The convolutional layers in the inner layer encode the multiple fourth feature maps, and use the block feature extractor H k Each third feature map is divided to obtain a plurality of fourth image blocks (also called query blocks). iel ,P ier ,P in ,P im For example, get the generator G k Encoder E k The convolution layers in the image processing unit encode the plurality of fifth feature maps (which are partial maps of each second feature map obtained by the optical face sample image through multiple convolution layers), and use the block feature extractor H kEach fifth feature map is divided to obtain a plurality of fifth image blocks, wherein the fifth image block at the same position in the feature map output by the same convolution layer as the third image block is used as a positive class block, and the fifth image block at a different position in the feature map output by the same convolution layer as the third image block is used as a negative class block. Therefore, based on the cross entropy loss function operation l defined above, and according to the second local pseudo sketch sample image S i ' iel ,S i ' ier ,S i ' in ,S i ' im The corresponding fourth image block and the local optical sample image P iel ,P ier ,P in ,P im The corresponding sixth image block constructs a global multi-layer block contrast loss function (i.e., the second type of multi-layer block contrast loss function), which can be written as:

[0099]

[0100] In the formula The second partial pseudo sketch image S of the nose part i ' n In encoder E in The feature map output by the mth convolutional layer is then extracted by the local block feature extractor H in The nth query block obtained after block division; It represents the local optical image P of the nose part in Via encoder E in The feature map output by the mth convolutional layer is then extracted by the local block feature extractor H in The nth query block obtained after block division; It represents the local optical image P of the nose part in Via encoder E in The feature map output by the mth convolutional layer is then extracted by the local block feature extractor H in After segmentation, blocks other than positive blocks (negative blocks) are obtained.

[0101] In addition, the embodiment of the present application also considers the second local pseudo sketch sample image S of the global pseudo sketch sample image generated by the global generator G i ' iel ,S i ' ier ,S i ' in ,S i ' im The fourth image block and the local generator G kThe first local pseudo sketch sample image S generated i ' el ,S i ' er ,S i ' n ,S i ' m The difference between the third image blocks is large, resulting in a global partial multi-layer block contrast loss L LpatchNCET The optimization effect of the global generator G is poor. Therefore, the embodiment of the present application also proposes to increase the local consistency loss function to reduce the second local pseudo sketch sample image S i ' iel ,S i ' ier ,S i ' in ,S i ' im With the first local pseudo sketch sample image S i ' el ,S i ' er ,S i ' n ,S i ' m Therefore, the expression of the local consistency loss function can be written as:

[0102] L1=|S i ' el -S i ' iel |+|S i ' el -S i ' ier |+|S i ' n -S i ' in |+|S i ' m -S i ' im |

[0103] Step S134, based on the global generation of adversarial loss function L GAN , global multi-layer block contrast loss function L patc h NCE , the first type of local generative adversarial loss function L LGANL , the second type of local generation adversarial loss function L LGANT , the first type of multi-layer block contrast loss function L LpatchNCEL , the second type of multi-layer block contrast loss function L LpatchNCETThe first objective loss function is constructed by combining the local consistency loss function L1. Therefore, the expression of the first objective loss function can be written as:

[0104] L total =aL LGANL +bL LpatchNCEL +cL LGANT +dL LpatchNCET +eL GAN +fL patchNCE +L1

[0105] Among them, the above a, b, c, d, e, and f all represent the coefficients of each loss function. In the embodiment of the present application, each parameter can be defined in advance as: c = d = 0, a = b = 0.25, e = f = 1, that is, the global generative adversarial loss L GAN Optimize the global generator G and the global discriminator D, using the global multi-layer block contrast loss L patchNCE Optimize the global generator G and the global patch feature extractor H to generate a local adversarial loss L LGANL Optimize the local generator G i With the local discriminator D k , with local multi-layer block contrast loss L LpatchNCEL Optimize the local generator G k With the local patch feature extractor H i , using the global part to generate adversarial loss L LGANT Optimize the local discriminator D k And the global generator G, and then define the parameters as: a = b = c = d = 0.25, e = f = 1, that is, the global multi-layer block contrast loss L LpatchNCET Optimize the global generator G and the local patch feature extractor H k .

[0106] The global sketch synthesis module in the sketch synthesis model constructed above generates a pseudo sketch face image, while the local sketch refinement module provides loss optimization for the local pseudo sketch image generated by the global sketch synthesis module, so as to obtain better local effects.

[0107] In addition, the embodiment of the present application adopts hierarchical optimization, that is, the global sketch synthesis module and the local sketch refinement module are first optimized simultaneously, so that the global sketch synthesis module can synthesize a better global pseudo-sketch face image, and after the global sketch synthesis module generates a better global pseudo-sketch face image, the global part generation adversarial loss and the global part multi-layer block contrast loss provided by the local sketch refinement module are introduced to optimize the global generator G for the local characterization.

[0108] As an optional implementation, in the technical solution provided in the above step S206, the training process of the sketch face recognition model includes the following steps S21-S23, wherein:

[0109] Step S21, obtaining a second training set.

[0110] The second training set includes a second training sample set and a second sample label set, and the second training sample set includes: the first training sample set and a plurality of global pseudo sketch face sample images obtained by processing the first training sample set using the sketch face synthesis model; and the second sample label set includes: a face label of each sketch face sample image, a face label of each optical face sample image, and a face label of each global pseudo sketch face sample image. That is, the first training set D1 is processed by the sketch face synthesis model. train The global pseudo sketch face sample images and corresponding labels generated by the optical face sample images in are added to the original first training set D1 train , forming the second training set.

[0111] Step S22, constructing an initial recognition model.

[0112] Specifically, the present embodiment considers that the current CNN-based SFR method cannot capture the global connection between features to enhance and extract cross-modal discriminative features, resulting in the problem that the recognition performance cannot be further improved. To this end, the structure of the initial recognition model proposed in the present embodiment is as follows: Figure 5 As shown, it includes: convolutional neural network, serialization module, multi-layer grouping Transformer network, generalized average pooling layer, domain adaptation network, where:

[0113] (1) Convolutional Neural Network

[0114] Feature extraction is performed on the optical face sample images and sketch face sample images in the second training sample set to obtain corresponding feature maps; and the serialization module is used to serialize the feature maps output by the convolutional neural network to obtain the corresponding visual marker sequence. It should be noted that since Transformer is a Seqence-to-Sequence framework, the feature maps output by the convolutional neural network can be serialized pixel by pixel into a visual marker sequence embedded in a grouped Transformer network, and the visual marker sequence includes visual markers corresponding to the optical face sample images and sketch face sample images in the second training sample set.

[0115] (2) Multi-layer Group Transformer Network

[0116] At present, the existing Transformer mainly includes MHSA (Multi-Head Self Attention) module and FFN module. The global correlation between tag features mainly relies on the MHSA module to perform self-attention calculation within the global tag range, and the calculation complexity is the square of the number of tags. Therefore, the existing Transformer often has the problem of high computational complexity. In addition, the serialized features obtained by the convolutional neural network and serialization module often contain a large number of tags, and the problem of high computational complexity is particularly serious.

[0117] Therefore, in order to avoid this problem, the embodiment of the present application adopts a grouped Transformer network as a feature extraction network for the sketch face recognition model, such as Figure 6 As shown, the multi-layer group Transformer network includes a multi-layer group Transformer network, and the group Transformer network includes a group multi-head self-attention module, a cross-group multi-head self-attention module and a feed-forward network (FFN) module.

[0118] Specifically, the grouped multi-head self-attention module is used to group the corresponding visual marker sequences in the sample set to obtain a first group with a preset number of markers, and perform self-attention operations in each first group to generate a grouped multi-head self-attention modeling result, and input the grouped multi-head self-attention modeling result into the cross-group multi-head self-attention module; the cross-group multi-head self-attention module is used to perform cross-group combination of the grouped multi-head self-attention modeling results to obtain a second group with a preset number of markers, and perform self-attention operations in each second group to generate a cross-group multi-head self-attention modeling result, and obtain the global connection between the visual markers corresponding to each training sample in the second training sample set, wherein the number of groups in the first group is equal to the preset number of markers in the second group, and the number of groups in the second group is equal to the preset number of markers in the first group.

[0119] It should be noted that the highly complex self-attention mechanism is optimized into two stages: group self-attention and cross-group self-attention through the grouped Transformer network. The group self-attention marks the groups to perform intra-group self-attention to improve efficiency, while the cross-group self-attention realizes cross-group interaction of different group labels through cross-group reorganization to learn global contextual connections, thereby capturing long-distance features and enhancing the key role of discriminative features in sketch face recognition, so as to effectively improve the accuracy of sketch face recognition.

[0120] (3) Domain Adaptive Network

[0121] The GeM pooling layer and the domain adaptation network can filter redundant feature information from the encoded output results of the grouped Transformer network and reduce the modality differences, obtaining the feature vectors corresponding to each optical face sample image and each sketch face sample image in the second training sample set, and projecting the feature vectors corresponding to each optical face sample image and sketch face sample image into a common subspace for paired recognition. Among them, by introducing the domain adaptation network, the impact of modality differences on the recognition performance can be reduced.

[0122] Step S23: Iteratively train the initial recognition model using the second training set to obtain a sketch face recognition model.

[0123] In the technical solution provided in the above step S23, considering the contradiction between the high complexity of the Transformer architecture and the small sample problem of sketch face recognition, directly training a Transformer network on small sketch face samples is prone to overfitting problems. Therefore, in order to effectively train a sketch face recognition model with a Transformer, in the embodiments of the present application, a meta-learning training strategy effective for small samples can be introduced to avoid model overfitting.

[0124] Therefore, the above step S23 can be implemented through the following steps: For each training batch in the iterative training process, randomly select a preset number of optical face sample images, sketch face sample images, and global pseudo-sketch face sample images from the second training sample set, and determine the face labels of the selected preset number of optical face sample images, sketch face sample images, and global pseudo-sketch face sample images from the second sample label set, obtaining multiple groups of second training samples. Among them, the face labels of the optical face sample images, sketch face sample images, and global pseudo-sketch face sample images in each group of second training samples are the same; use each group of second training samples corresponding to multiple training batches to train the initial recognition model to obtain a sketch face recognition model.

[0125] In the above embodiment, multiple meta-learning tasks "episodes" are generated according to a preset meta-learning strategy. Among them, the meta-learning strategy is to improve the learning level to the task. In each meta-learning task, the meta-learning strategy randomly samples M pairs (M < N) of sample pairs from the second training set with N pairs of images and resets their labels to {1, 2,..., M}. The goal of the meta-learning task is to correctly match these M pairs of samples as much as possible, where M is a hyperparameter of the number of batch sample pairs in the meta-learning strategy. The M pairs of samples selected form the query set of the meta-learning task "episode". In Q t In P t represents the optical face sample image support set, S trepresents the support set of sketch face sample images, S t ′ represents the global pseudo-sketch face sample image support set, and t={1,2…,T} represents the number of times the meta-learning task is executed. Among them, the face labels of the optical face sample images in the query set corresponding to each meta-learning task, the face labels of the sketch face sample images, and the face labels of the global pseudo-sketch face sample images correspond one to one. Perform multiple meta-learning tasks "episodes" to train the initial recognition model using the sample sets corresponding to each meta-learning task to obtain a sketch face recognition model.

[0126] It should be noted that in actual application scenarios, the selection of the number of samples M in each batch is crucial. A larger M can make the network converge better and faster, but the calculation will require a large amount of video memory resources. Although a smaller M avoids the problem of video memory overflow, the model cannot be effectively trained, resulting in a decrease in recognition accuracy. Therefore, the embodiment of the present application proposes a meta-learning sample pool, the essence of which is to train the sketch face synthesis model and the sketch face recognition model with different M, and update the two modules with different numbers of iterations within one epoch. The content is: the initial recognition model as a whole inputs the optical-sketch face sample image pair each time, and after passing through the sketch face synthesis model, the synthesized global pseudo sketch face sample image is injected into the meta-learning sample pool; and the sketch face recognition model needs to check whether the number of face labels in the meta-learning sample pool is M. Only when the number of data labels reaches M, the sketch face recognition module can obtain the M groups of pseudo sketch-optical-sketch face sample image triplets from the sample pool, and input them into the sketch face recognition model for iterative training, and finally clear the meta-learning sample pool, so as to ensure the use of meta-learning strategy to effectively train the sketch face recognition model, while avoiding the problem of insufficient video memory caused by large batch training of the overall network.

[0127] When the initial recognition model is trained using multiple groups of second training samples corresponding to multiple training batches, the initial recognition model can perform the following steps for any target image in each group of second training samples (i.e., any image in the triplet of pseudo-sketch-optical-sketch face sample images):

[0128] Step 1: Use a convolutional neural network to convolve the target image to obtain a low-resolution feature map.

[0129] Step 2: Use the serialization module to serialize the low-resolution feature map to obtain a visual marker sequence.

[0130] Step 3: Use a multi-head self-attention network to group the visual marker sequence to obtain multiple first groups, perform self-attention operations on all serialized features in each first group, and concatenate the self-attention modeling results obtained for each group to obtain the grouped multi-head self-attention modeling results corresponding to each first group.

[0131] The essence of the above-mentioned grouped multi-head self-attention module is to split the serialized features of each target image into multiple groups, and only calculate the association between the labeled contexts within the group, thereby reducing the number of tags when calculating the context association and improving the calculation efficiency.

[0132] Specifically, the visual marker sequence corresponding to each target image in the second training sample includes N(H×W)=x×y visual markers, where H and W represent the length and width of the feature map, respectively, x represents the number of groups, and y represents the number of markers in the group. in (T in =T+PE(T)), visual marking sequence = marking sequence + position code) is divided into x first groups according to the grouping strategy (i.e., grouping operation), each first group includes a marking sequence of y marks, which can be expressed as:

[0133] T i =G(T in ,y)∈{T1,T2,…,T x}

[0134] Where G(*,y) represents the grouping function that groups the visual marker sequence according to the number of markers y, and the x groups of serialized features obtained by grouping are sti∈{1,2,…,x}. x is a set of serialized features T i Multi-head self-attention calculations are completed in parallel to capture the contextual associations between tokens within each group.

[0135] In the multi-head self-attention of each set of labeled sequences, we first pass different transfer matrices W i q ,W i k ,W i v The embedded tag sequence is linearly mapped to h sets of query matrices, key matrices, and value matrices respectively. The mapping process can be expressed as follows:

[0136] Q ij ,K ij ,V ij =T i W j q ,T i Wj k ,T i W j v ,j∈{1,2,…,h}

[0137] In the formula, Q ij ,K ij ,V ij denote the query matrix, key matrix, and value matrix, respectively. h denotes the number of heads, d denotes the label dimension, and d q ,d k ,d v =d / h.

[0138] Next, the query matrix Q ij and key matrix K ij Performing a scaled dot product self-attention operation can obtain a single-head self-attention matrix between tokens within a group. After performing a softmax(·) normalization operation on the self-attention matrix, it is combined with the value matrix V ij Multiply to obtain the single-head self-attention modeling result of the group label, which can be written as:

[0139]

[0140] in Represents the single-head self-attention modeling result of the j-th head in the i-th first group in the grouped multi-head self-attention.

[0141] Then, the self-attention modeling results of all single heads in each group are concatenated along the label dimension to obtain the serialized feature T of the group i The multi-head self-attention modeling results:

[0142] GMSA(T i )=Concat(head i1 ,head i2 ,…head ih )

[0143] In the formula, GMSA(T i ) represents the multi-head self-attention modeling result of the i-th group of serialized features,

[0144] Finally, concatenating all grouped multi-head self-attention modeling results can obtain the grouped multi-head self-attention modeling results of the input serialization features:

[0145] T xy =Concat(GMSA(T1),GMSA(T2),…GMSA(T x ))

[0146] In the above formula, Concat(·) represents the concatenation function, T xy Represents the output of the grouped multi-head attention module, whose T xy There is a contextual connection between the tags within the group. However, due to the lack of attention interaction between the tags of different groups, it is impossible to establish a global contextual connection between the features. Therefore, in order to overcome the defects of group self-attention, the group Transformer then uses the group modeling tags T output by the group multi-head self-attention xy Cross-group interactions between markers of different groups are achieved through cross-group combination to capture the global connection between all markers.

[0147] Step 4: Use the cross-group multi-head self-attention network to perform cross-group combination of multiple group multi-head self-attention modeling results to obtain multiple second groups, and perform self-attention operations on all serialized features in each second group, and splice the self-attention modeling results obtained from each group to obtain the cross-group multi-head self-attention modeling results corresponding to each second group.

[0148] The essence of the cross-group multi-head self-attention network is to convert the output T of the grouped multi-head self-attention module xy Regroup by cross-group combination so that each new group contains T xy Each group has partial features, and the new group contains some tags of the original group, thereby realizing cross-group interaction of the original group tags, and then enabling the network to capture the global connection between all tags.

[0149] Specifically, the cross-group multi-head self-attention network follows the grouping strategy of N = y × x to group T xy After grouping, we get y groups of grouped serialized features with x tags:

[0150] T i ′=G(T xy ,x)∈{T1′,T2′,…,T y ′}

[0151] The grouping tag sequence sti∈{1,2,…,y}, and y groups of labeled features also perform multi-head attention calculations in parallel.

[0152] The tag sequence T obtained by grouping across groups i 'First, by grouping the transfer matrix W in the self-attention i q ,W i k ,W i v Projected to the new query matrix, key matrix and value matrix, the specific calculation method is as follows:

[0153] Q i ' j ,K i ' j ,V i ' j =T i 'W j q,T i 'W j k ,T i 'W j v ,j∈{1,2,…,h}

[0154] Next, similar to the operation in the grouped multi-head self-attention module, the query matrix Q′ ij and key matrix K i ' j Performing a scaled dot product self-attention operation can obtain a single-head self-attention matrix between tokens within a group. After performing a softmax(·) normalization operation on the self-attention matrix, it is combined with the value matrix V i ' j Multiply to obtain the single-head self-attention modeling result of the group label, which can be written as:

[0155]

[0156] Then, the self-attention modeling results of all single heads in each group are concatenated along the label dimension to obtain the serialized feature T of the group i The multi-head self-attention modeling result can be expressed as:

[0157] GMSA(T′ i ) = Concat(head′ i1 ,head′ i2 ,…head′ ih )

[0158] Finally, the multi-head self-attention modeling results of all groups are spliced ​​according to the number of tags. Since the reorganization of tags in the self-attention across groups changes the position of the tags in the space, direct splicing may cause the jump connection of the embedded tag sequence to be unreliable. Therefore, it is necessary to calibrate the spliced ​​tag sequence to reset the tag sequence in space. The calibrated serialized features are then passed through the parameter matrix W o The mapping can obtain the output T of the cross-group multi-head self-attention module yx :

[0159] T yx =Correct(Concat(GMSA(T1′),GMSA(T2′),…GMSA(T y ′)))Wo

[0160] In the above formula, Correct(·) represents a calibration function, which is used to calibrate the misaligned serialization features.

[0161] Furthermore, in order to alleviate the problem of gradient vanishing and gradient exploding, the output T of the cross-group multi-head self-attention module is yx and the serialization feature T in The skip connections between them are then processed by a normalization layer in the feedforward network module to obtain the modeling output of the grouped Transformer attention layer, which can be expressed as:

[0162] T′=LayerNormalization(T yx +T in )

[0163] T′ obtained by modeling group self-attention and cross-group self-attention ensures the association between tags within and between groups.

[0164] Step 5: Use a feedforward network to analyze multiple cross-group multi-head self-attention modeling results and visual tag sequences to obtain an image feature vector, and use a generalized average pooling layer to process the image feature vector to obtain the target image feature vector.

[0165] Consider that the feedforward network (FFN) in the original Transformer is composed of two layers of linear layers (MLP), which usually expands the dimension of the mark first and then compresses it by the same multiple to keep the dimension of the mark unchanged. However, when the dimension of the mark is too large, the expansion and compression of FFN will consume a lot of computing resources, which seriously affects the encoding process of Transformer and is not conducive to the establishment of associations between global labels. Therefore, in order to balance the computational consumption of the attention layer and FFN and further reduce the parameters of the model, the embodiment of the present application proposes an improved feedforward network, and the improved feedforward network compresses the dimension of the mark first and then expands it. Therefore, the expression of T′ after mapping and processing by the feedforward network module is as follows:

[0166] FFN(T′)=σ(T′F1)F2

[0167] In the above formula, F1 and F2 represent weight matrices, and σ represents the Relu activation function. FFN(T′) is skip-connected with the output T′ of the grouped Transformer, and then input into the second normalization layer for processing, and finally with the embedded marker T in Perform a skip connection to obtain the final encoding output of the grouped Transformer, expressed as:

[0168] T out=LayerNormalization(T′+FFN(T′))+T in

[0169] The feature map T after modeling out The GeMpooling operation and domain adaptation network are used to filter redundant feature information and reduce modality differences to obtain the feature vector of the target image.

[0170] Step 6: Use the domain adaptation network to determine the similarity between the target image feature vectors of each target image in each group of second training samples.

[0171] Specifically, the domain adaptation network can use the Euclidean distance to measure the similarity between the target image feature vectors of each target image in each group of second training samples, and convert the query set Q t The feature vector of the optical face sample image and the feature vector of the sketch face sample image f(S i ), the feature vector f(P i ) and the feature vector f(S i ′ ), the similarity between the feature vector of the sketch face sample image and the feature vector of the global pseudo sketch face sample image is expressed using the Euclidean distance as follows:

[0172] d(S i ,P i )=d(P i ,S i )=||f(P i )-f(S i )||

[0173] d(S i ′,P i )=d(P i ,S i ′)=||f(P i )-f(S i ′)||

[0174] d(S i ′,S i )=d(S i ,S i ′)=||f(S i )-f(S i ′)||

[0175] Here, || || represents the calculation of Euclidean distance, which is the operation of squaring first and then taking the square root. The smaller the distance between feature vectors, the greater the probability that the labels between images are the same. This section uses Euclidean distance to measure the similarity between images, and the distance between three images with the same subscript i should be smaller than the feature vectors of images with different subscripts.

[0176] Step 7: Construct a second target loss function based on the similarity between the target image feature vectors of each target image in multiple groups of second training samples, and use the second target loss function to adjust the model parameters of the initial recognition model to obtain a sketch face recognition model.

[0177] Specifically, for each group of optical face sample images P in the second training samples i , sketch face sample image S i , global pseudo sketch face sample image S i ', introduce domain alignment embedding loss to optimize the preset recognition model. The domain alignment embedding loss uses the Euclidean metric to measure the distance between features. The domain alignment embedding loss reduces the cross-entropy loss function on the negative Euclidean metric between cross-domain features to make the features of the same label across domains closer, thereby increasing the similarity between images within the class. Therefore, the embodiment of the present application can combine the alignment embedding loss of the optical image domain, the alignment embedding loss of the sketch image domain, and the alignment embedding loss of the pseudo-sketch image domain to construct the final domain alignment embedding loss, and by reducing the domain alignment embedding loss, the feature vectors of the optical image and the feature vectors of the sketch image with the same label in the common space are brought close to each other, so that the target recognition model can extract identification features that are helpful for recognition, thereby improving the accuracy of sketch face recognition.

[0178] Taking optical face images and sketch face images as an example, the Euclidean distance between all cross-domain optical face images and sketch face images in the same training batch is calculated as follows:

[0179]

[0180] Where P(S k / P i ) indicates that a given P i With S k The predicted probability of label consistency between (k=1,2,...,k), P(P k / S i ) is S i With P k (k=1,2,...,k) The predicted probability of the label being consistent between them. When k=i, that is, P(S i / P i ) and P(P k / S i) indicates that P i ,S i The probability of predicting the same label as the corresponding cross-domain image.

[0181] Based on the above prediction probability, the domain adaptation loss function between the three modalities can be calculated by the following formula, which can be written as:

[0182]

[0183] The final domain adaptation loss is the sum of the domain adaptation losses between the above three modalities. Therefore, the expression of the domain adaptation loss function (i.e., the second objective loss function) of the sketch face recognition model can be written as:

[0184] L=g*L PS +h*L SP +m*L S′P +n*L PS′ +q*L SS′ +r*L S′S

[0185] Among them, the above g, h, m, n, q, r represent the coefficients of each domain adaptive loss function. Considering that the optimization of the sketch face recognition model pays more attention to the matching between the real sketch and the real optical face image, the domain adaptive loss also needs to pay more attention to the two. Therefore, the coefficient of the domain adaptive loss function between the pseudo sketch face image and the optical face image and the real sketch face image is small, thereby giving the model fault tolerance, improving the generalization of the model, and improving the model's extraction effect on key features. In the embodiment of the present application, it can be set to g = h = 1, m = n = q = r = 0.1.

[0186] Step S135, using the first target loss function to adjust the model parameters of the initial synthesis model, to obtain a sketch face synthesis model. That is, by reducing the domain alignment embedding loss function L, the feature vectors of the optical image and the feature vectors of the sketch image with the same label in the common space are made close to each other, so that the network can extract identification information that is helpful for recognition, and finally a trained sketch face synthesis model is obtained.

[0187] The sketch face synthesis model and sketch face recognition model trained through the above steps have the following advantages in the relevant scenarios of sketch face recognition:

[0188] (1) In the case of asymmetric input (the category labels of the optical face image and the sketch face image are different), the global sketch synthesis module in the sketch face synthesis model is used to first generate a global pseudo sketch face image, and then the local sketch refinement module is used to refine the local features of the generated global pseudo sketch face image, so that the facial details of the final intermediate state (i.e., the pseudo sketch face image) are richer;

[0189] (2) In view of the problem that the CNN-based SFR method cannot capture the global connection between features to enhance and extract cross-modal discriminative features, resulting in the inability to further improve the recognition performance, the grouped Transformer network in the embodiment of the present application is used to capture the global connection between features, thereby improving the efficiency of Transformer self-attention execution;

[0190] (3) In view of the problem of modality difference between sketch images and optical images, the domain adaptation network of the embodiment of the present application can effectively reduce the impact of modality difference on recognition performance;

[0191] (4) In order to effectively train the recognition model with Transformer, the embodiment of the present application also introduces a meta-learning training strategy that is effective for small samples to avoid model overfitting.

[0192] The following ablation experiment results are given below to illustrate the advantages of the above technical solution compared with the existing technical solution.

[0193] (I) Verify the role of the sketch face synthesis model in the sketch face recognition task.

[0194] The test results obtained by inputting the triplet (pseudo sketch-optical-sketch face image) containing the pseudo sketch face image generated by the sketch face recognition model into the sketch face recognition model were compared with the test results obtained by inputting the binary (optical-sketch face image) not containing the pseudo sketch face image generated by the sketch face recognition model into the sketch face recognition model, and the ablation experiment results shown in Table 1 were obtained.

[0195] Table 1

[0196]

[0197]

[0198] From Table 1 above, it is not difficult to find that the test results obtained by inputting the pseudo sketch face image generated by the sketch face recognition model into the sketch face recognition model have an improvement of more than 10 points in Rank-1 in Set A, and also a significant improvement in Rank-10 and Rank-50, and the recognition accuracy is also improved in Set B. It is worth mentioning that the Rank-50 in both Sets SetA and SetB is close to 100%, and at the same time, each set has at least two groups of results with a Rank-50 of 100% in the five randomly conducted experimental results; and if the sketch face synthesis module is not included, the Rank-50 indicators in both SetA and SetB are less than 100%, which shows that the data expansion of the sketch face synthesis module has an effective effect on improving the accuracy of the sketch face recognition task. This is because the sketch face synthesis model is committed to generating an intermediate image between the optical face image and the sketch face image, or an intermediate state between the two modalities, and inputs the intermediate state into the sketch face recognition module as extended data. Although the image quality of the intermediate state is insufficient, there are still distinguishing features that cannot be ignored. Therefore, the method of the embodiment of the present application improves the sketch face recognition module's grasp of the essential distinguishing features.

[0199] (2) Verify the promoting role of the meta-learning sample pool in the sketch face recognition task.

[0200] Considering that the meta-learning sample pool is the input of the sketch face recognition model, if there is no meta-learning sample pool, the sketch face synthesis model and the sketch face recognition model cannot be effectively associated. Therefore, this ablation experiment focuses on testing the impact of the number of sample labels M contained in the meta-learning sample pool on the sketch face recognition accuracy. Affected by the limitation of video memory, the experiment in the embodiment of this application sets the number of sample labels in the meta-learning sample pool to 5, 10, 15, and 25 respectively. The specific test results are shown in Table 2 below.

[0201] Table 2

[0202]

[0203] The experimental results obtained in Table 2 above reflect to a certain extent that the more sample labels the meta-learning sample pool contains, the more sample labels need to be distinguished in each training, the greater the optimization intensity of the network, and the better the final network effect. When the number of sample labels contained is M=25, the test results achieve the best effect.

[0204] (iii) Verify the promoting role of the grouped Transformer network in the sketch face recognition task.

[0205] Considering that the number of input tag sequences of the input grouping transformer network is 256, the (number of groups, number of tags in a group) is divided into three cases: (4, 64), (8, 32), and (16, 16). At the same time, the experiment is set up without using the grouping strategy experiment for control. The specific test results are shown in Table 3 below.

[0206] Table 3

[0207]

[0208] It is not difficult to see from Table 3 that the experimental results using the grouping strategy are better than those without the grouping strategy, and the recognition effect is best when the (16,16) grouping strategy is used.

[0209] In addition, in order to verify the superiority of the sketch face recognition method proposed in the embodiment of the present application over the prior art, the following control experiment is also proposed. The sketch face recognition method proposed in the embodiment of the present application is compared with other sketch face recognition methods such as CNN (Convolutional Neural Network), DANN (Domain-Adversarial Neural Network), CDAN (Conditional Domain Adaptation Network), DAEN (Domain Adaptive Ensemble Network), SP-Net (Semantic Prior Network), LWVT-ResNet18 (Lightweight Vision Transformer ResNet18), etc. on the UoM-SGFS dataset (i.e., high-quality dataset) for recognition rate comparison, and the test results shown in Table 4 below are obtained.

[0210] Table 4

[0211]

[0212]

[0213] At the same time, in order to reflect the recognition ability of various methods for lower quality sketch data sets and fully reflect the generalization of the model, the sketch image synthesized by Cycle-GAN was used to test the sketch face recognition method proposed in the embodiment of the present application and other sketch face recognition methods, and the test results shown in Table 5 below were obtained.

[0214] Table 5

[0215]

[0216] Among them, the data in brackets in Tables 4 and 5 above are the recognition results of the test set using the Cycle-GAN synthetic sketch. Based on the above two tables, the sketch face recognition method of the embodiment of the present application achieved the best results under both high-quality data sets and low-quality synthetic data sets, fully demonstrating its good recognition performance and generalization.

[0217] Example 2

[0218] Based on Embodiment 1 of the present application, an embodiment of a sketch face recognition device is also provided, and the sketch face recognition method of the above embodiment is executed when the device is running. Figure 7 is a schematic structural diagram of an optional sketch face recognition device according to an embodiment of the present application, such as Figure 7 As shown, the sketch face recognition device at least includes an acquisition module, a synthesis module and a recognition module, wherein:

[0219] An acquisition module 71 is used to acquire a target sketch face image and an optical face image library to be matched, wherein the optical face image library includes a plurality of optical face images;

[0220] A synthesis module 72 is used to analyze each optical face image and a target sketch face image in the optical face image library using a pre-trained sketch face synthesis model to obtain a plurality of target pseudo sketch face images, wherein the sketch face synthesis model includes a global sketch synthesis module and a plurality of local sketch refinement modules, and the local sketch refinement module is used to perform local analysis on each facial part in the optical face image and the sketch face image to obtain a local pseudo sketch image;

[0221] The recognition module 73 is used to analyze the target sketch face image, multiple optical face images and the corresponding target pseudo sketch face image using a pre-trained sketch face recognition model, so as to determine a target optical face image from the multiple optical face images whose similarity with the target sketch face image is not less than a preset similarity threshold.

[0222] It should be noted that the various modules in the above-mentioned sketch face recognition device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0223] Example 3

[0224] According to an embodiment of the present application, a non-volatile storage medium is also provided, in which a program is stored, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the sketch face recognition method in Example 1.

[0225] Optionally, the device where the non-volatile storage medium is located implements the following steps by running the program: obtaining a target sketch face image and an optical face image library to be matched, wherein the optical face image library includes multiple optical face images; using a pre-trained sketch face synthesis model to analyze each optical face image and the target sketch face image in the optical face image library to obtain multiple target pseudo-sketch face images, wherein the sketch face synthesis model includes a global sketch synthesis module and multiple local sketch refinement modules, and the local sketch refinement module is used to perform local analysis on each facial part in the optical face image and the sketch face image to obtain a local pseudo-sketch image; using a pre-trained sketch face recognition model to analyze the target sketch face image, multiple optical face images and the corresponding target pseudo-sketch face image to determine a target optical face image from the multiple optical face images whose similarity with the target sketch face image is not less than a preset similarity threshold.

[0226] According to an embodiment of the present application, a computer program product is also provided, which includes a stored computer program, wherein when the computer program is executed by a processor, the sketch face recognition method in Example 1 is implemented.

[0227] Optionally, the computer program executes the following steps: obtaining a target sketch face image and an optical face image library to be matched, wherein the optical face image library includes multiple optical face images; using a pre-trained sketch face synthesis model to analyze each optical face image and the target sketch face image in the optical face image library to obtain multiple target pseudo-sketch face images, wherein the sketch face synthesis model includes a global sketch synthesis module and multiple local sketch refinement modules, and the local sketch refinement module is used to perform local analysis on each facial part in the optical face image and the sketch face image to obtain a local pseudo-sketch image; using a pre-trained sketch face recognition model to analyze the target sketch face image, multiple optical face images and the corresponding target pseudo-sketch face image to determine a target optical face image from the multiple optical face images whose similarity with the target sketch face image is not less than a preset similarity threshold.

[0228] According to an embodiment of the present application, a processor is also provided, which is used to run a program, wherein the sketch face recognition method in Example 1 is executed when the program is running.

[0229] Optionally, the following steps are executed when the program is running: obtaining a target sketch face image and an optical face image library to be matched, wherein the optical face image library includes multiple optical face images; using a pre-trained sketch face synthesis model to analyze each optical face image and the target sketch face image in the optical face image library to obtain multiple target pseudo-sketch face images, wherein the sketch face synthesis model includes a global sketch synthesis module and multiple local sketch refinement modules, and the local sketch refinement module is used to perform local analysis on each facial part in the optical face image and the sketch face image to obtain a local pseudo-sketch image; using a pre-trained sketch face recognition model to analyze the target sketch face image, multiple optical face images and the corresponding target pseudo-sketch face image to determine a target optical face image from the multiple optical face images whose similarity with the target sketch face image is not less than a preset similarity threshold.

[0230] According to an embodiment of the present application, an electronic device is also provided, wherein: Figure 8 is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application, such as Figure 8 As shown, the electronic device includes one or more processors; a memory for storing one or more programs, which, when the one or more programs are executed by the one or more processors, enables the one or more processors to run the programs, wherein the programs are configured to execute the sketch face recognition method in the above-mentioned embodiment 1 when running.

[0231] Optionally, the processor is configured to execute the following steps through a computer program: obtaining a target sketch face image and an optical face image library to be matched, wherein the optical face image library includes multiple optical face images; using a pre-trained sketch face synthesis model to analyze each optical face image and the target sketch face image in the optical face image library to obtain multiple target pseudo-sketch face images, wherein the sketch face synthesis model includes a global sketch synthesis module and multiple local sketch refinement modules, and the local sketch refinement module is used to perform local analysis on each facial part in the optical face image and the sketch face image to obtain a local pseudo-sketch image; using a pre-trained sketch face recognition model to analyze the target sketch face image, multiple optical face images and the corresponding target pseudo-sketch face image to determine a target optical face image from the multiple optical face images whose similarity with the target sketch face image is not less than a preset similarity threshold.

[0232] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0233] In the above embodiments of the present application, each embodiment is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. For details, please refer to the description of the above-mentioned related processing related embodiments, and no further description will be given here.

[0234] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0235] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0236] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0237] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or CD-ROM and other media that can store program codes.

[0238] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A sketch face recognition method, characterized in that: include: Acquire a target sketch face image and an optical face image library to be matched, wherein the optical face image library includes a plurality of optical face images; Analyzing each optical face image in the optical face image library and the target sketch face image using a pre-trained sketch face synthesis model to obtain a plurality of target pseudo sketch face images, wherein the sketch face synthesis model includes a global sketch synthesis module and a plurality of local sketch refinement modules, and the local sketch refinement modules are used to perform local analysis on each facial part in the optical face image and the sketch face image to obtain a local pseudo sketch image; The target sketch face image, the multiple optical face images and the corresponding target pseudo sketch face image are analyzed using a pre-trained sketch face recognition model to determine a target optical face image from the multiple optical face images whose similarity with the target sketch face image is not less than a preset similarity threshold.

2. The method according to claim 1, characterized in that The training process of the sketch face synthesis model includes: Acquire a first training set, wherein the first training set includes a first training sample set and a first sample label set, and the first training sample set includes a plurality of sketch face sample images and a plurality of optical face sample images, and the first sample label set includes a face label of each of the sketch face sample images and a face label of each of the optical face sample images; Build an initial synthesis model; The initial synthesis model is iteratively trained using the first training set to obtain the sketch face synthesis model.

3. The method according to claim 2, characterized in that The global sketch synthesis module and each of the local sketch refinement modules include a generator, a discriminator and a block feature extractor, wherein the initial synthesis model is iteratively trained using the first training set to obtain the sketch face synthesis model, including: For each training batch in the iterative training process, randomly selecting a preset number of sketch face sample images and a plurality of optical face sample images from the first training sample set, and determining face labels corresponding to the preset number of sketch face sample images and the plurality of optical face sample images from the first sample label set, to obtain a plurality of groups of first training samples; For each group of first training samples corresponding to each training batch, the following steps are performed: The optical face sample image is processed by a global generator to obtain a global pseudo sketch face sample image, and each face part in the optical face sample image is processed by multiple local generators to obtain multiple first local pseudo sketch sample images, wherein the global generator includes an encoder and a decoder, and the encoder includes at least a residual block; Dividing the global pseudo-sketch face sample image, the optical face sample image and the sketch face sample image according to face parts, respectively, to obtain a corresponding plurality of second local pseudo-sketch sample images, a plurality of local optical sample images and a plurality of local sketch sample images; Using a global discriminator to perform discriminant analysis on the global pseudo-sketch face sample image and the sketch face sample image to obtain a corresponding first authenticity discrimination result, and using multiple local discriminators to respectively perform discriminant analysis on the first local pseudo-sketch sample image and the local optical sample image, and the second local pseudo-sketch sample image and the local optical sample image of the same face part to obtain a corresponding second authenticity discrimination result and a third authenticity discrimination result; The global pseudo-sketch face sample image and the sketch face sample image are processed by a global block feature extractor to obtain a plurality of first image blocks corresponding to the global pseudo-sketch face sample image and a plurality of second image blocks corresponding to the sketch face sample image, and the plurality of first local pseudo-sketch sample images, a plurality of second local pseudo-sketch sample images and a plurality of local optical sample images are processed by a plurality of local block feature extractors to obtain a plurality of third image blocks corresponding to each of the first local pseudo-sketch sample images, a plurality of fourth image blocks corresponding to each of the second local pseudo-sketch sample images and a plurality of fifth image blocks corresponding to each of the local optical sample images, wherein the first image block, the second image block, the third image block, the fourth image block and the fifth image block are obtained by dividing the feature maps output by each convolutional layer in the encoder for the global pseudo-sketch face sample image, the sketch face sample image, each of the first local pseudo-sketch sample images, each of the second local pseudo-sketch sample images and each of the local optical sample images; Constructing a global generative adversarial loss function based on the first authenticity discrimination results of multiple groups of first training samples, the sketch face sample images, and the optical face sample images, and constructing a global multi-layer block contrast loss function based on the first image blocks and the second image blocks corresponding to the multiple groups of first training samples; Constructing a first type of local generative adversarial loss function based on the second authenticity discrimination results of the multiple groups of first training samples, the multiple first local pseudo-sketch sample images, and the multiple local optical sample images, and constructing a second type of local generative adversarial loss function based on the third authenticity discrimination results of the multiple groups of first training samples, the multiple local optical sample images, and the multiple second local pseudo-sketch sample images; Constructing a first type of multi-layer block contrast loss function based on the third image block and the fifth image block corresponding to the multiple groups of first training samples, and constructing a second type of multi-layer block contrast loss function based on the fourth image block and the fifth image block corresponding to the multiple groups of first training samples; Constructing a local consistency loss function based on a third image block corresponding to a first local pseudo sketch sample image and a fourth image block corresponding to the second local pseudo sketch sample image of the same face part in multiple groups of first training samples; Constructing a first target loss function based on the global generative adversarial loss function, the global multi-layer block contrast loss function, the first type of local generative adversarial loss function, the second type of local generative adversarial loss function, the first type of multi-layer block contrast loss function, the second type of multi-layer block contrast loss function and the local consistency loss function; The first objective loss function is used to adjust the model parameters of the initial synthesis model to obtain the sketch face synthesis model.

4. The method according to claim 3, characterized in that The global pseudo sketch face sample image and the sketch face sample image are processed by a global block feature extractor to obtain a plurality of first image blocks corresponding to the global pseudo sketch face sample image and a plurality of second image blocks corresponding to the sketch face sample image, including: Acquire multiple first feature maps obtained by encoding the global pseudo sketch face sample image using multiple convolutional layers in the encoder, and use a global block feature extractor to perform block processing on each of the first feature maps to obtain multiple first image blocks corresponding to each of the first feature maps; A plurality of second feature maps are obtained by encoding the optical face sample image using a plurality of convolutional layers in the encoder, and each of the second feature maps is processed into blocks using a global block feature extractor to obtain a plurality of second image blocks corresponding to each of the second feature maps, wherein a second image block at the same position in the feature map output by the same convolutional layer as the first image block is used as a positive block, and a second image block at a different position in the feature map output by the same convolutional layer as the first image block is used as a negative block.

5. The method according to claim 1, characterized in that The training process of the sketch face recognition model includes: Acquire a second training set, wherein the second training set includes a second training sample set and a second sample label set, and the second training sample set includes: the first training sample set and a plurality of global pseudo-sketch face sample images obtained by processing the first training sample set using the sketch face synthesis model, and the second sample label set includes: a face label of each of the sketch face sample images, a face label of each of the optical face sample images, and a face label of each of the global pseudo-sketch face sample images; Constructing an initial recognition model, wherein the initial recognition model at least includes: a convolutional neural network, a serialization module, a multi-layer grouped Transformer network, a generalized average pooling layer, and a domain adaptive network, and each layer of the grouped Transformer network includes a multi-head self-attention module, a cross-group multi-head self-attention module, and a feedforward network module; The initial recognition model is iteratively trained using the second training set to obtain the sketch face recognition model.

6. The method according to claim 5, characterized in that Iteratively training the initial recognition model using the second training set to obtain the sketch face recognition model includes: For each training batch in the iterative training process, a preset number of optical face sample images, sketch face sample images and global pseudo-sketch face sample images are randomly selected from the second training sample set, and face labels of the selected preset number of optical face sample images, sketch face sample images and global pseudo-sketch face sample images are determined from the second sample label set to obtain multiple groups of second training samples, wherein the face labels of the optical face sample images, the sketch face sample images and the global pseudo-sketch face sample images in each group of the second training samples are the same; The initial recognition model is trained using multiple groups of second training samples corresponding to multiple training batches to obtain the sketch face recognition model.

7. The method according to claim 6, characterized in that The initial recognition model is trained using multiple groups of second training samples corresponding to multiple training batches to obtain the sketch face recognition model, including: For any target image in each group of the second training samples, the following steps are performed: Using the convolutional neural network to perform convolution processing on the target image to obtain a low-resolution feature map; Utilizing the serialization module to perform serialization processing on the low-resolution feature map to obtain a visual marker sequence; The multi-head self-attention network is used to group the visual marker sequence to obtain multiple first groups, and self-attention operations are performed on all serialized features in each first group, and the self-attention modeling results obtained by each group are spliced ​​to obtain group multi-head self-attention modeling results corresponding to each first group; Using the cross-group multi-head self-attention network, a plurality of the grouped multi-head self-attention modeling results are cross-group combined to obtain a plurality of second groups, and self-attention operations are performed on all serialized features in each second group, and the self-attention modeling results obtained by each group are spliced ​​to obtain the cross-group multi-head self-attention modeling results corresponding to each of the second groups; Analyzing the multiple cross-group multi-head self-attention modeling results and the visual tag sequence using the feedforward network to obtain an image feature vector, and processing the image feature vector using the generalized average pooling layer to obtain a target image feature vector; Determining the similarity between the target image feature vectors of each target image in each group of second training samples using the domain adaptation network; A second target loss function is constructed based on the similarity between the target image feature vectors of each target image in multiple groups of the second training samples, and the model parameters of the initial recognition model are adjusted using the second target loss function to obtain the sketch face recognition model.

8. The method according to claim 7, characterized in that The feedforward network includes: a first normalization layer, a linear layer, and a second normalization layer, wherein the feedforward network is used to analyze a plurality of the cross-group multi-head self-attention modeling results and the visual tag sequence to obtain an image feature vector, including: Performing jump connections on the plurality of cross-group multi-head self-attention modeling results and the visual tag sequence to obtain corresponding first jump connection results, and performing normalization processing on the first jump connection results using the first normalization layer to obtain a first modeling result; Performing FFN mapping on the first modeling result by using the linear layer to obtain an FFN mapping result, and performing a jump connection between the FFN mapping result and the first modeling result to obtain a corresponding second jump connection result; Using the second normalization layer to normalize the second jump connection result to obtain a second modeling result; The second modeling result is skip-connected with the serialized feature to obtain the image feature vector.

9. The method according to claim 7, characterized in that: Constructing a second target loss function according to the Euclidean distances between target image feature vectors of each target image in the plurality of groups of the second training samples, including: Determine the probability distribution of the sketch face sample images in the plurality of groups of the second training samples in the plurality of groups of the second training samples according to the Euclidean distance; The second objective loss function is constructed according to the probability distribution.

10. A sketch face recognition device, characterized in that: include: An acquisition module, used for acquiring a target sketch face image and an optical face image library to be matched, wherein the optical face image library includes a plurality of optical face images; A synthesis module, used for analyzing each optical face image in the optical face image library and the target sketch face image by using a pre-trained sketch face synthesis model to obtain a plurality of target pseudo sketch face images, wherein the sketch face synthesis model includes a global sketch synthesis module and a plurality of local sketch refinement modules, and the local sketch refinement module is used for performing local analysis on each facial part in the optical face image and the sketch face image to obtain a local pseudo sketch image; A recognition module is used to analyze the target sketch face image, multiple optical face images and corresponding target pseudo sketch face images using a pre-trained sketch face recognition model, so as to determine a target optical face image from the multiple optical face images whose similarity with the target sketch face image is not less than a preset similarity threshold.

Images