Sketch face recognition method and device

Abstract

This specification relates to the field of artificial intelligence technology, and specifically discloses a sketch face recognition method and device, wherein the method includes: obtaining a training set; using the training set to train a preset recognition model to obtain a target recognition model to perform face recognition on a sketch image; the preset recognition model includes a convolutional neural network, a serialization module, a grouped Transformer network, a GeM pooling layer, and a domain adaptation network; the grouped multi-head self-attention module in the grouped Transformer network groups the visual marker sequences in the sample set, performs self-attention operations in each first group to generate grouped multi-head self-attention modeling results; the cross-group multi-head self-attention module combines the grouped multi-head self-attention modeling results across groups, performs self-attention operations in each second group to generate cross-group multi-head self-attention modeling results. The above scheme captures the global contextual connection of features and improves the accuracy of sketch face recognition.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to a sketch face recognition method and device. Background Art

[0002] With the advent of the information age, social security issues have attracted widespread attention from all sectors of society, and identity recognition technology has therefore received increasing attention. Within the social security sector, facial recognition technology, due to its intuitive, convenient, and contactless features, has been widely used in areas such as facial payment, security verification, and video analysis. With the development of deep learning technology, the increase in computing power, and the public availability of large datasets, facial recognition technology has become a research hotspot. However, in real-world scenarios, faces are represented in a variety of forms. In addition to the most common optical images, they also include sketches, near-infrared images, simple drawings, and cartoons. This has led to significant attention in heterogeneous face recognition. Due to its widespread application and urgent need in criminal investigation, sketch face recognition (SFR) has become a research hotspot. In real-world scenarios, the goal of sketch face recognition is to match the identity of a sketch with an optical image. A given sketch is compared with images containing identity information in an optical face database. The identity of the sketch is determined by identifying the most similar optical image, or a number of similar optical images can be identified to narrow the scope of investigation. However, in algorithm research, the sketch face dataset is small in scale and each identity has only one optical image and corresponding sketch. This small sample problem makes cross-modal recognition between sketches and optical images difficult to achieve.

[0003] Traditional sketch-based face recognition methods are mainly based on manually designed methods. However, due to the fixed feature extraction patterns of these methods, the extracted shallow features have limited expressive power and are unable to describe the highly nonlinear relationship between cross-modal images. This makes it difficult to improve the recognition rate. Moreover, these methods cannot automatically learn and extract features like neural networks, and have poor generalization capabilities.

[0004] With the development of deep learning technology, methods based on convolutional neural networks (CNNs) have become the mainstream of SFR. These methods can be roughly divided into two categories: intra-modality methods and inter-modality methods. These CNN-based methods primarily use CNNs as the backbone network to construct target models capable of extracting discriminative features between optical images and sketch images. However, because CNNs are local feature extractors, they are unable to capture long-range features and establish global connections between features. Therefore, these CNN-based methods cannot enhance the key role of discriminative features in recognition, resulting in an inability to further improve the accuracy of sketch face recognition.

[0005] To address the above issues, no effective solutions have been proposed so far. Summary of the Invention

[0006] The embodiments of this specification provide a sketch face recognition method and device to solve the problem of low accuracy of sketch face recognition in the prior art.

[0007] The embodiment of this specification provides a sketch face recognition method, including:

[0008] Acquire a training set; the training set includes a sample set and a label set; the sample set includes an optical image set and a sketch image set; the label set includes a label of each optical image in the optical image set and a label of each sketch image in the sketch image set; the label of each optical image corresponds one-to-one to the label of each sketch image;

[0009] The training set is used to train a preset recognition model to obtain a target recognition model; the target recognition model is used to perform face recognition on the sketch image;

[0010] Among them, the preset recognition model includes a convolutional neural network, a serialization module, a grouped Transformer network, a GeM pooling layer and a domain adaptation network; the grouped Transformer network includes a grouped multi-head self-attention module, a cross-group multi-head self-attention module and a feedforward network module; the convolutional neural network is used to extract features of the optical images and sketch images in the sample set to obtain corresponding low-resolution feature maps, and the serialization module is used to serialize the feature maps to obtain corresponding pixel-level visual marker sequences; the grouped multi-head self-attention module is used to group the corresponding visual marker sequences in the sample set to obtain the first group of the number of markers in the first preset group, and in each first Self-attention operation is performed within the group to generate a group multi-head self-attention modeling result; the cross-group multi-head self-attention module is used to perform cross-group combination on the group multi-head self-attention modeling result to obtain a second group with a second preset number of in-group labels; self-attention operation is performed within each second group to generate a cross-group multi-head self-attention modeling result, and a global connection between the visual labels corresponding to each sample in the sample set is obtained; the number of groups in the first group is equal to the number of labels in the second preset group, and the number of groups in the second group is equal to the number of labels in the first preset group; the domain adaptation network is used to project the cross-group multi-head self-attention modeling result after processing by the GeM pooling layer into a common subspace for pairing recognition.

[0011] In one embodiment, the preset recognition model is trained using the training set to obtain a target recognition model, including:

[0012] generating a plurality of meta-learning tasks based on a meta-learning strategy; randomly selecting a preset number of optical images and corresponding sketch images from the training set in each of the plurality of meta-learning tasks; generating a query set corresponding to each meta-learning task based on the selected optical images and sketch images, wherein the selected optical images and sketch images serve as a support set for optical images and a support set for sketch images, respectively; and having a one-to-one correspondence between labels of the optical images and labels of the sketch images in each meta-learning task;

[0013] The multiple meta-learning tasks are executed to train a preset recognition model using the sample sets corresponding to the meta-learning tasks to obtain a target training model.

[0014] In one embodiment, the grouped Transformer module is also used to perform jump connection and layer normalization processing on the cross-group multi-head self-attention modeling result and the visual marker sequence to obtain the modeling output result of the grouped Transformer self-attention layer; the feedforward network module is used to perform FFN mapping on the modeling output result of the grouped Transformer self-attention layer to obtain an FFN mapping result, jump connect the FFN mapping result with the modeling output result of the grouped Transformer self-attention layer, perform layer normalization, and then jump connect with the visual marker sequence output by the grouped Transformer self-attention layer to obtain the encoding output result of the grouped Transformer network; the FFN mapping is used to first reduce the dimension of the visual marker by a preset multiple and then expand the preset multiple.

[0015] In one embodiment, in the process of training the preset recognition model using the training set, the preset recognition model is trained and optimized using the domain alignment embedding loss function to obtain a target recognition model; the domain alignment embedding loss function includes the alignment embedding loss function of the optical image domain and the alignment embedding loss function of the sketch image domain.

[0016] In one embodiment, the preset recognition model includes a two-layer grouping Transformer network.

[0017] In one embodiment, after the preset recognition model is trained using the training set to obtain the target recognition model, the method further includes:

[0018] Acquire a set of images to be identified; the set of images to be identified includes sketch images to be identified and an optical image set;

[0019] Inputting the sketch image to be identified and the optical image set into a target recognition model to obtain similarities between the sketch image to be identified and each optical image in the optical image set;

[0020] The optical image whose similarity meets the preset conditions is used as the recognition result of the sketch image to be recognized.

[0021] The embodiment of this specification also provides a sketch face recognition device, comprising:

[0022] an acquisition module, configured to acquire a training set; the training set includes a sample set and a label set; the sample set includes an optical image set and a sketch image set; the label set includes a label for each optical image in the optical image set and a label for each sketch image in the sketch image set; the label for each optical image corresponds one-to-one to the label for each sketch image;

[0023] A training module is used to train a preset recognition model using the training set to obtain a target recognition model; the target recognition model is used to perform face recognition on the sketch image;

[0024] Among them, the preset recognition model includes a convolutional neural network, a serialization module, a grouped Transformer network, a GeM pooling layer and a domain adaptation network; the grouped Transformer network includes a grouped multi-head self-attention module, a cross-group multi-head self-attention module and a feedforward network module; the convolutional neural network is used to extract features of the optical images and sketch images in the sample set to obtain corresponding low-resolution feature maps, and the serialization module is used to serialize the feature maps to obtain corresponding pixel-level visual marker sequences; the grouped multi-head self-attention module is used to group the corresponding visual marker sequences in the sample set to obtain the first group of the number of markers in the first preset group, and in each first Self-attention operation is performed within the group to generate a group multi-head self-attention modeling result; the cross-group multi-head self-attention module is used to perform cross-group combination on the group multi-head self-attention modeling result to obtain a second group with a second preset number of in-group labels; self-attention operation is performed within each second group to generate a cross-group multi-head self-attention modeling result, and a global connection between the visual labels corresponding to each sample in the sample set is obtained; the number of groups in the first group is equal to the number of labels in the second preset group, and the number of groups in the second group is equal to the number of labels in the first preset group; the domain adaptation network is used to project the cross-group multi-head self-attention modeling result after processing by the GeM pooling layer into a common subspace for pairing recognition.

[0025] In one embodiment, the training module is specifically used to:

[0026] generating a plurality of meta-learning tasks based on a meta-learning strategy; randomly selecting a preset number of optical images and corresponding sketch images from the training set in each of the plurality of meta-learning tasks; generating a query set corresponding to each meta-learning task based on the selected optical images and sketch images, wherein the selected optical images and sketch images serve as a support set for optical images and a support set for sketch images, respectively; and having a one-to-one correspondence between labels of the optical images and labels of the sketch images in each meta-learning task;

[0027] The plurality of meta-learning tasks are executed to train a preset recognition model using the sample sets corresponding to the meta-learning tasks to obtain a target training model.

[0028] An embodiment of this specification also provides a computer device, including a processor and a memory for storing processor-executable instructions, wherein when the processor executes the instructions, the steps of the sketch face recognition method described in any of the above embodiments are implemented.

[0029] The embodiments of this specification also provide a computer-readable storage medium having computer instructions stored thereon, which, when executed, implement the steps of the sketch face recognition method described in any of the above embodiments.

[0030] In an embodiment of the present specification, a sketch face recognition method is provided, which can obtain a training set and use the training set to train a preset recognition model to obtain a target recognition model for face recognition. In this solution, the preset recognition model includes a grouped Transformer network, and the grouped Transformer network includes a grouped multi-head self-attention module, a cross-group multi-head self-attention module and a feedforward network module. The grouped multi-head self-attention module can group the corresponding visual marker sequences in the sample set to obtain a first group of markers in a first preset group, perform self-attention operations in each first group, generate grouped multi-head self-attention modeling results, and input the grouped multi-head self-attention modeling results into the cross-group multi-head self-attention module. The cross-group multi-head self-attention module can perform cross-group combination on the grouped multi-head self-attention modeling results to obtain a second group of markers in a second preset group, perform self-attention operations in each second group, and generate cross-group multi-head self-attention modeling results, so that the global connection between the visual markers corresponding to each sample in the sample set can be obtained. In the above scheme, the highly complex self-attention mechanism is optimized into two stages: group self-attention and cross-group self-attention through the grouped Transformer network. The group self-attention will mark the groups to perform intra-group self-attention to improve efficiency. The cross-group self-attention realizes cross-group interaction of different group labels through cross-group reorganization to learn global contextual connections, which can capture long-distance features and enhance the key role of identification features in sketch face recognition, thereby effectively improving the accuracy of sketch face recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings described herein are used to provide a further understanding of this specification, constitute a part of this specification, and do not constitute a limitation of this specification. In the accompanying drawings:

[0032] Figure 1 A flowchart of a sketch face recognition method according to an embodiment of the present specification is shown;

[0033] Figure 2 A schematic diagram of a target recognition model in an embodiment of the present specification is shown;

[0034] Figure 3 A schematic diagram of a packet Transformer network in an embodiment of this specification is shown;

[0035] Figure 4 A schematic diagram of a sketch face recognition device in an embodiment of this specification is shown;

[0036] Figure 5 A schematic diagram of a computer device in an embodiment of the present specification is shown. DETAILED DESCRIPTION

[0037] The principles and spirit of this specification will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided solely to enable those skilled in the art to better understand and implement this specification, and are not intended to limit the scope of this specification in any way. Rather, these embodiments are provided to make this specification more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0038] Those skilled in the art will appreciate that the embodiments of this specification may be implemented as a system, device, method, or computer program product. Therefore, the disclosure herein may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0039] In the prior art, in order to improve the accuracy of sketch face recognition, the intra-modal method and the inter-modal method are mainly used. The intra-modal method refers to the sketch face recognition algorithm based on CNN within the modality. This type of method converts the cross-modal image to the target modality through a CNN-based modality converter, and then calculates the similarity in the target modality to complete the identity recognition. However, the recognition effect of this type of method is directly related to the quality of the generated image, and when the modality gap is too large, it is difficult to describe the nonlinear relationship of the modal conversion. When the CNN-based modality converter generates an image of the target modality, due to the lack of global connection of the encoded features, the decoded target image is not structurally coherent and lacks detailed texture.

[0040] Intermodal methods refer to CNN-based methods for intermodal sketch face recognition. These methods project extracted cross-modal features into a common subspace to reduce modal differences. The goal is to learn a classifier that maximizes inter-class distance while minimizing intra-class distance, thereby extracting modality-invariant features for recognition. However, the difficulty of CNN-based intermodal methods lies in extracting discriminative features that are consistent across modalities. Due to the small size of sketch face datasets, the model is prone to overfitting. Furthermore, using CNN as a feature extraction network is limited to capturing local information and cannot establish interactive relationships between semantic concepts in space, resulting in insufficient recognition performance. Furthermore, semantic errors introduced during the sketching process can cause the feature extraction network to overlook key features.

[0041] In order to solve the problem that it is difficult to extract cross-modal identification features when the SFR dataset has small samples and the modality gap is too large, in the embodiments of this specification, the Transformer with global relationship modeling capabilities is introduced into sketch face recognition to learn the contextual connection between features, thereby highlighting the key role of identification features in the pairing process of optical images and sketch images. Visual Transformer usually constructs an end-to-end Transformer network architecture by embedding image patch sequences and stacking Transformer basic blocks, which has a high model complexity. Especially when inputting high-resolution images, the visual Transformer becomes extremely complex and parameterized. There is a contradiction between the high complexity of the Transformer architecture and the small sample problem of sketch face recognition. Directly training the Transformer network on a small sample of sketch faces is prone to overfitting problems. At the same time, the square ratio complexity of the self-attention mechanism in the Transformer module and the number of embedded tags will also affect the performance of the model. The embodiments of this specification focus on solving the problem of high complexity of the self-attention mechanism in the Transformer, and solving the problem of the applicability of the Transformer in small samples of sketch face recognition.

[0042] Based on this, an embodiment of this specification provides a sketch face recognition method. Figure 1A flowchart of a sketch face recognition method in one embodiment of this specification is shown. Although this specification provides method operation steps or device structures as shown in the following embodiments or drawings, more or fewer operation steps or module units may be included in the method or device based on routine or no creative labor. In the steps or structures where there is no necessary causal relationship logically, the execution order of these steps or the module structure of the device is not limited to the execution order or module structure described in the embodiments of this specification and shown in the drawings. When the method or module structure is applied to an actual device or terminal product, it can be connected in accordance with the method or module structure shown in the embodiments or drawings for sequential execution or parallel execution (for example, a parallel processor or multi-threaded processing environment, or even a distributed processing environment).

[0043] Specifically, if Figure 1 As shown, the sketch face recognition method provided by one embodiment of this specification may include the following steps:

[0044] Step S101, obtaining a training set; the training set includes a sample set and a label set; the sample set includes an optical image set and a sketch image set; the label set includes a label of each optical image in the optical image set and a label of each sketch image in the sketch image set; the label of each optical image corresponds one-to-one to the label of each sketch image.

[0045] The method in this embodiment can be applied to a server, which can be a single server, a cluster server, a cloud server, etc. A training set can be obtained. The training set can include a sample set and a label set. The sample set can include an optical image set and a sketch image set. The optical image set can include multiple optical images, where the optical images here can be facial images taken of actual faces by a camera. The sketch image set can be facial images obtained by sketching actual faces. The label set can include labels for each optical image in the optical image set, and can also include labels for each sketch image in the sketch image set. The labels of the optical images correspond one-to-one to the labels of the sketch images. That is, each optical image in the optical image set corresponds one-to-one to each sketch object in the sketch image set.

[0046] Step S102: using the training set to train a preset recognition model to obtain a target recognition model; the target recognition model is used to perform face recognition on the sketch image.

[0047] After obtaining the training set, the preset recognition model can be trained using the training set to obtain a target recognition model for performing sketch image face recognition.

[0048] In this embodiment, the preset recognition model includes a convolutional neural network, a serialization module, a grouped Transformer network, a GeM pooling layer and a domain adaptation network.

[0049] The convolutional neural network is used to extract features from the optical images and sketch images in the sample set to obtain corresponding feature maps. The serialization module is used to serialize the feature maps output by the convolutional neural network to obtain a corresponding visual marker sequence. In this embodiment, the convolutional neural network is used to extract features from the optical images and sketch images in the sample set to obtain corresponding feature maps. Since Transformer is a seqtoseq (sequence to sequence) framework, the feature maps are serialized pixel by pixel into a visual marker sequence embedded in a grouped Transformer network.

[0050] In this embodiment, the preset recognition model may include a grouped Transformer network. The input of the Transformer is a sequence, so the samples in the sample set need to be serialized to obtain a visual marker sequence. The visual marker sequence includes visual markers corresponding to each optical image and each sketch image in the sample set. The grouped Transformer network includes a grouped multi-head self-attention module, a cross-group multi-head self-attention module, and a feed-forward network (FFN) module.

[0051] The grouped multi-head self-attention module is used to group the corresponding visual marker sequences in the sample set to obtain first groups with a first preset number of markers. Self-attention operations are performed within each first group to generate grouped multi-head self-attention modeling results. The grouped multi-head self-attention modeling results are input into the cross-group multi-head self-attention module. The number of groups in the first group is equal to the second preset number of markers.

[0052] The cross-group multi-head self-attention module is used to combine the grouped multi-head self-attention modeling results across groups to obtain second groups with a second preset number of in-group markers. The number of groups in each second group is equal to the number of in-group markers in the first preset group. Self-attention operations are performed within each second group to generate cross-group multi-head self-attention modeling results, thereby obtaining global connections between the visual markers corresponding to each sample in the sample set.

[0053] For example, in one embodiment, the visual marker sequence corresponding to each sample in the sample set includes N=HW visual markers, where H and W are the length and width of the feature map, respectively. The grouped multi-head self-attention module can divide the N visual markers into x first groups, each of which includes y visual markers. The grouped multi-head self-attention modeling result output by the grouped multi-head self-attention module also includes N visual markers. The cross-group multi-head self-attention module can perform cross-group combination on the visual markers output by the grouped multi-head self-attention module, dividing the N visual markers into y second groups, each of which includes x visual markers. Among them, the cross-group combination can be to select a visual marker from each of the x first groups for combination, to obtain a second group containing x visual markers, and after y selections, y second groups can be obtained. In one embodiment, visual markers with the same relative position within the group in different first groups can be grouped together to obtain y second groups.

[0054] A self-attention operation is performed within each second group to generate a cross-group multi-head self-attention modeling result, so as to obtain a global connection between the visual tags corresponding to each sample in the sample set.

[0055] In this embodiment, the domain adaptation network is used to project the cross-group multi-head self-attention modeling results after processing by the GeM pooling layer to a common subspace for pairing recognition. The GeM pooling layer and the domain adaptation network are used to filter redundant feature information and reduce modal differences from the encoding output results of the grouped Transformer network to obtain feature vectors corresponding to each sample in the sample set. The GeMpooling operation and the domain adaptation network can be used to filter redundant feature information and reduce modal differences from the encoding output results of the grouped Transformer, and obtain feature vectors corresponding to each sample in the sample set, which may include feature vectors corresponding to each optical image and feature vectors corresponding to each sketch image. The feature vectors corresponding to each optical image and the feature vectors corresponding to each sketch image can be projected to a common subspace for pairing recognition. The introduction of a domain adaptation network can reduce the impact of modal differences on recognition performance.

[0056] In this embodiment, the highly complex self-attention mechanism is optimized into two stages: group self-attention and cross-group self-attention through the grouped Transformer network. The group self-attention groups the tags to perform intra-group self-attention to improve efficiency. The cross-group self-attention realizes cross-group interaction of different group tags through cross-group reorganization to learn global contextual connections, which can capture long-distance features and enhance the key role of identification features in sketch face recognition, thereby effectively improving the accuracy of sketch face recognition.

[0057] In some embodiments of the present specification, the grouped Transformer module can also be used to perform jump connection and layer normalization processing on the cross-group multi-head self-attention modeling structure and the visual marker sequence to obtain the modeling output result of the grouped Transformer self-attention layer; the grouped Transformer network also includes a feedforward network module, which is used to perform FFN mapping on the modeling output result of the grouped Transformer self-attention layer to obtain an FFN mapping result, jump connect the FFN mapping result with the modeling output result of the grouped Transformer self-attention layer, perform layer normalization, and then jump connect with the visual marker sequence to obtain the encoding output result of the grouped Transformer network; the FFN mapping is used to first reduce the dimension of the visual marker by a preset multiple, and then expand the preset multiple.

[0058] In this embodiment, to further reduce the number of model parameters and balance the computational cost of the attention layer and the FFN, an improved feedforward network module is proposed that first reduces the label dimension by a preset multiple and then expands it by a preset multiple. The preset multiple can be an integer greater than 1, for example, it can be set to 4. The FFN mapping result can be skip-connected with the modeling output of the self-attention layer, followed by layer normalization, and finally skip-connected with the input visual label sequence to obtain the encoded output of the grouped Transformer.

[0059] Considering the contradiction between the high complexity of the Transformer architecture and the small sample problem of sketch face recognition, directly training the Transformer network on a small sample of sketch faces is prone to overfitting problems. Therefore, in order to effectively train the recognition model with Transformer, a meta-learning training strategy that is effective for small samples can be introduced to avoid model overfitting. Therefore, in some embodiments of the present specification, using the training set to train the preset recognition model to obtain the target recognition model can include: generating multiple meta-learning tasks based on the meta-learning strategy; in each of the multiple meta-learning tasks, randomly selecting a preset number of optical images and corresponding sketch images from the training set, and generating query sets corresponding to the meta-learning tasks based on the selected optical images and sketch images, and the selected optical images and sketch images are respectively used as optical image support sets and sketch image support sets; the labels of the optical images in the query sets corresponding to the meta-learning tasks correspond one-to-one to the labels of the sketch images; executing the multiple meta-learning tasks to train the preset recognition model using the sample sets corresponding to the meta-learning tasks to obtain the target training model.

[0060] Specifically, this embodiment adopts the meta-learning training strategy that is currently effective for small samples. The specific strategy is: given a training set Dtrain ={p1,p2,...,p N ,s1,s2,...,s N},in represents an optical image, Denotes sketch images, i is their labels. The meta-learning training strategy raises the learning level to the task. In each meta-learning task, the meta-learning strategy first learns from the training set D train Randomly select K (K < N) pairs of sketch images and optical images, and then reset their labels to {1, 2, ..., K}. The goal of the meta-learning task is to match these K pairs of samples as accurately as possible, where K is a hyperparameter of the number of batch sample pairs in the meta-learning strategy. The selected K pairs of samples constitute the query set of the meta-learning task. In Q t In, P t As an optical image support set, S t As the support set of sketch images, t∈{1,2,…,T} represents the number of times the meta-learning task is executed. By executing multiple meta-learning tasks, the preset recognition model can be trained using the query sets corresponding to each meta-learning task, thereby obtaining the target training model. In this embodiment, by introducing a meta-learning training strategy to train the preset recognition model, the applicability of the Transformer to small sample sketch face recognition can be addressed, avoiding overfitting.

[0061] In some embodiments of the present specification, in the process of training a preset recognition model using the training set, the preset recognition model can be trained and optimized using a domain alignment embedding loss function to obtain a target recognition model; the domain alignment embedding loss function includes an alignment embedding loss function of the optical image domain and an alignment embedding loss function of the sketch image domain.

[0062] In this embodiment, domain alignment embedding loss can be introduced to optimize the preset recognition model. Domain alignment embedding loss uses Euclidean metric to measure the distance between features. Domain alignment embedding loss reduces the cross-entropy loss on the negative Euclidean metric between cross-domain features to make cross-domain features with the same label closer, thereby increasing the similarity between images within the class. The final domain alignment embedding loss can be constructed by combining the alignment embedding loss of the optical image domain and the alignment embedding loss of the sketch image domain. By reducing the domain alignment embedding loss, the feature vectors of the optical image and the feature vectors of the sketch image with the same label in the common space are brought close to each other, so that the target recognition model can extract identification features that are helpful for recognition, thereby improving the accuracy of sketch face recognition.

[0063] In some embodiments of the present specification, after using the training set to train the preset recognition model to obtain the target recognition model, it can also include: obtaining an image set to be recognized; the image set to be recognized includes a sketch image to be recognized and an optical image set; inputting the sketch image to be recognized and the optical image set into the target recognition model, obtaining the similarity between the sketch image to be recognized and each optical image in the optical image set, and taking the optical image whose similarity meets the preset conditions as the recognition result of the sketch image to be recognized.

[0064] After obtaining the target recognition model, a set of images to be recognized can be obtained. The sketch image to be recognized and the optical image set in the set of images to be recognized can be input into the target recognition model to obtain the similarity between the sketch image to be recognized and each optical image. Afterwards, the optical image whose similarity meets the preset conditions can be used as the recognition result of the sketch image to be recognized. The recognition results can be arranged from large to small according to the similarity. In one embodiment, the optical image corresponding to the maximum similarity can be determined as the recognition result of the sketch image to be recognized. In one embodiment, several recognition results with higher recognition rates can be used to narrow the reconnaissance range. Through this embodiment, an optical image that matches the sketch image to be recognized can be determined from the optical image set.

[0065] In some embodiments of this specification, the preset recognition model may include a two-layer grouped Transformer network. In this embodiment, considering that the original ResNet18 has two residual blocks in the last stage, only an equal number of modules are replaced in this embodiment to make the model comparison more convincing.

[0066] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. For details, please refer to the description of the aforementioned related processing embodiments, and no further description is given here.

[0067] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0068] The above method is described below with reference to a specific embodiment. However, it should be noted that this specific embodiment is only for better illustrating this specification and does not constitute an improper limitation to this specification.

[0069] In this specific embodiment, a sketch face recognition method is proposed. The method in this embodiment is based on a convolutional network and a grouped Transformer. The preset recognition model can be called GT-ResNet18. Please refer to Figure 2 , shows the framework diagram of GT-ResNet18 in this embodiment. Figure 2 As shown, GT-ResNet18 can be composed of a convolutional neural network, a two-layer grouped Transformer module, and a domain adaptation network. To address the problem that current CNN-based SFR methods cannot capture the global connections between features to enhance and extract cross-modal discriminative features, resulting in an inability to further improve recognition performance, this specific embodiment proposes a grouped Transformer module to capture the global connections between features. This module improves the efficiency of Transformer self-attention. To avoid the excessive complexity of the Transformer model caused by high-dimensional image embedding, the bottleneck block in the last stage of ResNet18 is replaced with a grouped Transformer module to construct a less complex GT-ResNet18 to capture the contextual connections of features. To address the modal differences between sketch images and optical images, a domain adaptation network is introduced to reduce the impact of modal differences on recognition performance. Finally, to effectively train GT-ResNet18 with Transformer, a meta-learning training strategy that is effective for small samples is introduced to avoid model overfitting.

[0070] In this embodiment, the feature extraction network is represented by f(· / w) parameterized by the GT-ResNet18 weight w, and the optical image p in the meta-task is i Or sketch images i After f(· / w), we can get the eigenvectors f(p i / w) and the eigenvector f(s) i / w), by introducing domain alignment embedding loss, the f(p i / w) and f(s) i / w) to optimize the feature extraction network.

[0071] In this embodiment, the meta-learning training strategy that is currently effective for small samples can be adopted. The specific strategy is: given a training set D train ={p1,p2,...,p N ,s1,s2,...,s N},in represents an optical image, Denotes sketch images, i is their labels. The meta-learning training strategy raises the learning level to the task. In each meta-learning task, the meta-learning strategy first learns from the training set D train Random selection

[0072] K (K < N) pairs of sketch images and optical images are selected, and their labels are reset to {1, 2, ..., K}. The goal of the meta-learning task is to match these K pairs of samples as accurately as possible, where K is a hyperparameter of the number of batch sample pairs in the meta-learning strategy. The selected K pairs of samples constitute the query set of the meta-learning task. In Q t In, P t As an optical image support set, S t As the support set of sketch images, t∈{1,2,…,T} represents the number of times the meta-learning task is performed.

[0073] The network structures of ResNet18 and GT-ResNet18 are shown in Table 1. The image sample x(x∈Q t ) After preprocessing, the size is 256×256. The size change of the feature map after downsampling by the residual network is shown in the output of Table 1. The input image passes through the convolutional network of the first four stages of GT-ResNet18 and outputs a low-resolution feature map:

[0074] X in =CNNs(x) (1)

[0075] Transformer is a seqtoseq framework, where the low-resolution feature map is serialized pixel by pixel into a token sequence embedded in the grouped Transformer:

[0076] T=R(X in ) (2)

[0077] Among them, R(·) is the reshaping operation, the feature map X in ∈R d×H×W The visual features are serialized into a visual tag sequence T∈R via R(·) HW×d The absolute position coding sequence can provide the position information of the marker in space. The commonly used absolute position coding can be represented by the sine / cosine function:

[0078]

[0079] Here pos represents the time sequence of the tag, d is the dimension of the tag, i∈{0, 1, …, d / 2} is used to calculate the value of the position encoding on each dimension. The dimension of the position encoding obtained by calculation is consistent with the tag dimension. The tag sequence plus the position encoding is the embedded tag sequence:

[0080] T in =T+PE(T) (4)

[0081] Table 1 Network structure of ResNet18 and GT-ResNet18

[0082]

[0083]

[0084] The original Transformer mainly includes a multi-head self-attention (MHSA) module and a feedforward network module. Transformer calculates the global connection between embedded tags through MHSA. Since MHSA performs self-attention calculation within the global tag range, it has a high computational complexity that is proportional to the square of the number of tags. In order to alleviate the high computational complexity of MHSA, the present invention proposes a new Transformer module called grouped Transformer. Grouped Transformer includes grouped multi-head self-attention, cross-group multi-head self-attention self-attention modules and an improved feedforward network module. Its structure is as follows Figure 2 As shown. The grouped multi-head self-attention in the grouped Transformer avoids the self-attention calculation on the global mark in the original Transformer MHSA by grouping the marks. Each group of marks completes the self-attention calculation within the group, which can reduce the computational complexity to a linear relationship. The interactive connection between different groups is realized by cross-group multi-head self-attention. The cross-group multi-head self-attention regroups the marks with the same relative position within the group in the grouped multi-head self-attention into a group to perform cross-group self-attention calculation, thereby realizing the self-attention calculation on the global mark of the grouped Transformer. The grouping form is defined as the grouping strategy, which satisfies that the number of marks N (N = HW) is equal to the number of groups multiplied by the number of marks within the group, and that the number of groups and the number of marks within the group are both greater than 1. Assuming that in the grouped multi-head self-attention, the mark sequence is grouped according to N = x × y, where x represents the number of groups and y is the number of marks within the group, then the computational complexity of the grouped multi-head self-attention is o(yNd). According to the regrouping mechanism of cross-group multi-head self-attention, the number of groups and the number of labels in each group are y and x respectively. The grouping strategy can be expressed as N=y×x, and the complexity of self-attention calculation is o(xNd). The total complexity of self-attention calculation in the grouped Transformer is O((x+y)Nd), which is much smaller than O(N 2 d) That is, the complexity of O(xyNd), the grouped Transformer can capture the global connection between embedded tokens with lower complexity. Please refer to Figure 3, shows a schematic diagram of the grouping transformer in the embodiment of this specification. Figure 3 As shown, the tag sequence with position encoding is divided into x groups of tag sequences with y tags through the grouping strategy N = x × y:

[0085] T i =G(T in ,y)∈{T1,T2,…,T x} (5)

[0086] Among them, G(·,y) represents the grouping function of the tag sequence according to the number of tags y, and the tag sequence obtained by grouping is T i ∈R y×d sti∈{1,2,…,x}, where x groups of tag sequences complete multi-head attention calculations in parallel to capture the contextual connections between tags within the group. In the multi-head attention of each group of tag sequences, first, different transfer matrices are used to calculate the attention. Linearly project the embedded token sequence into h sets of queries, keys, and values ​​respectively:

[0087]

[0088] Among them, query key Sum h is the number of heads, d q d k d v =d / h. Query Q ij and key K ij Performing self-attention of scaled dot products yields a single-head attention matrix between tokens within a group. The attention matrix is ​​normalized by the softmax(·) operation and then summed with V ij Multiply to obtain the single-head attention modeling results of the group mark:

[0089]

[0090] Among them, head ij ∈R y×d / h Represents the label modeling result of the jth head of the i-th group in the grouped self-attention. All single-head attention modeling results in each group are spliced ​​in the direction of the label dimension to obtain the group label sequence T i Multi-head attention modeling results:

[0091] GMSA(T i )=Concat(head i1 ,head i2 ,···head ih ) (8)

[0092] Among them, GMSA(T i )∈R HW×d / h The multi-head attention modeling results of the i-th group of labeled sequences are concatenated to obtain the grouped multi-head self-attention modeling results of the input sequence:

[0093] T xy =Concat(GMSA(T1),GMSA(T2),···GMSA(T x )) (9)

[0094] Among them, Concat(·) is the concatenation function; T xy There are contextual connections between tags within a group, but due to the lack of attention interaction between tags in different groups, it is impossible to establish global contextual connections between features.

[0095] In order to overcome the defects of group self-attention, the group Transformer then labels the group model output of the group multi-head self-attention as T xy Cross-group interaction between different groups of tags is achieved by combining them across groups to capture the global connection between all tags. Figure 3 As shown in the cross-group multi-head self-attention, according to the characteristics of cross-group grouping, cross-group grouping will follow the grouping strategy of N=y×x to group T xy After grouping, we get y groups of labeled sequences with x labels:

[0096] T i '=G(T xy ,x)∈{T1',T'2,…,T' y} (10)

[0097] Among them, the group tag sequence T i '∈R x×d st . i∈{1,2,…,y}, y groups of tag sequences also perform multi-head attention calculations concurrently. The tag sequence T obtained by cross-group grouping i 'By grouping the used Linearly project to query, key, and value respectively:

[0098]

[0099] Among them, query key Sum Here, the transfer matrix is ​​parameter-shared to avoid introducing more parameter pressure to the model. i , K' i and V'i Perform the self-attention operation to obtain the modeling result of the single-head self-attention within the group in the cross-group self-attention:

[0100]

[0101] All the single-head self-attention modeling results in the group are spliced ​​along the direction of the tag dimension to obtain the tag sequence T i 'The results of multi-head self-attention modeling within the group:

[0102] GMSA(T i ')=Concat(head' i1 ,head' i2 ,···head' ij ) (13)

[0103] Then, the multi-head self-attention modeling results of all groups are spliced ​​in the direction of the number of tags. Since the reorganization of tags in the cross-group self-attention causes the position of the tags in space to change, direct splicing may cause the jump connection of the embedded tag sequence to be unreliable. Therefore, the spliced ​​tags need to be calibrated to reset the tags in space. The calibrated tag sequence is passed through the parameter matrix W o ∈R d×d Mapping to obtain the group self-attention modeling result T xy The cross-group multi-head self-attention modeling results:

[0104] T yx =Correct(Concat(GMSA(T1'),GMSA(T'2),···GMSA(T' y )))W o (14)

[0105] Among them, Correct(·) is a calibration function used to correct the misplaced marker sequence. yx ∈R HW×d The modeling output of the grouped Transformer attention layer is obtained through skip connections between the embedded tokens and layer normalization:

[0106] T′=LayerNormalization(T yx +T in ) (15)

[0107] T′ obtained by group self-attention and cross-group self-attention modeling contains global information between visual tags.

[0108] In order to further reduce the number of model parameters and balance the computational consumption of the attention layer and FFN, an improved feedforward network is proposed to reduce the label dimension by 4 times and then expand it by 4 times. Its structure is as follows Figure 3 As shown. T′ is obtained through FFN mapping:

[0109] FFN(T′)=σ(T′F1)F2 (16)

[0110] Where F1∈R d×d / 4 , F2∈R d / 4×d is the weight matrix, and σ is the ReLU activation function. FFN(T′) is connected to the modeling output of the self-attention layer through a jump connection, and then the layer is normalized and finally embedded with the tag T in The skip connection of , obtains the encoding output of the grouped Transformer:

[0111] T out =LayerNormalization(T′+FFN(T′))+T in (17)

[0112] The feature map T after modeling out The redundant feature information is filtered and the modality difference is reduced through GeMpooling operation and domain adaptation network, and the feature vector of the image is obtained. The present invention introduces meta-learning training strategy and domain alignment embedding loss to train and optimize the proposed GT-ResNet18 model. In the meta-learning task, K pairs of samples are randomly selected as batch samples, and the set of these K pairs of samples is used as the query set Q of the meta-task. t ={p1,p2,…,p k ,s1,s2,...,s k}, Q t The optical image in the image is used as the optical image support set P t ={p1,p2,...,p k}, Q t The sketch image in the sketch image support set S t ={s1,s2,...,s k}.Q t The optical image p i (i∈{1,2,…,k}) and sketch image s i (i∈{1,2,…,k}) is extracted by GT-ResNet18 to obtain the optical image feature vector f(p i / w) and the sketch feature vector f(s) i / w). The domain alignment embedding loss uses the Euclidean metric to measure the distance between features. The query set Q t The characteristics of optical images in S t The distance between the features of the sketch image and the query set Q t The characteristics of sketch images and P tThe distance between features in an optical image can be expressed using the Euclidean metric between feature vectors:

[0113] d(p i ,s i )=||f(p i / w)-f(s i / w)|| (18)

[0114] d(s i ,p i )=||f(s i / w)-f(p i / w)|| (19)

[0115] Here, ||·|| represents the Euclidean metric. The domain alignment embedding loss reduces the cross-entropy loss on the negative Euclidean metric between cross-domain features to make the features of the same label across domains closer, thereby increasing the similarity between images within a class. First, the label of the input image is predicted by the softmax function on the negative Euclidean metric between cross-domain images. For the optical image p i and sketches i The label prediction can be obtained:

[0116]

[0117]

[0118] Among them, P(s k / p i ) indicates that given p i With s k (k=1,2,…,k) the probability of consistency, P(p k / s i ) indicates that given s i With p k (k=1,2,…,k) is consistent. When k=i, that is, P(s i / p i ) and P(p i / s i ) means to change p i and s i The probability of predicting the same label as the corresponding cross-domain image. t The negative logarithmic probability P(s) of all optical images in i / p i ) and sum to get the alignment embedding loss in the optical image domain:

[0119]

[0120] Similarly, Q tThe negative logarithmic probability P(p i / s i )Sum.

[0121] Get the alignment embedding loss of the sketch domain:

[0122]

[0123] The sum of the alignment embedding losses of the optical image domain and the sketch image domain gives the total domain alignment embedding loss:

[0124] L=L ps +L sp (twenty four)

[0125] By reducing the domain alignment embedding loss, the feature vectors of the optical image and the sketch image with the same label in the common space are brought close to each other, allowing the network to extract discriminative information that is helpful for recognition.

[0126] A specific embodiment is given below to illustrate the above technical solution. First, the various hyperparameters in this method are determined. K represents the batch sample pairs in each meta-learning task and is set to 80. h represents the number of heads of MHSA and is set to 8. The deep learning PyTorch library can be used to build the model network framework, and the GPU memory size used in the experiment is 10.9GB. The initialization parameters of the convolutional network in GT-ResNet18 are the parameters of the first 4 stages of ResNet18 pre-trained on ImageNet, and the subsequent two-layer grouped Transformer and the final cross-domain alignment network are trained from scratch after initialization. In this embodiment, the Adamw optimizer is used to iteratively update the parameters of the model, and the learning rate is set to 0.00015, (β1, β2, weight_decay) = (0.9, 0.999, 0.02). For training on the UoM-SGFSv2 and CUFSF datasets, according to experimental tests, a total of 100 epochs were trained on the UoM-SGFSv2 dataset and 80 epochs were trained on the CUFSF dataset, and each epoch included 100 meta-learning tasks.

[0127] Before the experiment began, both datasets were preprocessed. This embodiment used MTCNN for face detection and alignment, preserving key facial features that are useful for recognition. During the training and testing phases, the images were uniformly resized to 256×256, and various enhancement techniques were used, including image deformation, padding, random cropping, and horizontal flipping. During the training phase, the feature extraction network of this embodiment was represented by f(. / w), parameterized by the model weights w. Given a meta-learning task K for the optical photos p in the sample, i Or sketch images i, the model of the present invention embeds it into the feature f(p i / w) or f(s) i / w), by continuously reducing the f(p i / w) or f(s) i / w) and the cross-domain feature f(s) with the same label extracted from the support set i / w) or f(p i / w) to reduce the modal difference between sketch images and optical photographs. The resulting domain alignment embedding loss is used to update model parameters through backpropagation to optimize the feature extraction network f(. / w). After training, the converged model is saved as a pth file. In the testing phase, the model is loaded and the sketch and photo image set to be recognized are input. The model network extracts features from the sketch and each image to calculate similarity. The image with the highest similarity to the sketch is considered the sketch recognition result.

[0128] In this embodiment, the Transformer is effectively applied to sketch face recognition. By designing a new grouped Transformer module, the efficiency of the Transformer's self-attention execution is improved, thereby improving the Transformer's performance in small sample scenarios. In this embodiment, a low-complexity feature extraction network GT-ResNet18 is constructed by combining a convolutional network with a grouped Transformer module, effectively alleviating the contradiction between the high complexity of the Transformer and the small sample size. This provides a specific solution for introducing the Transformer on small datasets. The trained GT-ResNet18 can capture global information between features to extract discriminative features between cross-domain images to improve the recognition effect of sketch faces. The grouped Transformer can achieve the best performance when evenly grouped. On the UoM-SGFS dataset, a synthetic sketch face dataset with a large number of sketch-optical image samples, and the CUFSF dataset, a viewing sketch face dataset, the recognition accuracy of GT-ResNet18 is significantly improved compared to the latest sketch face recognition algorithms based on convolutional neural networks. The recognition rate of the present invention on UoM-SGFSv2 reaches 86.00%, and the recognition rate on CUFSF reaches 93.75%, which proves the effectiveness and superiority of the method of this embodiment.

[0129] In the embodiments of this specification, a new grouped Transformer module is proposed, which optimizes the highly complex self-attention mechanism into two stages: grouped self-attention and cross-group self-attention. The grouped self-attention groups the labels to perform intra-group self-attention to improve efficiency, and the cross-group self-attention realizes cross-group interaction of different group labels through cross-group reorganization to learn global contextual connections. By replacing some bottleneck blocks in the residual network with grouped Transformers, a cross-domain discriminative feature extraction network GT-ResNet18 with lower complexity is constructed. This model alleviates the contradiction between the high complexity of the visual Transformer network and the small sample size of sketch face recognition. By introducing meta-learning training strategies and domain alignment embedding losses to train and optimize GT-ResNet18, the trained network can effectively extract discriminative features between optical photos and sketch photos.

[0130] Based on the same inventive concept, a sketch face recognition device is also provided in the embodiments of this specification, as described in the following embodiments. Since the principle of solving the problem by the sketch face recognition device is similar to that of the sketch face recognition method, the implementation of the sketch face recognition device can refer to the implementation of the sketch face recognition method, and the repeated parts will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements the predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived. Figure 4 This is a structural block diagram of the sketch face recognition device according to the embodiment of this specification. Figure 4 As shown, it includes: an acquisition module 401 and a training module 402. The structure is described below.

[0131] The acquisition module 401 is used to acquire a training set; the training set includes a sample set and a label set; the sample set includes an optical image set and a sketch image set; the label set includes a label of each optical image in the optical image set and a label of each sketch image in the sketch image set; the label of each optical image corresponds one-to-one to the label of each sketch image.

[0132] The training module 402 is used to train a preset recognition model using the training set to obtain a target recognition model; the target recognition model is used to perform face recognition on the sketch image.

[0133] Among them, the preset recognition model includes a convolutional neural network, a serialization module, a grouped Transformer network, a GeM pooling layer and a domain adaptation network; the grouped Transformer network includes a grouped multi-head self-attention module, a cross-group multi-head self-attention module and a feedforward network module; the convolutional neural network is used to extract features of the optical images and sketch images in the sample set to obtain corresponding low-resolution feature maps, and the serialization module is used to serialize the feature maps to obtain corresponding pixel-level visual marker sequences; the grouped multi-head self-attention module is used to group the corresponding visual marker sequences in the sample set to obtain the first group of the number of markers in the first preset group, and in each first Self-attention operation is performed within the group to generate a group multi-head self-attention modeling result; the cross-group multi-head self-attention module is used to perform cross-group combination on the group multi-head self-attention modeling result to obtain a second group with a second preset number of in-group labels; self-attention operation is performed within each second group to generate a cross-group multi-head self-attention modeling result, and a global connection between the visual labels corresponding to each sample in the sample set is obtained; the number of groups in the first group is equal to the number of labels in the second preset group, and the number of groups in the second group is equal to the number of labels in the first preset group; the domain adaptation network is used to project the cross-group multi-head self-attention modeling result after processing by the GeM pooling layer into a common subspace for pairing recognition.

[0134] In some embodiments of the present specification, the training module is specifically used to: generate multiple meta-learning tasks based on a meta-learning strategy; in each of the multiple meta-learning tasks, randomly select a preset number of optical images and corresponding sketch images from the training set, and generate a query set corresponding to each meta-learning task based on the selected optical images and sketch images, and the selected optical images and sketch images serve as an optical image support set and a sketch image support set, respectively; the labels of the optical images in each meta-learning task correspond one-to-one to the labels of the sketch images; execute the multiple meta-learning tasks to train the preset recognition model using the sample sets corresponding to the meta-learning tasks to obtain a target training model.

[0135] In some embodiments of the present specification, the grouped Transformer module is also used to perform jump connection and layer normalization processing on the cross-group multi-head self-attention modeling result and the visual marker sequence to obtain the modeling output result of the grouped Transformer self-attention layer; the feedforward network is used to perform FFN mapping on the modeling output result of the grouped Transformer self-attention layer to obtain an FFN mapping result, jump connect the FFN mapping result with the modeling output result of the grouped Transformer self-attention layer, perform layer normalization, and then jump connect with the visual marker sequence output by the grouped Transformer self-attention layer to obtain the encoding output result of the grouped Transformer network; the FFN mapping is used to first reduce the dimension of the visual marker by a preset multiple, and then expand the preset multiple.

[0136] In some embodiments of the present specification, the training module is specifically used to use the domain alignment embedding loss function to train and optimize the preset recognition model to obtain a target recognition model; the domain alignment embedding loss function includes the alignment embedding loss function of the optical image domain and the alignment embedding loss function of the sketch image domain.

[0137] In some embodiments of the present specification, the preset recognition model includes a two-layer group Transformer network.

[0138] In some embodiments of the present specification, the sketch face recognition device also includes a recognition module, which can be used to: obtain a set of images to be recognized; the set of images to be recognized includes a sketch image to be recognized and an optical image set; input the sketch image to be recognized and the optical image set into a target recognition model to obtain the similarity between the sketch image to be recognized and each optical image in the optical image set; and use the optical image whose similarity meets a preset condition as the recognition result of the sketch image to be recognized.

[0139] From the above description, it can be seen that the embodiments of this specification achieve the following technical effects: through the grouping Transformer network, the highly complex self-attention mechanism is optimized into two stages: group self-attention and cross-group self-attention. The group self-attention groups the tags to perform intra-group self-attention to improve efficiency. The cross-group self-attention realizes cross-group interaction of different group tags through cross-group reorganization to learn global contextual connections, which can capture long-distance features and enhance the key role of identification features in sketch face recognition, thereby effectively improving the accuracy of sketch face recognition.

[0140] This specification also provides a computer device. Figure 5The schematic diagram of the computer device structure shown in FIG. 1 is based on the sketch face recognition method provided in an embodiment of this specification. The computer device may include an input device 51, a processor 52, and a memory 53. The memory 53 is used to store processor-executable instructions. When the processor 52 executes these instructions, the steps of the sketch face recognition method described in any of the above embodiments are implemented.

[0141] In this embodiment, the input device can specifically be one of the primary devices for exchanging information between a user and a computer system. The input device can include a keyboard, mouse, camera, scanner, light pen, handwriting input tablet, voice input device, etc.; the input device is used to input raw data and programs for processing these data into the computer. The input device can also receive data transmitted from other modules, units, and devices. The processor can be implemented in any appropriate manner. For example, the processor can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (such as software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers, etc. The memory can specifically be a memory device used to store information in modern information technology. The memory can include multiple levels. In digital systems, anything that can store binary data can be considered a memory device. In integrated circuits, a circuit with storage functionality that does not have a physical form is also called a memory device, such as a RAM or FIFO. In systems, a physical storage device is also called a memory device, such as a memory stick or a TF card.

[0142] In this embodiment, the specific functions and effects achieved by the computer device can be explained in comparison with other embodiments and will not be repeated here.

[0143] The present specification also provides a computer storage medium based on a sketch face recognition method, wherein the computer storage medium stores computer program instructions, and when the computer program instructions are executed, the steps of the sketch face recognition method described in any of the above embodiments are implemented.

[0144] In this embodiment, the storage medium includes, but is not limited to, random access memory (RAM), read-only memory (ROM), cache, hard disk drive (HDD), or memory card. The memory can be used to store computer program instructions. The network communication unit can be an interface configured in accordance with the standards specified by the communication protocol for network connection communication.

[0145] In this embodiment, the functions and effects specifically implemented by the program instructions stored in the computer storage medium can be explained in comparison with other embodiments and will not be repeated here.

[0146] Obviously, those skilled in the art should understand that the various modules or steps of the above-mentioned embodiments of this specification can be implemented using a general-purpose computing device, they can be concentrated on a single computing device, or distributed across a network composed of multiple computing devices. Alternatively, they can be implemented using program code executable by the computing device, so that they can be stored in a storage device and executed by the computing device. In some cases, the steps shown or described can be performed in a different order than herein, or they can be made into separate integrated circuit modules, or multiple modules or steps can be made into a single integrated circuit module for implementation. Thus, the embodiments of this specification are not limited to any specific combination of hardware and software.

[0147] It should be understood that the above description is intended to be illustrative and not limiting. Numerous embodiments and applications beyond the examples provided will be readily apparent to those skilled in the art upon reading the above description. Therefore, the scope of this specification should not be determined with reference to the above description, but rather with reference to the preceding claims, along with the full scope of equivalents to which such claims are entitled.

[0148] The above description is merely a preferred embodiment of this specification and is not intended to limit this specification. Those skilled in the art will readily appreciate that various modifications and variations to the embodiments of this specification are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this specification shall be within the scope of protection of this specification.

Claims

1. A sketch face recognition method, characterized in that: include: Get the training set; The training set includes a sample set and a label set; the sample set includes an optical image set and a sketch image set; the label set includes a label of each optical image in the optical image set and a label of each sketch image in the sketch image set; The labels of the optical images correspond one-to-one to the labels of the sketch images; The training set is used to train a preset recognition model to obtain a target recognition model; the target recognition model is used to perform face recognition on the sketch image; Among them, the preset recognition model includes a convolutional neural network, a serialization module, a grouped Transformer network, a GeM pooling layer and a domain adaptation network; the grouped Transformer network includes a grouped multi-head self-attention module, a cross-group multi-head self-attention module and a feedforward network module; the convolutional neural network is used to extract features of the optical images and sketch images in the sample set to obtain corresponding low-resolution feature maps, and the serialization module is used to serialize the feature maps to obtain corresponding pixel-level visual marker sequences; the grouped multi-head self-attention module is used to group the corresponding visual marker sequences in the sample set to obtain the first group of the number of markers in the first preset group, and in each first Self-attention operation is performed within the group to generate a group multi-head self-attention modeling result; the cross-group multi-head self-attention module is used to perform cross-group combination on the group multi-head self-attention modeling result to obtain a second group with a second preset number of in-group labels; self-attention operation is performed within each second group to generate a cross-group multi-head self-attention modeling result, and a global connection between the visual labels corresponding to each sample in the sample set is obtained; the number of groups in the first group is equal to the number of labels in the second preset group, and the number of groups in the second group is equal to the number of labels in the first preset group; the domain adaptation network is used to project the cross-group multi-head self-attention modeling result after processing by the GeM pooling layer into a common subspace for pairing recognition.

2. The sketch face recognition method according to claim 1, characterized in that: The preset recognition model is trained using the training set to obtain a target recognition model, including: generating a plurality of meta-learning tasks based on a meta-learning strategy; randomly selecting a preset number of optical images and corresponding sketch images from the training set in each of the plurality of meta-learning tasks; generating a query set corresponding to each meta-learning task based on the selected optical images and sketch images, wherein the selected optical images and sketch images serve as a support set for optical images and a support set for sketch images, respectively; and having a one-to-one correspondence between labels of the optical images and labels of the sketch images in each meta-learning task; The multiple meta-learning tasks are executed to train a preset recognition model using the sample sets corresponding to the meta-learning tasks to obtain a target training model.

3. The sketch face recognition method according to claim 1, characterized in that: The grouped Transformer network is also used to perform jump connection and layer normalization processing on the cross-group multi-head self-attention modeling result and the visual marker sequence to obtain the modeling output result of the grouped Transformer self-attention layer; the feedforward network module is used to perform FFN mapping on the modeling output result of the grouped Transformer self-attention layer to obtain an FFN mapping result, jump connect the FFN mapping result with the modeling output result of the grouped Transformer self-attention layer, perform layer normalization, and then jump connect with the visual marker sequence output by the grouped Transformer self-attention layer to obtain the encoding output result of the grouped Transformer network; the FFN mapping is used to first reduce the dimension of the visual marker by a preset multiple, and then expand the preset multiple to enhance the representation ability of the preset recognition model.

4. The sketch face recognition method according to claim 1, characterized in that: In the process of training the preset recognition model using the training set, the preset recognition model is trained and optimized using the domain alignment embedding loss function to obtain a target recognition model; the domain alignment embedding loss function includes the alignment embedding loss function of the optical image domain and the alignment embedding loss function of the sketch image domain.

5. The sketch face recognition method according to claim 1, characterized in that: The preset recognition model includes a two-layer grouping Transformer network.

6. The sketch face recognition method according to claim 1, characterized in that: After the preset recognition model is trained using the training set to obtain the target recognition model, the method further includes: Acquire a set of images to be identified; the set of images to be identified includes sketch images to be identified and an optical image set; Inputting the sketch image to be identified and the optical image set into a target recognition model to obtain similarities between the sketch image to be identified and each optical image in the optical image set; The optical image whose similarity meets the preset conditions is used as the recognition result of the sketch image to be recognized.

7. A sketch face recognition device, characterized in that: include: An acquisition module, configured to acquire a training set; the training set includes a sample set and a label set; the sample set includes an optical image set and a sketch image set; the label set includes a label of each optical image in the optical image set and a label of each sketch image in the sketch image set; The labels of the optical images correspond one-to-one to the labels of the sketch images; A training module is used to train a preset recognition model using the training set to obtain a target recognition model; the target recognition model is used to perform face recognition on the sketch image; Among them, the preset recognition model includes a convolutional neural network, a serialization module, a grouped Transformer network, a GeM pooling layer and a domain adaptation network; the grouped Transformer network includes a grouped multi-head self-attention module, a cross-group multi-head self-attention module and a feedforward network module; the convolutional neural network is used to extract features of the optical images and sketch images in the sample set to obtain corresponding low-resolution feature maps, and the serialization module is used to serialize the feature maps to obtain corresponding pixel-level visual marker sequences; the grouped multi-head self-attention module is used to group the corresponding visual marker sequences in the sample set to obtain the first group of the number of markers in the first preset group, and in each first Self-attention operation is performed within the group to generate a group multi-head self-attention modeling result; the cross-group multi-head self-attention module is used to perform cross-group combination on the group multi-head self-attention modeling result to obtain a second group with a second preset number of in-group labels; self-attention operation is performed within each second group to generate a cross-group multi-head self-attention modeling result, and a global connection between the visual labels corresponding to each sample in the sample set is obtained; the number of groups in the first group is equal to the number of labels in the second preset group, and the number of groups in the second group is equal to the number of labels in the first preset group; the domain adaptation network is used to project the cross-group multi-head self-attention modeling result after processing by the GeM pooling layer into a common subspace for pairing recognition.

8. The sketch face recognition device according to claim 7, characterized in that: The training module is specifically used for: generating a plurality of meta-learning tasks based on a meta-learning strategy; randomly selecting a preset number of optical images and corresponding sketch images from the training set in each of the plurality of meta-learning tasks; generating a query set corresponding to each meta-learning task based on the selected optical images and sketch images, wherein the selected optical images and sketch images serve as a support set for optical images and a support set for sketch images, respectively; and having a one-to-one correspondence between labels of the optical images and labels of the sketch images in each meta-learning task; The multiple meta-learning tasks are executed to train a preset recognition model using the sample sets corresponding to the meta-learning tasks to obtain a target training model.

9. A computer device, characterized in that: The method comprises a processor and a memory for storing processor-executable instructions, wherein the processor implements the steps of the method according to any one of claims 1 to 6 when executing the instructions.

10. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images