Sketch face recognition method, device and equipment based on lightweight semantic transformer model

Abstract

This specification provides a method, apparatus, and device for sketch face recognition based on a lightweight semantic Transformer model. The method includes: acquiring feature images of training samples; inputting the feature images into a Transformer model to obtain first and second layer abstract semantic labels, and then establishing globally related abstract semantic labels; determining a target refinement feature map based on the feature images and globally related abstract semantic labels; determining a target domain adaptation focus loss based on the target refinement feature map; optimizing the Transformer model based on the target domain adaptation focus loss to obtain a target Transformer model; acquiring a sketch image and an optical face photograph to be recognized, and inputting the sketch image and optical face photograph into the target Transformer model to obtain the sketch face recognition result. This method can improve the accuracy of sketch face recognition.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method, apparatus and device for sketching face recognition based on a lightweight semantic Transformer model. Background Technology

[0002] Sketch face recognition refers to using a provided sketch image as an example, comparing the sketch image with optical photographs containing identity information in an optical face database, and finding the most similar photograph as the matching image for the sketch to determine the person's identity information or to identify several similar photographs to narrow down the scope of investigation.

[0003] In sketch face datasets, a single optical photograph typically produces only one sketch image, making sketch face recognition characterized by cross-modal recognition and small sample sizes. Existing sketch face recognition methods are unable to effectively address the small sample sizes and semantic errors introduced during sketch generation, resulting in reduced accuracy.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This manual provides a method, apparatus, and device for sketch face recognition based on a lightweight semantic Transformer model, which can improve the accuracy of sketch face recognition.

[0006] On the one hand, embodiments of this specification provide a sketch face recognition method based on a lightweight semantic Transformer model, including:

[0007] Obtain feature images of training samples, wherein the training samples include multiple sets of optical face photos and multiple sets of sketch images corresponding to the optical face photos;

[0008] The feature image is input into a pre-constructed initial lightweight semantic Transformer model to obtain the first layer of abstract semantic tags and the second layer of abstract semantic tags;

[0009] Establish a global connection between the first-level abstract semantic tag and the second-level abstract semantic tag to obtain an abstract semantic tag with a global connection;

[0010] Based on the feature image and the abstract semantic tag with global connection, a target refinement feature map with semantic information is obtained; wherein, the target refinement feature map with semantic information includes the refinement feature map of the optical face photograph and the refinement feature map of the sketch image;

[0011] Based on the refined feature map of the optical face photograph and the refined feature map of the sketch image, the target domain adaptation focus loss is determined;

[0012] The initial lightweight semantic Transformer model is optimized based on the target domain adaptation focus loss to obtain the target lightweight semantic Transformer model;

[0013] Obtain the sketch image and optical face photo to be identified, input the sketch image and optical face photo to be identified into the target lightweight semantic Transformer model, obtain the target matching result of the sketch image and optical face photo to be identified, and use the target matching result as the sketch face recognition result.

[0014] Further, obtaining a refined feature map of the target with semantic information based on the feature image and the abstract semantic tag includes:

[0015] The abstract semantic tags with global relationships are subjected to hierarchical group linear transformation to obtain lightweight semantic tags;

[0016] The lightweight semantic tags are modeled using lightweight tags to obtain the modeling results of lightweight tags;

[0017] Based on the modeling results of the lightweight marker, determine the encoding output of the initial lightweight semantic Transformer model;

[0018] Based on the feature image and the encoding output, a target refinement feature map with semantic information is obtained.

[0019] Furthermore, the step of performing a hierarchical group linear transformation on the abstract semantic tags with global relationships to obtain lightweight semantic tags includes:

[0020] The lightweight semantic tags are obtained by performing a hierarchical group linear transformation according to the following formula:

[0021]

[0022] Tin′=H(Tin)LW r

[0023] Where H(·) represents the hierarchical group linear transformation, H(T) in ) l G(·) represents the output of the group linear transformation of the l-th layer, and M(·) represents the mixed input connection. l The number of groups representing the linear transformation of the l-th layer. Let l be the weight set of the linear transformation of the l-th layer. Let H(T) be the bias set of the linear transformation of the l-th layer group, Tin′ represent the lightweight semantic tag, and H(T) be the bias set of the linear transformation of the l-th layer group. in L∈R N×d It is the output of the last level of the group linear transformation in the group linear transformation, T in W is an abstract semantic tag with global connections. r ∈R d×do It is a dimension reduction linear transformation.

[0024] Further, the step of performing lightweight tag modeling on the lightweight semantic tag to obtain the lightweight tag modeling result includes:

[0025] Lightweight tagging modeling is performed according to the following formula to obtain the lightweight tagging modeling results:

[0026] Q i K i V i =T′ in W i q ,T′ in W i k ,T′ in W i v , i∈{1,2,…,h},

[0027]

[0028] MultiHead(T′ in = Concat(head1, ..., head) h W o ,

[0029] T′ out =Layer Normalization(T in +MultiHead(T′ in ))

[0030] Among them, Q i For query, K i As a key, V i For the value, W i q W i k W i v Let T′ be the projection weight matrix. in For lightweight semantic markup, head i This is the result of single-head modeling of a multi-head attention mechanism, SOFTMAX. N It is a normalization function, d kIt is K i The dimension of MultiHead(T) i ′ n The result is the calculation result of multi-head attention, where Concat represents the number of connections, h is the number of heads, and W is the number of heads. o The projection weight matrix T′ used to adjust the labeled dimension out It is the modeling result of lightweight labeling, while Layer Normalization is layer normalization.

[0031] Further, determining the encoding output of the initial lightweight semantic Transformer model based on the modeling results of the lightweight marker includes:

[0032] The encoding output of the initial lightweight semantic Transformer model is determined according to the following formula:

[0033] FFN(T′ out )=σ(T′ out F1)F2,

[0034] T out =Layer Normalization(T′) out +FFN(T′ out ))

[0035] Where FFN is a feedforward network, T′ out For the modeling results, T out For the encoded output, F1 and F2 are weight matrices, σ is the ReLU activation function, and Layer Normalization is the layer normalization.

[0036] Further, obtaining a target refinement feature map with semantic information based on the feature image and the encoding output includes:

[0037] Based on the feature image and the encoded output result, the feature image information is determined;

[0038] The feature image information and the feature image are fused to obtain a target refinement feature map with semantic information.

[0039] Further, determining the target domain adaptation focus loss based on the refined feature map of the optical face photograph and the refined feature map of the sketch image includes:

[0040] Based on the refined feature map of the optical face photograph, determine the photograph domain alignment embedding loss;

[0041] Based on the refined feature map of the sketch image, determine the sketch domain alignment embedding loss;

[0042] The target domain adaptation focus loss is determined based on the photo domain alignment embedding loss and the sketch domain alignment embedding loss.

[0043] On the other hand, embodiments of this specification also provide a sketch face recognition device based on a lightweight semantic Transformer model, including:

[0044] The acquisition module is used to acquire feature images of training samples, wherein the training samples include multiple sets of optical face photos and multiple sets of sketch images corresponding to the optical face photos;

[0045] The semantic tagging module is used to input the feature image into a pre-constructed initial lightweight semantic Transformer model to obtain the first layer of abstract semantic tags and the second layer of abstract semantic tags;

[0046] The refinement module is used to establish a global connection between the first-layer abstract semantic marker and the second-layer abstract semantic marker to obtain an abstract semantic marker with global connection; based on the feature image and the abstract semantic marker with global connection, a target refinement feature map with semantic information is obtained; wherein, the target refinement feature map with semantic information includes the refinement feature map of the optical face photograph and the refinement feature map of the sketch image;

[0047] The model optimization module is used to determine the target domain adaptation focus loss based on the refined feature map of the optical face photo and the refined feature map of the sketch image; and to optimize the initial lightweight semantic Transformer model based on the target domain adaptation focus loss to obtain the target lightweight semantic Transformer model.

[0048] The recognition module is used to acquire the sketch image and optical face photo to be recognized, input the sketch image and optical face photo to be recognized into the target lightweight semantic Transformer model, obtain the target matching result of the sketch image and optical face photo to be recognized, and use the target matching result as the sketch face recognition result.

[0049] Furthermore, this application also provides a sketch face recognition device based on a lightweight semantic Transformer model, including a processor and a memory for storing processor-executable instructions, wherein the processor implements the above-mentioned sketch face recognition method based on a lightweight semantic Transformer model when executing the instructions.

[0050] In another aspect, this application also provides a computer-readable storage medium storing computer instructions thereon, wherein the computer-readable storage medium implements the above-described sketch face recognition method based on a lightweight semantic Transformer model when the instructions are executed.

[0051] The sketch face recognition method, apparatus, and device based on a lightweight semantic Transformer model provided in this specification first acquire feature images of training samples, wherein the training samples include multiple sets of optical face photographs and multiple sets of sketch images corresponding to the optical face photographs; acquiring feature images lays the foundation for subsequent semantic tag generation, modeling, and reshaping; secondly, the feature images are input into a pre-constructed initial lightweight semantic Transformer model to obtain a first-layer abstract semantic tag and a second-layer abstract semantic tag; further, a global connection is established between the first-layer abstract semantic tag and the second-layer abstract semantic tag to obtain an abstract semantic tag with global connection; based on the feature images and the abstract semantic tags with global connection, a target refined feature map with semantic information is obtained; wherein the target refined feature map with semantic information includes refined feature maps of the optical face photographs and refined feature maps of the sketch images; by generating refined feature maps, it is advantageous to... To improve the recognition accuracy of sketched faces, the following steps are taken: First, based on the refined feature maps of the optical face photograph and the sketched image, a target domain adaptation focus loss is determined. This loss reduces pattern differences between the sketched image and the optical face photograph. Second, the initial lightweight semantic Transformer model is optimized based on the target domain adaptation focus loss to obtain a target lightweight semantic Transformer model. This model effectively addresses the issues of small sample sizes and semantic errors introduced during sketch generation, thus improving accuracy. Finally, the sketched image and optical face photograph to be recognized are acquired and input into the target lightweight semantic Transformer model to obtain a target matching result. This matching result is then used as the sketched face recognition result. This approach effectively extracts semantic features, improving the accuracy of sketched face recognition. Attached Figure Description

[0052] To more clearly illustrate the embodiments of this specification, the accompanying drawings used in the embodiments will be briefly introduced below. The drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a flowchart illustrating the sketch face recognition method based on a lightweight semantic Transformer model in an embodiment of the present invention.

[0054] Figure 2 This is an overall architecture diagram of the lightweight semantic Transformer model in this embodiment of the invention;

[0055] Figure 3 This is a structural diagram of the filter marker in an embodiment of the present invention;

[0056] Figure 4 This is a structural diagram of the cyclic marker in an embodiment of the present invention;

[0057] Figure 5 This is a schematic diagram of the structure of the standard Transformer module in an embodiment of the present invention;

[0058] Figure 6 This is a schematic diagram of the lightweight Transformer module in an embodiment of the present invention;

[0059] Figure 7 This is an overall flowchart of the hierarchical group linear transformation in an embodiment of the present invention;

[0060] Figure 8 This is a schematic diagram of the structural composition of the sketch face recognition device based on the lightweight semantic Transformer model in an embodiment of the present invention;

[0061] Figure 9 This is a schematic diagram of the structural composition of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0062] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0063] In real-world scenarios, sketch face recognition involves image pairing. Given a sketch image, it's compared to optical photographs containing identity information in an optical face database. The most similar photograph is chosen as the paired image to identify the person or narrow down the search by identifying several similar photographs. In algorithm research, sketch face recognition is a single-sample pairing problem. In sketch face datasets, an optical photograph typically only produces one sketch image, presenting two major challenges: cross-modal recognition and limited sample size. Furthermore, since sketch face images are generated based on semantic descriptions, semantic errors can easily be introduced during the description and rendering (or synthesis) process. For example, differences in drawing style can lead to different rendering results, affecting the accuracy of sketch face recognition.

[0064] Furthermore, traditional sketch face recognition methods are primarily based on manually designed techniques, such as Local SIFT, HOG, and LBP features. These methods use feature operators to describe image features and measure the degree of matching. However, because these methods have fixed feature extraction patterns and limited shallow feature representation capabilities, they struggle to describe the highly nonlinear relationships between cross-modal images, resulting in low recognition rates. Moreover, unlike neural networks, they cannot automatically learn and extract features, leading to poor generalization. With the development of deep learning, Convolutional Neural Networks (CNNs), as the backbone networks for various visual tasks, have driven advancements in computer vision with their larger scale, wider connectivity, and more complex convolutional forms, becoming mainstream in sketch face recognition. CNNs extract facial features from optical images and match them with sketch face features for recognition. However, existing deep learning-based methods primarily use CNNs as the network architecture to learn modality-invariant features. But CNNs, acting as local feature extractors, cannot establish the interactive relationships between semantic concepts in space, highlight the crucial role of enhancing discriminative features in recognition, or address the impact of semantic errors introduced during sketch generation on the accuracy of sketch face recognition. Existing CNN-based sketch face methods are limited by the inherent limitations of CNNs, making it difficult to further improve the performance of sketch face recognition.

[0065] Furthermore, existing intramodal sketch face recognition algorithms rely on high-quality synthetic target modality images. In real-world scenarios, these synthetic images suffer from artifacts and structural distortions, negatively impacting recognition performance. They also fail to address the influence of semantic errors introduced during face description and sketching processes on sketch face recognition. Existing intermodal sketch face recognition algorithms cannot effectively handle the small size of sketch face datasets, leading to model overfitting. Additionally, CNN-based feature extraction networks are limited to capturing local information and cannot establish the interactive relationships of semantic concepts in space, resulting in reduced recognition performance.

[0066] To address the aforementioned issues, this application proposes to introduce a sketch face recognition method based on a lightweight semantic Transformer model. This method extracts consistent semantic information between optical images and sketch images to improve the accuracy of sketch face recognition, while simultaneously alleviating the contradiction between the high complexity of the sketch face recognition model and the small sample size of sketch face recognition.

[0067] Based on the above approach, this specification proposes a sketch-based face recognition method using a lightweight semantic Transformer model. First, feature images of training samples are acquired, where the training samples include multiple sets of optical face photographs and sketch images corresponding to those photographs. Second, the feature images are input into a pre-constructed initial lightweight semantic Transformer model to obtain a first-layer abstract semantic label and a second-layer abstract semantic label. Further, a global connection is established between the first-layer and second-layer abstract semantic labels to obtain globally connected abstract semantic labels. Based on the feature images and the globally connected abstract semantic labels, a target refinement feature map with semantic information is obtained. The target refinement feature map of the information includes the refinement feature map of the optical face photograph and the refinement feature map of the sketch image; further, based on the refinement feature map of the optical face photograph and the refinement feature map of the sketch image, a target domain adaptation focus loss is determined; the initial lightweight semantic Transformer model is optimized based on the target domain adaptation focus loss to obtain a target lightweight semantic Transformer model; finally, the sketch image and the optical face photograph to be recognized are obtained, and the sketch image and the optical face photograph to be recognized are input into the target lightweight semantic Transformer model to obtain the target matching result of the sketch image and the optical face photograph to be recognized, and the target matching result is used as the sketch face recognition result. (See also...) Figure 1 As shown in the embodiments of this specification, a sketch-based face recognition method based on a lightweight semantic Transformer model is provided. In specific implementations, this method may include the following:

[0068] S101: Obtain feature images of training samples, wherein the training samples include multiple sets of optical face photos and multiple sets of sketch images corresponding to the optical face photos.

[0069] S102: Input the feature image into the pre-constructed initial lightweight semantic Transformer model to obtain the first layer abstract semantic label and the second layer abstract semantic label.

[0070] S103: Establish a global connection between the first-level abstract semantic tag and the second-level abstract semantic tag to obtain an abstract semantic tag with a global connection.

[0071] S104: Based on the feature image and the abstract semantic tag with global connection, obtain a target refinement feature map with semantic information; wherein, the target refinement feature map with semantic information includes the refinement feature map of the optical face photograph and the refinement feature map of the sketch image.

[0072] S105: Determine the target domain adaptation focus loss based on the refined feature map of the optical face photograph and the refined feature map of the sketch image.

[0073] S106: Optimize the initial lightweight semantic Transformer model based on the target domain adaptation focus loss to obtain the target lightweight semantic Transformer model.

[0074] S107: Obtain the sketch image and optical face photo to be identified, input the sketch image and optical face photo to be identified into the target lightweight semantic Transformer model, obtain the target matching result of the sketch image and optical face photo to be identified, and use the target matching result as the sketch face recognition result.

[0075] In some embodiments, reference may be made to Figure 2 As shown, Figure 2The overall architecture of the lightweight semantic Transformer model is shown. The lightweight semantic Transformer model can consist of a convolutional network (or convolutional neural network), two lightweight visual transformation modules (LWVT), and a domain adaptation module. The convolutional neural network can be composed of the first four convolutional layers of an 18-layer residual network (ResNet18) used to acquire feature images. For example, multiple sets of optical face photos and their corresponding sketch images can be input into the convolutional neural network to obtain feature images of the optical face photos and their corresponding sketch images. The LWVT module includes a labeler, a lightweight Transformer module, and an inverse transformer. The lightweight semantic Transformer model involves two LWVT layers. The lightweight Transformer modules in each LWVT layer can share parameters to reduce the overall model parameters and alleviate the training difficulty of LWVT-ResNet18 on the small sketch face dataset. The domain adaptation module is a fully connected layer with batch normalization, used to project cross-modal features into a common space.

[0076] In some embodiments, the overall process of obtaining the target lightweight semantic Transformer model can be as follows: First, a meta-learning training strategy effective for small samples is used to obtain K pairs of task-related training samples, such as multiple sets of optical face photos and multiple sets of sketch images corresponding to optical face photos; then, feature maps of the optical face photos and sketch images are extracted through a convolutional network; next, the feature maps are input into a stacked LWVT module (two layers of LWVT) to complete the generation, lightweighting, modeling, and reshaping of semantic tags (reshaping into more detailed feature maps, and then transforming the feature maps into feature vectors); finally, the cross-modal features (reshaped feature vectors) extracted by the network are projected onto a common subspace through a domain adaptation module, combined with a domain adaptation focus loss L... focal This approach reduces intra-class distance while increasing inter-class distance, minimizing the accuracy impact of cross-modal changes and improving model training performance, ultimately resulting in a well-trained lightweight semantic Transformer model. The specific process for training this lightweight semantic Transformer model will be explained later and will not be elaborated upon here.

[0077] In some embodiments, the above-mentioned meta-learning training strategy effective for small samples can specifically be: given a training set D train ={P,S}={p1,p2,…,p N ,s1,s2,…,s N},in Represents an optical face photo, represents a sketch image, i is their label, and there is a one-to-one correspondence between the optical face photo and the sketch image. In each meta-learning task, randomly select K (K < N) pairs of data where the optical face photo and the sketch have the same identity, and then set the label i of the K pairs of data to {1, 2, …, k}. The goal of each learning task is to correctly match these K pairs of samples as much as possible, where K is a hyperparameter of the meta-learning strategy. The set of these K pairs of data serves as the query set of the meta-task where Q t in P t serves as the optical face photo support set, S t serves as the sketch image support set, and T represents the number of meta-tasks. The query set Q of the meta-task obtained through the meta-learning training strategy can be t used as the above training samples to participate in subsequent model training.

[0078] In some embodiments, for the above-mentioned feature images of the acquired training samples, the following method can be adopted: input the training samples into the convolutional neural network module in the pre-constructed initial lightweight semantic Transformer model to obtain the feature images of the training samples, and the specific extraction process can be referred to the following formula:

[0079] X in = conv(x)(1)

[0080] where X in is the feature map of the query set image x is the query set image (x ∈ Q t ), and conv is the convolution operation.

[0081] In some embodiments, after obtaining the feature images, the first-layer abstract semantic tokens and the second-layer abstract semantic tokens can be obtained respectively according to the filtering tokenizer (T conv ) in the first layer of the LWVT module and the recurrent tokenizer (T rec ) in the second layer of the LWVT module. Refer to Figure 3 as shown Figure 3The diagram shows the structure of the filter tagger. After obtaining the feature image, the filter tagger can convert the feature images of the optical face photograph and the sketch image into semantic groups (it should be noted that, to distinguish them from subsequent semantic groups, this can also be represented as the first semantic group). Then, a spatial attention group is generated based on the semantic group (it should be noted that, to distinguish them from subsequent spatial attention groups, this can also be represented as the first spatial attention group). Finally, the feature image and the spatial attention group are multiplied to obtain the abstract semantic tag. The specific execution process of the filter tagger can be as follows: First, the feature image X... in Convolution with N convolution kernels of size 1×1 W A Perform pointwise convolution to reduce the number of channels to N; then, pass the SOFTMAX algorithm to each channel's pixels. HW (.) Generate N spatial attention groups; finally, the attention groups spatially aggregate pixels to obtain an abstract semantic tag with N tags. The form is shown in the following formula:

[0082] T in1 =T conv (X in =SOFTMAX HW (R(X in W A )) T R(X in (2)

[0083] Among them, T in1 Abstract semantic markup T conv For the filter marker, X in For the feature image, SOFTMAX_HW(.) is the normalization function for generating the spatial attention group. It is an N-channel semantic group obtained through convolution, and R(·) is a tensor recombination operation used to adjust the shape of the tensor, where, This represents the different dimensions in which a tensor exists.

[0084] In some embodiments, see Figure 4 As shown, Figure 4 The diagram shows the structure of the loop tagger. After obtaining the abstract semantic tags, the weight matrix (W) of the first layer of the LWVT module in the pre-constructed initial lightweight semantic Transformer model can be determined based on the abstract semantic tags. BThis can be denoted as the target weight matrix. After obtaining the target weight matrix, multiplying it with the first refined feature map output from the first layer of the LWVT module yields the second semantic group. Then, a second spatial attention group is generated based on the second semantic group. Finally, multiplying the first refined feature map with the second spatial attention group yields the second-layer abstract semantic label. The specific execution process of the recurrent labeler can be as follows: First, based on the first refined feature map generated by the first-layer LWVT... After weight matrix Generate semantic groups Secondly, the semantic group is processed by SOFTMAX. HW (.) Generate spatial attention groups. Finally, the spatial attention groups aggregate pixels to generate a second-layer abstract semantic tag with N LWVT tags. The process of generating tags using a cyclic tagger is shown in the following formula:

[0085] W B =(T in1 W A→B ) T

[0086] T in2 =T rec (X out1 =SOFTMAX HW (R(X out1 W B ) T R(X out1 (3)

[0087] Among them, W B Let W be the weight matrix. A For convolution, T in1 T is the first level of abstract semantic tagging. in2 T is the second-level abstract semantic marker. rec For the cycle marker, X out1 For the first refined feature map, SOFTMAX_HW(.) is the normalization function for generating the spatial attention group, and R(·) is the tensor recombination operation.

[0088] The weights W are further derived by using the abstract semantic tags generated by the previous LWVT layer. B , replacing convolution W A This method completes the transformation from features to labels, overcoming the limitations of filter labelers in classification tasks where the weights of the convolution kernel are fixed due to certain categories having specific semantic concepts.

[0089] In some embodiments, after obtaining the first-level abstract semantic tag and the second-level abstract semantic tag, a global connection can be established between the semantic tags, that is, a global connection can be established between the first-level abstract semantic tag and the second-level abstract semantic tag to obtain semantic tags with global connection, thereby laying a data foundation for subsequently obtaining a target refinement feature map with semantic information.

[0090] In some embodiments, the above-described method of obtaining a target refinement feature map with semantic information based on the feature image and the abstract semantic tag with global relationships may, in specific implementations, include:

[0091] S1: Perform a hierarchical group linear transformation on the abstract semantic tags with global relationships to obtain lightweight semantic tags;

[0092] S2: Perform lightweight tag modeling on the lightweight semantic tags to obtain the lightweight tag modeling results;

[0093] S3: Based on the modeling results of the lightweight marker, determine the encoding output of the initial lightweight semantic Transformer model;

[0094] S4: Based on the feature image and the encoding output, a target refinement feature map with semantic information is obtained.

[0095] In some embodiments, after obtaining the abstract semantic tags with global relationships, the process proceeds to the lightweight Transformer module. The lightweight Transformer module is used to model the semantic tag relationships, thereby enhancing the target identification information. See also... Figure 5 and Figure 6 As shown, Figure 5 A schematic diagram of the standard Transformer module is shown. Figure 6 A schematic diagram of the lightweight Transformer module is shown, wherein, see reference Figure 5 As shown, the standard Transformer module mainly includes a multi-head self-attention (MSA) module and a feed-forward network (FFN). Since the Transformer module has high complexity, and this complexity is directly related to the number and size of the labels, when the number of labels and the dimension are N and d respectively, the computational cost in MSA is O(4Nd) in relation to the number of labels and the dimension. 2 )+o(2N 2 d), where the computational cost of the self-attention layer is O(2N). 2d), the computational cost in FFN is related to the number of labels and the dimension as O(8Nd). 2 Therefore, while using a tagger to generate a smaller number of semantic tags, this specification also optimizes the MSA and FFN parts of the standard Transformer from the perspective of tag dimension, which is directly related to model complexity, to construct a lightweight Transformer. For details, please refer to... Figure 6 As shown, before completing the semantic tag modeling, firstly, the dimensions of the first and second layer abstract semantic tags are halved using hierarchical group linear transformation (HGLT) to reduce model complexity. Then, to achieve jump connections with the input tags (i.e., abstract semantic tags with global relationships are input into the Transformer module, collectively referred to as input tags), the modeled tags are remapped to the dimensions of the input tags. Finally, at FFN, the tag dimensions change from first expanding by a factor of 4 and then shrinking by a factor of 4 to first shrinking by a factor of 4 and then expanding by a factor of 4. This specification also uses parameter sharing between the two lightweight Transformers in the two LWVT layers (the two lightweight Transformers are located in different layers of the LWVT module) to reduce model parameters. The implementation details of the lightweight Transformer module are as follows:

[0096] First, the complexity of the multi-head self-attention module (MSA) in the Transformer can be reduced by using hierarchical group linear transformation to achieve lightweight semantic labeling, thus alleviating the contradiction between the high complexity of the Transformer and the small sample size problem of sketch face recognition.

[0097] For the overall process of Hierarchical Group Linear Transform (HGLT), please refer to [link / reference]. Figure 7 As shown, firstly, semantic tags are mapped to the initial dimension d of the group linear transformation through a linear layer. i The group linear transformation (GLT) in the first half of the HGLT progressively expands the dimension of the input labels to the maximum dimension d. m In the latter part of the group linear transformation, the dimensionality decreases layer by layer; finally, the semantic tags are projected onto half the dimension d of the generated tags. o (d o =d / 2), and after each GLT layer there is a mixer to complete the mixing connection of the GLT output label and the input label. Through the above operations, the complexity of the lightweight semantic Transformer model can be reduced, laying the foundation for subsequent training to obtain the final target lightweight semantic Transformer model.

[0098] In HGLT, the grouping variation of GLT follows the same expansion-shrinkage strategy as the features. The number of groups initially increases with each layer until it reaches the maximum number of groups, g_max, and then decreases layer by layer. The change in the number of GLT groups is consistent with the change in the label dimension. When the label dimension is expanded to a higher dimension, GLT can have more groups, resulting in fewer parameters for HGLT overall. The number of groups in each GLT layer and the dimension of the output labels in HGLT can be represented as:

[0099]

[0100]

[0101] Where l is the level of the current group linear transformation in the hierarchical group linear transformation, and g l This refers to the number of groups in the linear transformation of the l-th layer, d l This refers to the output dimension of the label, where L is the total number of GLT layers in HGLT, and d i It is the initial dimension, d m It is the maximum dimension of the marker, d L-l+1 It is the label dimension of the GLT output in the second half of HGLT, and it is symmetrical to the label dimension of the GLT output in the first half of HGLT.

[0102] In each layer of HGLT, in addition to group linear transformations, there is also a process of mixed input connection. For example... Figure 7 As shown in the mixer diagram, the mixed input features are obtained by equally dividing the output of the previous layer's group linear transformation and the original input along the dimensional direction according to the number of groups in the current group linear transformation. Then, the equally divided features are concatenated group by group, and finally, all groups are concatenated as the input to the current group linear transformation. Mixed input concatenation effectively promotes the reuse of input features, supports the expansion of HGLT network depth, and avoids network degradation and gradient vanishing problems. Here, the hierarchical group linear transformation (HGLT) is defined as:

[0103]

[0104] Where H(·) represents the hierarchical group linear transformation, H(T) in ) l G(·) represents the output of the group linear transformation of the l-th layer, and M(·) represents the mixed input connection. l The number of groups representing the linear transformation of the l-th layer. Let l be the weight set of the linear transformation of the l-th layer. Let w be the bias set of the linear transformation of the l-th layer group. l Medium weight and b l The offset number is consistent with the number of linear transformation groups in this layer group.

[0105] Finally, the lightweight semantic tagging obtained through HGLT can be represented by the following formula:

[0106] Tin′=H(Tin)LW r (7)

[0107] Where Tin′ represents a lightweight semantic tag, H(T) in L∈R N×d It is the output of the last level of group linear transformation (GLT) in hierarchical group linear transformation (HGLT), T in W is an abstract semantic tag with global connections. r ∈R d×do It is a dimension reduction linear transformation.

[0108] HGLT learns deeper and broader representations with fewer parameters through sparse connections using group linear transformation and an expansion-shrinking grouping strategy, effectively expanding the network's depth and width while avoiding the performance degradation that may result from label dimensionality reduction. Hierarchical group linear transformation reduces the label dimension embedded in the MSA to half the dimension of the labels generated by the labeler, significantly reducing the computational complexity and number of parameters in the Transformer's MSA for capturing global label relationships, thus alleviating the pressure on the Transformer for training with small sample sizes in sketch face recognition.

[0109] In some embodiments, after obtaining the lightweight semantic tags, a standard multi-head attention mechanism is needed to model the lightweight tags. The specific process can be as follows:

[0110] First, the multi-head attention mechanism uses different linear projections to apply lightweight semantic tags T i ′ n Linear projection onto h groups of queries respectively key Sum h is the number of heads, d q d k d v =d o / h. Then, Q in space i K i Parallel execution of scaled dot product self-attention and V i Multiplying them together yields the modeling result for the head.

[0111] Furthermore, the modeling results of all heads are connected via W o The projection is returned to dimension d to complete the skip connection. Finally, the multi-head attention calculation results (MultiHead(T)) are presented. i ′ n )) and T inAfter performing layer normalization, the modeling results of the relationships between the labels are obtained. The mechanism of MSA can be formally represented by the following formula:

[0112] Q i ,K i V i =T i ′ n W i q ,T i ′ n W i k ,T i ′ n W i v ,i∈{1,2,…,h},

[0113]

[0114] MultiHead(T i ′ n = Concat(head1,…,head) h W o ,

[0115] T o ′ ut =Layer Normalization(T in +MultiHead(T i ′ n ))(8)

[0116] Among them, Q i For query, K i As a key, V i For the value, W i q W i k W i v Let T be the projection weight matrix. i ′ n For lightweight semantic markup, head i This is the result of single-head modeling of a multi-head attention mechanism, SOFTMAX. N It is a normalization function, d k It is K i The dimension of MultiHead(T) i ′ n The result is the calculation result of multi-head attention, where Concat represents the number of connections, h is the number of heads, and W is the number of heads. oIt is the projection weight matrix used to adjust the labeled dimension, T o ′ ut It is the modeling result of lightweight labeling, while Layer Normalization is layer normalization.

[0117] In some embodiments, the above projection matrix The result of single-head modeling i The results of multi-head attention calculation can be obtained by stitching and then projecting the results. MSA modeling results T o ′ ut After FFN mapping, residual connections, and layer normalization, the encoded output of the Transformer is obtained. Formally, it can be represented by the following formula:

[0118] FFN(T o ′ ut )=σ(T o ′ ut F1)F2,

[0119] T out =Layer Normalization(T o ′ ut +FFN(T o ′ ut ))(9)

[0120] Where FFN is a feedforward network, T o ′ ut For the modeling results, T out For the encoded output, F1 and F2 are weight matrices, σ is the ReLU activation function, and Layer Normalization is the layer normalization.

[0121] Optimizing the bottleneck structure of FFN can improve the computational cost of the self-attention layer and thus enhance Transformer performance. In the Transformer, self-attention plays a crucial role in label relation modeling, with a computational complexity of O(2N). 2 d), while FFN is used to improve the nonlinear expressive power of the model, and its computational complexity is O(8Nd). 2)。For image and video tasks, N is generally very large, so the computational cost of MSA is more advantageous than that of FNN. However, in tasks such as text translation, the number of tokens N in the input sequence is usually small, and then FFN will consume more computations. In this specification, the embedding method of using a tokenizer to generate semantic tokens greatly reduces the size of N (N << d). At the same time, HGLT is used to reduce the dimension of the tokens embedded into the Transformer, and the complexity of the self-attention layer in the Transformer is consequently reduced to o(N 2 d). Based on the small number of tokens and the halving of the self-attention computational complexity in this specification, if the bottleneck design of the FFN in the original Transformer is adopted, it will cause a large amount of computational cost at the FFN compared to MSA. Considering that N << d in the scenario of sketch face semantic token modeling, in order to balance the computational resource consumption between MSA and FFN, this specification adjusts the dimensionality transformation of the FFN in the Transformer. The original FFN is as Figure 5 shown in the feed-forward network, where the token dimension is first expanded by 4 times and then reduced by 4 times. The optimized FFN structure is as Figure 6 shown in the feed-forward network, where the token dimension is first reduced to d / 4 and then expanded back to d.

[0122] On the basis of constructing a lightweight Transformer module by introducing HGLT and adjusting the structure of the FFN, a lightweight semantic Transformer architecture is constructed by making the lightweight Transformer parameters in two LWVT modules consistent based on the idea of parameter sharing. Cross-layer parameter sharing in the Transformer can effectively improve the efficiency of parameters. A Transformer model using parameter sharing can achieve an effect comparable to that of the basic model with fewer parameters, thereby effectively preventing the model from overfitting.

[0123] In some embodiments, after completing the modeling of semantic tokens, it can be passed through an inverse transformer (P rev ), and the modeled semantic tokens are reshaped into a feature map with more detailed features, that is, it can be reshaped into a pixel-level feature map, which can be used as the above-mentioned target refinement feature map with semantic information. By obtaining the target refinement feature map with semantic information, the accuracy of sketch face recognition can be improved. Among them, the specific execution process of the inverse transformer (P rev ) can be as follows: First, the feature image X in and the encoded output result T out are input into the inverse transformer (P rev ), and the feature map information X o ′ ut is obtained; then, the obtained feature map information X o ′ utWith feature image X in By fusing these features, we can obtain a refined pixel-level feature image or a refined target feature map X with semantic information. out It can be expressed by the following formula:

[0124]

[0125] X out =X in +X o ′ ut

[0126] Among them, X o ′ ut It is feature map information, P rev It is an inverse transformer, X in It is the input feature map (i.e., it can be the feature image of the training sample). T out It is the encoded output, softmax. N (.) is the normalized function for generating spatial attention groups, W q W is the learnable weight used to calculate the labeled sequence query. k W is a learnable weight used to calculate the key. v These are the learnable weights used for the values, where d is the number of channels in the feature map, and X is... out It is the output feature map (i.e., a refined feature map of the target with semantic information).

[0127] Among them, query X in W q ∈R HW×d It contains query information that requires features from semantic tags, key T out W k ∈R N×d It contains key information for N tags. X in W q and T out W k Perform scaling dot product attention to obtain the attention matrix. Then the attention matrix and encoded information T out W v ∈R N×d Multiplication generates pixel-level feature maps X out Finally, X in Skip connections can prevent network degradation and enable the model to converge faster.

[0128] In some embodiments, to train a better-performing lightweight semantic Transformer model, this specification employs a meta-learning training strategy for small samples. The meta-learning training strategy is similar to that described above and will not be repeated here. In the meta-learning training task, randomly selected data is highly likely to produce new classes, which can significantly enhance the model's generalization ability. It is generally considered that difficult-to-distinguish sample pairs are more effective for model training, but in the randomly selected K pairs of sample data, most of the photo sketches may match very well. In this case, although the cross-entropy loss of these easily matched data pairs is small, their cumulative loss may approach or even exceed that of the difficult-to-match data pairs, which is detrimental to the training of the sketch face recognition model.

[0129] Furthermore, in order to avoid the situation where a large number of easily matched photo sketch data pairs appear during random selection, which would lead to a deterioration in the model training effect, the lightweight semantic Transformer model constructed in this specification also includes a domain adaptation module. The domain adaptation module needs to calculate the domain adaptation focus loss. It can add a small weight to the loss of easily matched sample data pairs and a relatively large weight to the loss of difficult-to-match sample data pairs, thereby improving the training effect of the model.

[0130] In some embodiments, in each meta-task batch, the optical face photographs or sketch images in the query set are compared with the sketch image support set S. t Or, optical face photos support image collection. t The loss for domain embedding alignment of images can be represented by the cross-entropy loss on the negative Euclidean metric between photo features and corresponding sketch image features. The photo domain alignment embedding loss can be expressed by the following formula:

[0131]

[0132]

[0133] Wherein, P(s) k / p i ) represents a given p using a logistic regression function (softmax function) over the negative Euclidean distance between cross-domain features. i and s k The probability that the pairs (i = 1, 2, ..., k) are consistent is p i Given an optical image (which could be a refined feature map of an optical face photograph), s k It is used to predict all sketch images for a given optical image label, ||·|| represents the Euclidean distance metric between extracted cross-domain features, f(p i / w) is a feature vector extracted from a given optical image (which can be a feature vector extracted from a refined feature map of an optical face photograph), f(s) k / w) is a feature vector extracted from any sketch image, f(s) j / w) is also a feature vector extracted from the sketch image, P(p i / s i L is the probability of predicting the label of a given optical image as the label of its corresponding sketch image. ps It is the photo domain alignment embedding loss.

[0134] The sketch domain alignment embedding loss can be expressed by the following formula:

[0135]

[0136]

[0137] Wherein, P(p k / s i ) represents a given s using a logistic regression function (softmax function) over the negative Euclidean distance between cross-domain features. i and p k The probability s of the same (i = 1, 2, ..., k) i Given a sketch image (which may be a refined feature map of the sketch image), p k It is used to predict all optical images for a given sketch image label, ||·|| represents the Euclidean distance metric between extracted cross-domain features, f(s) i f(p) is a feature vector extracted from a given sketch image (which can be a feature vector extracted from a refined feature map of the sketch image), f(p) k f(p) is a feature vector extracted from any optical image. j / w) is also a feature vector extracted from the optical image, P(p i / s i L is the probability of predicting the label of a given sketch image as the label of its corresponding optical image. sp It is the sketch domain alignment embedding loss.

[0138] In some embodiments, feature vectors can be extracted from the refined feature map of the optical face photograph, and feature vectors can be extracted from the corresponding sketch image of the optical face photograph. Then, based on the extracted feature vectors from the refined feature map of the optical face photograph and the extracted feature vectors from the corresponding sketch image of the optical face photograph, the probability that the optical face photograph is predicted as the corresponding sketch image is determined. The calculation process can be referred to as the following formula:

[0139]

[0140] Then, based on the probability that an optical face photograph is predicted as a corresponding sketch image, the photograph domain alignment embedding loss is determined. The calculation process can be referred to as the following formula:

[0141]

[0142] In some embodiments, feature vectors can be extracted from the refined feature map of the sketch image and feature vectors can be extracted from the optical face photograph. Then, based on the feature vectors extracted from the refined feature map of the sketch image and the feature vectors extracted from the optical face photograph, the probability that the sketch image is predicted to be the corresponding optical face photograph can be determined. The calculation process can be referred to the following formula:

[0143]

[0144] Then, based on the probability that the sketch image is predicted to be the corresponding optical face photograph, the sketch domain alignment embedding loss is determined. The calculation process can be referred to as the following formula:

[0145]

[0146] In some embodiments, the target thinning feature map with semantic information obtained by the inverse transform can be reshaped by tensor to obtain the target feature vector. For example, the thinning feature map of an optical face photo can be reshaped by tensor to obtain the feature image corresponding to the thinning feature map of the optical face photo, and the thinning feature map of a sketch image can be reshaped by tensor to obtain the feature image corresponding to the thinning feature map of the sketch image. Finally, the target feature vector is projected into a common subspace through the domain adaptation module, and the pair of feature vectors generate a domain adaptation focus loss by measuring the distance, which can be used as the domain alignment embedding loss.

[0147] In some embodiments, after obtaining the photo domain alignment embedding loss and the pixel domain alignment embedding loss, the domain alignment embedding loss L can be adjusted. ps and L sp Multiply by the weighting factor (1-P(s)) i / p i )) γ or (1-P(p) i / s i )) γ To achieve the effect of adding different weights to the loss of samples with different recognition difficulties, the L with weighting factors can be used. ps and L sp The summation yields the target domain adaptation focus loss, which can be expressed by the following formula:

[0148]

[0149] Among them, L focal Domain-adaptive focus loss, where P represents the matching rate of data with the same label, and γ is an adjustable focusing parameter that controls the change in added weights. Indicates the photo domain alignment embedding loss. P(s) represents the sketch domain alignment embedding loss. i / p i P(p) is the probability of predicting the label of a given optical image as the label of its corresponding sketch image. i / s i ) is the probability of predicting the label of a given sketch image as the label of the corresponding optical image.

[0150] By obtaining the target domain adaptation focus loss, the initial lightweight semantic Transformer model can be further optimized. When the target domain adaptation focus loss achieves the effect of minimizing intra-class distance and maximizing inter-class distance, the final optimized target lightweight semantic Transformer model can be obtained. Obtaining the target lightweight semantic Transformer model can improve the recognition accuracy of sketched faces, while alleviating the contradiction between the high complexity of Transformer and the small sample size problem of sketched face recognition.

[0151] In some embodiments, the recognition accuracy of the aforementioned lightweight semantic Transformer model can be verified on a sketch face dataset, wherein the sketch face dataset can be the UoM-SGFSv2 or CUGSF dataset, and the model can be trained and tested according to the following parameters, such as:

[0152] First, the batch sample pairs (represented by K) in each meta-learning task can be set to 80; the number of labels generated by the labeler in the face sketch dataset (represented by N) can be set to 16; and the number of heads in the MSA (represented by h) can be set to 8. A deep learning library (PyTorch) can be used to build the model network framework, and the GPU memory size used in the experiment can be 10.9GB. The initialization parameters of the convolutional network in the lightweight semantic labeling Transformer can be the parameters of the first four stages of a ResNet18 pre-trained on ImageNet. The last two LMVT modules and the final cross-domain alignment network can be trained from scratch after initialization. The optimizer (Adamw) can be used to iteratively update the model parameters, and the learning rate can be set to 0.00015, (β1,β2,weight_decay)=(0.9,0.999,0.02). The UoM-SGFSv2 dataset mentioned above can be trained for a total of 80 epochs (one epoch refers to training once using all the samples in the training set), while the CUGSF dataset mentioned above can be trained for 150 epochs, with each epoch including a 100-dimensional learning task.

[0153] Secondly, the two datasets mentioned above (UoM-SGFSv2 and CUGSF) can be preprocessed. Multi-task Cascaded Convolutional Neural Networks (MTCNN) can be used for face detection and alignment, preserving facial key point information that is beneficial for recognition. During the training and testing phases, the images can be uniformly resized to 256×256, and various enhancement techniques are employed, including image warping, padding, random cropping, and horizontal flipping.

[0154] During the training phase, the feature extraction network in this paper can be represented by f(. / w) parameterized with model weights w, given the meta-learning task K for optical face photos p in the samples. i Or sketch images i The model embeds it into the feature f(p) i / w) or f(s) i / w), by continuously reducing f(p) extracted from the query set in each meta-task i / w) or f(s) i / w) and the cross-domain features f(s) with the same label extracted from the support set. i / w) or f(p) i The domain adaptation distance between / w) is used to reduce modal differences between sketched images and optical face photographs, resulting in a domain adaptation focus loss L. focalThe feature extraction network f(. / w) is optimized by updating the model parameters through backpropagation. After training, the converged model is saved as a pth file.

[0155] During the testing phase, the model can be loaded, and the sketch and photograph images to be recognized can be input. The best matching result between the sketch image and the optical face photograph, i.e., the result with the highest similarity, is determined as the target matching result, completing the screening and recognition of sketch faces. Through the above experiments, the recognition rate on the synthetic sketch face dataset UoM-SGFSv2 was improved by 7.6%, and the recognition rate on the viewed sketch face dataset CUXSF reached 92.59%. Therefore, good recognition results were achieved on both the viewed hand-drawn sketch face dataset CUXSF and the synthetic sketch face dataset UoM-SGFS. The above method has good generalization ability and can be applied to the recognition of different types of sketch face datasets or other cross-domain data.

[0156] The above method will be described below with reference to a specific embodiment. However, it is worth noting that this specific embodiment is only for better illustration of this application and does not constitute an improper limitation of this application.

[0157] In the specific implementation process, firstly, training samples are acquired, and feature images of the training samples are extracted using a convolutional network. The training samples include multiple sets of optical face photos and multiple sets of corresponding sketch images. Secondly, the feature images are input into a pre-constructed initial lightweight semantic Transformer model. Abstract semantic tags are obtained using the filter tagger in the initial lightweight semantic Transformer model, and a second-layer abstract semantic tag is obtained using the loop tagger in the initial lightweight semantic Transformer model. Next, lightweight semantic tags are determined based on the abstract semantic tags and the second-layer abstract semantic tags. Lightweight tag modeling is performed on the lightweight semantic tags to obtain the lightweight tag modeling result. The encoding output result of the initial lightweight semantic Transformer model is determined based on the lightweight tag modeling result. Feature image information is determined based on the feature images and the encoding output result. The feature image information and the feature images are fused to obtain a target refined feature map with semantic information. Furthermore, based on the refined feature map of the optical face photograph, a photographic domain alignment embedding loss is determined; based on the refined feature map of the sketch image, a sketch domain alignment embedding loss is determined; and based on the photographic domain alignment embedding loss and the sketch domain alignment embedding loss, a target domain adaptation focus loss is determined. Finally, the sketch image and the optical face photograph to be identified are obtained, and input into the target lightweight semantic Transformer model. The most similar result between the sketch image and the optical face photograph is used as the target matching result, and the target matching result is used as the final result of sketch face recognition.

[0158] Although this specification provides the following examples or appendices Figure 8 The methods, steps, or apparatus structures shown may include more or fewer combined operational steps or module units based on conventional or non-inventive methods. In steps or structures where there is no logically necessary causal relationship, the execution order of these steps or the module structure of the apparatus is not limited to the execution order or module structure shown in the embodiments or drawings of this specification. When the methods or module structures described are applied in actual devices, servers, or terminal products, they can be executed sequentially or in parallel according to the methods or module structures shown in the embodiments or drawings (e.g., in parallel processor or multi-threaded processing environments, or even distributed processing or server cluster implementation environments).

[0159] Based on the above-described sketch face recognition method based on a lightweight semantic Transformer model, this specification also proposes embodiments of a sketch face recognition device based on a lightweight semantic Transformer model. For example... Figure 8As shown, the device may specifically include the following modules:

[0160] The acquisition module 801 is used to acquire feature images of training samples, wherein the training samples include multiple sets of optical face photos and multiple sets of sketch images corresponding to the optical face photos;

[0161] The semantic tagging module 802 is used to input the feature image into a pre-constructed initial lightweight semantic Transformer model to obtain the first layer of abstract semantic tags and the second layer of abstract semantic tags;

[0162] The refinement module 803 is used to establish a global connection between the first-layer abstract semantic marker and the second-layer abstract semantic marker to obtain an abstract semantic marker with global connection; and to obtain a target refinement feature map with semantic information based on the feature image and the abstract semantic marker with global connection; wherein the target refinement feature map with semantic information includes the refinement feature map of the optical face photograph and the refinement feature map of the sketch image.

[0163] The model optimization module 804 is used to determine the target domain adaptation focus loss based on the refined feature map of the optical face photo and the refined feature map of the sketch image; and to optimize the initial lightweight semantic Transformer model based on the target domain adaptation focus loss to obtain the target lightweight semantic Transformer model.

[0164] The recognition module 805 is used to acquire the sketch image and optical face photo to be recognized, input the sketch image and optical face photo to be recognized into the target lightweight semantic Transformer model, obtain the target matching result of the sketch image and optical face photo to be recognized, and use the target matching result as the sketch face recognition result.

[0165] In some embodiments, the refinement module 803 can be specifically used to perform a hierarchical group linear transformation on the abstract semantic tags with global relationships to obtain lightweight semantic tags; to perform lightweight tag modeling on the lightweight semantic tags to obtain the modeling result of the lightweight tags; to determine the encoding output result of the initial lightweight semantic Transformer model based on the modeling result of the lightweight tags; and to obtain a target refined feature map with semantic information based on the feature image and the encoding output result.

[0166] In some embodiments, the refinement module 803 described above can be specifically used to determine lightweight semantic tags according to the following formula:

[0167]

[0168] Tin′=H(Tin)LWr

[0169] Where H(·) represents the hierarchical group linear transformation, H(T) in ) l G(·) represents the output of the group linear transformation of the l-th layer, and M(·) represents the mixed input connection. l The number of groups representing the linear transformation of the l-th layer. Let l be the weight set of the linear transformation of the l-th layer. Let H(T) be the bias set of the linear transformation of the l-th layer group, Tin′ represent the lightweight semantic tag, and H(T) be the bias set of the linear transformation of the l-th layer group. in L∈R N×d It is the output of the last level of the group linear transformation in the group linear transformation, T in W is an abstract semantic tag with global connections. r ∈R d×do It is a dimension reduction linear transformation.

[0170] In some embodiments, the refinement module 803 described above can be used to perform lightweight marker modeling according to the following formula to obtain the lightweight marker modeling result:

[0171] Q i K i V i =T i ′ n W i q T i ′ n W i k ,T i ′ n W i v , i∈{1,2,…,h},

[0172]

[0173] MultiHead(T i ′ n = Concat(head1, ..., head) h W o ,

[0174] T o ′ ut =Layer Normalization(T in +MultiHead(T i ′ n ))

[0175] Among them, Q i For query, Ki As a key, V i For the value, W i q W i k W i v Let T be the projection weight matrix. i ′ n For lightweight semantic markup, head i This is the result of single-head modeling of a multi-head attention mechanism, SOFTMAX. N It is a normalization function, d k It is K i The dimension of MultiHead(T) i ′ n The result is the calculation result of multi-head attention, where Concat represents the number of connections, h is the number of heads, and W is the number of heads. o It is the projection weight matrix used to adjust the labeled dimension, T o ′ ut It is the modeling result of lightweight labeling, while Layer Normalization is layer normalization.

[0176] In some embodiments, the refinement module 803 described above can be specifically used to determine the encoding output of the initial lightweight semantic Transformer model according to the following formula:

[0177] FFN(T o ′ ut )=σ(T o ′ ut F1)F2,

[0178] T out =Layer Normalization(T o ′ ut +FFN(T o ′ ut ))

[0179] Where FFN is a feedforward network, T o ′ ut For the modeling results, T out For the encoded output, F1 and F2 are weight matrices, σ is the ReLU activation function, and Layer Normalization is the layer normalization.

[0180] In some embodiments, the refinement module 803 can be specifically used to determine feature image information based on the feature image and the encoding output result; and to fuse the feature image information and the feature image to obtain a target refined feature map with semantic information.

[0181] In some embodiments, the model optimization module 804 described above can be specifically used to determine the photo domain alignment embedding loss based on the refined feature map of the optical face photo; determine the sketch domain alignment embedding loss based on the refined feature map of the sketch image; and determine the target domain adaptation focus loss based on the photo domain alignment embedding loss and the sketch domain alignment embedding loss.

[0182] It should be noted that the units, devices, or modules described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. For ease of description, the above devices are described by dividing them into various modules according to their functions. Of course, in implementing this specification, the functions of each module can be implemented in one or more software and / or hardware, or the module that implements the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection between the devices or units shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0183] As can be seen from the above, the sketch face recognition device based on the lightweight semantic Transformer model provided in the embodiments of this specification, on the one hand, by introducing Hierarchical Group Linear Transformation (HGLT) to balance the computational cost of Multi-Head Self-Attention (MSA) and Feedforward Network (FFN), and proposing the LWVT module, can alleviate the contradiction between the high complexity of Transformer and the small sample size problem of sketch face recognition. On the other hand, by proposing the lightweight semantic Transformer network, semantic features can be effectively extracted and simulated, improving the accuracy of sketch face recognition. Finally, by training the lightweight semantic Transformer network through a meta-learning strategy, a domain adaptation focus loss is proposed, which can reduce the pattern difference between sketches and photographs. The training effect of the hard sample highlighting loss in the meta-task samples is enhanced.

[0184] This specification also provides an electronic device, including a processor and a memory for storing processor-executable instructions. Specifically, the processor can perform the following steps according to the instructions: acquiring feature images of training samples, wherein the training samples include multiple sets of optical face photographs and multiple sets of sketch images corresponding to the optical face photographs; inputting the feature images into a pre-constructed initial lightweight semantic Transformer model to obtain a first-layer abstract semantic label and a second-layer abstract semantic label; establishing a global connection between the first-layer abstract semantic label and the second-layer abstract semantic label to obtain an abstract semantic label with global connection; and obtaining a target refined feature map with semantic information based on the feature images and the abstract semantic label with global connection. The semantically information-rich target refinement feature map includes the refinement feature map of the optical face photograph and the refinement feature map of the sketch image. Based on the refinement feature map of the optical face photograph and the refinement feature map of the sketch image, a target domain adaptation focus loss is determined. The initial lightweight semantic Transformer model is optimized based on the target domain adaptation focus loss to obtain a target lightweight semantic Transformer model. The sketch image and optical face photograph to be recognized are acquired, and the sketch image and optical face photograph to be recognized are input into the target lightweight semantic Transformer model to obtain the target matching result of the sketch image and optical face photograph to be recognized. The target matching result is used as the sketch face recognition result.

[0185] To execute the above instructions more accurately, please refer to... Figure 9 As shown in the embodiments of this specification, another specific electronic device is also provided, wherein the electronic device includes a network communication port 901, a processor 902, and a memory 903. The above structures are connected by internal cables so that the various structures can perform specific data interaction.

[0186] Specifically, the network communication port 901 can be used to acquire feature images of training samples, wherein the training samples include multiple sets of optical face photos and multiple sets of sketch images corresponding to the optical face photos.

[0187] The processor 902 can specifically be used to input the feature image into a pre-constructed initial lightweight semantic Transformer model to obtain a first-layer abstract semantic label and a second-layer abstract semantic label; establish a global connection between the first-layer abstract semantic label and the second-layer abstract semantic label to obtain an abstract semantic label with global connection; obtain a target refinement feature map with semantic information based on the feature image and the abstract semantic label with global connection; wherein the target refinement feature map with semantic information includes a refinement feature map of the optical face photograph and a refinement feature map of the sketch image; determine the target domain adaptation focus loss based on the refinement feature map of the optical face photograph and the refinement feature map of the sketch image; optimize the initial lightweight semantic Transformer model based on the target domain adaptation focus loss to obtain a target lightweight semantic Transformer model; acquire the sketch image and the optical face photograph to be recognized, input the sketch image and the optical face photograph to be recognized into the target lightweight semantic Transformer model to obtain the target matching result of the sketch image and the optical face photograph to be recognized, and use the target matching result as the sketch face recognition result.

[0188] The memory 903 can be used to store the corresponding instruction program.

[0189] In this embodiment, the network communication port 901 can be a virtual port bound to different communication protocols, thereby enabling the sending or receiving of different data. For example, the network communication port can be a port responsible for web data communication, a port responsible for FTP data communication, or a port responsible for email data communication. Furthermore, the network communication port can also be a physical communication interface or communication chip. For example, it can be a wireless mobile network communication chip, such as GSM or CDMA; it can also be a Wi-Fi chip; or it can be a Bluetooth chip.

[0190] In this embodiment, the processor 902 can be implemented in any suitable manner. For example, the processor can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers, etc. This specification is not limiting.

[0191] In this embodiment, the memory 903 may include multiple layers. In a digital system, anything that can store binary data can be a memory. In an integrated circuit, a circuit with storage function but no physical form is also called a memory, such as RAM, FIFO, etc. In a system, a storage device with a physical form is also called a memory, such as a memory stick, TF card, etc.

[0192] This specification also provides a computer storage medium for the sketch face recognition method based on the lightweight semantic Transformer model described above. The computer storage medium stores computer program instructions that, when executed, perform the following: acquiring feature images of training samples, wherein the training samples include multiple sets of optical face photographs and sketch images corresponding to the optical face photographs; inputting the feature images into a pre-constructed initial lightweight semantic Transformer model to obtain a first-layer abstract semantic label and a second-layer abstract semantic label; establishing a global connection between the first-layer abstract semantic label and the second-layer abstract semantic label to obtain an abstract semantic label with a global connection; and obtaining a face recognition method with a specific feature image based on the feature image and the globally connected abstract semantic label. A semantically information-rich target refinement feature map is generated, comprising a refinement feature map of the optical face photograph and a refinement feature map of the sketch image. A target domain adaptation focus loss is determined based on the refinement feature maps of the optical face photograph and the sketch image. The initial lightweight semantic Transformer model is optimized based on the target domain adaptation focus loss to obtain a target lightweight semantic Transformer model. A sketch image and an optical face photograph to be recognized are acquired, and the sketch image and optical face photograph are input into the target lightweight semantic Transformer model to obtain a target matching result for the sketch image and the optical face photograph. The target matching result is then used as the sketch face recognition result.

[0193] In this embodiment, the storage medium includes, but is not limited to, Random Access Memory (RAM), Read-Only Memory (ROM), cache, hard disk drive (HDD), or memory card. The memory can be used to store computer program instructions. The network communication unit can be an interface configured according to standards specified in the communication protocol for network connection communication.

[0194] While this specification provides the steps of operation for the methods described in the embodiments or flowcharts, more or fewer steps may be included based on conventional or non-inventive means. The order of steps listed in the embodiments is merely one possible order of execution among many steps and does not represent the only possible order. In actual device or client product execution, the methods shown in the embodiments or drawings may be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment, or even a distributed data processing environment). The terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitations, the presence of other identical or equivalent elements in a process, method, product, or apparatus that includes said elements is not excluded. The terms "first," "second," etc., are used to denote names and do not indicate any particular order.

[0195] Those skilled in the art will also know that, besides implementing the controller using purely computer-readable program code, the same functions can be achieved by logically programming the method steps, making the controller function as logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers (PLCs), and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the devices within it used to implement various functions can also be considered structures within that hardware component. Alternatively, the devices used to implement various functions can be considered as both software modules implementing the method and structures within a hardware component.

[0196] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0197] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this specification can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of this specification can essentially be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, mobile terminal, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments of this specification.

[0198] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. This specification can be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable electronic devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices, etc.

[0199] Although this specification has been described by way of examples, those skilled in the art will recognize that many variations of this specification are possible without departing from its spirit, and it is intended that the appended claims cover such variations without departing from the spirit of this specification.

Claims

1. A sketch-based face recognition method based on a lightweight semantic Transformer model, characterized in that, include: Obtain feature images of training samples, wherein the training samples include multiple sets of optical face photos and multiple sets of sketch images corresponding to the optical face photos; The feature image is input into a pre-constructed initial lightweight semantic Transformer model to obtain the first layer of abstract semantic tags and the second layer of abstract semantic tags; Establish a global connection between the first-level abstract semantic tag and the second-level abstract semantic tag to obtain an abstract semantic tag with a global connection; Based on the feature image and the abstract semantic tag with global connection, a target refinement feature map with semantic information is obtained; wherein, the target refinement feature map with semantic information includes the refinement feature map of the optical face photograph and the refinement feature map of the sketch image; Based on the refined feature map of the optical face photograph and the refined feature map of the sketch image, the target domain adaptation focus loss is determined; The initial lightweight semantic Transformer model is optimized based on the target domain adaptation focus loss to obtain the target lightweight semantic Transformer model; Obtain the sketch image and optical face photo to be identified, input the sketch image and optical face photo to be identified into the target lightweight semantic Transformer model, obtain the target matching result of the sketch image and optical face photo to be identified, and use the target matching result as the sketch face recognition result.

2. The method according to claim 1, characterized in that, Based on the feature image and the globally related abstract semantic markers, a refined feature map of the target with semantic information is obtained, including: The abstract semantic tags with global relationships are subjected to hierarchical group linear transformation to obtain lightweight semantic tags; The lightweight semantic tags are modeled using lightweight tags to obtain the modeling results of lightweight tags; Based on the modeling results of the lightweight marker, determine the encoding output of the initial lightweight semantic Transformer model; Based on the feature image and the encoding output, a target refinement feature map with semantic information is obtained.

3. The method according to claim 2, characterized in that, The abstract semantic tags with global relationships are subjected to a hierarchical group linear transformation to obtain lightweight semantic tags, including: The lightweight semantic tags are obtained by performing a hierarchical group linear transformation according to the following formula: Tin′=H(Tin)LW r Where H(·) represents the hierarchical group linear transformation, H(T) in ) l G(·) represents the output of the group linear transformation of the l-th layer, and M(·) represents the mixed input connection. l The number of groups representing the linear transformation of the l-th layer. Let l be the weight set of the linear transformation of the l-th layer. Let H(T) be the bias set of the linear transformation of the l-th layer group, Tin′ represent the lightweight semantic tag, and H(T) be the bias set of the linear transformation of the l-th layer group. in L∈R N×d It is the output of the last level of the group linear transformation in the group linear transformation, T in W is an abstract semantic tag with global connections. r ∈R d×do It is a dimension reduction linear transformation.

4. The method according to claim 2, characterized in that, The lightweight semantic tags are modeled using lightweight tags to obtain the modeling results of the lightweight tags, including: Lightweight tagging modeling is performed according to the following formula to obtain the lightweight tagging modeling results: MultiHead(T i ′ n )=Concat(head1,…,head h )W o , T o ′ ut =Layer Normalization(T in +MultiHead(T i ′ n )) Among them, Q i For query, K i As a key, V i For value, Let T be the projection weight matrix. i ′ n For lightweight semantic markup, head i This is the result of single-head modeling of a multi-head attention mechanism, SOFTMAX. N It is a normalization function, d k It is K i The dimension of MultiHead(T) i ′ n The result is the calculation result of multi-head attention, where Concat represents the number of connections, h is the number of heads, and W is the number of heads. o It is the projection weight matrix used to adjust the labeled dimension, T o ′ ut It is the modeling result of lightweight labeling, while Layer Normalization is layer normalization.

5. The method according to claim 2, characterized in that, Based on the modeling results of the lightweight marker, the encoding output of the initial lightweight semantic Transformer model is determined, including: The encoding output of the initial lightweight semantic Transformer model is determined according to the following formula: FFN(T o ′ ut )=σ(T o ′ ut F1)F2, T out =Layer Normalization(T o ′ ut +FFN(T o ′ ut )) Where FFN is a feedforward network, T o ′ ut For the modeling results, T out For the encoded output, F1 and F2 are weight matrices, σ is the ReLU activation function, and Layer Normalization is the layer normalization.

6. The method according to claim 2, characterized in that, Based on the feature image and the encoding output, a target refinement feature map with semantic information is obtained, including: Based on the feature image and the encoded output result, the feature image information is determined; The feature image information and the feature image are fused to obtain a target refinement feature map with semantic information.

7. The method according to claim 1, characterized in that, Based on the refined feature map of the optical face photograph and the refined feature map of the sketch image, the target domain adaptation focus loss is determined, including: Based on the refined feature map of the optical face photograph, determine the photograph domain alignment embedding loss; Based on the refined feature map of the sketch image, determine the sketch domain alignment embedding loss; The target domain adaptation focus loss is determined based on the photo domain alignment embedding loss and the sketch domain alignment embedding loss.

8. A sketch-based face recognition device based on a lightweight semantic Transformer model, characterized in that, include: The acquisition module is used to acquire feature images of training samples, wherein the training samples include multiple sets of optical face photos and multiple sets of sketch images corresponding to the optical face photos; The semantic tagging module is used to input the feature image into a pre-constructed initial lightweight semantic Transformer model to obtain the first layer of abstract semantic tags and the second layer of abstract semantic tags; The refinement module is used to establish a global connection between the first-layer abstract semantic marker and the second-layer abstract semantic marker to obtain an abstract semantic marker with global connection; based on the feature image and the abstract semantic marker with global connection, a target refinement feature map with semantic information is obtained; wherein, the target refinement feature map with semantic information includes the refinement feature map of the optical face photograph and the refinement feature map of the sketch image; The model optimization module is used to determine the target domain adaptation focus loss based on the refined feature map of the optical face photo and the refined feature map of the sketch image; and to optimize the initial lightweight semantic Transformer model based on the target domain adaptation focus loss to obtain the target lightweight semantic Transformer model. The recognition module is used to acquire the sketch image and optical face photo to be recognized, input the sketch image and optical face photo to be recognized into the target lightweight semantic Transformer model, obtain the target matching result of the sketch image and optical face photo to be recognized, and use the target matching result as the sketch face recognition result.

9. A sketch-based face recognition device based on a lightweight semantic Transformer model, characterized in that, include: It includes a processor and a memory for storing processor-executable instructions, wherein the processor, when executing the instructions, implements the steps of the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores computer instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7.

Images