Sketch-photo recognition method and system based on asymmetric decoupling and dynamic synthesis

Abstract

The application discloses a sketch-photo recognition method and system based on asymmetric decoupling and dynamic synthesis, proposes an asymmetric decoupling scheme, and decomposes photo modal features into sketch-related representation and sketch-irrelevant representation based on an automatic encoder. In order to obtain an information-symmetrical cross-modal feature embedding space, the application proposes to fuse sketch representation and sketch-irrelevant representation in the photo modal by using a knowledge transfer technology to obtain updated sketch representation which is symmetrical to the photo modal representation. Finally, the model performs metric learning on the aligned feature embedding space, thereby promoting modal-invariant feature learning. In addition, the application proposes to generate a dynamic and updateable and learnable auxiliary sketch (A-sketch) modal from the photo modal to guide the asymmetric decoupling of the photo modal. The application greatly improves the retrieval accuracy of the sketch-photo recognition task without increasing the reasoning time.

Description

Technical Field

[0001] This invention belongs to the field of image recognition and retrieval technology, and relates to a sketch-photo recognition method and system, specifically a sketch-photo recognition method and system based on asymmetric decoupling and dynamic synthesis. Background Technology

[0002] Sketch-photo recognition is an asymmetric, cross-modal matching problem where the query set consists of sketches drawn by artists or amateur painters. Sketch-photo recognition systems are widely used by law enforcement agencies ([References 1-3]) and commercial organizations ([References 4-6]). Given a target sketch image to be queried, the system can match it with photograph images of the same category. Depending on the sketching method, existing sketch-photo recognition research includes image retrieval based on freehand sketches ([Reference 3]) and pedestrian re-identification based on professional hand-drawn sketches ([Reference 5]).

[0003] The different imaging principles of sketches and photographs result in heterogeneity and information asymmetry between the two modalities, making the extraction of domain-invariant features extremely challenging. For example, sketches are highly abstract, containing only structural / outline information of the object but lacking detailed information about its appearance (such as color and texture). One intuitive approach to mitigating the impact of modal differences between sketches and photographs is to use image transformation networks to generate corresponding modal images to replace the original ones, such as sketch-to-photo or photo-to-sketch synthesis. However, due to sample diversity and domain differences, this fixed-parameter synthesis method inevitably introduces generation noise and struggles to achieve intermodal information alignment.

[0004] Other existing methods ([Reference 3]) aim to mine shared feature embedding spaces for sketch and photograph modalities through cross-modal attitude metric learning. They typically perform metric learning in the original modal representation space, with specific training and testing steps as follows:

[0005] 1. Prepare a dataset of sketch-photo images, dividing it into a training set and a test set; 2. Train a neural network by inputting the image pairs from the training set into a shared-weights neural network, including building a basic network for feature extraction of sketch and photo images, and optimizing the network using classification loss and metric loss;

[0006] 3. Sketches in the test set are considered the query set, and photographs are considered the image library. The best-performing network model during training is used to extract features from both the query set and the image library, and the retrieval accuracy of the query set is output.

[0007] However, due to information asymmetry between the two modalities, some modality-specific information (e.g., fine-grained appearance information in photographic images) increases the intra-class discrepancy between features of different modalities within the same category. Directly optimizing the original modal feature space limits the model's robustness to modality variations. Recently, Sain et al. ([Reference 2]) proposed a symmetric decoupling method that decomposes photographic and sketch features into content representations for retrieval (modality-shared) and style representations as interfering factors (modality-specific), respectively, to address the modality-style variation problem. This symmetric decoupling method typically focuses only on modeling modal content representations, which leads to the loss of modality information and limited feature discriminative power.

[0008] In summary, how to refine cross-modal differences without losing modal information, align cross-modal feature representations, and mitigate the impact of modal information asymmetry on model performance is an urgent problem to be solved in sketch-photo recognition tasks.

[0009] [Document 1] Ayan Kumar Bhunia, Pinaki Nath Chowdhury, AneeshanSain,Yongxin Yang, Tao Xiang, and Yi-Zhe Song. 2021. More Photos are All You Need: Semi-Supervised

[0010] Learning for Fine-Grained Sketch Based Image Retrieval. InProceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition. 4247–4256.

[0011] [Document 2] AneeshanSain, Ayan Kumar Bhunia, Yongxin Yang, Tao Xiang, andYi-ZheSong. 2021. Stylemeup: Towards style-agnostic sketch-based imageretrieval. InProceedings of the IEEE / CVF Conference on Computer Vision andPattern Recognition.8504–8513.

[0012] [Reference 3] Qian Yu, Jifei Song, Yi-Zhe Song, Tao Xiang, and Timothy MHospedales. 2021. Fine-Grained Instance-Level Sketch-Based Image Retrieval.International Journal of Computer Vision 129, 2 (2021), 484–500.

[0013] [Reference 4] ShaojunGui, Yu Zhu, Xiangxiang Qin, and Xiaofeng Ling. 2020.Learning multi-level domain invariant features for sketch re-identification.Neurocomputing 403 (2020), 294–303.

[0014] [Reference 5] Lu Pang, Yaowei Wang, Yi-Zhe Song, Tiejun Huang, and YonghongTian. 2018.Cross-domain adversarial feature learning for sketch re-identification. In Proceedings of the 26th ACM international conference onMultimedia. 609–617.

[0015] [Reference 6] Fan Yang, Yang Wu, Zheng Wang, Xiang Li, Sakriani Sakti, andSatoshi Nakamura. 2020. Instance-level heterogeneous domain adaptation forlimited-labeled sketch-to-photo retrieval. IEEE Transactions on Multimedia(2020). Summary of the Invention

[0016] To address the shortcomings of existing technologies, this invention provides a sketch-photo recognition (ADDS) method and system based on asymmetrical disentanglement and dynamic synthesis, which enables simultaneous modeling of modality sharing and modality-specific information, alleviates the impact of cross-modal information asymmetry on model optimization, and improves the retrieval accuracy of the sketch-photo recognition model.

[0017] The technical solution adopted by the method of the present invention is: a sketch-photo recognition method based on asymmetric decoupling and dynamic synthesis, comprising the following steps:

[0018] Step 1: Generate an auxiliary sketch image from the original photograph image;

[0019] Step 2: For the auxiliary sketch image generated in Step 1, the photo modality features are decomposed into sketch-related representations and sketch-independent representations; wherein, sketch-related representations describe the outline and structural information of the target object shared with the sketch modality, and sketch-independent representations describe the appearance details unique to the photo modality.

[0020] Step 3: For the sketch image, fuse the sketch-irrelevant representation in the photo modality obtained in Step 1 with the sketch image to obtain an updated sketch image that is symmetrical to the photo modality representation information;

[0021] Step 4: Compare and identify the updated sketch image obtained in Step 3 with the original photo image in Step 1 to obtain the identification result.

[0022] The technical solution adopted by the system of this invention is: a sketch-photo recognition system based on asymmetric decoupling and dynamic synthesis, comprising the following modules:

[0023] Module 1 is used to generate auxiliary sketch images from the original photograph images;

[0024] Module 2 is used to decompose the photo modality features into sketch-related representations and sketch-independent representations for the auxiliary sketch image generated in Module 1. The sketch-related representations describe the outline and structural information of the target object shared with the sketch modality, while the sketch-independent representations describe the appearance details unique to the photo modality.

[0025] Module 3 is used to fuse the sketch-image-independent representation in the photo modality obtained in Module 1 with the sketch image to obtain an updated sketch image that is symmetrical with the photo modality representation information.

[0026] Module 4 is used to compare and identify the updated sketch image obtained in Module 3 with the original photograph image in Module 1 to obtain the identification result.

[0027] The present invention has the following advantages:

[0028] (1) This invention addresses the impact of modal information asymmetry on model optimization in sketch-photo recognition tasks for the first time, proposing a novel asymmetric decoupling mechanism. By optimizing the aligned cross-modal feature embedding space, the model extracts more discriminative modal shared representations.

[0029] (2) This invention proposes a dynamic synthesis-assisted sketching method. Through an adaptive adjustment strategy, the generated assisted sketching modality possesses style information similar to the hand-drawn sketch modality, while retaining the same structural information as the photographic modality, thus effectively guiding the asymmetric decoupling of photographic representation.

[0030] (3) The method proposed in this invention does not incorporate any modules or learning mechanisms during the testing phase. Without increasing the computational load and inference time, the method of this invention effectively improves the retrieval accuracy of the model. Attached Figure Description

[0031] Figure 1 : A schematic diagram illustrating the dispensing principle of an embodiment of the present invention;

[0032] Figure 2 : A structural diagram of the photo characterization decoupling module in an embodiment of the present invention. Detailed Implementation

[0033] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to embodiments. It should be understood that the embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0034] The concept of this invention is based on the consideration that photographic modalities not only contain structural information corresponding to the sketch but also specific style information. The core idea is to construct cross-modal representations with information symmetry, without losing any information when learning the shared feature embedding space. Specifically, this invention proposes an asymmetric decoupling scheme that decomposes photographic modal features into sketch-related and sketch-independent representations based on an autoencoder. The sketch-related representations describe the outline and structural information of the target object shared with the sketch modality, while the sketch-independent representations describe the appearance details (such as color, texture, and other style information) unique to the photographic modality. To obtain an information-symmetric cross-modal feature embedding space, this invention proposes to use knowledge transfer techniques to fuse the sketch representation and the sketch-independent representation in the photographic modality, resulting in an updated sketch representation that is information-symmetric to the photographic modality representation. Finally, our model performs metric learning on the aligned feature embedding space, thereby promoting modality-invariant feature learning.

[0035] Furthermore, considering the difficulty in extracting sketch-related representations from photographic images due to the inability to directly obtain structural information of the target object, this invention proposes generating a dynamically updatable and learnable auxiliary sketch (A-sketch) modality from the photographic modality to guide the asymmetric decoupling of the photographic modality. The A-sketch modality contains stylistic information similar to the hand-drawn sketch modality while maintaining the same structural information as the photographic modality. This invention employs a task-driven attention mechanism to model the spatial relationship between the photographic modality representation and the A-sketch modality representation, thereby decomposing sketch-related representations from the photographic modality.

[0036] During inference, this invention uses only a modal feature extractor and does not employ an auxiliary sketch generator or asymmetric decoupling mechanism. Therefore, without increasing the test setup or inference time, this invention significantly improves the retrieval accuracy of the sketch-photo recognition task.

[0037] Please see Figure 1 The present invention provides a sketch-photo recognition method based on asymmetric decoupling and dynamic synthesis, characterized by comprising the following steps:

[0038] Step 1: Generate an auxiliary sketch image from the original photograph image;

[0039] This embodiment proposes a dynamically updated auxiliary sketch generation method, which achieves asymmetric decoupling of photograph representation under the guidance of this auxiliary sketch. Specifically, the auxiliary sketch generation method employs an existing sketch generation network and initializes it using pre-trained parameters from the Scibble dataset. Then, this sketch generation network is jointly trained with a recognition task network (including a sketch-related representation encoder, a sketch-independent representation encoder, an image fusion module, and a contrast recognition module) to achieve dynamic updates of the sketch generation network parameters. This makes the generated auxiliary sketch more aligned with the needs of the recognition task, with the following update loss function:

[0040]

[0041]

[0042] in, This represents the identity loss, which can be either cross-entropy loss or triplet loss. Indicates hyperparameters; Style binding loss; and These represent the initial A-sketch image and the refined A-sketch image, respectively. This indicates the number of pixels in the auxiliary sketch image. Denotes the Frobenius norm;

[0043] Step 2: For the auxiliary sketch image generated in Step 1, the photo modality features are decomposed into sketch-related representations and sketch-independent representations based on the autoencoder. The sketch-related representations describe the outline and structural information of the target object shared with the sketch modality, while the sketch-independent representations describe the appearance details (such as color, texture and other style information) unique to the photo modality.

[0044] Because sketches are more abstract, photographic images and sketches of the same category... Local-level features often exhibit significant differences. However, global-level features, such as the correlation between the shape of the target object and local information, show better discriminative power in sketch-photo recognition. Therefore, this embodiment utilizes the Vision Transformer (ViT) network to achieve long-distance interaction between modal local features and mine global-level modal feature representations:

[0045] ;

[0046] In the formula, This represents the additional learnable parameters used to obtain the final image representation. A linear projection layer is used to convert image patches into one-dimensional vectors. PE represents the positional encoding of the image patch.

[0047] To address the information asymmetry between photographic and sketch modalities and to learn discriminative modal-shared feature representations, this invention proposes an asymmetric decoupling mechanism. This mechanism creates a cross-modal feature embedding space with information symmetry by supplementing missing information in the sketch image, thereby guiding the model to focus on modal-shared feature representations. The mechanism comprises two steps: photographic representation decoupling and information transfer.

[0048] Photo representation decoupling aims to decompose photo features into sketch-related and sketch-independent representations based on the generated A-sketch modality. Considering that the A-sketch modality and the photo modality have the same spatial structure but different color and intensity information, this invention designs a task-driven attention mechanism that decouples photo representations by modeling the spatial relationship between the A-sketch modality and the photo modality. In this embodiment, the photo modality features are decomposed into sketch-related and sketch-independent representations based on an autoencoder and a generated auxiliary sketch modality.

[0049] The process of modal feature decomposition of a photograph is as follows:

[0050] ;

[0051] ;

[0052] ;

[0053] Among them, the mask matrix Middle elements Describes the photo modality. line characteristics With auxiliary sketching modality Column features Similarity between This indicates similarity calculation; and These represent the sketch-related and sketch-unrelated representations of the photographic modality, respectively. , , and These represent linear transformation layers with non-shared weights, used to map corresponding features to different representation subspaces. and These represent the encoder that is related to the sketch and the encoder that is not related to the sketch, respectively.

[0054] Please see Figure 2 The sketch-related representation encoder in this embodiment consists of two linear projection layers, including a first linear fully connected layer and a second linear fully connected layer; the first linear fully connected layer and the second linear fully connected layer are connected sequentially.

[0055] The first linear fully connected layer maps the photographic representation to different feature spaces; the mask matrix is ​​obtained by modeling the spatial similarity between the photograph and the auxiliary sketch representation. The matrix multiplication is performed between the matrix and the output of the first linear fully connected layer to achieve weighted correction of the photo representation, enhancing the structure-related representations in the photo features. The result is used as the input to the second linear fully connected layer, which includes a fully connected layer and a dropout layer.

[0056] Please see Figure 2 The sketch-independent representation encoder in this embodiment consists of two linear projection layers, including a first fully connected linear layer and a second fully connected linear layer. The first and second fully connected linear layers are connected sequentially.

[0057] The first linear fully connected layer maps the photographic representation to different feature spaces; the mask matrix is ​​obtained by modeling the spatial similarity between the photograph and the auxiliary sketch representation. ,Will The matrix is ​​multiplied with the output of the first linear fully connected layer to perform a weighted correction on the image representation, enhancing the structure-independent representations in the image features. The result is used as the input to the second linear fully connected layer, which consists of a fully connected layer and a dropout layer.

[0058] Step 3: For the sketch image, use the knowledge transfer method to fuse the sketch-irrelevant representation in the photo modality obtained in Step 1 with the sketch image to obtain an updated sketch image that is symmetrical with the photo modality representation information;

[0059] The main idea of ​​information transfer is to update the feature representation of sketch modality through knowledge transfer, make up for the missing information in the sketch representation, and thus obtain a cross-modal feature embedding space with information alignment.

[0060] In this embodiment, the knowledge transfer method is used to fuse the sketch-irrelevant representation in the photo modality obtained in step 1 with the sketch image;

[0061] ;

[0062] in, and These represent the sketch representations before and after the update, respectively. and These represent the mean and standard deviation, respectively, used for standardizing modal features in sketching. and These represent the representations in the photographic modality that are unrelated to the sketch. Learnable affine parameters; It represents a small quantity.

[0063] Step 4: Compare and identify the updated sketch image obtained in Step 3 with the original photo image in Step 1 to obtain the identification result.

[0064] In this embodiment, the sketch generation network and the recognition task network are pre-trained networks, and the overall loss function for network training is... It includes the following four parts:

[0065]

[0066]

[0067]

[0068]

[0069] in, The loss representing identity classification is derived from sketch modal features. Auxiliary sketch modal features and photo modal features Cross-entropy loss function The sum of ( ); Since the number of positive samples in sketch-photo recognition datasets is usually small and the intra-class differences are large, existing triplet loss functions for hard sample mining struggle to learn sufficiently small intra-class distances. To address this issue, this invention proposes a contrastive triplet loss function. The aim is to increase the distance between similar samples and increase the distance between dissimilar samples. Indicates the first The distance between a sample and its negative sample t Indicates sample The sum of distances between it and all its positive samples; Indicates sample The number of negative samples. This loss function combines the distances of all positive sample pairs. And with all negative samples By making comparisons, we can effectively enhance modal feature representation, optimize the modal shared feature embedding space, and improve the robustness of the model to cross-modal changes. This represents a decoupling spatial feature optimization function, designed to show how to shorten the distance between cross-modal aligned features. This represents the smooth L1 loss function. The first term of this function... Explicit zoom-in photograph representation and updated sketch representation The distance between them, the second item explicitly shortens the sketch representation. and decomposition characterization The distance between them. and Both represent hyperparameters.

[0070] During the testing phase, the proposed method uses only the pre-trained cross-modal feature extractor, without employing an auxiliary sketch generator or asymmetric decoupling mechanisms. Therefore, compared to the baseline network model, the proposed method effectively improves the model's retrieval accuracy while maintaining unchanged test settings and inference time.

[0071] This implementation example uses PyTorch as the deep learning framework, version 1.7.0, with CUDA version 11.2. The hardware environment for the experiment is an NVIDIA GeForce RTX 1080 graphics card and an Intel(R) Xeon(R) Gold 6240C processor.

[0072] First, an asymmetric decoupling and dynamic synthesis sketch-photo recognition network is constructed.

[0073] In the experiment, the VisionTransformer (ViT) network was used as the cross-modal feature extractor, and the existing sketch synthesis network U-Net was used as the auxiliary sketch generator. An asymmetric decoupling mechanism was added at the end of the feature extraction network for implementation. Identity classification loss, contrast triplet loss, style binding loss and decoupling spatial feature optimization loss were used to jointly optimize the feature extractor, sketch synthesis network and asymmetric decoupling mechanism end-to-end.

[0074] Then, network training is performed.

[0075] Sketch and photograph images were divided into training and testing sets. Pairs of sketch and photograph images were fed into an asymmetric decoupled and dynamically synthesized sketch-photo recognition network for training. The network parameters were optimized and updated using forward and backward propagation.

[0076] Finally, conduct network testing.

[0077] The sketch images from the test set are used as the query set, and the photographic images are used as the image library. The best-performing model during training is used for inference to obtain the final retrieval results on the test set. The evaluation metric is Rank-... k Matching accuracy, which reflects the probability of correctly retrieving the cross-modal image.

[0078] This invention was experimentally validated on three fine-grained sketch-based datasets: PKU-Sketch, QMUL-ShoeV2 (ShoeV2), and QMUL-ChairV2 (ChairV2). The PKU-Sketch dataset is the first sketch-based pedestrian re-identification dataset in a surveillance system. It consists of 200 pedestrians, each with two photographs and one professional hand-drawn sketch. The ShoeV2 and ChairV2 datasets are two representative freehand sketch datasets used in commercial applications, containing 2000 shoe photographs and 400 chair photographs, respectively. Each photograph corresponds to at least three sketch images. Finally, there are a total of 1275 chair sketches and 6730 shoe sketches.

[0079] This invention employs a ViT network with a patch size of 16 as the backbone feature extraction network, and initializes the network parameters using parameters pre-trained on ImageNet. The batch size is 16, with each identity containing one photograph, one sketch, and one auxiliary sketch. During training, random cropping, padding, and horizontal flipping are used as data augmentation operations. The proposed network is trained using a stochastic gradient descent (SGD) optimizer. Furthermore, the initial learning rate is set to 0.008, and cosine learning rate decay is employed. On the PKU Sketch and ChairV2 datasets, the parameters... and They were set to 5.0 and 10.0 respectively. On the ShoeV2 dataset, the parameters... and All were set to 10.0.

[0080] To verify the effectiveness of this invention, this section compares the retrieval results of this invention with existing sketch-photo recognition methods, which mainly include:

[0081] (1) Triple SN: Qian Yu, Feng Liu, Yi-Zhe Song, Tao Xiang, Timothy MHospedales, and Chen Change Loy. 2016. Sketch me that shoe. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 799–807.

[0082] (2)GN Siamese:PatsornSangkloy, Nathan Burnell, Cusuh Ham, and JamesHays. 2016. Thesketchy database: learning to retrieve badly drawn bunnies.ACM Transactions

[0083] on Graphics 35, 4 (2016), 1–12.

[0084] (3)CD-AFL: Lu Pang, Yaowei Wang, Yi-Zhe Song, Tiejun Huang, andYonghong Tian. 2018. Cross-domain adversarial feature learning for sketch re-identification. In Proceedings of the 26th ACM international conference onMultimedia. 609–617.

[0085] (4) LMDI: ShaojunGui, Yu Zhu, Xiangxiang Qin, and Xiaofeng Ling.2020. Learning multi-level domain invariant features for sketch re-identification. Neurocomputing 403(2020), 294–303.

[0086] (5) Triplet RL: Ayan Kumar Bhunia, Yongxin Yang, Timothy MHospedales, Tao Xiang, andYi-Zhe Song. 2020. Sketch less for more: On-the-flyfine-grained sketch-based image retrieval. In Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition. 9779–9788.

[0087] (6) Edgemap: Filip Radenovic, GiorgosTolias, and Ondrej Chum. 2018.Deep shape matching.In Proceedings of the European Conference on ComputerVision. 751–767.

[0088] (7) SSL: Ayan Kumar Bhunia, Pinaki Nath Chowdhury, AneeshanSain,Yongxin Yang, TaoXiang, and Yi-Zhe Song. 2021. More Photos are All You Need:Semi-SupervisedLearning for Fine-Grained Sketch Based Image Retrieval. InProceedings of theIEEE / CVF Conference on Computer Vision and PatternRecognition. 4247–4256.

[0089] (8) Stylemeup: AneeshanSain, Ayan Kumar Bhunia, Yongxin Yang, TaoXiang, and Yi-ZheSong. 2021. Stylemeup: Towards style-agnostic sketch-basedimage retrieval. InProceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.8504–8513.

[0090] The test was conducted on the PKU-Sketch dataset, and the results are shown in Table 1:

[0091] Table 1

[0092]

[0093] Tests were conducted on the QMUL-ShoeV2 and QMUL-ChairV2 datasets, and the results are shown in Table 2.

[0094] Table 2

[0095]

[0096] As shown in Tables 1 and 2, compared with recent sketch-photo recognition methods, the method proposed in this invention achieves improved retrieval accuracy on both professional and freehand sketch datasets. On the PKU-Sketch dataset, the proposed method outperforms the current state-of-the-art method LMDI by 35.6% in Rank-1. On the ChairV2 and ShoeV2 datasets, the proposed method outperforms the current state-of-the-art method Stylemeup by 18.8% and 2.2% in Rank-1, respectively. Experimental results on three fine-grained sketch datasets demonstrate the effectiveness and generalization ability of the proposed method.

[0097] It should be understood that the above description of the implementation examples is quite detailed, but it should not be considered as a patent protection system for this invention. Those skilled in the art, under the guidance of this invention, can make substitutions or modifications without departing from the scope of protection of the claims of this invention, and all such substitutions or modifications fall within the scope of protection of this invention. The scope of protection of this invention should be determined by the appended claims.

Claims

1. A sketch-photograph recognition method based on asymmetric decoupling and dynamic synthesis, characterized in that, Includes the following steps: Step 1: Generate an auxiliary sketch image from the original photograph image; Step 2: For the auxiliary sketch image generated in Step 1, the photo modality features are decomposed into sketch-related representations and sketch-independent representations; wherein, sketch-related representations describe the outline and structural information of the target object shared with the sketch modality, and sketch-independent representations describe the appearance details unique to the photo modality. Among them, the photo modality features are decomposed into sketch-related representations and sketch-independent representations based on the autoencoder and the generated auxiliary sketch modality. The process of modal feature decomposition of a photograph is as follows: ； ； ； in, Represents the original features of a photographic modality; mask matrix medium elements Describes the photo modality. line characteristics With auxiliary sketching modality Column features Similarity between This indicates similarity calculation; and These represent the sketch-related and sketch-unrelated representations of the photographic modality, respectively. , , and These represent linear transformation layers with non-shared weights, used to map corresponding features to different representation subspaces; and These represent encoders that are related to the sketch and encoders that are unrelated to the sketch, respectively. Step 3: For the sketch image, fuse the sketch-irrelevant representation in the photo modality obtained in Step 1 with the sketch image to obtain an updated sketch image that is symmetrical to the photo modality representation information; Specifically, a knowledge transfer method is used to fuse the sketch-independent representations in the photo modality obtained in step 1 with the sketch image; ； in, and These represent the sketch representations before and after the update, respectively. and These represent the mean and standard deviation, respectively, used for standardizing modal features in sketching. and These represent the representations in the photographic modality that are unrelated to the sketch. Learnable affine parameters; Indicates a small quantity; Step 4: Compare and identify the updated sketch image obtained in Step 3 with the original photo image in Step 1 to obtain the identification result.

2. The sketch-photo recognition method based on asymmetric decoupling and dynamic synthesis according to claim 1, characterized in that: In step 1, an auxiliary sketch image is generated using a sketch generation network, which is a pre-trained network. During training, the network is initialized using pre-trained parameters on the Scibble dataset. Then, the sketch generation network and the recognition task network are jointly trained to achieve dynamic updates of the sketch generation network parameters, thereby making the generated auxiliary sketches more in line with the needs of the recognition task. The recognition task network includes a sketch-related representation encoder, a sketch-independent representation encoder, an image fusion module, and a contrast recognition module; The loss function used to update the sketch generation network is: in, This represents the identity loss, which can be either cross-entropy loss or triplet loss. Indicates hyperparameters; Style binding loss; and These represent the initial A-sketch image and the refined A-sketch image, respectively. This indicates the number of pixels in the auxiliary sketch image. This represents the Frobenius norm.

3. The sketch-photograph recognition method based on asymmetric decoupling and dynamic synthesis according to claim 1, characterized in that: In step 2, the Vision Transformer network is used to realize long-distance interaction between local modal features and to mine global modal feature representations. Given a sketch / photograph image First, it is divided into non-overlapping fixed-size image blocks. Then the ViT input sequence : ； In the formula, This represents the additional learnable parameters used to obtain the final image representation; The linear projection layer is used to convert image patches into one-dimensional vectors; PE represents the positional encoding of the image patch.

4. The sketch-photograph recognition method based on asymmetric decoupling and dynamic synthesis according to claim 1, characterized in that: The sketch-related representation encoder consists of two linear projection layers, including a first fully connected linear layer and a second fully connected linear layer; the first fully connected linear layer and the second fully connected linear layer are connected sequentially. The first linear fully connected layer is used to map the image representation to different feature spaces; The mask matrix is ​​obtained by modeling the spatial similarity between photographs and auxiliary sketch representations. The matrix multiplication is performed between the matrix and the output of the first linear fully connected layer to achieve weighted correction of the photo representation, enhancing the structure-related representations in the photo features. The result is used as the input to the second linear fully connected layer, which includes a fully connected layer and a dropout layer.

5. The sketch-photo recognition method based on asymmetric decoupling and dynamic synthesis according to claim 1, characterized in that: The sketch-independent representation encoder consists of two linear projection layers, including a first fully connected linear layer and a second fully connected linear layer; the first fully connected linear layer and the second fully connected linear layer are connected sequentially. The first linear fully connected layer is used to map the image representation to different feature spaces; The mask matrix is ​​obtained by modeling the spatial similarity between photographs and auxiliary sketch representations. ,Will The matrix is ​​multiplied with the output of the first linear fully connected layer to perform a weighted correction on the image representation, enhancing the structure-independent representations in the image features. The result is used as the input to the second linear fully connected layer, which includes a fully connected layer and a dropout layer.

6. The sketch-photo recognition method based on asymmetric decoupling and dynamic synthesis according to claim 2, characterized in that: The sketch generation network and the recognition task network are trained networks, and the overall loss function of the network training is... It includes the following four parts: in, The loss representing identity classification is derived from sketch modal features. Auxiliary sketch modal features and photo modal features Cross-entropy loss function The sum of ( ); triplet loss function The aim is to increase the distance between similar samples and increase the distance between dissimilar samples. Indicates the first The distance between a sample and its negative sample t Indicates sample The sum of distances between it and all its positive samples; Indicates sample The number of negative samples; This represents a decoupling spatial feature optimization function, designed to show how to shorten the distance between cross-modal aligned features. This represents the smooth L1 loss function; the first term of this function... Explicit zoom-in photograph representation and updated sketch representation The distance between them, the second item explicitly shortens the sketch representation. and decomposition characterization The distance between; and Both represent hyperparameters.

7. A sketch-photo recognition system based on asymmetric decoupling and dynamic synthesis, characterized in that, Includes the following modules: Module 1 is used to generate auxiliary sketch images from the original photograph images; Module 2 is used to decompose the photo modality features into sketch-related representations and sketch-independent representations for the auxiliary sketch image generated in Module 1. The sketch-related representations describe the outline and structural information of the target object shared with the sketch modality, while the sketch-independent representations describe the appearance details unique to the photo modality. Among them, the photo modality features are decomposed into sketch-related representations and sketch-independent representations based on the autoencoder and the generated auxiliary sketch modality. The process of modal feature decomposition of a photograph is as follows: ； ； ； in, Represents the original features of a photographic modality; mask matrix medium elements Describes the photo modality. line characteristics With auxiliary sketching modality Column features Similarity between This indicates similarity calculation; and These represent the sketch-related and sketch-unrelated representations of the photographic modality, respectively. , , and These represent linear transformation layers with non-shared weights, used to map corresponding features to different representation subspaces; and These represent encoders that are related to the sketch and encoders that are unrelated to the sketch, respectively. Module 3 is used to fuse the sketch-image-independent representation in the photo modality obtained in Module 1 with the sketch image to obtain an updated sketch image that is symmetrical with the photo modality representation information. Specifically, a knowledge transfer method is used to fuse the sketch-independent representations in the photo modality obtained in step 1 with the sketch image; ； in, and These represent the sketch representations before and after the update, respectively. and These represent the mean and standard deviation, respectively, used for standardizing modal features in sketching. and These represent the representations in the photographic modality that are unrelated to the sketch. Learnable affine parameters; Indicates a small quantity; Module 4 is used to compare and identify the updated sketch image obtained in Module 3 with the original photograph image in Module 1 to obtain the identification result.

Images