A text-driven semantic consistency zero-shot sketch retrieval method

Abstract

This invention belongs to the field of image and graphics processing technology and discloses a text-driven, semantically consistent zero-shot sketch retrieval method. The method includes: constructing a semantic representation library using the CLIP model and calculating the semantic consistency index of the image to quantify the semantic clarity of the samples; embedding the sketch and image into a unified semantic space using a dual-branch ViT encoder with shared parameters; introducing the semantic consistency index into the triplet loss for dynamic reweighting to adaptively adjust the sample supervision intensity; generating a semantic mask through text labels to drive the base model to remove image noise, and designing a cross-domain Transformer decoder to achieve local region reconstruction of the sketch and mask images, enhancing fine-grained geometric alignment. During the inference phase, only the trained and converged dual encoder is loaded for cosine similarity retrieval, requiring no additional computational overhead. This invention effectively suppresses noise sample interference and improves the accuracy and robustness of zero-shot sketch retrieval.

Description

Technical Field

[0001] This invention belongs to the field of image and graphics processing technology, and particularly relates to a text-driven semantic consistency zero-sample sketch retrieval method. Background Technology

[0002] In multimedia retrieval systems, sketches, as an intuitive visual interaction method, are gradually becoming an important tool for image retrieval. However, sketches only contain abstract line outlines, exhibiting significant cross-domain semantic differences compared to the rich texture, color, and background information of natural images. During the training phase, because a single sketch may correspond to multiple different images, the model is highly susceptible to semantic ambiguity and inter-class confusion, leading to a decrease in retrieval accuracy during inference. Existing zero-shot sketch retrieval methods often ignore the semantic clarity differences between sketches and natural images, failing to effectively distinguish between easily confused and clear samples. Furthermore, they easily introduce irrelevant background noise during cross-domain feature alignment, limiting further improvements in retrieval performance. Summary of the Invention

[0003] To overcome the aforementioned problems, this invention proposes a text-driven, semantically consistent zero-shot sketch retrieval method for the training phase, aiming to improve the semantic consistency between sketches and images through "semantic text-driven" approaches. This framework utilizes the text encoder of the CLIP model to construct a semantic representation library as an intermediate text representation, and calculates the semantic consistency index between sketches and images based on the correlation between images and text to quantify the semantic clarity of samples. Furthermore, this invention proposes a semantically aware dynamic reweighting strategy and a semantically cleansing local region reorganization mechanism: the former adaptively adjusts training weights based on the semantic consistency index to suppress noise interference in samples, while the latter uses a base model to generate a semantic mask to remove irrelevant regions and strengthens the fine-grained geometric alignment between sketches and mask images through a cross-domain Transformer decoder, thereby achieving more accurate and robust zero-shot sketch retrieval.

[0004] To achieve the above objectives, the specific technical solution of the text-driven semantic consistency zero-shot sketch retrieval method of the present invention is as follows:

[0005] A text-driven, semantically consistent zero-shot sketch retrieval method includes the following steps:

[0006] Step 1: Automatically generate semantic cue words for all semantic labels in the training dataset, and generate a set of semantic cue word embedding vectors through the text encoder of the CLIP vision-language pre-trained model to obtain the semantic representation library required for all natural images in the dataset;

[0007] Step 2: For a given input image, generate its visual embedding vector through the visual encoder of the CLIP vision-language pre-trained model, and calculate its similarity with each semantic text embedding in the semantic representation library to obtain the semantic association vector between the image and the semantic representation library.

[0008] Step 3: Calculate the Softmax function for the semantic relevance vector and extract the corresponding values ​​of the semantic relevance vector according to the index of the image label to generate the semantic consistency index;

[0009] Step 4: During the training phase, feature embedding is performed on the input sketch and image data using a shared parameter encoder architecture, embedding the sketch and image features into a unified semantic space;

[0010] Step 5: Introduce the semantic consistency index into the triplet loss function for sketch and image matching for dynamic reweighting;

[0011] Step 6: Generate a semantic mask by driving the base model through text semantic labels, remove image noise unrelated to the sketch, and perform self-supervised local region reconstruction on the sketch and mask images through a cross-domain Transformer decoder;

[0012] Step 7: Utilize the dual encoder trained in the semantically consistent latent space to directly calculate the similarity between the sketch and the image, thus completing the sketch image retrieval.

[0013] Furthermore, in step 1, a fixed text template is manually constructed for each category to form a category-specific set of prompt words. For datasets with uneven category distribution, additional general prompt words are introduced. The text encoder is used to encode the set of prompt words to generate a text feature sequence, which serves as the text representation benchmark for zero-shot retrieval.

[0014] Furthermore, in step 2, the visual representation of the input image is extracted by the image encoder of the CLIP vision-language pre-trained model, the inner product of the visual representation and each text feature is calculated to obtain a similarity vector, and Softmax normalization with a temperature parameter is applied to the similarity vector to obtain the normalized confidence score of each category.

[0015] Furthermore, in step 3, given the true label index of the image, the confidence level corresponding to the true category index is defined as the semantic consistency index; the semantic consistency index reflects the confidence level that the sample is correctly assigned to its true category. The closer the semantic consistency index is to 0, the higher the noise of the image sample and the easier it is to be confused. The closer the index is to 1, the clearer the semantics of the sample.

[0016] Furthermore, in step 4, the encoder architecture with shared parameters adopts a dual-branch ViT structure, which receives sketch and natural image inputs respectively. The two branches share parameters, forcibly aligning features of different modalities into a unified semantic space. The encoder output includes local feature sequences and global category label representations.

[0017] Furthermore, in step 5, the semantic consistency index is first normalized based on the images in the image library. The marginal value is dynamically calculated based on the normalized semantic consistency index and substituted into the triplet loss function to obtain the semantic perception loss. When the semantic consistency index approaches 0, the marginal value automatically decreases, and when the semantic consistency index approaches 1, the marginal value increases.

[0018] Furthermore, in step 5, the semantic perception loss uses the global representation of the sketch as the anchor point, the global representation of images of the same category as positive samples, and the global representation of images of different categories as negative samples. The loss value is calculated based on the distance between the anchor point and the positive sample, the distance between the anchor point and the negative sample, and the dynamically calculated marginal value. The distance is calculated based on the inner product similarity of normalized vectors.

[0019] Furthermore, in step 6, generating the semantic mask specifically includes: generating a semantic mask by driving the Grounded SAM basic model through text semantic labels, locating and retaining the image foreground region consistent with the text semantic labels; downsampling the semantic mask from the feature map to an image block-level binary vector, filtering the effective region through average pooling and thresholding, filtering image features accordingly, and retaining only the features corresponding to the effective image blocks with mask values, thereby obtaining a mask image feature sequence related to the sketch semantics.

[0020] Furthermore, in step 6, the local feature sequence of the sketch is randomly divided into several segments and the order is shuffled to obtain the rearranged sketch feature sequence; the rearranged sketch feature sequence and the mask image feature sequence are input into a multi-layer Transformer decoder, and the semantic relevance is evaluated through a cross-attention mechanism; the local region reconstruction includes two optimization objectives: binary classification pairing loss and reconstruction loss. The binary classification pairing loss is used to determine whether they are matching pairs, and the reconstruction loss is used to predict the spatial arrangement order of sketch image blocks for positive sample pairs.

[0021] Furthermore, in step 7, the retrieval process is completely decoupled from the semantic consistency evaluation and local region reconstruction module in the training phase. During inference, only the dual encoder that has converged during training is loaded to extract the semantic embedding vectors of the sketch to be retrieved and the database image, respectively. The retrieval results are output by calculating the cosine similarity and sorting them.

[0022] The text-driven semantic consistency zero-sample sketch retrieval method of the present invention has the following advantages:

[0023] 1. Semantic Explicitness Quantification and Noise Interference Suppression: The semantic consistency index of the image is calculated by the CLIP model to quantify the confidence that the sample is correctly classified, accurately identify easily confused and high-noise samples, provide a reliable basis for subsequent adaptive training, and effectively alleviate the inter-class confusion problem caused by the semantic ambiguity of the sketch.

[0024] 2. Dynamically reweighted loss, adaptively adjusting supervision intensity: The semantic consistency index is introduced into the triplet loss, and the margin boundary is dynamically adjusted according to the semantic clarity of the samples. More stringent constraints are imposed on high-consistency samples to enhance the discriminative power of clear samples; the boundary is automatically relaxed for low-consistency samples to avoid the model overfitting noise, significantly improving training stability and generalization ability.

[0025] 3. Semantic mask purification to remove background noise: The Grounded SAM basic model is driven by text labels to generate semantic masks, accurately locate the foreground region consistent with the text semantics, and filter out irrelevant background noise, effectively solving the problem of interference from complex backgrounds in natural images on cross-domain matching.

[0026] 4. Local region recombination to enhance fine-grained geometric alignment: By interacting the shuffled sketch features with the mask image features through a cross-domain Transformer decoder, and combining two self-supervised tasks, pairing discrimination and spatial order prediction, the model is forced to learn the spatial correspondence between the shape structure of the sketch and the local regions of the image, thereby improving the semantic understanding of abstract lines.

[0027] 5. Highly efficient inference and strong zero-shot generalization ability: The retrieval stage is completely decoupled from the auxiliary module during training, and only the dual encoder is loaded to calculate cosine similarity. It achieves efficient inference while ensuring high retrieval accuracy, making it suitable for practical deployment on large-scale datasets. It also has good zero-shot generalization ability for new categories that do not appear in the training set. Attached Figure Description

[0028] Figure 1 A flowchart for calculating the text-driven sketch-natural image semantic consistency index;

[0029] Figure 2 A quantization sketch of the semantic consistency index - example image pair;

[0030] Figure 3 This is a schematic diagram of a semantic awareness matching module based on a semantic consistency index.

[0031] Figure 4(a) shows the T-SNE clustering diagram of the semantic space of natural images;

[0032] Figure 4(b) shows the T-SNE clustering diagram of the sketch semantic space;

[0033] Figure 5This is a schematic diagram of sketch-based image retrieval results for zero samples that do not appear in the training set categories. Detailed Implementation

[0034] To better understand the purpose, structure, and function of this invention, the following detailed description of a text-driven semantic consistency zero-sample sketch retrieval method based on the accompanying drawings is provided.

[0035] The present invention provides a text-driven, semantically consistent zero-sample sketch retrieval method, comprising the following steps:

[0036] Step 1: Automatically generate semantic cue words for all semantic labels in the training dataset, and generate a set of semantic cue word embedding vectors through the text encoder of the CLIP vision-language pre-trained model to obtain the semantic representation library required for all natural images in the dataset.

[0037] The purpose of automatically generating semantic cue words is to build a text representation benchmark covering all categories, providing a unified semantic anchor for subsequent zero-shot retrieval.

[0038] First, for the dataset For each of the three categories, the algorithm manually constructs a fixed text template, "a photo of [category]", to form a category-specific set of prompt words. Furthermore, to improve the model's generalization ability on datasets with imbalanced class distributions (such as the "person" class, which is significantly more frequent), the generic cue word "a photo of person" is introduced. Subsequently, the CLIP text encoder is used. The above prompt words are encoded to generate corresponding text feature sequences. As a text representation benchmark for zero-shot retrieval, its mathematical expression is as follows:

[0039]

[0040] in This indicates the CLIP text encoder. For the first Text prompts for each category, This is the corresponding text feature vector.

[0041] Step 2: For a given input image, generate its visual embedding vector through the visual encoder of the CLIP model, and calculate its similarity with each semantic text embedding in the semantic representation library to obtain the semantic association vector between the image and the semantic representation library.

[0042] The purpose of calculating the semantic relevance vector is to quantify the degree of matching between the input image and all candidate semantic labels, providing basic data for semantic consistency evaluation.

[0043] For a given input image via CLIP image encoder Extract its visual representation ,in for A 3D feature tensor is then used. Subsequently, this visual representation is computed with each text feature. The inner product of these terms yields the similarity vector. ,in for dimensional feature tensor, where Further, a Softmax normalization with a temperature parameter is applied to the similarity vector to obtain the normalized confidence score for each category, mathematically expressed as follows:

[0044]

[0045] in Temperature is a parameter used to control the smoothness of the probability distribution; in this method, it is taken as... , For image Belongs to the Normalized confidence scores for each category.

[0046] Step 3: Calculate the Softmax function for the semantic relevance vector and extract the corresponding values ​​of the semantic relevance vector according to the index of the image label to generate the semantic consistency index.

[0047] The purpose of generating the semantic consistency index is to quantify the semantic clarity of samples and identify easily confused samples.

[0048] Given image Real Tag Index Define the semantic consistency index This is the confidence level corresponding to the true class index. This index reflects the confidence level that a sample has been correctly assigned to its true class. The closer to 0, the higher the noise level of the image sample and the easier it is to be confused. The closer the index is to 1, the clearer the semantic meaning of the sample. Compared to manually constructing a large number of fixed cue word combinations to calculate absolute consistency, the semantic consistency index reflects the relative consistency of samples within the dataset, and can better adapt to the differences in the size and distribution of different datasets.

[0049] Step 4: During the training phase, feature embedding is performed on the input sketch and image data using a shared parameter encoder architecture. The sketch and image features are embedded into a unified semantic space to ensure that the sketch and image have comparable global representations.

[0050] The goal of the shared-parameter encoder architecture is to map heterogeneous sketch line features and natural image texture features to the same semantic subspace, ensuring that the two are directly comparable at the global representation level.

[0051] The encoder architecture with shared parameters adopts a dual-branch ViT structure, receiving sketches separately. With natural images Input. The two branches share parameters, forcing features from different modalities to align into a unified semantic space. The encoder output contains a sequence of local features. and global class token representation ,in express 3D feature tensor For sequence length, For the embedding dimension, the mathematical expression is as follows:

[0052]

[0053] in and For ViT encoders with shared parameters, and These are the global semantic representations of the sketch and the image, respectively. and It is a local feature sequence.

[0054] Step 5: Introduce the semantic consistency index into the triplet loss function of sketch and image matching for dynamic reweighting, thereby suppressing the interference of noisy samples and enhancing the supervision signal of high-quality and reliable samples.

[0055] Furthermore, in step 5, the purpose of semantic consistency matching is to adaptively adjust the margin boundary of the triplet loss based on the semantic clarity of the samples, suppress the interference of noisy samples, and enhance the learning effect of high-quality and reliable samples.

[0056] First, the semantic consistency index is normalized based on the images in the image library, resulting in... ,in and These represent the maximum and minimum values ​​of the semantic consistency index in the dataset, respectively. This normalization operation prevents the marginal values ​​from being too low overall in noisy datasets, thus avoiding stagnation in the loss function optimization. Subsequently, the marginal values ​​are dynamically calculated based on the normalized semantic consistency index. Substituting this into the triplet loss function yields the semantic-aware loss function. The mathematical expression is as follows:

[0057]

[0058]

[0059] in For curve parameters, this invention uses 10. It is the basic marginal parameter, with a value range of 0-1; , , These represent the global representations of the anchor point sketch, positive sample image, and negative sample image, respectively. The distance function is calculated as follows: .when When approaching 0, Automatic reduction allows the model to handle noisy sample pairs more flexibly, when When it approaches 1, Increase the number of samples to force stricter separation constraints on high-quality samples.

[0060] Step 6: The base model is further driven to generate a semantic mask by text semantic labels, which removes image noise that is not related to the sketch. The sketch and mask images are then reconstructed by a cross-domain Transformer decoder to enhance the fine-grained local matching between the sketch and the natural image.

[0061] Furthermore, in step 6, the purpose of semantic sanitization matching is to remove background noise from the image that is irrelevant to the sketch and to enhance the local geometric alignment between the sketch and the natural image through fine-grained interaction.

[0062] First, semantic masks are generated by driving the Grounded SAM base model through text semantic labels. It locates and preserves the image foreground region that is consistent with the text semantic label, and removes irrelevant noise.

[0063] semantic mask From feature maps Downsampling to patch-level binary vectors Valid regions are selected by average pooling and threshold determination: , where the threshold Reshape the mask to And filter image features accordingly. Only patches with a mask value of 1 are retained to obtain a mask image feature sequence related to the sketch semantics. ,in Then, the sketch features Randomly divided into The segments are shuffled to obtain rearranged sketch features. .Will and Input multi-layer Transformer decoder Semantic relevance is evaluated using a cross-attention mechanism, mathematically expressed as follows:

[0064]

[0065] in , , These are the weight matrices corresponding to the query, key, and value, respectively. This task has two optimization objectives: one is the binary classification pairing loss. ,judge Is it a matching pair?

[0066]

[0067] in To match real tags, The first is the probability prediction. The second is the recombination loss, which only applies to positive sample pairs and predicts the spatial order of the sketch patch using multi-class cross-entropy. Relative to the actual position Arrangement:

[0068]

[0069] in The cross-entropy function is used for multi-class classification. This auxiliary task enhances the model's shape-aware reasoning ability by enforcing spatial consistency between the sketch and the image. The final total loss is a joint optimization of the semantic-aware loss function and the local reconstruction loss function.

[0070] .

[0071] Step 7: Utilize the dual encoder trained in the semantic consistency latent space to directly calculate the similarity between sketches and images, without the need for additional semantic consistency index calculation or cross-domain interaction, thereby achieving accurate sketch image retrieval.

[0072] The purpose of the inference phase is to achieve efficient sketch-image retrieval without increasing computational overhead while ensuring zero-sample retrieval accuracy.

[0073] The retrieval process is completely decoupled from the semantic consistency evaluation and local region reconstruction modules during the training phase. During inference, only the converged dual encoders are loaded to extract the sketch to be retrieved. Images in the database The semantic embedding vectors are used to calculate the cosine similarity between the two vectors and output the retrieval results in a sorted manner. No additional semantic consistency index calculation or cross-domain interaction is required. The mathematical expression is as follows:

[0074]

[0075] in, These are the global feature vectors of the sketch and the image during the reasoning process, respectively. This represents the L2 norm. Cosine similarity is used to calculate the matching degree between the sketch and the image, thus achieving efficient and accurate zero-shot sketch image retrieval.

[0076] like Figure 1 As shown in this embodiment, a text-driven semantic consistency zero-shot sketch retrieval method includes the following steps:

[0077] Step 1: Semantic Cue Word Construction and Semantic Consistency Index Calculation

[0078] For all categories in the dataset, fixed-format text prompts are manually constructed and transformed into text feature sequences using the CLIP text encoder, serving as semantic anchors for zero-shot retrieval. Figure 1 This paper demonstrates the generation process of the semantic consistency index. For any input image, its visual representation is extracted using the CLIP image encoder, and the inner product similarity between this visual representation and the text features of all categories is calculated. Subsequently, a Softmax normalization with a temperature parameter is applied to the inner product result to obtain the confidence scores for each category. Based on the ground truth label index of the image, the confidence score at the corresponding position is extracted and defined as the semantic consistency index. Figure 2 This demonstrates that the semantic consistency index can quantify the semantic clarity of a sample; the smaller the value, the greater the sample noise, providing a basis for subsequent adaptive training.

[0079] Step 2: Feature extraction from the shared encoder and semantic-aware matching

[0080] A parameter-sharing dual-branch ViT encoder is used to process sketch and natural image inputs respectively, mapping them to a unified semantic subspace, and outputting feature representations containing local patch sequences and global class tokens. Figure 3The process of flexible matching is demonstrated. During the training phase, the margin boundary of the triplet loss is dynamically adjusted based on the semantic consistency index calculated in step 1. Specifically, the semantic consistency index is first normalized, and then a soft margin parameter is generated through an index mapping function: when the sample semantic consistency index is low, the margin boundary is automatically reduced, allowing the model to flexibly handle easily confused samples; when the sample semantic consistency index is high, the margin boundary is increased, forcing stricter separation constraints on clear samples, thereby suppressing noise interference and improving optimization stability.

[0081] Step 3: Language-driven mask generation and local region reconstruction task

[0082] A semantic mask is generated using a text-based semantic label-driven segmentation model. This mask is then downsampled into patch-level binary vectors using average pooling and thresholding. Image feature sequences are filtered based on this mask, retaining only foreground regions semantically relevant to the sketch and removing background noise. A local region reconstruction task is then performed: sketch features are randomly divided into segments and shuffled. These segments, along with the masked image features, are input into a multi-layer Transformer decoder, where cross-attention is used to evaluate their semantic relevance. The local region reconstruction task has two optimization objectives: a binary pairing loss to determine if the shuffled sketch and mask image are a matching pair; and a reconstruction loss to predict the original spatial arrangement of sketch patches, only for positive pairs. By jointly optimizing these two objectives, the model is forced to learn local geometric alignment and shape-aware reasoning capabilities between the sketch and the image.

[0083] Step 4: Joint Training and Loss Optimization

[0084] During training, the model achieves end-to-end learning by jointly optimizing the two loss functions in steps 2 and 3. The semantic-aware loss adaptively adjusts the supervision strength based on the noise level of the samples, balancing the learning contribution of easy and difficult samples. The local region reconstruction loss strengthens cross-domain local structure alignment through pairwise discrimination and spatial order prediction. The two types of losses work synergistically, enabling the model to be optimized simultaneously at both the global semantic matching and fine-grained geometric alignment levels. The entire training process requires no manual annotation of additional correspondences, relying entirely on language-driven masking and self-supervised reconstruction tasks to achieve effective learning in zero-shot scenarios.

[0085] Step 5: Efficient Retrieval in the Reasoning Stage

[0086] During the inference phase, the model is completely decoupled from the semantic consistency evaluation, semantic-aware matching, and semantic sanitization matching modules during training, loading only the shared dual encoder that converged during training in step 2. Global semantic embedding vectors are extracted from both the sketch to be retrieved and the natural images in the database. The cosine similarity between the two is directly calculated, and the retrieval results are sorted and output. This process eliminates the need to recalculate the semantic consistency index, generate masks, or perform cross-domain interactions, ensuring zero-shot retrieval accuracy while avoiding additional computational overhead. It enables efficient and real-time sketch image retrieval on large-scale datasets. Figures 4(a) and 4(b) show the T-SNE clustering diagrams of the natural image semantic space and the sketch semantic space computed by the model in a zero-shot scenario. Figure 5 The retrieval results during the inference phase of the model are displayed.

[0087] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A text-driven based semantic consistency zero-shot sketch retrieval method, characterized in that, Includes the following steps: Step 1: Automatically generate semantic cue words for all semantic labels in the training dataset, and generate a set of semantic cue word embedding vectors through the text encoder of the CLIP vision-language pre-trained model to obtain the semantic representation library required for all natural images in the dataset; Step 2: For a given input image, generate its visual embedding vector through the visual encoder of the CLIP vision-language pre-trained model, and calculate its similarity with each semantic text embedding in the semantic representation library to obtain the semantic association vector between the image and the semantic representation library. Step 3: Calculate the Softmax function for the semantic relevance vector and extract the corresponding values ​​of the semantic relevance vector according to the index of the image label to generate the semantic consistency index; Step 4: During the training phase, feature embedding is performed on the input sketch and image data using a shared parameter encoder architecture, embedding the sketch and image features into a unified semantic space; Step 5: Introduce the semantic consistency index into the triplet loss function for sketch and image matching for dynamic reweighting; Step 6: Generate a semantic mask by driving the base model through text semantic labels, remove image noise unrelated to the sketch, and perform self-supervised local region reconstruction on the sketch and mask images through a cross-domain Transformer decoder; Step 7: Utilize the dual encoder trained in the semantically consistent latent space to directly calculate the similarity between the sketch and the image, thus completing the sketch image retrieval.

2. The semantic consistency zero-shot sketch retrieval method according to claim 1, characterized in that, In step 1, a fixed text template is manually constructed for each category to form a category-specific set of prompt words. For datasets with uneven category distribution, additional general prompt words are introduced. The text encoder is used to encode the set of prompt words to generate a text feature sequence, which serves as the text representation benchmark for zero-shot retrieval.

3. The semantically consistent zero-sample sketch retrieval method according to claim 1, characterized in that, In step 2, the visual representation of the input image is extracted by the image encoder of the CLIP vision-language pre-trained model. The inner product of the visual representation and each text feature is calculated to obtain a similarity vector. Softmax normalization with a temperature parameter is applied to the similarity vector to obtain the normalized confidence score for each category.

4. The semantically consistent zero-sample sketch retrieval method according to claim 1, characterized in that, In step 3, given the true label index of the image, the confidence level corresponding to the true category index is defined as the semantic consistency index. The semantic consistency index reflects the confidence level that the sample is correctly assigned to its true category. The closer the semantic consistency index is to 0, the higher the noise of the image sample and the easier it is to be confused. The closer the index is to 1, the clearer the semantics of the sample.

5. The semantically consistent zero-sample sketch retrieval method according to claim 1, characterized in that, In step 4, the encoder architecture with shared parameters adopts a dual-branch ViT structure, which receives sketch and natural image inputs respectively. The two branches share parameters, forcibly aligning features of different modalities into a unified semantic space. The encoder output includes local feature sequences and global category label representations.

6. The semantically consistent zero-sample sketch retrieval method according to claim 1, characterized in that, In step 5, the semantic consistency index is first normalized based on the images in the image library. The marginal value is then dynamically calculated based on the normalized semantic consistency index and substituted into the triplet loss function to obtain the semantic perception loss. When the semantic consistency index approaches 0, the marginal value automatically decreases, and when the semantic consistency index approaches 1, the marginal value increases.

7. The semantically consistent zero-sample sketch retrieval method according to claim 6, characterized in that, In step 5, the semantic perception loss uses the global representation of the sketch as the anchor point, the global representation of images of the same category as positive samples, and the global representation of images of different categories as negative samples. The loss value is calculated based on the distance between the anchor point and the positive sample, the distance between the anchor point and the negative sample, and the dynamically calculated marginal value. The distance is calculated based on the inner product similarity of normalized vectors.

8. The semantically consistent zero-sample sketch retrieval method according to claim 1, characterized in that, In step 6, generating the semantic mask specifically includes: generating a semantic mask by driving the Grounded SAM basic model through text semantic labels, locating and retaining the image foreground region consistent with the text semantic labels; downsampling the semantic mask from the feature map to an image block-level binary vector, filtering the effective region through average pooling and thresholding, filtering the image features accordingly, and retaining only the features corresponding to the effective image blocks with mask values, thereby obtaining a mask image feature sequence related to the sketch semantics.

9. The semantically consistent zero-sample sketch retrieval method according to claim 1, characterized in that, In step 6, the local feature sequence of the sketch is randomly divided into several segments and the order is shuffled to obtain the rearranged sketch feature sequence. The rearranged sketch feature sequence and the mask image feature sequence are input into the multi-layer Transformer decoder, and the semantic relevance is evaluated through the cross-attention mechanism. The local region reconstruction includes two optimization objectives: binary classification pairing loss and reconstruction loss. The binary classification pairing loss is used to determine whether they are matching pairs, and the reconstruction loss is used to predict the spatial arrangement order of sketch image blocks for positive sample pairs.

10. The semantically consistent zero-sample sketch retrieval method according to claim 1, characterized in that, In step 7, the retrieval process is completely decoupled from the semantic consistency evaluation and local region reconstruction module in the training phase. During inference, only the dual encoder that has converged during training is loaded to extract the semantic embedding vectors of the sketch to be retrieved and the database image, respectively. The retrieval results are output by calculating the cosine similarity and sorting them.

Images