A zero-shot sketch retrieval method based on mask and matching

Abstract

The present invention belongs to the technical field of zero-shot sketch retrieval, and discloses a zero-shot sketch retrieval method based on mask and matching. First, a visual-cross-language sampler is designed. This module uses semantic labels to generate masks, shielding semantic information in the image that is irrelevant to the sketch target, and optimizing the semantic alignment effect of cross-domain matching. Then, a purification mask matching module is designed, which includes two parts: feature reconstruction and semantic interaction. It suppresses redundant semantics by forcing the image encoder to reconstruct masked features, and uses a transformer decoder to promote cross-domain interaction between sketch and image features, thereby achieving more refined semantic matching. Finally, the training mechanism of triplet loss, reconstruction loss and interaction loss is combined to enable the model to significantly improve the retrieval accuracy in the purified semantic space. This method masks the interference elements in the image through masks and achieves pure sketch-image matching, thereby effectively solving the problem of semantic differences between sketches and natural images.

Description

Technical Field

[0001] The present invention belongs to the technical field of zero-shot sketch retrieval, and in particular relates to a zero-shot sketch retrieval method based on masking and matching. Background Art

[0002] Sketches are becoming an increasingly important tool for image retrieval and generation in multimedia systems as a new form of communication alternative to natural language. Sketches, characterized by their simple line visual cues, differ significantly from natural images, which contain complex elements such as background, foreground, and texture. This difference poses a challenge for zero-shot sketch retrieval, as the mismatch in semantic alignment between sketches and natural images can lead to distracting models and ambiguous semantics. Summary of the Invention

[0003] The present invention aims to provide a zero-shot sketch retrieval method based on mask and matching to solve the above technical problems.

[0004] To solve the above technical problems, the specific technical solution of the zero-shot sketch retrieval method based on masking and matching of the present invention is as follows:

[0005] A zero-shot sketch retrieval method based on mask and matching includes the following steps:

[0006] Step 1: Use the Vision Transformer encoder architecture to embed features of the input sketch and image data, thereby providing a unified semantic space representation for subsequent cross-domain matching;

[0007] Step 2: During the training phase, the semantic features of the image are filtered through a visual-cross-lingual sampler. A language mask is generated based on the semantic labels to mask irrelevant features in the natural image, reduce semantic interference, and ensure that the model focuses on important semantic information.

[0008] Step 3: The generated mask image features are input into the cleansing mask matching module, and the masked image features are reconstructed through the image encoder to reduce redundant semantics while retaining the original context information;

[0009] Step 4: Use the Transformer decoder to perform cross-domain semantic interaction between sketch features and mask image features to enhance the fine semantic matching between sketch and image;

[0010] Step 5: The model is trained using three loss functions: triplet loss, reconstruction loss, and interaction loss. The triplet loss is used to optimize the semantic alignment between the sketch and the image; the reconstruction loss ensures high-quality reconstruction of the mask features; and the interaction loss is used to improve the accuracy of cross-domain semantic interaction.

[0011] Step 6: In the inference phase, the trained dual encoder is used to calculate the similarity between sketches and images without the need for additional mask generation or complex cross-domain interaction steps, thereby achieving efficient sketch image retrieval.

[0012] Furthermore, in step 1, for each sketch and image in the dataset, its features are first embedded through a dual encoder architecture, that is, the input sketch and image are processed by the sketch encoder and image encoder respectively. The sketch encoder and the image encoder use the same embedding space and extract the global features of the sketch and image through the multi-head self-attention mechanism. After feature embedding, the feature vectors of the sketch and image will be mapped to a shared semantic space, making the sketch and image comparable in the space, which is convenient for subsequent matching and retrieval.

[0013] Furthermore, in step 1, first, for the sketch, the algorithm uses the Vision Transformer-based encoder to block the sketch image, dividing the input sketch into multiple feature segments, each representing a different part of the sketch; then, the image encoder uses the same method to block the input image and extract the global features of the image. The two encoders share the same semantic space and optimize the matching of the sketch and image through similarity measurement. In this way, the feature vectors of the sketch and image can be closely connected in a common space, which can be mathematically expressed as follows:

[0014] Embedding(S)=f sketch (S)

[0015] Embedding(I)=f image (I)

[0016] where f sketch and f image The encoders S and I represent sketches and images respectively. They represent the input sketches and images respectively. Through training, sketches and images of the same type are represented closer in the semantic space, while sketches and images of different types are separated as much as possible.

[0017] Furthermore, in step 2, first, the feature representation of the input image is extracted by the image encoder of the CLIP model. Then, the semantic label of the image is processed using the text encoder of CLIP to obtain the visual feature representation of the image. The semantic label of the image is processed using the text encoder to convert the label into text features. On this basis, the algorithm calculates the similarity between the image features and the text features, generates a semantic mask by setting a threshold, and filters out the semantically relevant parts of the image. For irrelevant areas, they are removed from the image features by generating a mask. In this step, the mask ratio is set to 70%, and the mathematical expression is as follows:

[0018] Masked Image Features=M(Image Features)

[0019] Among them, M represents the mask function generated by text features, Image Features is the features extracted by the image encoder, and the masked image features only retain the parts related to the sketch target, and other irrelevant parts are blocked.

[0020] Furthermore, in step 3, the purpose of the purification mask matching module is to optimize the cross-domain matching between the image and the sketch through two stages: feature reconstruction and semantic interaction. The image encoder reconstructs the masked image features to ensure that the reconstructed image features maintain the original semantics as much as possible while removing redundant interference information. By minimizing the reconstruction loss, the algorithm makes the features output by the image encoder as close to the masked features as possible.

[0021] Furthermore, in step 4, the transformer decoder module is used to interact with the features of the sketch and image. The features of the sketch and image are randomly shuffled to enhance the model's learning of fine-grained semantics. The decoder establishes an effective semantic connection between the sketch and image features through the cross-attention mechanism, thereby enhancing the matching effect between the two. The mathematical expression is as follows:

[0022]

[0023] L interaction =Cross-Attention(f sketch ,f masked image )

[0024] in, Represents the reconstructed image features, L reconstruction is the reconstruction loss, L interaction is the semantic interaction loss.

[0025] Furthermore, in step 5, a joint optimization strategy of triplet loss, reconstruction loss, and interaction loss is adopted during training to ensure that the model can work effectively in zero-shot scenarios. The representation of sketches and images should be close to images and sketches of the same type in the semantic space and away from images and sketches of different types. By minimizing the triplet loss, the similarity of samples of the same type is maximized and the similarity of samples of different types is minimized. The reconstruction loss suppresses redundant semantic information in the image by minimizing the difference between masked image features and reconstructed image features. The interaction loss further improves the accuracy of cross-domain matching by minimizing the interaction error between sketches and images in the semantic space. The mathematical expression is as follows:

[0026] Ltotal =L triplet +λ rec L reconstruction +λ int L interaction

[0027] Among them, L total is the total loss function, L triplet is the triplet loss, λ rec and λ int are the weights of reconstruction loss and interaction loss, respectively.

[0028] Furthermore, in step 6, the inference phase no longer uses masking or interactive operations of image features, but directly uses the trained dual encoder to calculate the similarity between the sketch and the image.

[0029] During the inference process, the similarity between sketch features and image features in the semantic space is calculated using cosine similarity, which allows the image that best matches the sketch to be quickly retrieved from a large-scale image library. This process requires no additional computational overhead, ensuring the efficiency of the retrieval process. The mathematical expression is as follows:

[0030] Similarity=CosineSimilarity(f sketch ,f image )

[0031] The matching degree between sketch and image is calculated by cosine similarity, thus achieving efficient zero-shot sketch image retrieval.

[0032] A zero-shot sketch retrieval method based on mask and matching of the present invention has the following advantages: the framework of the present invention introduces a visual-language cross-sampler to generate a language mask based on semantic labels to shield insignificant features in the image and reduce semantic noise. Furthermore, the present invention proposes the concept of purified mask matching, which includes two core processes: reconstruction and interaction. The reconstruction process forces the image encoder to reconstruct the masked image features, thereby enhancing the understanding of key information; the interaction process uses the transform decoder to process the sketch and the masked image features, explore the cross-domain relationship between the two, and achieve more effective matching. Finally, the training mechanism of triplet loss, reconstruction loss and interaction loss is combined, so that the model can significantly improve the retrieval accuracy in the purified semantic space. The present invention effectively solves the problem of semantic differences between sketches and natural images. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Flowchart of a zero-shot sketch retrieval method based on mask and matching;

[0034] Figure 2Schematic diagram of generating semantic masks for visual-cross-lingual sampler;

[0035] Figure 3 Schematic diagram of the model training framework;

[0036] Figure 4 This is the result of zero-sample sketch retrieval; DETAILED DESCRIPTION

[0037] In order to better understand the purpose, structure and function of the present invention, the following further describes in detail a zero-shot sketch retrieval method based on masking and matching of the present invention in conjunction with the accompanying drawings.

[0038] like Figure 1 As shown, a zero-shot sketch retrieval method based on mask and matching of the present invention comprises the following steps:

[0039] Step 1: The input sketch and image data are respectively embedded using the Vision Transformer encoder architecture to provide a unified semantic space representation for subsequent cross-domain matching.

[0040] In step 1, each sketch and image in the dataset is first feature-embedded using a dual-encoder architecture. The dual-encoder architecture aims to map sketches and images into a common semantic space. Specifically, the input sketch and image are processed by a sketch encoder and an image encoder, respectively. The sketch encoder and image encoder use the same embedding space and extract global features of the sketch and image through a multi-head self-attention mechanism. After feature embedding, the feature vectors of the sketch and image are mapped into a shared semantic space, making them comparable within this space and facilitating subsequent matching and retrieval.

[0041] First, for the sketch, the algorithm blocks the sketch image through an encoder based on Vision Transformer (ViT), dividing the input sketch into multiple feature segments, each of which represents a different part of the sketch. Then, the image encoder uses a similar method to block the input image and extract the global features of the image. Preferably, the feature dimension of the sketch and image is set to 768 dimensions. To ensure the semantic alignment of the sketch and image, the two encoders share the same semantic space and optimize the matching of the sketch and image through a similarity metric (such as cosine similarity). In this way, the feature vectors of the sketch and image can be closely connected in a common space, which can be mathematically expressed as follows:

[0042] Embedding(S)=f sketch (S)

[0043] Embedding(I)=fimage (I)

[0044] where f sketch and f image The encoders S and I represent sketches and images respectively. S and I represent the input sketches and images respectively. Through training, sketches and images of the same type are closer in representation in the semantic space, while sketches and images of different types are separated as much as possible.

[0045] Step 2: During the training phase, the semantic features of the image are screened through a visual-cross-lingual sampler, and a language mask is generated based on the semantic labels to mask irrelevant features in natural images, reduce semantic interference, and ensure that the model focuses on important semantic information.

[0046] like Figure 2 As shown in Figure 2, in step 2, the visual-cross-lingual sampler aims to remove distracting information from the image that is irrelevant to the sketch semantics and retain image features that are relevant to the sketch target.

[0047] First, the CLIP model's image encoder extracts a feature representation of the input image. Subsequently, the CLIP model's text encoder processes the image's semantic labels to obtain a visual feature representation of the image. The text encoder then processes the image's semantic labels and converts them into text features. Based on this, the algorithm calculates the similarity between the image and text features and generates a semantic mask by setting a threshold to filter out semantically relevant portions of the image. Irrelevant regions are removed from the image features by generating a mask. In this step, the mask ratio is set to 70% to ensure that key semantic regions are preserved in the image.

[0048] This method effectively filters the semantic information in the image, reduces the model's dependence on interfering features, and improves the accuracy of cross-domain matching. The mathematical expression is as follows:

[0049] Masked Image Features=M(Image Features)

[0050] Where M represents the mask function generated by the text features, and Image Features is the features extracted by the image encoder. Masked Image Features only retains the parts related to the sketch target, and other irrelevant parts are blocked.

[0051] Step 3: The generated mask image features are input into the purification mask matching module, and the masked image features are reconstructed through the image encoder to minimize redundant semantics while retaining the original context information.

[0052] In step 3, the cleansing mask matching module optimizes the cross-domain matching between the image and the sketch through two stages: feature reconstruction and semantic interaction. The image encoder reconstructs the masked image features, ensuring that the reconstructed image features retain the original semantics as much as possible while removing unnecessary noise. By minimizing the reconstruction loss, the algorithm ensures that the features output by the image encoder are as close as possible to the masked features.

[0053] Step 4: Perform cross-domain semantic interaction between sketch features and mask image features through the Transformer decoder to enhance the fine semantic matching between sketch and image.

[0054] In step 4, the transformer decoder module is used to interact with the features of the sketch and image. The features of the sketch and image are randomly shuffled to enhance the model's learning of fine-grained semantics. The decoder uses a cross-attention mechanism to establish effective semantic connections between the sketch and image features, thereby enhancing the matching effect between the two. The mathematical expression is as follows:

[0055]

[0056] L interaction =Cross-Attention(f sketch ,f masked image )

[0057] in, Represents the reconstructed image features, L reconstruction is the reconstruction loss, L interaction is the semantic interaction loss. By jointly optimizing these two loss functions, the algorithm can effectively improve the semantic matching between sketches and images.

[0058] Step 5: The model is trained using three loss functions: triplet loss, reconstruction loss, and interaction loss. The triplet loss optimizes the semantic alignment between the sketch and the image; the reconstruction loss ensures high-quality reconstruction of the mask features; and the interaction loss improves the accuracy of cross-domain semantic interaction.

[0059] like Figure 3 As shown in Figure 5, in step 5, a joint optimization strategy of triplet loss, reconstruction loss, and interaction loss is adopted during training to ensure that the model can work effectively in zero-shot scenarios.

[0060] Sketches and images should be represented in semantic space close to images and sketches of the same type and away from images and sketches of different types. By minimizing the triplet loss, the similarity of samples of the same type is maximized and the similarity of samples of different types is minimized. The reconstruction loss suppresses redundant semantic information in the image by minimizing the difference between masked image features and reconstructed image features. The interaction loss further improves the accuracy of cross-domain matching by minimizing the interaction error between sketches and images in semantic space. The mathematical expression is as follows:

[0061] L total =L triplet +λ rec L reconstruction +λ int L interaction

[0062] Among them, L total is the total loss function, L triplet is the triplet loss, λ rec and λ int are the weights of reconstruction loss and interaction loss, respectively.

[0063] Step 6: In the inference phase, the trained dual encoder is used to calculate the similarity between sketches and images without the need for additional mask generation or complex cross-domain interaction steps, thereby achieving efficient sketch image retrieval.

[0064] In step 6, the inference phase no longer uses masking or interaction operations of image features, but directly utilizes the trained dual encoder to calculate the similarity between the sketch and the image.

[0065] During the inference process, the similarity between sketch features and image features in semantic space is calculated using cosine similarity, allowing the image that best matches the sketch to be quickly retrieved from a large-scale image library. This process requires no additional computational overhead, ensuring high retrieval efficiency. The mathematical expression is as follows:

[0066] Similarity=CosineSimilarity(f sketch ,f image )

[0067] The matching degree between sketch and image is calculated by cosine similarity, thus achieving efficient zero-shot sketch image retrieval.

[0068] It will be understood that the present invention is described by way of some embodiments, and it will be appreciated by those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are intended to be protected by the present invention.

Claims

1. A zero-shot sketch retrieval method based on mask and matching, characterized in that: The following steps are involved: Step 1: Use the Vision Transformer encoder architecture to embed features of the input sketch and image data, thereby providing a unified semantic space representation for subsequent cross-domain matching; Step 2: During the training phase, the semantic features of the image are filtered through a visual-cross-lingual sampler. A language mask is generated based on the semantic labels to mask irrelevant features in the natural image, reduce semantic interference, and ensure that the model focuses on important semantic information. Step 3: The generated mask image features are input into the cleansing mask matching module, and the masked image features are reconstructed through the image encoder to reduce redundant semantics while retaining the original context information; Step 4: Use the Transformer decoder to perform cross-domain semantic interaction between sketch features and mask image features to enhance the fine semantic matching between sketch and image; Step 5: The model is trained using three loss functions: triplet loss, reconstruction loss, and interaction loss. The triplet loss is used to optimize the semantic alignment between the sketch and the image; the reconstruction loss ensures high-quality reconstruction of the mask features; and the interaction loss is used to improve the accuracy of cross-domain semantic interaction. Step 6: In the inference phase, the trained dual encoder is used to calculate the similarity between sketches and images without the need for additional mask generation or complex cross-domain interaction steps, thereby achieving efficient sketch image retrieval.

2. The zero-shot sketch retrieval method based on mask and matching according to claim 1, characterized in that: In step 1, for each sketch and image in the dataset, its features are first embedded through a dual encoder architecture. That is, the input sketch and image are processed by the sketch encoder and image encoder respectively. The sketch encoder and image encoder use the same embedding space and extract the global features of the sketch and image through the multi-head self-attention mechanism. After feature embedding, the feature vectors of the sketch and image will be mapped to a shared semantic space, making the sketch and image comparable in this space, which is convenient for subsequent matching and retrieval.

3. The zero-shot sketch retrieval method based on mask and matching according to claim 1, characterized in that: In step 1, the algorithm first blocks the sketch image through the Vision Transformer-based encoder, dividing the input sketch into multiple feature segments, each representing a different part of the sketch. Then, the image encoder uses the same method to block the input image and extract the global features of the image. The two encoders share the same semantic space and optimize the matching of the sketch and image through similarity measurement. In this way, the feature vectors of the sketch and image can be closely connected in a common space, which can be expressed mathematically as follows: Embedding(S)=f sketch (S) Embedding(I)=f image (I) where f sketch and f image The encoders S and I represent sketches and images respectively. They represent the input sketches and images respectively. Through training, sketches and images of the same type are represented closer in the semantic space, while sketches and images of different types are separated as much as possible.

4. The zero-shot sketch retrieval method based on mask and matching according to claim 1, characterized in that: In step 2, first, the feature representation of the input image is extracted by the image encoder of the CLIP model. Then, the semantic label of the image is processed using the text encoder of CLIP to obtain the visual feature representation of the image. The semantic label of the image is processed using the text encoder to convert the label into text features. On this basis, the algorithm calculates the similarity between the image features and the text features, generates a semantic mask by setting a threshold, and filters out the semantically relevant parts of the image. For irrelevant areas, they are removed from the image features by generating a mask. In this step, the mask ratio is set to 70%, and the mathematical expression is as follows: Masked Image Features=M(Image Features) Among them, M represents the mask function generated by text features, Image Features is the features extracted by the image encoder, and the masked image features only retain the parts related to the sketch target, and other irrelevant parts are blocked.

5. The zero-shot sketch retrieval method based on mask and matching according to claim 1, characterized in that: In step 3, the purpose of the purification mask matching module is to optimize the cross-domain matching between the image and the sketch through two stages: feature reconstruction and semantic interaction. The image encoder reconstructs the masked image features to ensure that the reconstructed image features maintain the original semantics as much as possible while removing redundant interference information. By minimizing the reconstruction loss, the algorithm makes the features output by the image encoder as close as possible to the masked features.

6. The zero-shot sketch retrieval method based on mask and matching according to claim 1, characterized in that: In step 4, the transformer decoder module is used to interact with the features of the sketch and image. The features of the sketch and image are randomly shuffled to enhance the model's learning of fine-grained semantics. The decoder establishes an effective semantic connection between the sketch and image features through the cross-attention mechanism, thereby enhancing the matching effect between the two. The mathematical expression is as follows: L interaction =Cross-Attention(f sketch ,f maskedimage ) in, Represents the reconstructed image features, L reconstruction is the reconstruction loss, L interaction is the semantic interaction loss.

7. The zero-shot sketch retrieval method based on mask and matching according to claim 1, characterized in that: In step 5, a joint optimization strategy of triplet loss, reconstruction loss, and interaction loss is adopted during the training process to ensure that the model can work effectively in zero-shot scenarios. The representation of sketches and images should be close to images and sketches of the same type in the semantic space and away from images and sketches of different types. By minimizing the triplet loss, the similarity of samples of the same type is maximized and the similarity of samples of different types is minimized. The reconstruction loss suppresses redundant semantic information in the image by minimizing the difference between masked image features and reconstructed image features. The interaction loss further improves the accuracy of cross-domain matching by minimizing the interaction error between sketches and images in the semantic space. The mathematical expression is as follows: L total =L triplet +λ rec L reconstruction +λ int L interaction Among them, L total is the total loss function, L triplet is the triplet loss, λ rec and λ int are the weights of reconstruction loss and interaction loss, respectively.

8. The zero-shot sketch retrieval method based on mask and matching according to claim 1, characterized in that: In step 6, the inference phase no longer uses masking or interactive operations of image features, but directly uses the trained dual encoder to calculate the similarity between the sketch and the image. During the inference process, the similarity between sketch features and image features in the semantic space is calculated using cosine similarity, which allows the image that best matches the sketch to be quickly retrieved from a large-scale image library. This process requires no additional computational overhead, ensuring the efficiency of the retrieval process. The mathematical expression is as follows: Similarity=CosineSimilarity(f sketch ,f image ) The matching degree between sketch and image is calculated by cosine similarity, thus achieving efficient zero-shot sketch image retrieval.

Images