A zero-shot sketch image retrieval method based on class-surrogate distillation

Abstract

The present invention belongs to the technical field of cross-modal retrieval, and relates to a zero-sample sketch image retrieval method based on class proxy distillation; obtaining a sketch sample and a corresponding picture sample; using a pre-trained teacher model to extract a first semantic feature vector and a first visual feature vector of the sketch sample, a second semantic feature vector and a second visual feature vector of the picture sample; fusing the first semantic feature vector and the second visual feature vector to obtain a target first fused feature vector, and fusing the second semantic feature vector and the first visual feature vector to obtain a target second fused feature vector; inputting the sketch sample and the picture sample into a student model, and outputting a sketch feature vector and a picture feature vector; constructing a loss function and iteratively training the student model until the value of the loss function is minimized, and obtaining a trained student model. By fusing semantic features and visual features, more abundant feature information is provided for the matching of sketches and pictures, and the accuracy of retrieval results is improved.

Description

Technical Field

[0001] The present invention relates to the field of cross-modal retrieval technology, and in particular to a zero-sample sketch image retrieval method based on class proxy distillation. Background Art

[0002] Sketch image retrieval is an important task in the field of computer vision and image processing. Its purpose is to retrieve real images using abstract hand-drawn sketches. This task is widely used in design and art creation, search engines, and image and database management. For example, in the field of design and art creation, designers can quickly retrieve similar real pictures or design examples based on sketches; in the field of search engines, users can directly upload sketches to retrieve corresponding pictures without entering picture descriptions or keywords. With the increasing application of sketch image retrieval, how to improve the accuracy of sketch image retrieval results is a problem that needs attention.

[0003] In the prior art, a sketch image retrieval method based on knowledge distillation is usually used, which uses a pre-trained teacher model to preliminarily extract the visual features of the sketch and the visual features of the picture, and then performs deep feature extraction on the visual features of the sketch and the visual features of the picture to obtain high-level semantic features of the sketch and the high-level semantic features of the picture. A loss function is constructed based on the similarity between the high-level semantic features of the sketch extracted by the teacher model and the sketch features extracted by the student model, and the similarity between the high-level semantic features of the picture extracted by the teacher model and the picture features extracted by the student model, so as to guide the student model to learn, so that the sketch feature representation and the picture feature representation output by the student model are as similar as possible to the high-level semantic features of the sketch and the high-level semantic features of the picture output by the teacher model; when performing sketch image retrieval afterwards, the student model is directly used to extract the feature representations of the sketch to be matched and each picture in the database, so as to find the target picture with the highest similarity or the same category as the sketch to be matched.

[0004] Since the picture itself has rich color and texture information, the framework information of the sketch will also be affected by the painter and present different visual effects. These visual features can directly present the shape and posture of the object, thereby providing more effective feature information for the sketch image retrieval task. However, in the existing sketch image retrieval method, the teacher model only focuses on the high-level semantic features contained in the visual features when extracting features, and ignores the low-level visual features. As a result, the student model also pays more attention to high-level semantic features when extracting features, and ignores the visual features of sketches and pictures, resulting in the inability to accurately match the subtle differences between sketches and pictures during retrieval, thereby reducing the accuracy of the retrieval results. Summary of the invention

[0005] To this end, the technical problem to be solved by the present invention is to overcome the problem that the sketch image retrieval method in the prior art only matches sketches and images through high-level semantic features, ignoring the visual features of sketches and images, resulting in the inability to accurately match the subtle differences between sketches and images, thereby reducing the accuracy of the retrieval results.

[0006] To solve the above technical problems, the present invention provides a zero-shot sketch image retrieval method based on class proxy distillation, comprising:

[0007] Obtain sketch samples in a training set and image samples corresponding to the sketch samples;

[0008] Inputting the sketch sample and the picture sample into a pre-trained teacher model for feature extraction and fusion, and outputting a target first fused feature vector and a target second fused feature vector; which specifically includes:

[0009] Input the sketch sample into a first encoder, and output a first semantic feature vector and a first visual feature vector; input the picture sample into a second encoder, and output a second semantic feature vector and a second visual feature vector;

[0010] Input the first semantic feature vector into a first mapping layer, input the output of the first mapping layer and the second visual feature vector into a first cross-modal fusion module, and output a first fused feature vector; input the first fused feature vector into a second mapping layer, and output a target first fused feature vector;

[0011] Input the second semantic feature vector into a third mapping layer, input the output of the third mapping layer and the first visual feature vector into a second cross-modal fusion module, and output a second fused feature vector; input the second fused feature vector into a fourth mapping layer, and output a target second fused feature vector;

[0012] Input the sketch sample into the third encoder in the student model, and output a sketch feature vector; input the picture sample into the fourth encoder in the student model, and output a picture feature vector;

[0013] Based on the first target fused feature vector, the second target fused feature vector, the sketch feature vector and the picture feature vector, a loss function is constructed; which specifically includes:

[0014] Mapping the target first fused feature vector and the target second fused feature vector respectively to obtain an output probability distribution of the teacher model for the sketch sample and an output probability distribution of the teacher model for the picture sample;

[0015] Mapping the sketch feature vector and the picture feature vector respectively to obtain an output probability distribution of the student model for the sketch sample and an output probability distribution of the picture sample;

[0016] Based on the similarity between the output probability distribution of the teacher model for the sketch sample and the output probability distribution of the student model for the sketch sample, construct a first knowledge distillation loss function; based on the similarity between the output probability distribution of the teacher model for the picture sample and the output probability distribution of the student model for the picture sample, construct a second knowledge distillation loss function;

[0017] Classifying the sketch feature vector and the picture feature vector respectively, and outputting the predicted category of the sketch sample and the predicted category of the picture sample by the student model;

[0018] Based on the predicted category of the sketch sample by the student model and the true category of the sketch sample, a first classification loss function is constructed; based on the predicted category of the picture sample by the student model and the true category of the picture sample, a second classification loss function is constructed;

[0019] Summing the first knowledge distillation loss function, the second knowledge distillation loss function, the first classification loss function, and the second classification loss function to obtain a loss function;

[0020] The student model is iteratively trained until the value of the loss function is minimized, thereby obtaining a trained student model; wherein the step of using the trained student model to perform sketch image retrieval includes:

[0021] Input the sketch to be matched into the third encoder in the trained student model, and output the sketch feature vector;

[0022] Input the pictures in the database into the fourth encoder in the trained student model respectively, and output the picture feature vector of each picture;

[0023] Classifying the sketch feature vector to obtain the category of the sketch to be matched; classifying the picture feature vectors of each picture to obtain the category of each picture;

[0024] Using a picture of the same category as the sketch to be matched as a target picture of the sketch to be matched;

[0025] Among them, the pre-training process of the teacher model includes:

[0026] Obtain sketch samples in a training set and image samples corresponding to the sketch samples;

[0027] Input the sketch sample into a first encoder in the teacher model, and output a first semantic feature vector and a first visual feature vector; input the picture sample into a second encoder in the teacher model, and output a second semantic feature vector and a second visual feature vector;

[0028] Input the first semantic feature vector into a first mapping layer in a teacher model, input the output of the first mapping layer and the second visual feature vector into a first cross-modal fusion module, output a first fused feature vector, input the first fused feature vector into a second mapping layer in a teacher model, and output a target first fused feature vector;

[0029] Input the second semantic feature vector into the third mapping layer in the teacher model, input the output of the third mapping layer and the first visual feature vector into the second cross-modal fusion module, output a second fused feature vector, input the second fused feature vector into the fourth mapping layer in the teacher model, and output a target second fused feature vector;

[0030] Based on the target first fused feature vector and the target second fused feature vector, a teacher model pre-training loss function is constructed; the teacher model is iteratively trained using sketch samples and picture samples in the training set until the value of the teacher model pre-training loss function is minimized, thereby obtaining a pre-trained teacher model; wherein the process of constructing the teacher model pre-training loss function includes:

[0031] Based on the similarity between the first target fused feature vector and the second target fused feature vector, a pre-trained contrast loss function is constructed;

[0032] Classify the target first fused feature vector and the target second fused feature vector respectively, and output the predicted category of the teacher model for the sketch sample and the predicted category for the picture sample;

[0033] Based on the predicted category of the sketch sample by the teacher model and the true category of the sketch sample, a first pre-training classification loss function is constructed;

[0034] Based on the predicted category of the picture sample and the true category of the picture sample by the teacher model, a second pre-training classification loss function is constructed; the pre-training contrast loss function, the first pre-training classification loss function and the second pre-training classification loss function are summed to obtain the teacher model pre-training loss function.

[0035] Preferably, the first knowledge distillation loss function is expressed as:

[0036] ,

[0037] in, represents the first knowledge distillation loss function; represents the output probability distribution of the student model for the sketch sample; represents the output probability distribution of the teacher model for the sketch sample; Indicates the distillation temperature; Represents a dynamic queue for expanding the number of sketch sample comparisons; Represents a dynamic queue Comparison samples in ;

[0038] ,

[0039] ,

[0040] in, represents normalization; represents a linear layer; represents the sketch feature vector; represents the first fused feature vector of the target; Factor representing the smoothed probability distribution;

[0041] The second knowledge distillation loss function is expressed as:

[0042] ,

[0043] in, represents the second knowledge distillation loss function; Represents the output probability distribution of the student model for the image sample; Represents the output probability distribution of the teacher model for the image sample; Represents a dynamic queue for expanding the number of image sample comparisons; Represents a dynamic queue Comparison samples in ;

[0044] ,

[0045] ,

[0046] in, Represents the image feature vector; Represents the target second fused feature vector.

[0047] Preferably, the first classification loss function is expressed as:

[0048] ,

[0049] in, represents the first classification loss function; Indicates the number of categories; Indicates that the sketch sample belongs to The true probability of each category; Indicates that the sketch sample output by the student model belongs to The predicted probability of each category;

[0050] The second classification loss function is expressed as:

[0051] ,

[0052] in, represents the second classification loss function; Indicates that the image sample belongs to The true probability of each category; Indicates that the image sample output by the student model belongs to The predicted probability of each category;

[0053] The loss function is expressed as:

[0054] ,

[0055] in, represents the loss function; represents the first knowledge distillation loss function; represents the second knowledge distillation loss function.

[0056] Preferably, after obtaining the loss function, the following steps are also included:

[0057] For any sketch sample in the training set, a quadruple loss function is constructed based on the Euclidean distance between a sketch feature vector of the sketch sample and a picture feature vector of a picture sample corresponding to the sketch sample, the Euclidean distance between the sketch feature vector of the sketch sample and a picture feature vector of a target picture sample, and the Euclidean distance between the sketch feature vector of the sketch sample and a sketch feature vector of a target sketch sample; wherein the target picture sample is a picture sample of the same category as the sketch sample in the training set, and the target sketch sample is a sketch sample of the same category as the sketch sample in the training set;

[0058] Cluster the sketch samples and picture samples of the same category in the training set respectively to obtain the sketch cluster center and picture cluster center of the current category; construct the center alignment loss function based on the Euclidean distance between the sketch cluster center and the picture cluster center of each category;

[0059] The loss function, the quadruple loss function and the center alignment loss function are weightedly summed to obtain a target loss function.

[0060] Preferably, the quadruple loss function is expressed as:

[0061] ,

[0062] in, represents the quadruple loss function; Indicates the number of target image samples or target sketch samples in the training set; Indicates boundary value; The Euclidean distance between a sketch feature vector of a sketch sample and a picture feature vector of a picture sample corresponding to the sketch sample; The sketch feature vector representing the sketch sample is The Euclidean distance between the image feature vectors of target image samples; The sketch feature vector representing the sketch sample is The Euclidean distance between the sketch feature vectors of target sketch samples;

[0063] The center alignment loss function is expressed as:

[0064] ,

[0065] in, represents the center alignment loss function; Indicates the number of categories; Indicates The sketch cluster centers of each category; Indicates The cluster centers of the images of each category;

[0066] The objective loss function is expressed as:

[0067] ,

[0068] in, represents the target loss function; represents the weight of the center alignment loss function; Represents the weight of the loss function; represents the loss function.

[0069] Preferably, the pre-training contrast loss function is expressed as:

[0070] ,

[0071] in, represents the pre-training contrast loss function; represents the first fused feature vector of the target; represents the second fused feature vector of the target; Represents a dynamic queue for expanding the number of sketch sample comparisons; Represents a dynamic queue Comparison samples in ; represents the temperature control embedding distribution coefficient; Represents a dynamic queue for expanding the number of image sample comparisons; Represents a dynamic queue Comparison samples in ;

[0072] The first pre-training classification loss function is expressed as:

[0073] ,

[0074] in, represents the first pre-training classification loss function; Indicates the number of categories; Indicates that the sketch sample belongs to The true probability of each category; Indicates that the sketch sample output by the teacher model belongs to The predicted probability of each category;

[0075] The second pre-training classification loss function is expressed as:

[0076] ,

[0077] in, represents the second pre-training classification loss function; Indicates that the image sample belongs to The true probability of each category; Indicates that the picture sample output by the teacher model belongs to The predicted probability of each category;

[0078] The teacher model pre-training loss function is expressed as:

[0079] ,

[0080] in, Represents the teacher model pre-training loss function.

[0081] The sketch image retrieval method based on class-agent distillation provided in the present application designs an encoder for feature extraction and a cross-modal fusion module for feature fusion in the teacher model. The encoder is first used to extract the visual features and semantic features of the two modal samples, and then the semantic features of each modal sample are mapped. Since there are large differences in details and textures between the two modalities of sketch and picture, directly fusing the visual features and semantic features of a single modality cannot utilize the complementarity between the sketch and the picture, nor can it accurately capture the correlation between the sketch and the picture. Therefore, the cross-modal fusion module is used to fuse the mapped semantic features of the current modal sample with the visual features of the other modal sample to obtain a first fused feature including the semantic features of the sketch sample and the visual features of the picture sample, and a second fused feature including the semantic features of the picture sample and the visual features of the sketch sample. Finally, the first fused feature and The second fused features are mapped back to the sketch domain and the image domain respectively to obtain the target first fused features corresponding to the sketch samples and the target second fused features corresponding to the image samples; not only the visual features and semantic features of the samples are effectively utilized, but also the cross-fusion of the visual and semantic features of the two modalities can better align the similarities between the sketches and the images, so that the features output by the teacher model contain both the visual features and semantic features of the two modalities, and reduce the negative impact of the differences between the sketches and the images on the retrieval results; finally, a loss function is constructed based on the target first fused features, the target second fused features and the sketch features and image features extracted by the student model, and the student model is trained so that the student model can extract the visual features and semantic features of the samples under the guidance of the teacher model, thereby providing richer feature information for the matching of sketches and images and improving the accuracy of the retrieval results. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein

[0083] Figure 1 Flowchart of the sketch image retrieval method based on class agent distillation provided for this application;

[0084] Figure 2 A framework diagram of the sketch image retrieval method based on class agent distillation provided for this application;

[0085] Figure 3 Schematic diagram of the knowledge distillation principle of the teacher model and student model provided for this application. DETAILED DESCRIPTION

[0086] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0087] See also Figure 1 , Figure 1 A flowchart of a sketch image retrieval method based on class agent distillation provided in this application, the method specifically includes:

[0088] S10: Obtain sketch samples in the training set and image samples corresponding to the sketch samples;

[0089] S20: Input the sketch sample and the picture sample into the pre-trained teacher model for feature extraction and fusion, and output the target first fused feature vector and the target second fused feature vector; which specifically includes:

[0090] S200: Inputting the sketch sample into a first encoder, outputting a first semantic feature vector and a first visual feature vector; inputting the picture sample into a second encoder, outputting a second semantic feature vector and a second visual feature vector;

[0091] S201: inputting the first semantic feature vector into the first mapping layer, inputting the output of the first mapping layer and the second visual feature vector into the first cross-modal fusion module, and outputting a first fused feature vector; inputting the first fused feature vector into the second mapping layer, and outputting a target first fused feature vector;

[0092] S202: input the second semantic feature vector into the third mapping layer, input the output of the third mapping layer and the first visual feature vector into the second cross-modal fusion module, and output a second fused feature vector; input the second fused feature vector into the fourth mapping layer, and output a target second fused feature vector;

[0093] S30: Input the sketch sample into the third encoder in the student model, and output the sketch feature vector; input the picture sample into the fourth encoder in the student model, and output the picture feature vector;

[0094] S40: constructing a loss function based on the target first fused feature vector, the target second fused feature vector, the sketch feature vector and the picture feature vector; and iteratively training the student model until the value of the loss function is minimized, thereby obtaining a trained student model;

[0095] Among them, the construction process of the teacher model pre-training loss function includes:

[0096] Based on the similarity between the first target fused feature vector and the second target fused feature vector, a pre-trained contrast loss function is constructed;

[0097] Classify the target first fused feature vector and the target second fused feature vector respectively, and output the predicted category of the teacher model for the sketch sample and the predicted category for the picture sample;

[0098] Based on the predicted category of the sketch sample by the teacher model and the true category of the sketch sample, a first pre-training classification loss function is constructed; based on the predicted category of the picture sample by the teacher model and the true category of the picture sample, a second pre-training classification loss function is constructed;

[0099] The pre-training contrast loss function, the first pre-training classification loss function and the second pre-training classification loss function are summed to obtain the teacher model pre-training loss function.

[0100] The sketch image retrieval method based on class-agent distillation provided in the present application designs an encoder for feature extraction and a cross-modal fusion module for feature fusion in the teacher model. The encoder is first used to extract the visual features and semantic features of the two modal samples, and then the semantic features of each modal sample are mapped. Since there are large differences in details and textures between the two modalities of sketch and picture, directly fusing the visual features and semantic features of a single modality cannot utilize the complementarity between the sketch and the picture, nor can it accurately capture the correlation between the sketch and the picture. Therefore, the cross-modal fusion module is used to fuse the mapped semantic features of the current modal sample with the visual features of the other modal sample to obtain a first fused feature including the semantic features of the sketch sample and the visual features of the picture sample, and a second fused feature including the semantic features of the picture sample and the visual features of the sketch sample. Finally, the first fused feature and The second fused features are mapped back to the sketch domain and the image domain respectively to obtain the target first fused features corresponding to the sketch samples and the target second fused features corresponding to the image samples; not only the visual features and semantic features of the samples are effectively utilized, but also the cross-fusion of the visual and semantic features of the two modalities can better align the similarities between the sketches and the images, so that the features output by the teacher model contain both the visual features and semantic features of the two modalities, and reduce the negative impact of the differences between the sketches and the images on the retrieval results; finally, a loss function is constructed based on the target first fused features, the target second fused features and the sketch features and image features extracted by the student model, and the student model is trained so that the student model can extract the visual features and semantic features of the samples under the guidance of the teacher model, thereby providing richer feature information for the matching of sketches and images and improving the accuracy of the retrieval results.

[0101] Optionally, before using the encoder to extract features from the sketch and the picture, the sketch and the picture may be preprocessed to improve the accuracy and efficiency of feature extraction. The preprocessing process specifically includes:

[0102] For the sketch sample, since there are a lot of blanks in the image, which will lead to the extraction of sparse features, the blanks in the image are removed, that is, the anchor frame of the sketch sample body is obtained according to the black pixels in the image, the tendencies outside the anchor frame are cropped, and then the sketch sample in the anchor frame is scaled to obtain ;in, and Indicates the height and width of the scaled image;

[0103] Scale the image sample to get ;

[0104] Finally and Convert it into a tensor and perform mean-variance normalization to obtain the preprocessed sketch samples and image samples.

[0105] like Figure 2 The figure shows a framework diagram of a sketch image retrieval method based on class agent distillation provided by the present application. Due to the huge modality gap between the sketch domain and the picture domain, it is necessary to assign an independent encoder to each modality. Specifically, in the teacher model, the first encoder is used to extract features of the sketch sample, and the second encoder is used to extract features of the picture sample. In the student model, the third encoder is used to extract features of the sketch sample, and the fourth encoder is used to extract features of the picture sample.

[0106] As a specific example of this application, a vision transformer (ViT) is used as the backbone network of the encoder. This is because the self-attention mechanism of ViT can effectively model the global relationship of the input image and capture complex long-range dependencies. Specifically, the specific process of the encoder extracting features from the input image includes:

[0107] For a given sketch sample And picture samples ,First, the patch embedding operation is performed to divide the sketch samples and the picture samples into patches;

[0108] Since the sketch sample is a frame diagram of the object, it is necessary to reflect effective information as a whole. If it is divided into local diagrams, the feature representation ability of the lines will be lost. Therefore, in the embodiment of the present application, for the sketch sample, a convolution stack is used to capture the relationship between the current local view and the surrounding views. Specifically, the convolution stack includes a convolution layer and a ReLu layer, and its formula is expressed as follows: ,in represents the convolution operation, represents the ReLu activation function, Indicates the number of convolutions;

[0109] Furthermore, ViT is used for feature encoding to obtain the visual features and semantic features of the sketch sample, as well as the visual features and semantic features of the image sample:

[0110]

[0111]

[0112] in, , The category tokens represent sketch samples and image samples respectively, which contain the information of all patches and can be regarded as semantic features; , The visual tokens representing the sketch samples and the image samples are obtained from each local view and can be regarded as visual features; represents the first encoder; represents the second encoder; Indicates a Patch embed operation.

[0113] Furthermore, for cross-modal retrieval tasks, cross-modal feature fusion can reduce the gap between modalities and promote feature alignment. However, if only the semantic features of the sketch sample and the picture sample are fused, some visual features will be lost, and although some attributes or features of the sketch sample and the picture sample may be semantically consistent, the visual features of the two will present different feature representations due to the specific modality; fusing the visual features of the sketch sample and the picture sample can reduce the impact of these factors, but since the sketch sample itself has deformation and deviates from the features of the picture domain, if the visual features of the sketch sample and the picture sample are directly fused, it is difficult to align the same semantic information of the two. Based on this, in the embodiment of the present application, a cross-modal fusion module is designed in the teacher model to obtain multimodal feature representation by combining the semantic features of the sketch sample and the visual features of the picture sample, as well as the semantic features of the picture sample and the visual features of the sketch sample.

[0114] Specifically, Figure 2 As shown in , the semantic feature representation of the sketch sample , first use the first mapping layer to map it to the image domain to obtain ,use and visual feature representation of image samples Do self-attention, get the attention score, and use the attention score to and Weighted fusion is performed to fuse the visual features of the image sample into the semantic features of the sketch sample to enhance the object features related to the retrieval target and reduce the negative impact of the abstract sketch; then, the fused features are mapped back to the sketch domain through the second mapping layer to obtain the target first fused feature vector corresponding to the sketch sample . For the semantic feature representation of the image sample , first use the third mapping layer to map it to the sketch domain to obtain ,use and visual feature representation of sketch samples Do self-attention, get the attention score, and use the attention score to and Weighted fusion is performed to fuse the visual features of the sketch sample into the semantic features of the image sample; then, the fused features are mapped back to the image domain through the fourth mapping layer to obtain the target second fused feature vector corresponding to the image sample .

[0115] In the zero-shot sketch image retrieval task, it is necessary to enhance the model's ability to generalize the knowledge learned from the visible class to the unseen class. In the prior art, a loss function is constructed by the similarity of the feature representations extracted by the two models, so that the feature representation extracted by the student model is as similar as possible to the teacher model. However, this method involves multi-level internal features and requires mining a large number of intrinsic category differences. However, the number of sketch samples and image samples is limited. Therefore, a distillation method based on soft labels is proposed in an embodiment of the present application to avoid processing complex multi-layer features, such as Figure 3 Shown is a schematic diagram of the knowledge distillation principle of the teacher model and the student model provided in this application;

[0116] Specifically, in step S40, based on the target first fused feature vector, the target second fused feature vector, the sketch feature vector and the picture feature vector, constructing the loss function includes:

[0117] S400: Mapping the target first fused feature vector and the target second fused feature vector respectively to obtain the output probability distribution of the teacher model for the sketch sample and the output probability distribution of the teacher model for the picture sample;

[0118] S401: Mapping the sketch feature vector and the picture feature vector respectively to obtain the output probability distribution of the student model for the sketch sample and the output probability distribution for the picture sample;

[0119] S402: constructing a first knowledge distillation loss function based on the similarity between the output probability distribution of the teacher model for the sketch sample and the output probability distribution of the student model for the sketch sample; constructing a second knowledge distillation loss function based on the similarity between the output probability distribution of the teacher model for the picture sample and the output probability distribution of the student model for the picture sample;

[0120] By using a linear layer of a specific modality to map the feature vector output by the teacher model and the feature vector output by the student model, the corresponding proxy is obtained. , that is, the category probability output by the teacher model for the sketch sample or the picture sample, and the category probability output by the student model for the sketch sample or the picture sample, and use the modality-aware proxy similarity distillation loss to make the category probability output by the student model for the same sample as similar as possible to that of the teacher model;

[0121] Specifically, the first knowledge distillation loss function is expressed as:

[0122] ,

[0123] in, represents the first knowledge distillation loss function; represents the output probability distribution of the student model for the sketch sample; represents the output probability distribution of the teacher model for the sketch sample; Indicates the distillation temperature; Represents a dynamic queue for expanding the number of sketch sample comparisons; Represents a dynamic queue Comparison samples in ;

[0124] ,

[0125] ,

[0126] in, represents normalization; represents a linear layer; represents the sketch feature vector; represents the first fused feature vector of the target; Factor representing the smoothed probability distribution;

[0127] The second knowledge distillation loss function is expressed as:

[0128] ,

[0129] in, represents the second knowledge distillation loss function; Represents the output probability distribution of the student model for the image sample; Represents the output probability distribution of the teacher model for the image sample; Represents a dynamic queue for expanding the number of image sample comparisons; Represents a dynamic queue Comparison samples in ;

[0130] ,

[0131] ,

[0132] in, Represents the image feature vector; Represents the target second fused feature vector.

[0133] S403: classifying the sketch feature vector and the picture feature vector respectively, and outputting the predicted category of the sketch sample and the predicted category of the picture sample by the student model;

[0134] S404: constructing a first classification loss function based on the predicted category of the sketch sample by the student model and the true category of the sketch sample; constructing a second classification loss function based on the predicted category of the picture sample by the student model and the true category of the picture sample;

[0135] In order to make the category probability distribution implied by the features output by the student model reach a stable state, in addition to the knowledge distillation loss, the classification loss is also introduced in the embodiment of the present application to constrain the output of the student model;

[0136] Specifically, the first classification loss function is expressed as:

[0137] ,

[0138] in, represents the first classification loss function; Indicates the number of categories; Indicates that the sketch sample belongs to The true probability of each category; Indicates that the sketch sample output by the student model belongs to The predicted probability of each category;

[0139] The second classification loss function is expressed as:

[0140] ,

[0141] in, represents the second classification loss function; Indicates that the image sample belongs to The true probability of each category; Indicates that the image sample output by the student model belongs to The predicted probability of each class.

[0142] S405: summing the first knowledge distillation loss function, the second knowledge distillation loss function, the first classification loss function, and the second classification loss function to obtain a loss function;

[0143] The embodiment of the present application uses the category confidence integrated with multiple modalities to construct a loss function, thereby helping the student model to establish the promotion of the visible class to the invisible class in two modalities. Specifically, the loss function is expressed as:

[0144] ,

[0145] in, represents the loss function; represents the first knowledge distillation loss function; represents the second knowledge distillation loss function.

[0146] Optionally, in some embodiments of the present application, since sketch samples usually have structural information of objects, while image samples contain richer information such as color, texture, and background, this difference makes it difficult to match the same object from the two domains. Therefore, in order to retrieve the correct image, the feature representation of the positive sample image should be close to the feature representation of the sketch query. Triple loss is a distance metric learning method that minimizes the distance between sketches and similar images and pushes different types of images away. In order to effectively utilize computing resources, an embodiment of the present application proposes an improved version of triple loss, namely quadruple loss, which uses negative sample sketches to perform modal alignment on the features extracted by the student model.

[0147] At the same time, directly using the four-reconstruction loss may cause the student model to ignore similarities between seen categories and unseen categories. For example, horses and zebras should be closer than horses and cars. In addition, there may be visual deviations between certain instances and samples of the same category. In order to reduce the impact of individuality on commonality, samples of the same category can be clustered. However, due to the existence of modal gaps, the feature representations of sketches and pictures are difficult to completely overlap in the feature space. Therefore, the embodiment of the present application performs intra-modal clustering on samples of each category, and constructs a center alignment loss function based on the distance between the two modal clustering centers of each category. By shortening the distance between the two modal clustering centers of the same category, cross-modal soft alignment at the category level is achieved, so that the student model can extract more comprehensive, consistent and rich feature information, thereby improving the performance and generalization ability of the model.

[0148] Specifically, after step S405, the following steps are further included:

[0149] S406: For any sketch sample in the training set, construct a quadruple loss function based on the Euclidean distance between the sketch feature vector of the sketch sample and the picture feature vector of the picture sample corresponding to the sketch sample, the Euclidean distance between the sketch feature vector of the sketch sample and the picture feature vector of the target picture sample, and the Euclidean distance between the sketch feature vector of the sketch sample and the sketch feature vector of the target sketch sample; wherein the target picture sample is a picture sample of the same category as the sketch sample in the training set, and the target sketch sample is a sketch sample of the same category as the sketch sample in the training set;

[0150] Specifically, the quadruple loss function is expressed as:

[0151] ,

[0152] in, represents the quadruple loss function; Indicates the number of target image samples or target sketch samples in the training set; Indicates boundary value; The Euclidean distance between a sketch feature vector of a sketch sample and a picture feature vector of a picture sample corresponding to the sketch sample; The sketch feature vector representing the sketch sample is The Euclidean distance between the image feature vectors of target image samples; The sketch feature vector representing the sketch sample is The Euclidean distance between the sketch feature vectors of target sketch samples;

[0153] A quadruple loss function is used to ensure that the distance between sketch samples and picture samples of the same category extracted by the student model is smaller than the distance between sketch samples and picture samples of different categories.

[0154] S407: clustering sketch samples and picture samples of the same category in the training set respectively to obtain sketch cluster centers and picture cluster centers of the current category; constructing a center alignment loss function based on the Euclidean distances between sketch cluster centers and picture cluster centers of each category;

[0155] Specifically, we create centers on the sketch domain and the image domain for each category. , the center of each category is iteratively updated using momentum update. Specifically, the update formula of the sketch cluster center of each category is expressed as:

[0156] ,

[0157] ,

[0158] in, represents the momentum parameter; express Sketch cluster centers of categories; is a mapper, which contains a linear layer that transforms the sketch samples The feature representation is mapped into the clustering space; express The number of sketch samples of the category;

[0159] The update formula of the image cluster center of each category is expressed as:

[0160] ,

[0161] ,

[0162] in, express The image cluster center of the category; is a mapper, which contains a linear layer that transforms the image samples The feature representation is mapped into the clustering space; express The number of image samples of the category;

[0163] Furthermore, the center alignment loss function is expressed as:

[0164] ,

[0165] in, represents the center alignment loss function; Indicates the number of categories; Indicates The sketch cluster centers of each category; Indicates The cluster centers of the images of each category;

[0166] S408: performing weighted summation on the loss function, the quadruple loss function and the center alignment loss function to obtain a target loss function;

[0167] Specifically, the objective loss function is expressed as:

[0168] ,

[0169] in, represents the target loss function; represents the weight of the center alignment loss function; Represents the weight of the loss function.

[0170] Specifically, since the feature representation output by the teacher model is used to guide the student model to learn, the student model can extract rich feature information similar to that of the teacher model through only a simple feature extraction network. Therefore, before training the student model, the teacher model and the reverse pre-training are required to ensure the accuracy of the feature information extracted by the teacher model;

[0171] Specifically, the pre-training process of the teacher model includes:

[0172] Obtain sketch samples in the training set and image samples corresponding to the sketch samples;

[0173] Input the sketch sample into the first encoder in the teacher model, and output a first semantic feature vector and a first visual feature vector; input the picture sample into the second encoder in the teacher model, and output a second semantic feature vector and a second visual feature vector;

[0174] Inputting the first semantic feature vector into the first mapping layer in the teacher model, inputting the output of the first mapping layer and the second visual feature vector into the first cross-modal fusion module, outputting a first fused feature vector, inputting the first fused feature vector into the second mapping layer in the teacher model, and outputting a target first fused feature vector;

[0175] Input the second semantic feature vector into the third mapping layer in the teacher model, input the output of the third mapping layer and the first visual feature vector into the second cross-modal fusion module, output the second fused feature vector, input the second fused feature vector into the fourth mapping layer in the teacher model, and output the target second fused feature vector;

[0176] Based on the target first fused feature vector and the target second fused feature vector, a teacher model pre-training loss function is constructed; the teacher model is iteratively trained using sketch samples and image samples in the training set until the value of the teacher model pre-training loss function is minimized, thereby obtaining a pre-trained teacher model.

[0177] Specifically, constructing the teacher model pre-training loss function includes:

[0178] Constructing a pre-training contrast loss function based on the similarity between the target first fused feature vector and the target second fused feature vector;

[0179] Since the purpose of modality fusion is to make sketches and pictures of the same category have similar feature representations, ideally, the fused sketch features and picture features should be similar. Therefore, the embodiment of the present application trains the teacher model by constructing a contrast loss function; specifically, the pre-training contrast loss function is expressed as:

[0180] ,

[0181] in, represents the pre-training contrast loss function; represents the first fused feature vector of the target; Represents the target second fused feature vector; Represents a dynamic queue for expanding the number of sketch sample comparisons; Represents a dynamic queue Comparison samples in ; represents the temperature control embedding distribution coefficient; Represents a dynamic queue for expanding the number of image sample comparisons; Represents a dynamic queue Comparison samples in ;

[0182] Classify the target first fused feature vector and the target second fused feature vector respectively, and output the predicted category of the teacher model for the sketch sample and the predicted category for the picture sample;

[0183] Based on the predicted category of the sketch sample by the teacher model and the true category of the sketch sample, a first pre-training classification loss function is constructed; based on the predicted category of the picture sample by the teacher model and the true category of the picture sample, a second pre-training classification loss function is constructed;

[0184] Based on the target first fused feature vector of the sketch sample and the target second fused feature vector of the picture sample extracted by the teacher model, the sketch sample and the picture sample should be accurately classified. Therefore, the embodiment of the present application trains the teacher model by constructing a classification loss; specifically, the first pre-training classification loss function is expressed as:

[0185] ,

[0186] in, represents the first pre-training classification loss function; Indicates the number of categories; Indicates that the sketch sample belongs to The true probability of each category; Indicates that the sketch sample output by the teacher model belongs to The predicted probability of each category;

[0187] The second pre-training classification loss function is expressed as:

[0188] ,

[0189] in, represents the second pre-training classification loss function; Indicates that the image sample belongs to The true probability of each category; Indicates that the picture sample output by the teacher model belongs to The predicted probability of each category;

[0190] The pre-training loss function, the first pre-training classification loss function and the second pre-training classification loss function are summed to obtain the teacher model pre-training loss function;

[0191] Specifically, the teacher model pre-training loss function is expressed as:

[0192] ,

[0193] in, Represents the teacher model pre-training loss function.

[0194] Specifically, the steps of using the trained student model to retrieve sketch images include:

[0195] Input the sketch to be matched into the third encoder in the trained student model, and output the sketch feature vector;

[0196] Input the pictures in the database into the fourth encoder in the trained student model respectively, and output the picture feature vector of each picture;

[0197] Classify the sketch feature vector to obtain the category of the sketch to be matched; classify the image feature vectors of each image to obtain the category of each image;

[0198] A picture of the same category as the sketch to be matched is used as the target picture of the sketch to be matched.

[0199] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0200] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0201] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0202] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1The steps for the functions specified in one or more boxes.

[0203] Obviously, the above embodiments are merely examples for the purpose of clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A zero-shot sketch image retrieval method based on class proxy distillation, characterized in that: include: Obtain sketch samples in a training set and image samples corresponding to the sketch samples; Input the sketch sample and the picture sample into a pre-trained teacher model for feature extraction and fusion, and output a target first fused feature vector and a target second fused feature vector; Specifically include: Input the sketch sample into a first encoder, and output a first semantic feature vector and a first visual feature vector; input the picture sample into a second encoder, and output a second semantic feature vector and a second visual feature vector; Input the first semantic feature vector into a first mapping layer, input the output of the first mapping layer and the second visual feature vector into a first cross-modal fusion module, and output a first fused feature vector; input the first fused feature vector into a second mapping layer, and output a target first fused feature vector; Input the second semantic feature vector into a third mapping layer, input the output of the third mapping layer and the first visual feature vector into a second cross-modal fusion module, and output a second fused feature vector; input the second fused feature vector into a fourth mapping layer, and output a target second fused feature vector; Input the sketch sample into the third encoder in the student model, and output a sketch feature vector; input the picture sample into the fourth encoder in the student model, and output a picture feature vector; Based on the first target fused feature vector, the second target fused feature vector, the sketch feature vector and the picture feature vector, a loss function is constructed; which specifically includes: Mapping the target first fused feature vector and the target second fused feature vector respectively to obtain an output probability distribution of the teacher model for the sketch sample and an output probability distribution of the teacher model for the picture sample; Mapping the sketch feature vector and the picture feature vector respectively to obtain an output probability distribution of the student model for the sketch sample and an output probability distribution of the picture sample; Based on the similarity between the output probability distribution of the teacher model for the sketch sample and the output probability distribution of the student model for the sketch sample, construct a first knowledge distillation loss function; based on the similarity between the output probability distribution of the teacher model for the picture sample and the output probability distribution of the student model for the picture sample, construct a second knowledge distillation loss function; Classifying the sketch feature vector and the picture feature vector respectively, and outputting the predicted category of the sketch sample and the predicted category of the picture sample by the student model; Based on the predicted category of the sketch sample by the student model and the true category of the sketch sample, a first classification loss function is constructed; based on the predicted category of the picture sample by the student model and the true category of the picture sample, a second classification loss function is constructed; Summing the first knowledge distillation loss function, the second knowledge distillation loss function, the first classification loss function, and the second classification loss function to obtain a loss function; The student model is iteratively trained until the value of the loss function is minimized, thereby obtaining a trained student model; wherein the step of using the trained student model to perform sketch image retrieval includes: Input the sketch to be matched into the third encoder in the trained student model, and output the sketch feature vector; Input the pictures in the database into the fourth encoder in the trained student model respectively, and output the picture feature vector of each picture; Classifying the sketch feature vector to obtain the category of the sketch to be matched; classifying the picture feature vectors of each picture to obtain the category of each picture; Using a picture of the same category as the sketch to be matched as a target picture of the sketch to be matched; Among them, the pre-training process of the teacher model includes: Obtain sketch samples in a training set and image samples corresponding to the sketch samples; Input the sketch sample into a first encoder in the teacher model, and output a first semantic feature vector and a first visual feature vector; input the picture sample into a second encoder in the teacher model, and output a second semantic feature vector and a second visual feature vector; Input the first semantic feature vector into a first mapping layer in a teacher model, input the output of the first mapping layer and the second visual feature vector into a first cross-modal fusion module, output a first fused feature vector, input the first fused feature vector into a second mapping layer in a teacher model, and output a target first fused feature vector; Input the second semantic feature vector into the third mapping layer in the teacher model, input the output of the third mapping layer and the first visual feature vector into the second cross-modal fusion module, output a second fused feature vector, input the second fused feature vector into the fourth mapping layer in the teacher model, and output a target second fused feature vector; Based on the target first fused feature vector and the target second fused feature vector, a teacher model pre-training loss function is constructed; the teacher model is iteratively trained using sketch samples and picture samples in the training set until the value of the teacher model pre-training loss function is minimized, thereby obtaining a pre-trained teacher model; wherein the process of constructing the teacher model pre-training loss function includes: Based on the similarity between the first target fused feature vector and the second target fused feature vector, a pre-trained contrast loss function is constructed; Classify the target first fused feature vector and the target second fused feature vector respectively, and output the predicted category of the teacher model for the sketch sample and the predicted category for the picture sample; Based on the predicted category of the sketch sample by the teacher model and the true category of the sketch sample, a first pre-training classification loss function is constructed; based on the predicted category of the picture sample by the teacher model and the true category of the picture sample, a second pre-training classification loss function is constructed; the pre-training contrast loss function, the first pre-training classification loss function and the second pre-training classification loss function are summed to obtain the teacher model pre-training loss function.

2. The zero-shot sketch image retrieval method based on class proxy distillation according to claim 1, characterized in that: The first knowledge distillation loss function is expressed as: ， in, represents the first knowledge distillation loss function; represents the output probability distribution of the student model for the sketch sample; represents the output probability distribution of the teacher model for the sketch sample; Indicates the distillation temperature; Represents a dynamic queue for expanding the number of sketch sample comparisons; Represents a dynamic queue Comparison samples in ; ， ， in, represents normalization; represents a linear layer; represents the sketch feature vector; represents the first fused feature vector of the target; Factor representing the smoothed probability distribution; The second knowledge distillation loss function is expressed as: ， in, represents the second knowledge distillation loss function; Represents the output probability distribution of the student model for the image sample; Represents the output probability distribution of the teacher model for the image sample; Represents a dynamic queue for expanding the number of image sample comparisons; Represents a dynamic queue Comparison samples in ; ， ， in, Represents the image feature vector; Represents the target second fused feature vector.

3. The zero-shot sketch image retrieval method based on class proxy distillation according to claim 1, characterized in that: The first classification loss function is expressed as: ， in, represents the first classification loss function; Indicates the number of categories; Indicates that the sketch sample belongs to The true probability of each category; Indicates that the sketch sample output by the student model belongs to The predicted probability of each category; The second classification loss function is expressed as: ， in, represents the second classification loss function; Indicates that the image sample belongs to The true probability of each category; Indicates that the image sample output by the student model belongs to The predicted probability of each category; The loss function is expressed as: ， in, represents the loss function; represents the first knowledge distillation loss function; represents the second knowledge distillation loss function.

4. The zero-shot sketch image retrieval method based on class proxy distillation according to claim 1, characterized in that: After obtaining the loss function, it also includes: For any sketch sample in the training set, a quadruple loss function is constructed based on the Euclidean distance between a sketch feature vector of the sketch sample and a picture feature vector of a picture sample corresponding to the sketch sample, the Euclidean distance between the sketch feature vector of the sketch sample and a picture feature vector of a target picture sample, and the Euclidean distance between the sketch feature vector of the sketch sample and a sketch feature vector of a target sketch sample; wherein the target picture sample is a picture sample of the same category as the sketch sample in the training set, and the target sketch sample is a sketch sample of the same category as the sketch sample in the training set; Cluster the sketch samples and picture samples of the same category in the training set respectively to obtain the sketch cluster center and picture cluster center of the current category; construct the center alignment loss function based on the Euclidean distance between the sketch cluster center and the picture cluster center of each category; The loss function, the quadruple loss function and the center alignment loss function are weightedly summed to obtain a target loss function.

5. The zero-shot sketch image retrieval method based on class proxy distillation according to claim 4, characterized in that: The quadruple loss function is expressed as: ， in, represents the quadruple loss function; Indicates the number of target image samples or target sketch samples in the training set; Indicates boundary value; The Euclidean distance between a sketch feature vector of a sketch sample and a picture feature vector of a picture sample corresponding to the sketch sample; The sketch feature vector representing the sketch sample is The Euclidean distance between the image feature vectors of target image samples; The sketch feature vector representing the sketch sample is The Euclidean distance between the sketch feature vectors of target sketch samples; The center alignment loss function is expressed as: ， in, represents the center alignment loss function; Indicates the number of categories; Indicates The sketch cluster centers of each category; Indicates The cluster centers of the images of each category; The objective loss function is expressed as: ， in, represents the target loss function; represents the weight of the center alignment loss function; Represents the weight of the loss function; Represents the loss function.

6. The zero-shot sketch image retrieval method based on class proxy distillation according to claim 1, characterized in that: The pre-training contrast loss function is expressed as: ， in, represents the pre-training contrast loss function; represents the first fused feature vector of the target; represents the second fused feature vector of the target; Represents a dynamic queue for expanding the number of sketch sample comparisons; Represents a dynamic queue Comparison samples in ; represents the temperature control embedding distribution coefficient; Represents a dynamic queue for expanding the number of image sample comparisons; Represents a dynamic queue Comparison samples in ; The first pre-training classification loss function is expressed as: ， in, represents the first pre-training classification loss function; Indicates the number of categories; Indicates that the sketch sample belongs to The true probability of each category; Indicates that the sketch sample output by the teacher model belongs to The predicted probability of each category; The second pre-training classification loss function is expressed as: ， in, represents the second pre-training classification loss function; Indicates that the image sample belongs to The true probability of each category; Indicates that the picture sample output by the teacher model belongs to The predicted probability of each category; The teacher model pre-training loss function is expressed as: ， in, Represents the teacher model pre-training loss function.

Images