Image retrieval method, device, equipment and storage medium based on sensory features

Abstract

The present invention discloses a sensory feature-based image retrieval method, apparatus, device, and storage medium, wherein the method includes the following steps: optimizing a visual Transformer model based on the sensory features and non-sensory features of RGB images in a training set; extracting features from a collected sketch using the visual Transformer model to obtain features of the sketch; extracting sensory features from each RGB image in a test set using the visual Transformer model, wherein the sensory features include structural features and texture features; and comparing the features of the sketch with the sensory features of each image in the test set to retrieve images that match the sketch. This application can improve image retrieval accuracy and reduce the difference between the sketch modality and the image modality, thereby effectively alleviating the negative impact of modality differences on training.

Description

Technical Field

[0001] The present invention relates to the field of image retrieval technology, and in particular to a sensory feature-based image retrieval method, device, equipment and storage medium. Background Art

[0002] Sketch-based image retrieval (SBIR) aims to retrieve images that are most similar to an input sketch from a color image database [1]. Zero-shot sketch image retrieval (Zero-shot SBIR) builds on sketch image retrieval technology by requiring that the categories of the test samples and the training samples are unique, making training more difficult [2]. However, in real-world applications, it is often difficult to cover all possible query categories during the training phase. Therefore, zero-shot sketch image retrieval technology has higher practical application value and commercial value.

[0003] Traditional methods currently extract the structural features of objects in real photos, matching the structure of sketches with that of real photos to achieve the goal of retrieving real photos based on sketches. It should be noted that traditional methods only focus on the structural features of objects and ignore texture features, which can lead to large errors in retrieval. For example, it is difficult to distinguish between objects with similar structures but different textures, such as horses and zebras. Another example is that hot air balloons and pizzas have similar structures but different textures.

[0004] Therefore, how to improve the retrieval accuracy of images is a technical problem that needs to be solved urgently. Summary of the Invention

[0005] The main purpose of the present invention is to provide an image retrieval method, apparatus, device and storage medium based on sensory features, which can improve the retrieval accuracy of images and reduce the difference between sketch modality and image modality, thereby effectively alleviating the negative impact of modality differences on training.

[0006] In a first aspect, the present application provides an image retrieval method based on sensory features, wherein the method comprises the steps of:

[0007] Optimize the visual Transformer model based on the sensory and non-sensory features of the RGB images in the training set;

[0008] Using the visual Transformer model, feature extraction is performed on the collected sketch to obtain features of the sketch;

[0009] Extracting sensory features of each RGB image in the test set using the visual Transformer model, wherein the sensory features include: structural features and texture features;

[0010] The features of the sketch are compared with the sensory features of each image in the test set to retrieve images that match the sketch.

[0011] In combination with the first aspect above, as an optional implementation method, the adversarial decoupling method CannyAD is used to extract the contour map corresponding to each image in the training set;

[0012] Input each image and its corresponding contour map into the visual Transformer model to obtain the features of the image and the features of the contour map corresponding to the image;

[0013] Inputting the features of the image and the features of the contour map corresponding to the image into the attention model to obtain the sensory features of the image;

[0014] Decoupling the image to obtain non-sensory features of the image;

[0015] According to the sensory features and the non-sensory features, adversarial features are obtained, and the visual Transformer model is optimized based on the adversarial features.

[0016] In combination with the first aspect above, as an optional implementation method, according to the formula: f ps =E(α×V(f p ))+f p , calculate the sensory features of the image, where f ps is the sensory feature of the image, E(.) represents a fully connected layer and a dropout operation, α is the mask matrix composed of the image and the corresponding contour map of the image, and V(.) represents the weight W v Perform linear transformation operation, f p is the extracted image features;

[0017] According to the formula: f pn =f p -f ps , calculate the non-sensory features of the image;

[0018] According to the formula: f' p =f ps +γf pn , calculate the adversarial feature, where γ is the adversarial coefficient, and the adversarial feature is a high-dimensional vector, which is used to enhance the robustness of the sensory features of the image;

[0019] According to the formula: Calculate the loss function of the adversarial feature, where CE(.) and Represent the cross entropy loss function and the fully connected layer respectively, and N1 represents the number of categories of objects in the training data;

[0020] Optimizing the visual Transformer model using adversarial feature loss functions.

[0021] In combination with the first aspect above, as an optional implementation, the attention model is: in Represents the features of the i-th real photo in a batch of data, Represents the features of the jth contour map in a batch of data. S(.) represents the similarity between two features. and Represents the weight matrix W used in the attention module k and the weight matrix W q right and Perform linear transformation operations.

[0022] In combination with the first aspect above, as an optional implementation method, representative modality prototypes are synthesized by averaging the sketch modality and image modality of the same type, wherein the modality prototypes include: a sketch modality prototype and an image modality prototype, each prototype representing a category;

[0023] The modality prototype is optimized using a contrastive loss function to maximize the similarity between sketches and images of the same category and minimize the similarity between sketches and images of different categories.

[0024] In combination with the first aspect above, as an optional implementation, according to the formula: Optimizing the modal prototype, wherein is the contrast loss function, N1 represents the total number of categories in the training set, N b represents the number of categories in a batch, represents the sensory feature prototype of the jth category of the entire training set, represents the prototype of the sketch feature of the kth category in a batch of data, and τ represents a hyperparameter.

[0025] In combination with the first aspect above, as an optional implementation, based on the features of the sketch and the sensory features of each image in the test set, the Euclidean distance between the sketch and each image in the test set is calculated;

[0026] The pictures are sorted in order from near to far by Euclidean distance, and the pictures arranged before the preset critical point are selected as outputs matching the sketch according to the requirements.

[0027] In a second aspect, the present application provides an image retrieval device based on sensory features, the device comprising:

[0028] An optimization module, which is used to optimize the visual Transformer model based on the sensory and non-sensory features of the RGB images in the training set;

[0029] An extraction module, which is used to extract features from the collected sketch using the visual Transformer model to obtain features of the sketch;

[0030] The extraction module is further configured to extract sensory features of each RGB image in the test set using the visual Transformer model, wherein the sensory features include structural features and texture features;

[0031] A retrieval module is used to compare the features of the sketch with the sensory features of each image in a test set to retrieve images that match the sketch.

[0032] In a third aspect, the present application further provides an electronic device comprising: a processor; and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the method described in any one of the first aspects is implemented.

[0033] In a fourth aspect, the present application further provides a computer-readable storage medium storing computer program instructions, which, when executed by a computer, enables the computer to execute any one of the methods described in the first aspect.

[0034] The present application provides a sensory feature-based image retrieval method, apparatus, device, and storage medium, wherein the method includes the following steps: optimizing a visual Transformer model based on the sensory features and non-sensory features of RGB images in a training set; extracting features from a collected sketch using the visual Transformer model to obtain features of the sketch; extracting sensory features from each RGB image in a test set using the visual Transformer model, wherein the sensory features include structural features and texture features; and comparing the features of the sketch with the sensory features of each image in the test set to retrieve images that match the sketch. The present application can improve the retrieval accuracy of images, and can also reduce the difference between the sketch modality and the image modality, thereby effectively alleviating the negative impact of modality differences on training.

[0035] It should be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0037] Figure 1 Flowchart of the sensory feature-based image retrieval method, apparatus, device, and storage medium provided in the embodiments of the present application;

[0038] Figure 2 Schematic diagram of the sensory feature-based image retrieval device, apparatus, equipment, and storage medium provided in the embodiments of the present application;

[0039] Figure 3 This is a schematic diagram of the optimization of the visual Transformer model provided in the embodiments of this application;

[0040] Figure 4 A schematic diagram of an electronic device provided in an embodiment of the present application;

[0041] Figure 5 A schematic diagram of a computer-readable program medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, like numbers in different figures represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present invention, as detailed in the appended claims.

[0043] Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Some of the blocks shown in the drawings are functional entities that do not necessarily correspond to physically or logically separate entities.

[0044] The embodiments of the present application are further described in detail below with reference to the accompanying drawings.

[0045] Reference Figure 1 , Figure 1 The flowchart of the image retrieval method based on sensory features provided by the present invention is shown as follows: Figure 1 As shown, the method includes the steps of:

[0046] Step S101: Optimize the visual Transformer model based on the sensory features and non-sensory features of the RGB images in the training set.

[0047] Specifically, the adversarial decoupling method CannyAD is used to extract the contour map corresponding to each image in the training set; each image and its corresponding contour map are input into the visual Transformer model to obtain the features of the image and the features of the contour map corresponding to the image; the features of the image and the features of the contour map corresponding to the image are input into the attention model to obtain the sensory features of the image; the image is decoupled to obtain the non-sensory features of the image; adversarial features are obtained based on the sensory features and non-sensory features, and the visual Transformer model is optimized based on the adversarial features.

[0048] Optional, according to the formula: f ps =E(α×V(f p ))+f p , calculate the sensory features of the image, where f ps is the sensory feature of the image, E(.) represents a fully connected layer and a dropout operation, α is the mask matrix composed of the image and the corresponding contour map of the image, and V(.) represents the weight W v Perform linear transformation operation, f p is the extracted image feature; according to the formula: f pn =f p -f ps , calculate the non-sensory features of the image; according to the formula: f' p =f ps +γf pn , calculate the adversarial feature, where γ is the adversarial coefficient. The adversarial feature is a high-dimensional vector that is used to enhance the robustness of the sensory features of the image; according to the formula: Calculate the loss function of the adversarial feature, where CE(.) and They represent the cross entropy loss function and the fully connected layer respectively, N1 represents the number of object categories in the training data; the loss function of the adversarial feature is used to optimize the visual Transformer model.

[0049] For example, the areas with drastic pixel changes in the image are more likely to contain the structural features and significant texture features of the object. The mathematical principle of the Canny operator is to calculate the maximum value of the grayscale gradient. Therefore, this project uses the Canny operator to extract the corresponding contour map for each real picture. These contour maps will serve as clues for mining structural and texture features, and compare them with real pictures and sketches. Figure 1 For the convenience of description, this project defines the union of structural features and texture features as sensory features.

[0050] In order to decouple sensory features and non-sensory features from photo features, this paper introduces an attention module to assist the decoupling process.p , a sketch x s and a contour map x c First, input them into the encoder of the visual transformer at the same time to obtain the features f corresponding to the three modalities p ∈R M×C , f s ∈R M×D and f c ∈R M×D . Where M represents the number of image blocks and D represents the dimension of the feature. Then f p and f c As the input of the attention module, a mask matrix is learned to construct the relationship between the two modalities. The matrix element α i,j represents the i-th row and j-th column element of the mask matrix α, which is defined as: in Represents the characteristics of the i-th real photo in a batch of data. Similarly Represents the features of the jth contour map in a batch of data. S(.) represents the similarity between two features. and Represents the weight matrix W used in the attention module k and the weight matrix W q right and Perform linear transformation operation. Sensory feature f ps ∈R M×D and non-sensory characteristics f pn ∈R M×D The definition of f is: ps =E(α×V(f p ))+f p , f pn =f p -f ps , where E(.) represents a fully connected layer and a dropout operation, α is a mask matrix. V(.) represents the weight W v Perform linear transformation operations. A residual connection operation is used in the last layer of the attention module to avoid the gradient disappearance problem.

[0051] After obtaining the sensory characteristics f ps and non-sensory characteristics f pn Afterwards, the adversarial coefficient γ is used to generate adversarial features, adversarial features f' p The definition of f' p =f ps +γf pn , where the value of the adversarial coefficient γ changes during the training process and obeys the normal distribution: Where μ represents the expectation of the normal distribution, and the standard deviation σ is a hyperparameter used to control the variation of the noise. Finally, the adversarial decoupling loss function is constructed and defined as follows:

[0052] Among them, CE(.) and Denote the cross-entropy loss function and the fully connected layer, respectively, and N1 denotes the number of object categories in the training data. During training, the change of γ will destroy the robustness of non-sensory features, ensuring that the adversarial decoupling method can mine the most discriminative sensory features.

[0053] It should be noted that the anti-decoupling method (Canny-cued Adversarial Disentanglement, CannyAD), the attention model can be understood as the mask matrix α constructed by the RGB image and its corresponding contour map, f p and f c It can be understood as a high-dimensional vector, non-sensory features can be understood as features other than structural features and texture features, which can be other features or noise, and adversarial features can be understood as high-dimensional vectors, which are used to enhance the robustness of the sensory features of the image. In this application, the image can be understood as an RGB image.

[0054] It is understandable that CannyAD uses a visual transformer framework pre-trained on ImageNet to extract modal features. Visual transformers have a larger receptive field than convolutional neural networks. Secondly, because areas with drastic grayscale changes in the image (i.e., overall structure and detailed texture) will leave a strong impression on the painter, it is hoped that the model will pay as much attention to these pixels as possible to improve the model's ability to extract discriminative information. CannyAD uses the traditional canny algorithm to obtain contour maps corresponding to the photo modality. These contour maps will serve as auxiliary modalities to guide the model to decouple sensory features from photo features. In addition, the adversarial decoupling mechanism in the CannyAD method changes the size of non-sensory features during training and regards them as an adversarial interference, thereby enhancing the robustness of sensory features.

[0055] In one embodiment, representative modality prototypes are synthesized by averaging sketch modalities and image modalities of the same type, wherein the modality prototypes include: a sketch modality prototype and an image modality prototype, each prototype representing a category; the modality prototypes are optimized using a contrast loss function to maximize the similarity between sketches and images of the same category, and minimize the similarity between sketches and images of different categories.

[0056] For ease of understanding, we give an example. In the training set, data of the same category inevitably have individual differences. This invention hopes to eliminate such differences during the training phase, making it easier for the model to extract the most representative and stable features of each category. Each batch of data contains N b ×m×3 images (photos, sketches, and outlines), where N b , m, and 3 represent the number of object categories, the number of images per category, and the number of modalities in a batch of images, respectively. Each batch of data is randomly sampled from the training set. We average the features of each object of the same category to synthesize prototypes. The prototype synthesized using sensory features is defined as follows: where p ps ∈R M×D and y represent the prototype and category labels of sensory features, Represents the sensory features extracted from the i-th photo in a batch of data of the same category. P represents the prototype of the kth category. ps Represents the set of all category prototypes in a batch of data. The same aggregation method is used to synthesize the prototype set of the sketch modality

[0057] To ensure that the model can extract highly discriminative sensory features and eliminate modality differences, this paper designs a prototype contrast loss function. It uses a set of synthesized modality prototypes to maximize the similarity between the same categories of the two modalities and maximize the difference from other different categories. The prototype contrast function is defined as follows: Where N1 represents the total number of categories in the training set, N b Indicates the number of categories in a batch. represents the sensory feature prototype of the jth category of the entire training set, represents the prototype of the sketch feature of the kth category in a batch of data, τ represents a hyperparameter, and its value range is 0 to 1, which is set to 0.3 in the present invention.

[0058] It's understandable that modal prototypes can eliminate modal differences between images and sketches, resulting in higher similarity between samples of the same category and more distinct differences between different categories. This can also be considered an optimization of the visual Transformer model. The most representative prototypes are synthesized for the sketch and photo modalities, respectively. Each prototype represents a category, forming two sets of prototypes. These two sets of prototypes are optimized using a contrastive loss function, maximizing the similarity between samples of the same category and minimizing the similarity between samples of different categories, thereby narrowing the modal gap.

[0059] In one embodiment, the network is optimized simultaneously by constructing a classification loss and a triplet loss. It is the cross entropy loss function without label smoothing, and for a triplet {a, p, n}, the triplet loss function is defined as:

[0060] According to the formula: The total loss function of the entire network is calculated to optimize the global situation, where λ1 and λ2 are hyperparameters, both of which are set to 0.1 in the present invention.

[0061] Optionally, a Vision Transformer (ViT) network encoder is used as the feature extractor for the sketch, photo, and silhouette modalities, with all three modalities sharing a single encoder. Features from the photo and silhouette modalities are fed into an adversarial attention module, which uses the silhouette as a cue to decouple sensory features from the photo modality. The sensory and sketch features are used to synthesize two sets of modality prototypes, using a prototype contrast loss to reduce modality differences. The end-to-end feature extractor is jointly optimized using an adversarial decoupling loss, a prototype contrast loss, a classification loss, and a triplet loss.

[0062] Optionally, the TU-Berlin and Sketchy datasets are divided into training and test sets. For all RGB images in the dataset, the Canny algorithm is used to obtain their corresponding contour maps. The upper and lower thresholds in the Canny algorithm are optimized after multiple experiments. Training data is fed into the network: All sketches, photos, and contour maps are fed into a sensory feature-based sketch-photo retrieval model for training. Network optimization and parameter updates: A stochastic gradient descent optimizer with momentum is used to update network parameters. Forward propagation calculates various losses, and backpropagation updates network parameters. After multiple iterations, the final network model is obtained. Stochastic gradient descent selects a small batch of data at a time, rather than all samples, and uses gradient descent to update model parameters. With the addition of momentum, parameter updates subtract not only the gradient of the current iteration but also the weighted sum of the gradients of the previous iteration. In stochastic gradient descent with momentum, both the gradient of the current iteration and the accumulated gradient of previous iterations influence parameter updates.

[0063] Optionally, during the testing phase, network training and parameter updates are omitted. The trained model is used for inference to obtain the final retrieval results on the test set. Evaluation metrics include the average precision of the top 100 items (Prec@100) and the average retrieval precision (mAP@all), which reflect the probability of retrieval of correct cross-modal images and the average accuracy rate, respectively.

[0064] Alternatively, use the TU-Berlin and Sketchy datasets. TU-Berlin originally had 250 categories, containing 13,419 natural images and 20,000 sketches. This dataset was later expanded with 191,067 new natural images. Sketchy originally contained 75,471 sketches and 12,500 natural images in 125 categories. This dataset was later expanded with 60,502 more natural images.

[0065] Optionally, a ViT network is used to extract features of the three modalities. ViT is initialized using parameters pre-trained on ImageNet. Each batch of data includes 16 randomly selected categories, each category contains 4 photos, 4 sketches and 4 outlines. And each photo is scaled to 256×128 pixels. During the training phase, the present invention uses random cropping, padding and horizontal flipping, channel adaptation and other operations as data enhancement. In adversarial decoupling, the standard deviation σ was initially set to 0.025, and we adjusted it to 0.01 after experiments. The present invention uses a stochastic gradient descent optimizer to train the model for a total of 15 rounds, with an initial momentum of 0.9 and a decay weight of 1e-4. The learning rate is 0.008 and adjusted to 0.0003 after experiments. The Euclidean distance is used to measure the similarity between features in the testing phase.

[0066] Step S102: Using the visual Transformer model, feature extraction is performed on the collected sketch to obtain features of the sketch.

[0067] Specifically, the collected sketch is input into the optimized visual Transformer model to extract features from the sketch. It should be noted that the features of the sketch can be understood as high-dimensional vectors.

[0068] Step S103: Using the visual Transformer model to extract sensory features of each RGB image in the test set, wherein the sensory features include structural features and texture features.

[0069] Specifically, all RGB images in the test set are fed one by one into an optimized visual Transformer model to extract sensory features. Sensory features include structural and texture features. Taking horses and zebras as examples, the horse only includes structural features, while the zebra includes both structural and texture features. It can be understood that the structure of an object and the details of its texture are related. Structure represents the relationship between different parts of the image, while texture refers to the differences between them. This paper defines such structural and texture information as sensory features.

[0070] Step S104: Compare the features of the sketch with the sensory features of each image in the test set to retrieve images that match the sketch.

[0071] Specifically, based on the features of the sketch and the sensory features of each image in the test set, the Euclidean distance between the sketch and each image in the test set is calculated; the images are sorted in order from near to far according to the Euclidean distance, and the images arranged before the preset critical point are selected as the output matching the sketch according to the needs.

[0072] For easy understanding, we use the visual Transformer model to extract the features of the sketch. s , then the model extracts the sensory features f of the real pictures stored in the database ps , calculate f for each image ps With sketch feature f s The Euclidean distance D between Sort the images in ascending order according to the Euclidean distance D, and take the first K images as the search results, where K is an adjustable value determined according to user needs.

[0073] Reference Figure 2 , Figure 2 FIG. 1 is a schematic diagram of an image retrieval method based on sensory features provided by the present invention. Figure 2 As shown, the device includes:

[0074] Optimization module 201: It is used to optimize the visual Transformer model based on the sensory features and non-sensory features of the RGB images in the training set.

[0075] Extraction module 202: It is used to extract features from the collected sketch through the visual Transformer model to obtain features of the sketch.

[0076] The extraction module is further used to extract sensory features of each RGB image in the test set using the visual Transformer model, where the sensory features include structural features and texture features.

[0077] Retrieval module 203: It is used to compare the features of the sketch with the sensory features of each image in the test set to retrieve images that match the sketch.

[0078] Furthermore, in a possible implementation, the extraction module is further configured to extract a contour map corresponding to each image in the training set using the adversarial decoupling method CannyAD;

[0079] Input each image and its corresponding contour map into the visual Transformer model to obtain the features of the image and the features of the contour map corresponding to the image;

[0080] Inputting the features of the image and the features of the contour map corresponding to the image into the attention model to obtain the sensory features of the image;

[0081] Decoupling the image to obtain non-sensory features of the image;

[0082] The optimization module is further used to obtain adversarial features based on the sensory features and non-sensory features, and optimize the visual Transformer model based on the adversarial features.

[0083] Furthermore, in a possible implementation, according to the formula: ps =E(α×V(f p ))+f p , calculate the sensory features of the image, where f ps is the sensory feature of the image, E(.) represents a fully connected layer and a dropout operation, α is the mask matrix composed of the image and the corresponding contour map of the image, and V(.) represents the weight W v Perform linear transformation operation, f p is the extracted image features;

[0084] According to the formula: f pn =f p -f ps , calculate the non-sensory features of the image;

[0085] According to the formula: f' p =f ps +γf pn , calculate the adversarial feature, where γ is the adversarial coefficient, and the adversarial feature is a high-dimensional vector, which is used to enhance the robustness of the sensory features of the image;

[0086] According to the formula: Calculate the loss function of the adversarial feature, where CE(.) and Represent the cross entropy loss function and the fully connected layer respectively, and N1 represents the number of categories of objects in the training data;

[0087] Optimizing the visual Transformer model using adversarial feature loss functions.

[0088] Furthermore, in a possible implementation, the attention model is: in Represents the features of the i-th real photo in a batch of data, Represents the features of the jth contour map in a batch of data. S(.) represents the similarity between two features. and Represents the weight matrix W used in the attention module k and the weight matrix W q right and Perform linear transformation operations.

[0089] Furthermore, in a possible implementation, a synthesis module is further included, which is used to synthesize representative modality prototypes by averaging the sketch modality and image modality of the same type, respectively, wherein the modality prototypes include: a sketch modality prototype and an image modality prototype, each prototype representing a category;

[0090] The optimization module is further configured to optimize the modal prototype using a contrast loss function to maximize the similarity between sketches and images of the same category and minimize the similarity between sketches and images of different categories.

[0091] Furthermore, in a possible implementation, according to the formula: Optimizing the modal prototype, wherein is the contrast loss function, N1 represents the total number of categories in the training set, N b represents the number of categories in a batch, represents the sensory feature prototype of the jth category of the entire training set, represents the prototype of the sketch feature of the kth category in a batch of data, and τ represents a hyperparameter.

[0092] Furthermore, in a possible implementation, the retrieval module is further configured to calculate the Euclidean distance between the sketch and each image in the test set based on the features of the sketch and the sensory features of each image in the test set;

[0093] The pictures are sorted in order from near to far by Euclidean distance, and the pictures arranged before the preset critical point are selected as outputs matching the sketch according to the requirements.

[0094] Reference Figure 3 , Figure 3 The figure shows the optimization diagram of the visual Transformer model provided by the present invention. Figure 3 As shown:

[0095] The first step is to conduct adversarial decoupling. The CannyAD algorithm is used to extract the corresponding contour map for each real picture. These contour maps will serve as clues for mining structure and texture features, and compare them with the real picture (RGB image) and sketch. Figure 1The same image is input into the visual Transformer model, which extracts the features of the image and the corresponding contour map, i.e., the high-dimensional vector, and transforms f p and f c As the input of the attention module, a mask matrix is learned to construct the relationship between the two modalities. The mask matrix here can be understood as an attention model. Using the contour map, the sensory features f of the image are mined. ps , and then use the image feature f p Subtract the sensory features of the image to obtain the non-sensory features f of the image pn , then f pn Multiply by a resistance coefficient and combine with f ps Get the adversarial feature f' p , the adversarial features are optimized using the loss function, and the optimized features are used to optimize the visual Transformer model.

[0096] There are inevitably individual differences in the data of the same category in the training set. The present invention hopes to eliminate such differences in the training stage, so that the model can more easily extract the most representative and stable features of each category. Each batch of data contains N_b×m×3 images (photos, sketches, and outlines), where N_b, m, and 3 represent the number of object categories in a batch of images, the number of images contained in each category, and the number of modalities, respectively. Each batch of data is randomly sampled from the training set. We average and aggregate the features of each object in the same category to synthesize the prototype. Using the synthesized set of modal prototypes, we maximize the similarity between the same categories of the two modalities and maximize the difference from other different categories.

[0097] It is understandable that the purpose of prototype learning is to eliminate the modal differences between sketches and images, and effectively alleviate the negative impact of modal differences on training.

[0098] Refer to the following Figure 4 An electronic device 400 according to this embodiment of the present invention will be described. Figure 4 The electronic device 400 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0099] like Figure 4 As shown, electronic device 400 is implemented as a general-purpose computing device. Components of electronic device 400 may include, but are not limited to, at least one processing unit 410, at least one storage unit 420, and a bus 430 connecting various system components (including storage unit 420 and processing unit 410).

[0100] The storage unit stores program codes, which can be executed by the processing unit 410, so that the processing unit 410 performs the steps according to various exemplary embodiments of the present invention described in the above “Example Method” section of this specification.

[0101] The storage unit 420 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 421 and / or a cache memory unit 422 , and may further include a read-only memory unit (ROM) 423 .

[0102] The storage unit 420 may also include a program / utility 424 having a set (at least one) of program modules 425, such program modules 425 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0103] Bus 430 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0104] The electronic device 400 may also communicate with one or more external devices (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 400, and / or any device that enables the electronic device 400 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication may occur via an input / output (I / O) interface 450. Furthermore, the electronic device 400 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 460. As shown, the network adapter 460 communicates with other modules of the electronic device 400 via a bus 430. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 400, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0105] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0106] According to the solution of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above-mentioned method of this specification is stored. In some possible implementations, various aspects of the present invention may also be implemented in the form of a program product, which includes program code. When the program product is executed on a terminal device, the program code is used to cause the terminal device to perform the steps according to various exemplary embodiments of the present invention described in the "Exemplary Methods" section of this specification.

[0107] refer to Figure 5 As shown, a program product 500 for implementing the above method according to an embodiment of the present invention is described. The program product 500 may be a portable compact disc read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0108] The program product may be implemented in any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0109] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0110] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0111] The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, and the like, as well as conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0112] Furthermore, the above-described figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention and are not intended to be limiting. It is readily understood that the processes illustrated in the above-described figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0113] The foregoing is merely a list of specific embodiments of the present application, intended to enable those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the broadest scope consistent with the principles and novel features of the present application.

[0114] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

Claims

1. An image retrieval method based on sensory features, characterized in that: include: Optimize the visual Transformer model based on the sensory and non-sensory features of the RGB images in the training set; Using the visual Transformer model, feature extraction is performed on the collected sketch to obtain features of the sketch; Extracting sensory features of each RGB image in the test set using the visual Transformer model, wherein the sensory features include: structural features and texture features; comparing features of the sketch with sensory features of each image in a test set to retrieve images that match the sketch; Among them, the adversarial decoupling method CannyAD is used to extract the contour map corresponding to each image in the training set; Input each image and its corresponding contour map into the visual Transformer model to obtain the features of the image and the features of the contour map corresponding to the image; Inputting the features of the image and the features of the contour map corresponding to the image into the attention model to obtain the sensory features of the image; Decoupling the image to obtain non-sensory features of the image; Obtaining adversarial features according to the sensory features and the non-sensory features, and optimizing a visual Transformer model based on the adversarial features; According to the formula: f ps =E(α×V(f p ))+f p , calculate the sensory features of the image, where f ps is the sensory feature of the image, E(.) represents a fully connected layer and a dropout operation, α is the mask matrix composed of the image and the corresponding contour map of the image, and V(.) represents the weight W v Perform linear transformation operation, F p is the extracted image features; According to the formula: F pn =F p -F ps , calculate the non-sensory features of the image; According to the formula: F p , =f ps +γf pn , calculate the adversarial feature, where γ is the adversarial coefficient, and the adversarial feature is a high-dimensional vector, which is used to enhance the robustness of the sensory features of the image; According to the formula: Calculate the loss function of the adversarial feature, where CE(.) and Represent the cross entropy loss function and the fully connected layer respectively, and N1 represents the number of categories of objects in the training data; Optimizing the visual Transformer model using adversarial feature loss functions.

2. The method according to claim 1, characterized in that Also includes: The attention model is: in Represents the features of the i-th real photo in a batch of data, represents the feature of the jth contour map in a batch of data, S(.) represents the similarity between the two features, and Represents the weight matrix W used in the attention module k and the weight matrix W q right and Perform linear transformation operations.

3. The method according to claim 1, characterized in that Also includes: Representative modality prototypes are synthesized by averaging the sketch modality and image modality of the same type, where the modality prototypes include sketch modality prototypes and image modality prototypes, and each prototype represents a category; The modality prototype is optimized using a contrastive loss function to maximize the similarity between sketches and images of the same category and minimize the similarity between sketches and images of different categories.

4. The method according to claim 3, wherein: According to the formula: Optimizing the modal prototype, wherein is the contrast loss function, N1 represents the total number of categories in the training set, N b represents the number of categories in a batch, represents the sensory feature prototype of the jth category of the entire training set, represents the prototype of the sketch feature of the kth category in a batch of data, and τ represents a hyperparameter.

5. The method according to claim 1, wherein The step of comparing the features of the sketch with the sensory features of each image in a test set to retrieve an image matching the sketch includes: Calculating the Euclidean distance between the sketch and each image in the test set based on the features of the sketch and the sensory features of each image in the test set; The pictures are sorted in order from near to far by Euclidean distance, and the pictures arranged before the preset critical point are selected as outputs matching the sketch according to the requirements.

6. An image retrieval device based on sensory features, characterized in that: include: An optimization module, which is used to optimize the visual Transformer model based on the sensory and non-sensory features of the RGB images in the training set; An extraction module, which is used to extract features from the collected sketch using the visual Transformer model to obtain features of the sketch; The extraction module is further configured to extract sensory features of each RGB image in the test set using the visual Transformer model, wherein the sensory features include structural features and texture features; a retrieval module for comparing the features of the sketch with the sensory features of each image in a test set to retrieve an image that matches the sketch; The extraction module is also used to extract the contour map corresponding to each image in the training set using the adversarial decoupling method CannyAD; Input each image and its corresponding contour map into the visual Transformer model to obtain the features of the image and the features of the contour map corresponding to the image; Inputting the features of the image and the features of the contour map corresponding to the image into the attention model to obtain the sensory features of the image; Decoupling the image to obtain non-sensory features of the image; The optimization module is further configured to obtain adversarial features based on the sensory features and the non-sensory features, and optimize the visual Transformer model based on the adversarial features; According to the formula: f ps =E(α×V(f p ))+f p , calculate the sensory features of the image, where f ps is the sensory feature of the image, E(.) represents a fully connected layer and a dropout operation, α is the mask matrix composed of the image and the corresponding contour map of the image, and V(.) represents the weight W v Perform linear transformation operation, f p is the extracted image features; According to the formula: f pn =f p -f ps , calculate the non-sensory features of the image; According to the formula: f p , =f ps +γf pn , calculate the adversarial feature, where γ is the adversarial coefficient, and the adversarial feature is a high-dimensional vector, which is used to enhance the robustness of the sensory features of the image; According to the formula: Calculate the loss function of the adversarial feature, where CE(.) and Represent the cross entropy loss function and the fully connected layer respectively, and N1 represents the number of categories of objects in the training data; Optimizing the visual Transformer model using adversarial feature loss functions.

7. An electronic device, characterized in that: The electronic device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, characterized in that The computer program instructions are stored therein, and when the computer program instructions are executed by a computer, the computer is caused to execute the method according to any one of claims 1 to 5.

Images