Fine-grained image retrieval method based on compact representation modeling and semantic label guidance

Abstract

The present application relates to a fine-grained image retrieval method based on compact representation modeling and semantic label guidance, and belongs to the technical field of fine-grained image analysis, solves the problem that it is difficult to efficiently mine significant and compact intermediate features of images in the prior art, comprising: step S1: preparing a pre-processed image data set; step S2: constructing a feature extraction network to extract visual word vectors; step S3: generating a hash coding center; step S4: constructing a compact coding mapping unit; step S5: training the constructed feature extraction network and compact coding mapping unit to obtain a trained feature extraction network and compact coding mapping unit; step S6: fine-grained image retrieval, matching the obtained to-be-detected code with the codes in the retrieval code library, and the sample with the smallest distance is the final retrieval result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of fine-grained image analysis, and particularly relates to a fine-grained image retrieval method based on compact representation modeling and semantic label guidance. BACKGROUND

[0002] Fine-grained image retrieval task aims to accurately retrieve images belonging to fine-grained sub-categories rather than traditional general categories. This task has a wide range of application scenarios in real life, such as intelligent monitoring, intelligent retail, and smart transportation, and therefore has received more and more attention in the field of artificial intelligence technology.

[0003] Existing traditional fine-grained image retrieval work can be divided into fine-grained content-based image retrieval (FG-CBIR) and fine-grained sketch-based image retrieval (FG-SBIR) according to the modality of the query image. In the aspect of FG-CBIR, the selective convolutional descriptor aggregation (SCDA) method first attempts to use a pre-trained convolutional neural network (CNN) to select meaningful deep descriptors by locating salient objects, without unordered image labels or bounding box annotations. Subsequently, supervised methods are developed using metric learning and additional sub-modules customized for fine-grained objects, effectively breaking through the accuracy limit of unsupervised retrieval. The decorrelated global-aware centralized ranking loss (DGCRL) method eliminates the gap between inner product and Euclidean distance by adding a normalization scale layer at the training and testing stages to enhance intra-class separability and inter-class compactness. While FG-SBIR aims to use a retrieved sketch as a query to match specific image instances, existing methods usually design a joint embedding space for training to bridge the gap between sketch-image domains. The above fine-grained image retrieval methods all focus on establishing discriminative fine-grained representations, but ignore retrieval time and feature storage consumption, thus greatly hindering retrieval applications on large databases.

[0004] Learning-based fine-grained hashing coding can effectively solve the above problems, which maps the high-dimensional real-valued features of images into low-dimensional binary coding, so that the search results in the coding space are as close as possible to the results in the original space, thereby significantly reducing the storage cost and at the same time speeding up the retrieval process. The general hashing algorithm focuses the attention of the network on the global features of the object, however, with the emergence of large-scale fine-grained datasets, most general hashing methods are difficult to meet the needs of fine-grained image analysis tasks due to the lack of mining of fine-grained features of objects. Therefore, many hashing methods for fine-grained samples are kicked out to effectively solve the high computational cost while extracting fine-grained features in large-scale fine-grained datasets. The ExchNet method adds an additional module to extract local features to represent each part of the visual object, and fuses global and local features to generate unified binary coding. The SEMICON method proposes a suppression-enhancing mask based attention operation to maintain the relationship between different activation regions, and combines it with a two-step interactive channel transformation module to calculate the attention between different channels, to effectively capture the correlation between fine-grained parts, and finally generate binary coding for the task. The AGMH method groups and mines class-specific visual attributes, and embeds them into multiple descriptors, optimizes them through attention dispersion loss to generate comprehensive feature representation, and proposes step-by-step interactive external attention to mine key attributes in each descriptor, while building the correlation between fine-grained attributes and objects.

[0005] However, the above methods all use local attention mechanisms based on CNN to extract fine-grained discriminative visual features of images, which is difficult to efficiently mine significant and compact intermediate features of images, difficult to build long-distance dependencies between features in the backbone network, and difficult to learn fine-grained semantics in the hashing task head for distinguishing fine-grained categories. Therefore, there is a need in the art for an improved image retrieval method that overcomes the above deficiencies. SUMMARY

[0006] In view of the technical defects existing in the above fine-grained image retrieval method, the purpose of the present application is to propose a fine-grained image retrieval method based on compact representation modeling and semantic label guidance. The method is based on the ViT (Visual Transformer) backbone network, and two representation learning branches are subsequently divided, namely a global branch for learning target level features and a local branch for learning partial level features, and are further integrated to generate an overall compact representation. However, the traditional ViT is designed for general visual tasks, and it is difficult to explore and integrate more subtle visual regions. In order to overcome this limitation, a MUSTA module (Multi-Level Salient Token Aggregating) is constructed in the local level feature learning branch, according to the attention weights of the low-level and high-level visual word vectors in the backbone network to select the most significant visual word vectors, and further aggregate them to the last Transformer layer, thereby significantly enriching the multi-layer local visual information. In addition, a semantic-guided fine-grained binary coding (SFBC) module is established, which guides the generation of the encoding center by considering the similarity between the fine-grained semantic class labels, and weights the center loss function with the similarity, so that the model can pay more attention to the significant differences between the samples belonging to similar categories.

[0007] The fine-grained image retrieval method based on compact representation modeling and semantic label guidance according to the embodiment of the present application comprises the following steps:

[0008] Step S1, preparing an image dataset containing fine-grained categories, performing data preprocessing and data division to obtain a preprocessed image dataset;

[0009] Step S2, constructing a feature extraction network, including constructing a multi-layer visual Transformer-based feature extraction network as a backbone network for visual feature extraction of images in the preprocessed image dataset to obtain visual word vectors, the feature extraction network further includes a global level feature learning branch and a local level feature learning branch, wherein the local level feature learning branch includes a MUSTA module;

[0010] Step S3, generating a hash encoding center, constructing a bag-of-words model according to the image fine-grained label, for counting the frequency of each word appearing in the category label as the word frequency, calculating the semantic similarity matrix between all category labels according to the statistical word frequency, and finally generating an encoding center for each category with the semantic similarity matrix as the weight;

[0011] Step S4, constructing a compact coding mapping unit, inputting the visual word vector extracted via the feature extraction network into the constructed coding mapping unit, and using an activation function in the coding mapping unit to map the high-dimensional real value feature of the visual word vector to a low-dimensional binary code to obtain a compact representation of the image, and constructing a retrieval code library with the obtained compact representation of the image;

[0012] Step S5, training the constructed feature extraction network and compact coding mapping unit to obtain a trained feature extraction network and compact coding mapping unit;

[0013] Step S6, performing fine-grained image retrieval, inputting the image to be detected into the trained feature extraction network and compact coding mapping unit to obtain the high-dimensional feature of the image to be detected, mapping the high-dimensional feature of the image to be detected to a low-dimensional code as a detection code to be detected, and matching the obtained detection code to be detected with the codes in the retrieval code library. The sample with the smallest distance is the final retrieval result.

[0014] Optionally, the step S2 specifically comprises the following steps:

[0015] Step S2.1, constructing a feature extraction network, first constructing a multi-layer visual Transformer including a global level feature learning branch and a local level feature learning branch as a backbone network of the feature extraction network, wherein the local level feature learning branch includes a MUSTA module;

[0016] Step S2.2, inputting the pre-processed image data set, and using the constructed multi-layer visual Transformer Φ ViT (·) extracting a global level visual word vector sequence from the input image I in the pre-processed image data set:

[0017] T gl = Φ ViT (I);

[0018] Step S2.3, using the set MUSTA module in parallel to select a number of more significant visual word vectors from each attention head of each Transformer layer in all Transformer layers of the multi-layer visual Transformer except the last layer, and aggregate all significant visual word vectors selected from each layer Transformer layer except the last layer into a final input sequence Z′ L-1 , wherein L represents the total number of Transformer layers included in the multi-layer visual Transformer;

[0019] Step S2.4, inputting the aggregated final input sequence Z′ L-1The input is fed into the last Transformer layer in the local-level feature learning branch, and after processing, a sequence of local-level visual word vectors is obtained:

[0020]

[0021] Optionally, step S2.3 specifically includes the following steps:

[0022] Step S2.3.1: The total number of Transformer layers in the multi-layer visual Transformer is L layers, and the input of the last Transformer layer in the local level feature learning branch is Z′. L-1 The attention weights of the first L-1 Tranformer layers, each with M attention heads, are expressed as follows:

[0023]

[0024] Among them, A l This represents the attention weight of the l-th Transformer layer, where l represents the current Transformer layer number and 1 ≤ l ≤ L-1. A represents l The attention weight of the m-th attention head. express The Nth T The attention weights corresponding to the visual word vectors, where m represents the current attention head number of this Tranformer layer, and N... T This represents the number of non-categorical visual word vectors for each attention head;

[0025] Step S2.3.2: In the MUSTA module, the attention weights of the Lth Tranformer layer and preceding layers are multiplied layer by layer to obtain the fusion attention weights used to record the forward propagation of visual word vectors from the input layer to higher layers.

[0026]

[0027] Where a′ represents the fusion attention weight, w represents the Transformer layer number, and A w This represents the attention weight of the w-th Tranformer layer, where 1 ≤ w ≤ l;

[0028] Step S2.3.3: Select the attention weights corresponding to the category visual word vectors from the fused attention weights a′:

[0029]

[0030] And select the indices corresponding to the n' visual word vectors with the highest weight values ​​from each attention head:

[0031]

[0032] wherein v l represents the fusion attention weight corresponding to the category visual word vector in the l-th layer of the Transformer layer, represents the attention weight of the m-th attention head in v l , K l represents the index value corresponding to the visual word vector with the highest fusion attention weight value in the l-th layer of the Transformer layer, represents the visual word vector index value of the m-th attention head in K l , and the TopSelection(·) function represents the use of the max(·) function to filter the index corresponding to the highest attention weight;

[0033] Step S2.3.4, in the output sequence of each Transformer layer of the first L-1 layers of the Transformer layer, find the visual word vector corresponding to the filtered index value K l of the layer:

[0034] T s,l = T l [K l ]

[0035] wherein T l represents the output sequence of the l-th layer of the Transformer layer;

[0036] Step S2.3.4, aggregate the visual word vectors of the output sequences of the Transformer layers of the first L-1 layers of the Transformer layer obtained above and the category visual word vectors of the original input sequence of the L-th layer of the Transformer layer through the MUSTA module as the final input sequence of the last layer of the Transformer layer in the local level feature learning branch:

[0037]

[0038] wherein, represents the category visual word vector of the original input sequence of the L-th layer of the Transformer layer.

[0039] Optionally, the step S3 specifically comprises the following steps:

[0040] Step S3.1, generating a set of binary centers containing q bits for at least distance d between K subcategories:

[0041]

[0042] where k denotes the fine-grained class index, c k represents the hash code center of the k-th class, d is the minimum distance between any two representative centers obtained based on Gilbert-Varshamov bound theory, K represents the total number of fine-grained classes, and q represents the hash code length.

[0043] Step S3.2, constructing a class semantic similarity matrix, adopting a bag-of-words model containing w' words, and representing the K fine-grained labels as a feature matrix V based on word occurrence probability, and the similarity matrix is represented as:

[0044] S = V · V T ;

[0045] Based on the class similarity matrix S, the minimum distance d above is replaced by the fine-grained weighted distance constraint d i,j between two classes:

[0046] d i,j = (1 + S i,j ) · d;

[0047] Subsequently, the hash code centers of each fine-grained class are generated by solving the following optimization problem:

[0048]

[0049] s.t. ||c i -c j || H ≥ d i,j , 1≤i,j≤K, i≠j,

[0050] where ||·|| H is the Hamming distance, i and j represent the fine-grained class index, c K , c i , and c j represent the hash code centers of the K-th, i-th, and j-th classes, respectively, d i,j represents the fine-grained weighted distance constraint between the i-th and j-th classes, and S i,j represents the similarity of the i-th and j-th labels.

[0051] Optionally, the step S4 specifically comprises the following steps:

[0052] Step S4.1, obtaining global-level real-valued features x gl and local-level real-valued features x loc by applying global average pooling on the global-level visual word vector sequence T gl and the local-level visual word vector sequence T loc respectively.

[0053] Step S4.2, using a global-level linear mapping matrix W gl and a local-level linear mapping matrix W loc , respectively, map the high-dimensional global-level real-valued features x gl and the high-dimensional local-level real-valued features x loc to low-dimensional global-level real-valued features h gl and low-dimensional local-level real-valued features h loc ;

[0054] Step S4.3, map the low-dimensional global-level real-valued features h gl and the low-dimensional local-level real-valued features h loc to low-dimensional compact codes b = {b gl , b loc}, to obtain the compact representation of the image, wherein b gl represents the global-level low-dimensional compact code, and b loc represents the local-level low-dimensional compact code.

[0055] Step S4.4, store the low-dimensional compact codes corresponding to the samples of the images in the pre-processed image data set in each image gallery according to the image sample numbers, and construct an index set Ω to build a retrieval code library.

[0056] Optionally, the step S5 specifically comprises the following steps:

[0057] Step S5.1, embed the similarity matrix S into the calculation of the compact code b n of the nth sample of the kth class and the corresponding kth class hash code center c k , to obtain the similarity of the compact code and the corresponding hash code center:

[0058]

[0059] wherein Sim(·,·) represents the scaled cosine similarity calculation, P n,k represents the similarity of the compact code b n and the corresponding kth class hash code center c k , t represents the fine-grained class number, K represents the total number of fine-grained classes, S k,t represents the similarity of the kth class and the tth class label, and c t represents the hash code center of the tth class.

[0060] Step S5.2, establish a semantic similarity weighted center loss function:

[0061]

[0062] wherein Z n,kis an indicator factor, if the nth input image I n is compactly encoded b n belongs to the kth class, then Z n,k = 1, otherwise Z n,k = 0;

[0063] Step S5.3, establish the quantization loss and the hash code loss function:

[0064]

[0065] Wherein, λ1, λ2 represent loss term coefficients, u represents image sample number in the gallery, v represents query sample number, Ω represents index set of gallery samples in the training data, Γ represents index set of query samples in the training data, b u , b v Respectively, the low-dimensional hash code of sample u, v, h u Respectively, the low-dimensional real value code of sample u, L uv Respectively, the pair of sample supervision information, and L uv ∈{-1,+1} (Q×D) , Q represents the number of query samples, D represents the number of gallery samples, and the First term is the quantization loss of the real value low-dimensional feature vector h u Binary, and the second term is the general semantic reservation loss;

[0066] Step S5.4, combine the quantization loss and the hash code loss function With the semantic similarity weighted center loss function Finally, the total loss function of optimizing the whole network is obtained:

[0067]

[0068] Step S5.5 the final total loss function contains the loss of the feature extraction network and the compact encoding mapping unit, in the gradient back propagation process, the network parameters of the feature extraction network and the compact encoding mapping unit are updated simultaneously until the loss of the loss function converges, and the trained feature extraction network and compact encoding mapping unit are obtained.

[0069] Compared with the prior art, the fine-grained image retrieval method based on compact representation modeling and semantic label guidance provided by the embodiment of the application has at least the following beneficial effects.

[0070] 1. The fine-grained image retrieval method based on compact representation modeling and semantic label guidance provided by the application makes up for the defect that the existing algorithm does not sufficiently model the significant visual information, and expands the idea of fine-grained image retrieval.

[0071] 2. The image salient visual features are extracted by a Multi-Level Salient Token Aggregating (MUSTA) module, which enables the backbone network to capture rich fine-grained visual information and improves the image representation ability of the network.

[0072] 3. The similarity information between fine-grained class labels is integrated into the encoding center generation and center loss function by a Semantic-Guided Fine-Grained Binary Coding (SFBC) module, so that the encoding mapping layer can pull away the encodings of samples belonging to similar fine-grained classes, and reduce the difficulty of distinguishing difficult samples. BRIEF DESCRIPTION OF DRAWINGS

[0073] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced below. The features and advantages of the present application can be more clearly understood by referring to the drawings. The drawings are schematic and should not be understood as any limitation on the present application. For those skilled in the art, other drawings can be obtained without creative labor on the basis of these drawings.

[0074] Figure 1 The flowchart of the fine-grained image retrieval method based on compact representation modeling and semantic label guidance according to the embodiment of the present application.

[0075] Figure 2 The overall structure diagram of the model constructed by the fine-grained image retrieval method based on compact representation modeling and semantic label guidance according to the embodiment of the present application.

[0076] Figure 3 The MUSTA module structure diagram in the fine-grained image retrieval method based on compact representation modeling and semantic label guidance according to the embodiment of the present application.

[0077] Figure 4 The SFBC module structure diagram in the fine-grained image retrieval method based on compact representation modeling and semantic label guidance according to the embodiment of the present application.

[0078] Figure 5 The MUSTA module specific flowchart in the fine-grained image retrieval method based on compact representation modeling and semantic label guidance according to the embodiment of the present application. DETAILED DESCRIPTION

[0079] In order to enable a more clear understanding of the above-mentioned objects, features and advantages of the present application, the present application will be further described below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0080] In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the present application, but the present application can also be implemented in other manners different from those described herein, and therefore, the protective scope of the present application is not limited to the specific embodiments disclosed below.

[0081] The compact representation-based modeling and semantic label-guided fine-grained image retrieval method provided by the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0082] Symbol explanation:

[0083] I represents an image in the input preprocessed image data set;

[0084] T gl represents a global-level visual word vector sequence;

[0085] L represents the total number of Transformer layers included in the multi-layer visual Transformer;

[0086] Z′ L-1 represents the final input sequence obtained by aggregating all the significant visual word vectors selected in each layer Transformer layer except the last layer;

[0087] M represents the number of attention heads;

[0088] A l represents the attention weight of the lth Transformer layer;

[0089] l represents the current Transformer layer number, 1≤l≤L-1;

[0090] represents the attention weight of the mth attention head in A l

[0091] represents the attention weight corresponding to the N T th visual word vector in A

[0092] m represents the current attention head number in the current Transformer layer;

[0093] N T represents the number of non-class visual word vectors of each attention head; ​

[0094] a′ represents the fusion attention weight;

[0095] A w represents the attention weight of the wth Transformer layer;

[0096] w represents the number of Transformer layers, 1≤w≤l;

[0097] v l represents the fusion attention weight corresponding to the category visual word vector in the lth Transformer layer;

[0098] represents the attention weight of the mth attention head in v l ;

[0099] K l represents the index value corresponding to the visual word vector with the highest fusion attention weight value in the lth Transformer layer;

[0100] represents the visual word vector index value of the mth attention head in K l ;

[0101] The TopSelection(·) function represents the use of the max(·) function to filter the index corresponding to the highest attention weight;

[0102] n’ represents the number of visual word vectors with the highest weight value in each attention head;

[0103] T l represents the output sequence of the lth Transformer layer;

[0104] represents the category visual word vector of the original input sequence of the Lth Transformer layer;

[0105] T loc represents the local-level visual word vector sequence;

[0106] Z L-1 represents the original complete input sequence;

[0107] c k represents the hash encoding center of the kth category;

[0108] k represents the number of fine-grained categories;

[0109] ‖·‖ H is the Hamming distance;

[0110] K represents the total number of fine-grained categories;

[0111] i, j represent the fine-grained class number;

[0112] c K , c i , c j represent the hash code center of the Kth, ith, jth class respectively;

[0113] q represents the length of hash code;

[0114] d i,j represents the fine-grained weighted distance constraint between the ith and jth class;

[0115] S i,j represents the similarity of the ith and jth class labels;

[0116] x gl represents the high-dimensional global-level real-valued feature;

[0117] x loc represents the high-dimensional local-level real-valued feature;

[0118] W gl represents the global-level linear mapping matrix;

[0119] W loc represents the local-level linear mapping matrix;

[0120] h gl represents the low-dimensional global-level real-valued feature;

[0121] h loc represents the low-dimensional local-level real-valued feature;

[0122] b represents the low-dimensional compact code;

[0123] b gl represents the global-level low-dimensional compact code;

[0124] b loc represents the local-level low-dimensional compact code;

[0125] S represents the similarity matrix;

[0126] Sim(·,·) represents the scaled cosine similarity calculation;

[0127] P n,k represents the similarity of the compact code b n and its corresponding hash code center c k ;

[0128] t represents the fine-grained class number;

[0129] S k,t represents the similarity of the kth and tth class labels; ​

[0130] c t Represents the hash code center of class t;

[0131] Z n,k As an indicator factor;

[0132] I n This represents the nth input image;

[0133] b n This represents the compact encoding of the nth sample in the kth class;

[0134] c k′ Represents the encoding center of other categories that are semantically similar to the k-th category;

[0135] This represents the semantic similarity weighted center loss function;

[0136] Represents the hash encoding loss function;

[0137] Represents the total loss function;

[0138] λ1 and λ2 represent the coefficients of the loss term;

[0139] u represents the image sample number in the image library;

[0140] v represents the query sample number;

[0141] Ω represents the set of indices for the library samples in the training data;

[0142] Γ represents the set of indices for the query samples in the training data;

[0143] b u b v These represent the low-dimensional hash codes of samples u and v, respectively;

[0144] h u This represents the low-dimensional real-valued encoding of sample u;

[0145] L uv Indicates sample supervision information;

[0146] Q represents the number of query samples;

[0147] D represents the number of samples in the image library.

[0148] like Figure 1 As shown, the fine-grained image retrieval method based on compact representation modeling and semantic tag guidance provided according to an embodiment of the present invention includes the following steps.

[0149] Step S1, prepare an image dataset containing fine-grained categories, perform data preprocessing and data division to obtain a preprocessed image dataset. Optionally, the image dataset containing fine-grained categories in this step can be selected from a public database, the data preprocessing method includes random cropping, random flipping, random scaling, etc., and the data division follows the data division rules of each public dataset.

[0150] Step S2, construct a feature extraction network. As shown in Figure 2 , the feature extraction network is constructed, including a multi-layer visual Transformer based on ViT (visual Transformer) as the backbone network, and the visual features of the preprocessed image dataset are extracted to obtain visual word vectors. The visual features include global level features (i.e., target features) and local level features (i.e., component features). The multi-layer visual Transformer of the feature extraction network includes a global level feature learning branch and a local level feature learning branch, which respectively model the global level features and the local level features. In addition, the MUSTA module (multi-level salient visual word vector aggregation module) is also included in the local level feature learning branch, as shown in Figure 3 , the MUSTA module is used to filter the salient visual word vectors of several Transformer layers of the visual Transformer, and the filtered salient visual word vectors are fused as the input of the last Transformer layer to enhance the saliency of the features in the local level feature learning branch.

[0151] Further, the feature extraction network constructed in the step S2 is a visual Transformer containing a MUSTA module. Specifically, for fine-grained tasks, both global level features and local level features are very important. As shown in Figure 2 , Figure 3 and Figure 5 , the step S2 specifically includes the following steps.

[0152] Step S2.1, construct a feature extraction network, first construct a multi-layer visual Transformer including a global level feature learning branch and a local level feature learning branch as the backbone network of the feature extraction network, wherein the MUSTA module is included in the local level feature learning branch.

[0153] Step S2.2, input the preprocessed image dataset, and use the constructed multi-layer visual Transformer, φ ViT (·), to extract the global level visual word vector sequence of the input preprocessed image dataset: T gl = Φ ViT (I).

[0154] Step S2.3, simultaneously and in parallel, using the MUSTA module set, select several more salient visual word vectors from each attention head of each of the Transformer layers of the multi-layer visual Transformer, except the last Transformer layer, and aggregate all the salient visual word vectors selected from each of the Transformer layers except the last Transformer layer into a final input sequence Z' L-1 , where L represents the total number of Transformer layers included in the multi-layer visual Transformer.

[0155] Further, the MUSTA module of the local-level feature learning branch in the step S2 extracts and aggregates the most salient fine-grained features (i.e., more salient visual word vectors) in all low layers and high layers of the multi-layer visual Transformer network except the last Transformer layer, so as to enhance the saliency of the features learned by the local-level feature learning branch and reduce the redundant information in the visual word vector sequence.

[0156] Step S2.3 specifically includes the following steps.

[0157] Step S2.3.1, in order to make full use of the attention information and input more salient information to the last Transformer layer, the attention weights of each Transformer layer are fused and assigned to each visual word vector in the sequence in this step to establish the correlation between the attention weights and the importance of the word vectors. The input of the last Transformer layer in the local-level feature learning branch is set as Z' L-1 , then the attention weight of each of the first L-1 Transformer layers each having M attention heads is represented as:

[0158]

[0159] where M represents the number of attention heads, A1 represents the attention weight of the lth Transformer layer, represents A l , m represents the attention weight of the mth attention head in A1, represents , N T represents the attention weight corresponding to the Nth visual word vector in A1, l represents the current Transformer layer number and 1≤l≤L-1, m represents the current attention head number in the Transformer layer, and N T represents the number of non-class visual word vectors of each attention head.

[0160] Step S2.3.2, in the MUSTA module, multiply the first l Transformer layers and the attention weights before them layer by layer to obtain the fusion attention weight for recording how the visual word vector is forward propagated from the input layer to the higher layer:

[0161]

[0162] wherein a' represents the fusion attention weight, w represents the number of the Transformer layer, A w represents the attention weight of the wth Transformer layer and 1≤w≤l.

[0163] Step S2.3.3, the original single-layer attention weight a is more suitable for selecting the discriminative region. Since the category visual word vector can represent the global level features of the image, the weight corresponding to the category visual word vector is selected from the fusion attention weight a':

[0164]

[0165] and the index corresponding to the n' visual word vectors with the highest weight value in each attention head is screened out:

[0166]

[0167] wherein v l represents the fusion attention weight of the lth Transformer layer corresponding to the category visual word vector, represents the attention weight of the mth attention head in v l , K l represents the index value corresponding to the n' visual word vectors with the highest fusion attention weight value in the lth Transformer layer, represents the visual word vector index value of the mth attention head in K l , and the TopSelection(·) function represents using the max(·) function to screen the indexes corresponding to the n' highest attention weights.

[0168] Step S2.3.4, then, as shown in Figure 2 and Figure 5 , find the visual word vector corresponding to the index value corresponding to the n' visual word vectors with the highest fusion attention weight value in the output sequence of each Transformer layer of the first L-1 Transformer layers:

[0169] T s,l = T l [K l ]

[0170] wherein T l denotes the output sequence of the l-th Transformer layer.

[0171] Step S2.3.4, by the MUSTA module, aggregating the visual word vectors of the output sequences of the respective Transformer layers of the first L-1 Transformer layers and the category visual word vector of the original input sequence of the L-th Transformer layer as the final input sequence of the last Transformer layer in the local-level feature learning branch:

[0172]

[0173] wherein, denotes the category visual word vector of the original input sequence of the L-th Transformer layer.

[0174] by replacing the original complete input sequence Z L-1l with the above obtained final input sequence of the most significant visual word vector sequence Z' L-1 , and connecting the category word vector of the original input sequence of the L-th Transformer layer as the input to the last Transformer layer, not only the global-level information can be retained, but also the subtle differences between different sub-categories can be captured, while removing the redundant information such as common features belonging to the same parent category.

[0175] Step S2.4, subsequently, inputting the aggregated final input sequence Z' L-1 into the last Transformer layer in the local-level feature learning branch, and obtaining the local-level visual word vector sequence

[0176] Step S3, generating the hash coding center. As Figure 1As shown, the SFBC module (Semantic-Guided Fine-Grained Binary Coding) first constructs a bag-of-words model according to the image fine-grained labels, which is used to count the frequency of each word appearing in the class label. Then, the semantic similarity matrix between all class labels is calculated according to the word frequency, and finally the encoding center of each class is generated with the semantic similarity matrix as the weight, so that the distance between similar class encoding centers is larger. The sum of the generated encoding center of each class is the hash encoding center set. Wherein, the image fine-grained label is all the class labels contained in the fine-grained dataset, the word is all the words in all the class labels contained in the fine-grained dataset, and the class label is all the class labels contained in the fine-grained dataset, which are all information provided by the dataset. The label can be directly loaded from each dataset folder.

[0177] Further, the hash encoding center in step S3 is a global similarity measure, which is used to promote the hash encoding of similar images to approach its common hash center and keep distance from other hash encoding centers. Step S3 specifically includes the following steps.

[0178] Step S3.1, first generate a set of binary centers containing q bits for at least distance d between K subcategories:

[0179]

[0180] Wherein, k represents the fine-grained class number, c k represents the hash encoding center of the kth class. And wherein, d is the minimum distance between any two representation centers obtained based on Gilbert-Varshamov boundary theory.

[0181] Step S3.2, then, by pulling the compact encoding b n from the nth image of the kth class to its corresponding class center c k and at the same time pushing away other class centers, so as to realize the learning of compact encoding.

[0182] In addition, in order to more accurately distinguish difficult samples, i.e. samples from different fine-grained classes of the same parent class with a large number of similar features, a semantic-guided fine-grained compact representation learning module can be constructed, which applies stricter distance constraints and weights to semantically similar classes than to those dissimilar classes, in order to enhance the discriminative compact representation learned for fine-grained retrieval. The sample refers to the input image to be processed, i.e. the image in the preprocessed image dataset.

[0183] The step S3.2 can further include the following steps. Firstly, in order to obtain the inter-class semantic similarity, a class semantic similarity matrix is constructed, a bag-of-words model containing w' words is adopted, and the K fine-grained labels are represented as a feature matrix V based on the word occurrence probability. Therefore, the similarity matrix is represented as

[0184] S = V · V T .

[0185] Based on the class similarity matrix S, the original distance constraint d is replaced by a fine-grained weighted distance constraint

[0186] d i,j = (1 + S i,j ) · d

[0187] Subsequently, the hash coding centers of the respective fine-grained classes are generated by solving the following optimization problem:

[0188]

[0189] s.t. ||c i -c j || H ≥ d i,j , 1≤i,j≤K, i≠j,

[0190] wherein ‖·‖ H is the Hamming distance, K represents the total number of fine-grained classes, i and j represent the fine-grained class numbers, c K , c i , and c j represent the hash coding centers of the Kth, ith, and jth classes respectively, q represents the hash coding length, d i,j represents the fine-grained weighted distance constraint between the ith and jth classes, and S i,j represents the similarity of the ith and jth labels. Through the above calculation formula, the classes with similar semantics are given greater distance constraints, thereby forcing the corresponding coding centers to be further pushed away.

[0191] Step S4, constructing a compact coding mapping unit. As shown in Figure 1 and Figure 4 , all images in the preprocessed image data set are respectively input into the global level feature learning branch and the local level feature learning branch of the feature extraction network, the visual word vectors are extracted, the extracted visual word vectors are input into the coding mapping unit, and the high-dimensional real value features of the visual word vectors are mapped to low-dimensional binary codes using an activation function to obtain the compact representation of the images, and a retrieval code library is constructed by using the obtained compact representation of the images.

[0192] Further, the global-level visual word vector sequence and the local-level visual word vector sequence extracted from all the images in the retrieval library through the step S2 are mapped into low-dimensional binary codes by the compact coding mapping unit in the step S4, and a retrieval code library is constructed. Referring to Figure 4 Specifically, the step S4 includes the following steps.

[0193] In the step S4.1, the high-dimensional global-level real value feature x gl and the high-dimensional local-level real value feature x loc are obtained by applying the global average pooling on the global-level visual word vector sequence T gl and the local-level visual word vector sequence T loc respectively.

[0194] In the step S4.2, the high-dimensional global-level real value feature x gl and the high-dimensional local-level real value feature x loc are mapped into low-dimensional global-level real value feature h gl and low-dimensional local-level real value feature h loc respectively by using the global-level linear mapping matrix W gl and the local-level linear mapping matrix W loc .

[0195] In the step S4.3, the low-dimensional global-level real value feature h gl and the low-dimensional local-level real value feature h loc are mapped into low-dimensional compact codes b = {b gl , b loc}. Wherein, b gl represents the global-level low-dimensional compact code, and b loc represents the local-level low-dimensional compact code. In this step S4.3, the low-dimensional global / local-level real value feature is mapped into the global / local-level low-dimensional compact code by the activation function. Optionally, the activation function is the Tanh function.

[0196] In the step S4.4, the low-dimensional compact codes corresponding to the samples of the images in the input preprocessed image data set are stored according to the image sample numbers in the image library, and a retrieval code library with an index set Ω is constructed.

[0197] In the step S5, the constructed feature extraction network and the compact coding mapping unit are trained to obtain the trained feature extraction network and the compact coding mapping unit. The loss function is set, the back propagation algorithm is adopted, and the network parameters are iteratively updated and optimized until the loss of the loss function converges.

[0198] Further, the loss function in the step S5 includes a semantic similarity weighted center loss function, a quantization loss and a hash coding loss. The semantic similarity weighted center loss function is used to pull the distance to the subordinate sub-class coding center and pull the distance to other sub-class coding centers, the quantization loss is used to reduce the difference between high-dimensional real value features and low-dimensional compact representations, and the hash coding loss is used to pull the intra-class coding distance and pull the inter-class coding distance. The step S5 specifically includes the following steps.

[0199] Step S5.1, specifically, in order to further retain the fine-grained semantic information of the learned mechanism coding, the similarity matrix S is embedded into the calculation of the compact coding b n and the corresponding k-th class hash coding center c k , to obtain the similarity of the compact coding and the corresponding hash coding center:

[0200]

[0201] wherein Sim(·,·) represents the scaled cosine similarity calculation, P n,k represents the similarity of the compact coding b n and the corresponding k-th class hash coding center, t represents the fine-grained class number, K represents the total number of fine-grained classes, S k,t represents the similarity of the k-th class and the t-th class label, c t represents the hash coding center of the t-th class.

[0202] Step S5.2, a semantic similarity weighted center loss function is established and represented as:

[0203]

[0204] wherein Z n,k is an indication factor, if the compact coding b n of the n-th input image I n belongs to the k-th class, then Z n,k =1, otherwise Z n,k =0. When minimizing the center loss, the compact coding b n and the coding center c k′ of other classes with semantic similarity to the k-th class are given higher penalty weights, so as to force them to be pushed away, which greatly helps to learn more discriminative compact coding to separate the input samples from the fine-grained classes.

[0205] Step S5.3, a quantization loss and a hash coding loss function are established:

[0206]

[0207] wherein, λ1, λ2 represent loss term coefficients, u represents image sample number in the gallery, Ω represents index set of gallery samples in the training data, v represents query sample number, Γ represents index set of query samples in the training data, b u 、b v respectively represent low-dimensional hash encoding of samples u, v, h u represents low-dimensional real value encoding of sample u, L uv represents sample supervision information, L uv ∈{-1,+1} (Q×D) represents paired supervision information, Q represents query sample number, D represents gallery sample number. The first term is the quantization loss of binaryzation of real value low-dimensional feature vector h u The second term is the general semantic preserving loss. λ1 and λ2 can be set to 1 and 200 respectively.

[0208] Step S5.4, combine the quantization loss and the hash encoding loss function with the semantic similarity weighted center loss function (i.e. fine-grained semantic guided loss) to obtain the total loss function for optimizing the entire network:

[0209]

[0210] Step S5.5, the final total loss function contains the loss of the feature extraction network and the compact encoding mapping unit, in the process of gradient back propagation, the network parameters of the feature extraction network and the compact encoding mapping unit are updated simultaneously until the loss of the loss function converges, obtaining the trained feature extraction network and compact encoding mapping unit. The updated network parameters can include the parameters of the Transformer layer structure in the feature extraction network, the MUSTA module, and the fully connected layer of the compact encoding mapping unit.

[0211] Step S6, perform fine-grained image retrieval, input the image to be detected into the trained feature extraction network and compact encoding mapping unit to obtain the high-dimensional feature of the image to be detected, map the high-dimensional feature of the image to be detected into low-dimensional encoding as the detection code to be detected, and match the obtained detection code to be detected with the encoding in the retrieval code library. The sample with the smallest distance is the final retrieval result. The method of the present application can be applied to search engines, e-commerce, social media, etc. The final retrieval result obtained is the sample with the highest similarity to the image to be detected (i.e. the sample with the smallest hash encoding distance), as the retrieval result in the application, thereby providing efficient and accurate image retrieval function.

[0212] Embodiment 1

[0213] One specific embodiment 1 of the fine-grained image retrieval method based on compact representation modeling and semantic label guidance according to the present application is applied as follows, comprising the following steps.

[0214] S1, data set preparation. Perform data set selection, data preprocessing and data division.

[0215] S1.1, in this example, large-scale fine-grained image CUB200-2011 data set, Aircraft data set, Food101 data set, NABirds data set and VegFru data set are selected as the fine-grained image retrieval data set for verifying the invention.

[0216] S1.2, data preprocessing includes image enhancement and image normalization. Specifically, image enhancement includes resizing, random inversion and random cropping, etc., and in this example, two enhancement methods of resizing the image to 224x224 pixels and 50% probability random horizontal inversion are selected.

[0217] S1.3, data division is divided according to the standard given by each data set. CUB200-2011 data set contains 11788 images from 200 bird species, 5994 images for training and 5794 images for testing; Aircraft data set contains 10000 images from 100 aircraft, 6667 images for training and 3333 images for testing; Food101 data set contains 101000 images from 101 food, 750 images for training and 250 images for testing per class; NABirds data set contains 48562 images of North American birds from 555 categories, 23929 images for training and 24633 images for testing. VegFru data set contains 200 vegetables and 92 fruits, 29200 images for training and 116931 images for testing. For the first three data sets, 2000 images are randomly selected for training per iteration, and for the last two data sets, 4000 images are randomly selected for training per iteration.

[0218] S2, construct a feature extraction network. Used to extract global and local level features of images in the training set.

[0219] S2.1, ViT-B / 16 is used as the feature extraction network, wherein the number of network layers, the number of attention heads and the dimension of hidden layer feature vectors are 12, 12 and 768 respectively. After preprocessing the input training data, the image is divided into a sequence of non-overlapping image blocks of 16x16 pixels, which is spliced and fused with the position coding embedding to form an input visual word vector module as the input of the backbone network of ViT (visual Transformer).

[0220] S2.2, for the global level feature learning branch of the backbone network, the visual word vector is directly input into the original ViT (Visual Transformer) network, and after passing through 12 Transformer layers, the global visual word vector sequence is obtained.

[0221] S2.3, for the local level feature learning branch of the backbone network, the MUSTA module is introduced into the original ViT network structure to filter 3 most significant visual word vectors according to the attention weight of the visual word vector in each Transformer layer and aggregate them as the input sequence of the last layer. The specific model design is described in the foregoing, which will not be repeated here.

[0222] S3, generate the hash encoding center. First, according to the fine-grained class labels of the dataset, a bag-of-words model of the statistical word frequency is constructed, then the fine-grained semantic similarity matrix is calculated according to the word frequency, and finally the encoding center generation process is weighted to obtain the semantic-guided encoding center matrix, where the number of rows corresponds to the number of classes of the dataset, and the number of columns corresponds to the final encoding length of the current instance, which can be set to 12 bits, 24 bits, 32 bits and 48 bits.

[0223] S4, construct the compact encoding mapping unit. The visual word vectors extracted by the global level feature learning branch and the local level feature learning branch of all images in the training dataset are input into the encoding mapping unit, and the high-dimensional real-valued features are mapped to low-dimensional binary codes using the activation function to obtain the compact representation of the image and construct the retrieval code library.

[0224] S5, train the feature extraction network and the compact encoding mapping unit. The semantic similarity weighted center loss, quantization loss and hash coding loss are used to constrain the model training. The specific constraint design is described in the foregoing, which will not be repeated here. The backpropagation algorithm is used to update and optimize the network parameter weight until the model loss area converges. In this example, the image inpainting model training and evaluation are completed on the Pytorch platform. The model is trained on a single GeForce RTX3090Ti GPU (24GB), and the batch size is set to 16. The network is optimized using the SGD optimizer with a learning rate of 2.5x10 -4 , and the weight decay coefficient and momentum coefficient are set to 1x10 -4 and 0.91 respectively, and the learning rate is reduced by 10% after the first iteration. For the dataset with less than 20,000 images, the training is performed for 40 epochs, and each epoch is iterated for 30 rounds, while for other datasets, the training is performed for 50 epochs, and each epoch is iterated for 30 rounds. The results of the best model on the test set are finally reported.

[0225] S6, the model training is completed, fine-grained image retrieval is performed, test data sets of each data set are input into the model obtained by training, and the output result of the model is the retrieval result. In this example, mAP is used as the final evaluation index, and the code lengths are 12 bits, 24 bits, 32 bits and 48 bits. For the CUB200-2011 data set, the retrieval accuracies under the four length encodings are 83.76%, 88.92%, 89.37% and 90.28% respectively; for the Aircraft data set, the retrieval accuracies under the four length encodings are 75.32%, 85.91%, 86.47% and 86.99% respectively; for the Food101 data set, the retrieval accuracies under the four length encodings are 85.22%, 88.97%, 89.11% and 89.96% respectively; for the NABirds data set, the retrieval accuracies under the four length encodings are 67.83%, 79.61%, 82.85% and 85.27% respectively; for the VegFru data set, the retrieval accuracies under the four length encodings are 88.70%, 92.95%, 93.86% and 93.27% respectively. The results show that the fine-grained image retrieval task can be effectively completed by using the present application, and good performance is shown under multiple length encodings.

[0226] All the optional technical solutions described above can be combined to form optional embodiments of the present application, and will not be repeated here.

[0227] It should be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0228] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A fine-grained image retrieval method based on compact representation modeling and semantic label guidance, characterized in that, The method comprises the following steps: Step S1, preparing an image data set containing fine-grained categories, performing data preprocessing and data division to obtain a preprocessed image data set; Step S2, constructing a feature extraction network, including constructing a feature extraction network based on a multi-layer visual Transformer as a backbone network for visual feature extraction of images in the preprocessed image data set to obtain visual word vectors, the feature extraction network further comprising a global level feature learning branch and a local level feature learning branch, wherein the local level feature learning branch comprises a MUSTA module; Step S3, generating a hash coding center, constructing a bag-of-words model according to the image fine-grained labels, counting the frequency of each word appearing in the category label as the word frequency, calculating the semantic similarity matrix between all category labels according to the statistical word frequency, and finally generating the coding center of each category with the semantic similarity matrix as the weight; Step S4, constructing a compact coding mapping unit, inputting the visual word vectors extracted by the feature extraction network into the constructed coding mapping unit, and using the activation function in the coding mapping unit to map the high-dimensional real value features of the visual word vectors to low-dimensional binary codes to obtain the compact representation of the image, and constructing a retrieval code library with the obtained compact representation of the image; Step S5, training the constructed feature extraction network and compact coding mapping unit to obtain a trained feature extraction network and compact coding mapping unit; Step S6, performing fine-grained image retrieval, inputting the image to be detected into the trained feature extraction network and compact coding mapping unit to obtain the high-dimensional features of the image to be detected, mapping the high-dimensional features of the image to be detected to low-dimensional codes as the detection code, and matching the obtained detection code with the codes in the retrieval code library, and the sample with the smallest distance is the final retrieval result; The step S2 specifically comprises the following steps: Step S2.1, constructing a feature extraction network, first constructing a multi-layer visual Transformer comprising a global level feature learning branch and a local level feature learning branch as the backbone network of the feature extraction network, wherein the local level feature learning branch comprises a MUSTA module; Step S2.2, input the pre-processed image dataset, use the constructed multi-layer visual Transformer Φ ViT (·) Extract the global-level visual word vector sequence from the image I in the input pre-processed image dataset: T gl = Φ ViT (I); Step S2.3, in parallel, using the set MUSTA module, selecting several more salient visual word vectors from each attention head of each Transformer layer of all but the last Transformer layer of the multi-layer visual Transformer, and aggregating all the salient visual word vectors selected from the layers other than the last layer into a final input sequence Z' L-1 wherein L represents the total number of Transformer layers included in the multi-layer visual Transformer; Step S2.4, aggregate the final input sequence Z′ L-1 The input is fed into the last Transformer layer in the local-level feature learning branch, and after processing, a sequence of local-level visual word vectors is obtained:

2. The method according to claim 1, wherein, The step S2.3 specifically comprises the following steps: Step S2.3.1, the total number of layers of the Transformer layers included in the multi-layer visual Transformer is L layers, and the input of the last layer of the Transformer layer in the local-level feature learning branch is Z' L-1 The attention weight of each layer of the first L-1 layers of the Transformer layer, each having M attention heads, is represented as: wherein A l represents the attention weight of the lth Transformer layer, l represents the current Transformer layer number and 1≤l≤L-1, represents A l represents the attention weight of the mth attention head in A represents represents the attention weight corresponding to the N T th visual word vector in A T represents the number of non-class visual word vectors of each attention head; Step S2.3.2, in the MUSTA module, multiplying the attention weights of the first l Transformer layers and before them layer by layer to obtain the fusion attention weight for recording the visual word vector from the input layer forward propagation to a higher layer: wherein a' represents the fusion attention weight, w represents the number of the Transformer layer, A w represents the attention weight of the wth Transformer layer and 1≤w≤l; Step S2.3.3, selecting the attention weight corresponding to the category visual word vector from the fusion attention weight a′: And selecting the index corresponding to the n' visual word vectors with the highest weight value from each attention head: where v l denotes the fusion attention weight corresponding to the category visual word vector in the l-th layer of the Transformer layer, denotes the attention weight of the m-th attention head in v l l denotes the index value corresponding to the visual word vector with the highest fusion attention weight value in the l-th layer of the Transformer layer, denotes the visual word vector index value of the m-th attention head in K l TopSelection(·) function represents the use of max(·) function to filter the index corresponding to the highest attention weight;​ Step S2.3.

4. In the output sequence of each of the previous L-1 layer Transformer layers, find the index value K selected by each of the previous L-1 layer Transformer layers l Corresponding visual word vector: T s,l = T l [K l ] wherein T l represents the output sequence of the l-th Transformer layer; Step S2.3.5, aggregating the visual word vectors of the output sequence of each Transformer layer of the first L-1 layers of Transformer layers and the class visual word vector of the original input sequence of the Lth layer of Transformer layers by the MUSTA module as the final input sequence of the last layer of Transformer layer in the local level feature learning branch: wherein, represents the class visual word vector of the original input sequence of the Lth layer of the Transformer layer.

3. The compact representation based modeling and semantic label guided fine-grained image retrieval method according to claim 2, characterized in that, The step S3 specifically comprises the following steps: Step S3.1, generating a set of binary centers containing q bits for at least distance d between K sub-categories: wherein k represents a fine-grained class number, c k represents the hash coding center of the kth class, d is the minimum distance between any two representation centers obtained based on Gilbert-Varshamov boundary theory, K represents the total number of fine-grained classes, and q represents the hash coding length. Step S3.2, constructing a category semantic similarity matrix, using a bag-of-words model containing w' words, and representing K fine-grained labels as a feature matrix V based on word occurrence probability, and the similarity matrix is represented as: S = V · V T ; Based on the category similarity matrix S, the minimum distance d above is replaced by the fine-grained weighted distance constraint d between two categories i,j : d i,j = (1 + S i,j ) · d; Subsequently, the hash coding center of each fine-grained category is generated by solving the following optimization problem: s.t. ||c i -c j || H ≥d i,j ,1≤i,j≤K,i≠j, where ||·|| is the Hamming distance, i, j represent the fine-grained class number, c h is the Hamming distance, i, j represent the fine-grained class number, c K , c i , c j respectively represent the K, i, j class hash code center, d i,j represents the fine-grained weighted distance constraint between the i-th class and the j-th class, S i,j represents the similarity of the i-th class and the j-th class label.

4. The method of claim 3, wherein the compact representation based modeling and semantic label guided fine-grained image retrieval method is characterized by, The step S4 specifically comprises the following steps: Step S4.1, obtaining global-level real-valued features x gl and local-level real-valued features x loc by applying global average pooling on the global-level visual word vector sequence T gl and the local-level visual word vector sequence T loc respectively; Step S4.2, using a global-level linear mapping matrix W gl and a local-level linear mapping matrix W loc , respectively, map the high-dimensional global-level real-valued feature x gl and the high-dimensional local-level real-valued feature x loc to a low-dimensional global-level real-valued feature h gl and a low-dimensional local-level real-valued feature h loc , respectively; Step S4.3, mapping the low-dimensional global-level real-valued features h gl and the low-dimensional local-level real-valued features h loc to low-dimensional compact codes b = {b gl , b loc}, to obtain the compact representation of the image, where b gl represents the low-dimensional compact codes of the global level, and b loc represents the low-dimensional compact codes of the local level; Step S4.4, storing the low-dimensional compact codes corresponding to the sample pairs of images in all input preprocessed image data sets according to the image sample numbers in each gallery, and constructing an index set as the retrieval code library Ω.

5. The method of claim 4, wherein the compact representation based modeling and semantic label guided fine-grained image retrieval method is characterized by, The step S5 comprises the following steps: Step S5.1, embedding the similarity matrix S into calculating the compact code b of the nth sample of the kth class n and the corresponding hash code center c of the kth class k , obtaining the similarity of the compact code and its corresponding hash code center wherein Sim(·,·) represents the scaled cosine similarity calculation, P n,k denotes the compact code b n and the corresponding k-th hash code center c k of the similarity, t denotes the fine-grained class number, K denotes the total number of fine-grained classes, S k,t denotes the similarity between the k-th and t-th labels, c t denotes the hash code center of the t-th class; Step S5.2, establishing a semantic similarity weighted center loss function: wherein Z n,k is an indicator factor, Z n = 1 if the compact code b n of the nth input image I n,k belongs to the kth class, and Z n,k = 0 otherwise. Step S5.3, establishing a quantization loss and a hash coding loss function: wherein λ1, λ2 represent loss term coefficients, u represents image sample number in the gallery, v represents query sample number, Ω represents index set of gallery samples in the training data, Γ represents index set of query samples in the training data, b u , b v respectively represent low-dimensional hash coding of samples u, v, h u represents low-dimensional real value coding of sample u, L uv represents pair-wise sample supervision information and L uv ∈{-1, +1} (Q×D) , Q represents query sample number, D represents gallery sample number, and the first term is quantization loss of binarizing real value low-dimensional feature vector h u , and the second term is general semantic preservation loss; Step S5.4, the quantization loss and the hash coding loss function with semantic similarity weighted center loss function In combination, the total loss function for optimizing the entire network is ultimately obtained: Step S5.5 The final total loss function contains the loss of the feature extraction network and the compact coding mapping unit. In the gradient backpropagation process, the network parameters of the feature extraction network and the compact coding mapping unit are updated simultaneously until the loss of the loss function converges, and the trained feature extraction network and compact coding mapping unit are obtained.

Images