A weakly supervised hash image retrieval system with knowledge graph embedding attention mechanism

Abstract

The application discloses a kind of weakly annotated hash image retrieval architecture of knowledge graph embedding attention mechanism, and the architecture is integrated into an end-to-end image retrieval architecture by feature learning, hash learning, knowledge graph embedding, cross-modal feature fusion, including image visual feature extraction network, text feature extraction model, the decoder that is executed cross attention to image feature and text feature, the noise mask matrix of embedding self-attention layer, the hash layer of generating compact semantics, single modal visual model and the knowledge distillation mechanism for the transferability of multi-modal model to single modal model.The architecture has certain promotion to the image retrieval performance under weakly annotated environment.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image retrieval, and particularly relates to a weakly labeled hash image retrieval system with a knowledge graph embedding attention mechanism. BACKGROUND

[0002] Many learning-based hash methods have been proposed by researchers and used in image retrieval, and the goal of these methods is to learn a compact similarity-preserving representation method, so that similar images can be mapped to adjacent binary hash codes in the Hamming space. With the help of manually labeled images, supervised hash learning methods have achieved remarkable results. However, the acquisition cost of large-scale labeled image data is high, which limits the use of such algorithms.

[0003] In real environments, there are many Web image libraries, among which weakly labeled image data is very rich. Such Web images usually contain useful information image labels, but such weak labels are often accompanied by a large amount of noise labels.

[0004] With the advent of the Transformer architecture based on the self-attention mechanism, many studies have begun to apply Transformer to the visual field. In addition, with the development of knowledge graphs, more and more researchers are trying to use knowledge information learned from knowledge graphs to improve the performance of some knowledge-driven tasks.

[0005] How to use knowledge graphs as auxiliary knowledge to reduce the influence of noise labels in the attention mechanism under the joint driving of data and knowledge is currently less studied in the field of image retrieval.

[0006] Through the research of the applicant, the following problems exist in the prior art:

[0007] (1) Supervised learning relies on manually annotated data sets, but the acquisition cost of large-scale labeled image data is high, which limits the use of such algorithms.

[0008] (2) In real environments, there is a large amount of weakly labeled data, which contains useful information image labels and is accompanied by a large amount of noise labels.

[0009] (3) The image visual features and text features are not only inconsistent in representation, but also have different attributes. This heterogeneity makes it very difficult to measure the correlation across modalities, and it is difficult to achieve unified representation and comprehensive utilization of cross-modal data. SUMMARY

[0010] In view of the defects or deficiencies in the prior art, the purpose of the present application is to provide a weakly labeled hash image retrieval architecture with a knowledge graph embedding attention mechanism.

[0011] In order to achieve the above-mentioned tasks, the present application adopts the following technical solutions:

[0012] A weakly annotated hash image retrieval architecture of a knowledge graph embedding attention mechanism, characterized in that the architecture integrates feature learning, hash learning, knowledge graph embedding and cross-modal feature fusion into an image retrieval architecture, including the following structures:

[0013] Structure (1): image visual feature extraction network;

[0014] Structure (2): text feature extraction model;

[0015] Structure (3): decoder for performing cross-attention on image features and text features;

[0016] Structure (4): noise mask matrix of embedded self-attention layer;

[0017] Structure (5): hash layer for generating compact semantics;

[0018] Structure (6): single-modal visual model;

[0019] Structure (7): knowledge distillation mechanism for transferring the multi-modal model to a single-modal model.

[0020] According to the present application, the noise mask matrix, in particular the prior knowledge provided by the knowledge graph, can provide the similarity between words for the text, and then form a correlation matrix, and form a mask for words with low correlation. When calculating the self-attention of the word, the attention between the mark and the noise mark is blocked, thereby reducing the influence of noise or error marks on the self-attention score.

[0021] Specifically, the knowledge distillation mechanism is specifically: the model generated by structures (1)-(5) takes both image and text as input, the purpose is to obtain visual semantic information while introducing rich text semantics, and the strong visual text model is formed on the basis of reducing text noise, and the visual text model is not retrievable, because the input of the image retrieval model in the retrieval stage is only image. Therefore, the knowledge distillation mechanism adopted by structure (7) introduces a retrievable visual model to learn the powerful but non-retrievable visual text model, and the performance of the visual text model is migrated to a simple and efficient retrievable single-modal visual model through learning distribution.

[0022] Compared with the prior art, the weakly annotated hash image retrieval architecture of the knowledge graph embedding attention mechanism of the present application has the following technical effects:

[0023] (1) Feature learning, hash learning, knowledge graph embedding and cross-modal feature fusion are integrated into an end-to-end image retrieval architecture.

[0024] (2) In view of the great semantic difference between language and vision, cross-attention Transformer is adopted to fuse the features of the two, so as to realize unified representation across modalities;

[0025] (3) Prior knowledge of the knowledge graph is used as auxiliary knowledge, and a noise mask matrix marked with occlusion noise is introduced when calculating the text self-attention, so as to reduce the influence of errors or noise labels on the semantic mining of images and texts under the joint driving of data and knowledge;

[0026] (4) A distillation mechanism is adopted to compress the powerful but non-retrievable visual-text model into an indexable visual model suitable for image retrieval tasks;

[0027] (5) Experiments on the NUS-WIDE benchmark dataset show that the architecture has certain improvement on the image retrieval performance in the weakly labeled environment. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 The weakly labeled hash image retrieval architecture of the knowledge graph embedding attention mechanism of the application adopts an overall structural block diagram;

[0029] Figure 2 The Attention Mask principle diagram is formed for the knowledge graph embedding attention mechanism;

[0030] The application will be further described below in combination with the drawings and examples. DETAILED DESCRIPTION

[0031] The design idea of the application is: in a real environment, that is, in the case of a large number of noise and fuzzy labels, the prior knowledge provided by the knowledge graph is embedded into the attention mechanism, and under the joint driving of data and knowledge, the influence of errors or noise labels on the semantic mining of images and texts is reduced, and the performance and generalization ability of the model in the real environment are improved. In addition, the cross-attention mechanism built into the Transformer is used to make the visual features and text word features perform cross-attention, so as to perform fine-grained level feature learning and realize cross-modal feature fusion. Finally, the features are mapped to the Hamming space to learn more compact hash codes, so as to improve the efficiency in large-scale image retrieval.

[0032] The embodiment provides a weakly labeled hash image retrieval architecture of a knowledge graph embedding attention mechanism, which integrates feature learning, hash learning, knowledge graph embedding and cross-modal feature fusion into an image retrieval architecture, and specifically includes the following structures:

[0033] Structure (1): image visual feature extraction network;

[0034] Structure (2): text feature extraction model;

[0035] Structure (3): decoder performing cross-attention between image features and text features;

[0036] Structure (4): noise mask matrix embedded in the self-attention layer;

[0037] Structure (5): hash layer generating compact semantics;

[0038] Structure (6): single-modal visual model;

[0039] Structure (7): knowledge distillation mechanism for transferring the multi-modal model to the single-modal model.

[0040] In this embodiment, the image visual feature extraction network refers to any publicly disclosed neural network model, such as ResNet, VGG16 or ViT, and the selection of the neural network model is not limited to the image feature extraction network. Therefore, this neural network model is universal, and it can integrate any advanced deep model, so that these models can also be trained on weakly labeled data, thereby improving their performance. The applicant used the ResNet50 network model as the image visual feature extraction network model in the subsequent experiment.

[0041] In this embodiment, the text feature extraction model uses a pre-trained word2vec model to convert text into a feature vector.

[0042] The decoder performing cross-attention between image features and text features is specifically a decoder (Decoder) in the Transformer structure performing cross-attention.

[0043] In the cross-attention layer of the traditional decoder Decoder, the query, key, and value all come from the features of a single modality. In this embodiment, the query and key come from the visual features, and the value comes from the text features. The process of calculating the attention of the three completes the cross-attention between the visual features and the text features, and realizes the fusion of cross-modal features.

[0044] The noise mask matrix embedded in the self-attention layer is used to determine the relevance between words based on the prior knowledge of the knowledge graph for the text information corresponding to a sample. The attention score calculated between unrelated words will be noise and affect the experimental results. Therefore, a noise mask matrix is introduced in the embedded self-attention layer. When calculating the self-attention layer in the decoder, the scores between unrelated words are "masked" and not included in the final result, thereby reducing the influence of noise on the results.

[0045] The knowledge graph uses the publicly available text knowledge graph ConcepNet5, which is constructed from relational knowledge in the form of triples. It adopts an informal description that is closer to natural language and focuses on the relationships between words.

[0046] The hash layer that generates compact semantics specifically refers to mapping image features into compact binary hash codes in the final inference stage, thereby improving the efficiency of image retrieval.

[0047] The unimodal visual model specifically refers to a unimodal model that can be retrieved by simply inputting an image during the next stage of knowledge distillation. In this embodiment, the structure of the unimodal visual model is to add an activation layer Sigmoid and a hash layer to the image visual feature extraction network in structure (1). The purpose of this design is to share training parameters with structure (1).

[0048] The knowledge distillation described herein is as follows: Image retrieval takes an image as input to the model, and the model returns multiple images that are as similar as possible. The model requires not only images but also text as input. The knowledge distillation mechanism described in this embodiment is as follows: The model generated by structures (1)-(5) takes both images and text as input. The purpose is to acquire visual semantic information while also introducing rich text semantics. Based on reducing text noise, a powerful visual text model is formed. However, the visual text model is not searchable because the input of the image retrieval model is only an image during the retrieval stage. Therefore, the knowledge distillation mechanism adopted by structure (7) is to introduce a searchable visual model to learn a powerful but unsearchable visual text model. The performance of the visual text model is transferred to a simple, efficient, and searchable single-modal visual model through a learning distribution. Finally, the performance of the single-modal visual model is distilled into a deep model that only requires a single-modal input and is small and fast.

[0049] The following is the specific implementation process.

[0050] See Figure 1 , Figure 1 A block diagram of a weakly labeled hash image retrieval architecture with a knowledge graph embedding attention mechanism is presented. The specific construction includes the following steps:

[0051] First, assume the dataset consists of image-text-label ( x i , T i , l i ) composed of, where x i Indicates the first i Images of a sample, T iThis represents the corresponding text set. l i This indicates the actual label.

[0052] Step 1, Feature Extraction Stage. This step includes the extraction of visual and textual features. Specifically, given a sample pair ( x For images (T), x Spatial features of images are extracted using a CNN backbone network. ,in HW This represents the product of the height and width of the feature map. d The dimension representing the feature. For a text T consisting of K words, each word is transformed into a dimensional vector using any word embedding model (word2vec, GRU network, or BERT), ultimately forming a word embedding vector. .

[0053] Step 2, the tag embedding vector obtained in step 1 The visual features of the image serve as the decoder for the transformer, which is the input to the query. F These are then used as both key and value inputs to the decoder, where a multi-layer transformer performs cross-attention computation on the spatial features of the image and the word features of the text. The standard transformer decoder structure consists of a self-attention module, a cross-attention module, and a feedforward network (FNN). i Queries of each decoder layer Q i It will be based on the output of the previous layer. Q i-1 Update as follows:

[0054]

[0055]

[0056]

[0057] In the formula, Q The wavy line above indicates the result after position embedding. and There are two intermediate variables. MultiHead(query, key, value) and FNN ( x These are multi-head attention mechanisms and feedforward networks, respectively. The same mechanism is used in both the self-attention module and the modal cross-attention module. MuliHead Function implementation. The difference lies in where the keys and values ​​come from; in the self-attention module, query, key, value All of these originate from the label embedding vectors, while in the modal cross-attention module,keye, value The image space features become image space features. The intuitive description of the modal cross-attention is that the features of each label Query all image space features, select visual vectors with high correlation, and combine them. Each word vector will obtain an image-related feature and update itself, so It is updated layer by layer and gradually injects semantic information from the image through cross-attention, thereby realizing cross-modal fusion of image space features and text information.

[0058] Step 3, since there may be some noise labels in the text features that are not related to the real semantics, when performing self-attention, these noise labels will also be cross-calculated with the visual features and injected into the final result. In order to reduce the influence of these error or noise labels, the embodiment introduces a knowledge graph as auxiliary knowledge to construct a semantic graph in the self-attention stage in step 2, and uses the correlation provided by the knowledge graph to mask the attention calculation between unrelated words.

[0059] Specifically, the knowledge graph ConcepNet is constructed with words as vertices, correlation weights as edges, and (start point, end point, relationship) triples as semantic graphs. The edge with negative or smaller correlation represents that the two words are not related or have low correlation. Therefore, for any word w i and w j , define the Attention Mask matrix M as:

[0060]

[0061] In the formula, r ij represents the correlation score between words w i and w j , and epsilon represents a correlation threshold. When the correlation score between two words is lower than epsilon , it is considered that the two words are not related.

[0062] In order to prevent the influence of noise words on the self-attention mechanism, a mask self-attention is introduced based on the matrix M, which is an extension of the self-attention mechanism. Specifically, in the Attention calculation process, a noise mask matrix M is introduced:

[0063]

[0064] where Q, K, V are query, key and value respectively, is a scaling factor, and M is the noise mask matrix. Intuitively, if a certain word in query w i is not relevant to a certain word in key w j , then M ij will set the attention score to 0, that is, two words w i that are not relevant to each other w j will not affect the value, thus reducing the impact of noise labels on the final result calculation.

[0065] As shown in Figure 2 , sky is not visible to bay because the correlation between the two is low in the semantic graph provided by the knowledge graph, which means that sky and bay are not relevant, so the attention score is 0 and will not participate in the softmax calculation. Sky is visible to sunset, clouds, etc., so it will get the attention score between them and be considered in the softmax calculation. The benefit of this process is that when calculating the attention of sky and the rest of the words, the noise bay will not participate in the softmax calculation, thus reducing the impact of noise on the real semantics.

[0066] Step 4, use the average pooling operation to obtain the image-level semantic representation from the user label, which can also reduce the impact of noise and ambiguous labels. The specific formula is:

[0067]

[0068] where represents the cross-modal fusion feature of the text feature and the image visual feature of the i th sample after cross attention, K i represents the number of words corresponding to the i th sample. Thus, a feature matrix of Z epsilon R N×d dimensions is obtained, where N is the size of the mini-batch.

[0069] Step 5, map the feature matrix obtained in step 4 to the n-bit Hamming space through the fully connected layer and use the sigmoid activation, the specific formula is:

[0070]

[0071] where,h i is the hash code of the i-th sample, is a sigmoid activation function.

[0072] It is worth mentioning that since the sign function sgn() is not derivable, the hash code generated in the training stage is a real-valued hash code, and in the inference stage, a binary hash code is generated by the following formula:

[0073]

[0074] wherein, hi represents a real-valued hash code, sgn() represents a sign function, bi represents a binary hash code.

[0075] Step 6, for the weakly supervised task, since there is no strong label constraint, information guiding the model training needs to be mined from the weak label. TripletLoss is a common ranking loss, which makes the similarity between positive sample pairs greater than the similarity between negative sample pairs:

[0076]

[0077] wherein, , S is the cosine similarity between sample pairs, S n and S p respectively represent the similarity of the negative sample pair {x a , x n} and the similarity of the positive sample pair {x a , x p}. The triplet loss considers S n + lambda > S p sample pairs as valid sample pairs to be added to the loss function, while S n + lambda ≤ S p sample pairs are considered to contain less information and are discarded.

[0078] For the mining of positive and negative sample pairs, first, the text features are generated by the pre-trained word2vec model and are averaged, and then the cosine similarity score between texts is calculated S T , and a threshold threshold is set, then the positive sample pair set , and the negative sample pair set , wherein threshold is a hyperparameter.

[0079] Since the mining of positive sample pairs is relatively easy, and the mining of negative sample pairs is relatively difficult, on the basis of the triplet loss, a loss function for the imbalance of positive and negative sample pairs is proposed:

[0080]

[0081] Wherein, N is the size of the mini-batch, The cosine similarity between the visual real value hash codes obtained in step 5 is represented, gamma is an adjustable factor, and is regarded as a hyperparameter. threshold Actually reflects the closeness between samples.

[0082] For samples with a text similarity greater than threshold , it is considered that they will also be more similar in visual features, so for easy-to-mine positive sample pairs, (1- threshold ) is closer to a smaller value, and for difficult-to-mine negative sample pairs, threshold is closer to a larger value. Compared with the traditional triplet loss, which treats positive and negative sample pairs equally, the loss changes less for mining more accurate positive sample pairs, and changes more for difficult-to-mine or inaccurate negative sample pairs. In short, it is equivalent to increasing the weight of difficult-to-mine sample pairs in the loss function, so that the loss function tends to difficult-to-mine samples, which helps to improve the accuracy of difficult-to-mine samples.

[0083] Step 7, although the cross-modal cross-fusion model has high accuracy, due to the large model and the need for text features, in real demand, only the input picture is needed to execute the image retrieval function, so the invention finally uses the above-mentioned model based on knowledge graph embedding attention mechanism as the Teachter model, and another smaller and faster CNN network-based model as the Student model, to extract the knowledge of the cross-modal cross-fusion model into the Student model.

[0084] Specifically, the Student uses the same Backbone as the Teachter model, and the parameters of both are shared, and the Backbone parameters are fixed when training the Student model. The structure thereafter follows the convention of the Backbone classification network, except that the final fully connected layer used for classification is removed, and a fully connected mapping layer is added to map it to the same nbits bits as the output of the Teacher model, and a sigmoid activation is used. Since there is no strong label, the traditional distillation loss form cannot be used, and relative entropy is used as the distillation loss function to measure the distance between the distributions of the two models.

[0085] First, assuming for any sample i , the hash code output by the Teachter model is h itech , the hash code output by the Student model is h istud , then the distillation loss is:

[0086]

[0087] where p i and q i respectively represent the probability distribution of the corresponding sample i in the two models, tau is the temperature parameter in knowledge distillation.

[0088] Step 8, in order to evaluate the proposed weakly supervised hash learning framework, the inventors compared it with several currently leading hash methods on the NUS-WIDE dataset, including LSH, DSH and DeepBit, WDHT, SCADH, SGDH, WDH, etc.

[0089] The inventors followed the image retrieval evaluation index convention and used mAP@5000 to evaluate the performance of the model, and the experimental results are as follows in Table 1:

[0090] Table 1

[0091]

[0092] This model can combine weakly labeled text information with high relevance visual information through attention mechanism, realize feature fusion of two different modalities, and the prior knowledge provided by the knowledge graph provides conditions for forming AttentionMask, and helps to reduce the influence of noisy labels on retrieval results. From the results in Table 1, it can be seen that Attention Mask plays a significant role between 12bit and 32bit, and it decreases in 64bit. The reason may be that as the number of hash codes increases, the model will also become more complex, and the improvement will gradually tend to saturation, which may require more model optimization measures and training time.

Claims

1. A weakly supervised hash image retrieval system with knowledge graph embedding attention mechanism, characterized in that, The system integrates feature learning, hash learning, knowledge graph embedding and cross-modal feature fusion into an end-to-end image retrieval architecture, including the following structures: Structure 1: image visual feature extraction network; Structure 2: text feature extraction model; Structure 3: decoder performing cross-attention on image features and text features; Structure 4: noise mask matrix of embedded self-attention layer; Structure 5: hash layer generating compact semantics; Structure 6: single-modal visual model; Structure 7: knowledge distillation mechanism for transferring the performance of the multi-modal model to the single-modal model; The knowledge distillation mechanism is as follows: the model generated by structures 1 to 5 takes both image and text as input, aiming to obtain visual semantic information while introducing text semantics, and the resulting powerful visual text model is based on reduced text noise. However, the visual text model is not retrievable because the input of the image retrieval model in the retrieval stage is only an image. Therefore, the knowledge distillation mechanism adopted by structure 7 introduces a retrievable visual model to learn the powerful but non-retrievable visual text model, and transfers the performance of the visual text model to a simple and efficient retrievable single-modal visual model through learning distribution. 2.The weakly supervised hash image retrieval system with knowledge graph embedding attention mechanism of claim 1, wherein, The noise mask matrix, specifically the prior knowledge provided by the knowledge graph, can provide word-to-word similarity for the text, thereby forming a correlation matrix. The matrix masks words with low correlation, and in the calculation of the self-attention of the words, the attention between the mask marks and the noise marks is calculated, thereby reducing the influence of noise or error marks on the self-attention score.

Images