Knowledge relationship-based retrieval enhancement method, model, device and storage medium

Abstract

The embodiment of the application relates to the technical field of large model migration learning, in particular to a knowledge relationship retrieval enhancement method, a model, equipment and a storage medium, the method comprising the following steps: firstly, constructing a memory; the memory comprises an external memory and an enhanced memory; then, using a downstream data set as a query to retrieve knowledge from the external memory; using the retrieved knowledge as a retrieval query to retrieve semantic text features from the enhanced memory; next, based on the semantic text features and original image features, obtaining final enhanced features of image classification; finally, based on the final enhanced features, enhancing the original image features. The knowledge relationship retrieval enhancement method provided by the embodiment of the application uses a large-scale external data set as a medium to retrieve more effective knowledge from the memory to enhance the original features, so as to improve the classification performance of the model.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of large model transfer learning, in particular to a knowledge relationship retrieval enhancement method, model, device and storage medium. BACKGROUND

[0002] The visual language base model learns a large amount of knowledge from large-scale pre-training data, and how to transfer these knowledge to downstream visual tasks is worth studying. The traditional fine-tuning strategy will inevitably lose the knowledge useful for downstream tasks. At present, the existing retrieval enhancement method utilizes the internal memory of data through element addressable content, which can alleviate the knowledge loss problem of the base model and help them better transfer knowledge to improve performance. However, this method directly utilizes the feature similarity between samples to retrieve knowledge, and effective knowledge may be missed. SUMMARY

[0003] Embodiments of the present application provide a knowledge relationship retrieval enhancement method, model, device and storage medium, which utilizes large-scale external data sets as a medium to retrieve more effective knowledge from memory to enhance the original features, so as to improve the classification performance of the model.

[0004] To solve the above technical problems, in a first aspect, embodiments of the present application provide a knowledge relationship retrieval enhancement method, comprising the following steps: first, constructing a memory; the memory includes an external memory and an enhanced memory; then, using a downstream data set as a query to retrieve knowledge from the external memory; using the retrieved knowledge as a retrieval query to retrieve semantic text features from the enhanced memory; next, based on the semantic text features and the original image features, obtaining the final enhanced features for image classification; finally, enhancing the original image features based on the final enhanced features.

[0005] In some example embodiments, based on the semantic text features and the original image features, the final enhanced features for image classification are obtained, comprising: using the encoder of the transformer to fuse the semantic text features to obtain the final response features; and obtaining the final enhanced features for image classification by combining the original image features and the final response features.

[0006] In some example embodiments, the memory is constructed, comprising: constructing the external memory and the enhanced memory respectively; the external memory is composed of features extracted by a large model from a large-scale data set; the enhanced memory retrieves knowledge from the external memory by using each downstream data sample as a query, and stores the retrieved knowledge as a key value in the enhanced memory, and the value corresponding to each key value is generated by the label text feature of the corresponding query.

[0007] In some example embodiments, retrieving the knowledge as a retrieval query, retrieving the semantic text features from the enhanced memory includes: comparing the retrieved knowledge as a retrieval query with each key value in the enhanced memory to obtain a similarity matrix; and obtaining the retrieved effective semantic text features from the enhanced memory according to the similarity matrix.

[0008] In some example embodiments, the similarity calculation formula is as follows:

[0009]

[0010] wherein i represents a sample index, and p and q represent indices of Q and K respectively.

[0011] In a second aspect, the embodiments of the present application also provide a knowledge relationship retrieval enhancement model, which includes a memory construction module, a knowledge relationship retrieval module, and an image feature enhancement module. The memory construction module includes an external memory construction module and an enhanced memory construction module, and is configured to construct an external memory and an enhanced memory respectively. The knowledge relationship retrieval module includes a first retrieval module and a second retrieval module. The first retrieval module is configured to retrieve knowledge from the external memory and use the retrieved knowledge as a retrieval query in the enhanced memory. The second retrieval module is configured to retrieve semantic text features from the enhanced memory. The image feature enhancement module is configured to obtain final enhanced features of image classification according to the semantic text features and original image features, and enhance the original image features based on the final enhanced features.

[0012] In some example embodiments, the external memory module extracts image features from a large-scale data set by using a visual encoder pre-trained on the large-scale data set.

[0013] In some example embodiments, after extracting the image features from the large-scale data set, the extracted image features are stored in the external memory module as knowledge.

[0014] In addition, the present application also provides an electronic device, which includes at least one processor, and a memory in communication connection with the at least one processor. The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above-mentioned knowledge relationship retrieval enhancement method.

[0015] In addition, the present application also provides a computer readable storage medium storing a computer program, which is executed by a processor to implement the above-mentioned knowledge relationship retrieval enhancement method.

[0016] The technical solutions provided by the embodiments of the present application have at least the following advantages:

[0017] The embodiment of the present application provides a knowledge relationship retrieval enhancement method, model, device and storage medium. The method comprises the following steps: firstly, constructing a memory; the memory comprises an external memory and an enhanced memory; then, using a downstream data set as a query to retrieve knowledge from the external memory; using the retrieved knowledge as a retrieval query to retrieve semantic text features from the enhanced memory; next, based on the semantic text features and original image features, obtaining final enhanced features of image classification; finally, based on the final enhanced features, enhancing the original image features.

[0018] The knowledge relationship retrieval enhancement method provided by the present application uses a large-scale data set to construct a memory, and uses a large model to extract features and store them in the memory, so as to utilize the rich knowledge in the large-scale data set through the memory. The features retrieved from the memory by the present application for generating response features are text features generated by using the labels associated with downstream samples. Compared with typical features fused by using downstream image data, the method used by the present application has stronger semantics and reduces confusion, thereby improving the performance of the model.

[0019] In addition, the present application uses a transformer encoder to summarize the retrieved text features to obtain final response features, so as to enhance the original image features. This way does not limit the number of texts, can not waste the retrieved text features, and better generates effective response features. At the same time, the present application uses an external data set as a bridge connecting downstream data sets, compares the similarity between the retrieved knowledge to obtain more effective memory slots. In addition, the external memory is used as a medium for establishing knowledge relationship, which can realize plug and play without the need of reorganizing different data sets. BRIEF DESCRIPTION OF DRAWINGS

[0020] One or more embodiments are illustrated by way of example in the accompanying drawings that are not intended to be limiting of the embodiments, unless otherwise explicitly stated. The drawings depict one or more embodiments by way of example, not by way of limitation. Limitation of the accompanying drawings is not intended to constitute a proportion.

[0021] Figure 1 A flowchart of a knowledge relationship retrieval enhancement method provided by an embodiment of the present application is shown in the figure;

[0022] Figure 2 A structure diagram of a knowledge relationship retrieval enhancement model provided by an embodiment of the present application is shown in the figure;

[0023] Figure 3 A knowledge relationship retrieval example provided by an embodiment of the present application is shown in the figure;

[0024] Figure 4 An architecture diagram of a knowledge relation retrieval enhanced base model provided for an embodiment of the present application is provided.

[0025] Figure 5 A qualitative comparison diagram of knowledge relations and representation relations provided for an embodiment of the present application is provided.

[0026] Figure 6 An influence diagram of hyperparameters k on model performance on CIFAR100 provided for an embodiment of the present application is provided.

[0027] Figure 7 A structure diagram of an electronic device provided for an embodiment of the present application is provided. DETAILED DESCRIPTION

[0028] As known from the background art, in the existing retrieval enhancement method, since the knowledge is directly retrieved by using the feature similarity between samples, there is a problem that effective knowledge is missed.

[0029] Vision and text are two main forms of human perception. Large-scale models trained using large-scale image-text datasets using contrastive learning can learn rich knowledge and extract powerful features. Applying such large models to various downstream vision tasks has achieved significant improvement.

[0030] Large models store rich knowledge using parameters. How to use the knowledge in these large models is worth studying. The common method is to use the fine-tuning strategy to adapt the pre-trained model to the specific distribution of the downstream task. However, this method will inevitably lead to the occurrence of catastrophic forgetting. On the contrary, using large models to extract the knowledge of large data, storing these knowledge element by element in an addressable memory space, and using a way to read and write knowledge information from the memory space can more effectively alleviate catastrophic forgetting and better transfer knowledge from large models to perform downstream tasks. Extracting features from large-scale datasets and storing them element by element in an external repository, and explicitly retrieving beneficial knowledge to enhance downstream data, is similar to the human memory system used to store, retrieve, and replay knowledge in cognitive neuroscience and related disciplines, and is worth studying.

[0031] Memory-augmented networks have been applied to various fields such as memory network problems, meta-learning, data generation, and language modeling. The use of memory-augmented networks for large model knowledge transfer has also achieved significant improvement in the NLP field. Recently, for image classification, a typical class feature memory-augmented module has been designed, and the memory memory stores the typical features of each class. Such typical features are obtained through a moving average method. Then, through an attention-based retrieval mechanism, the relevant typical class features in the memory memory are retrieved using the downstream dataset to generate response features to enhance the original features. The final generated enhanced features are used for the final image classification.

[0032] Another related technology proposes a retrieval augmented classification (Retrieval Augmented Classification) network, which consists of an image encoder and a parallel retrieval augmentation branch. The image encoder branch is an image pre-training model based on a CLIP large model that extracts image original features from the downstream dataset images. The downstream data samples are used as queries (Query), and the image features in the memory memory are used as keys (Key), and the corresponding text is used as value (Value), that is, the memory memory is stored in the form of key-value (Key-Value). The retrieval branch uses the query (Query) and the key (Key) to calculate the similarity, then retrieves the top K text entries according to the similarity, then concatenates the K text entries into a total text entry, and then inputs the total text entry into a text pre-training model based on a CLIP large model to generate response features to enhance the original features. The final generated enhanced features are used for the final image classification.

[0033] In addition, there is also a related technology that first classifies an external dataset, then extracts features from the downstream data samples, uses the extracted features as queries (Query) to retrieve similar knowledge samples from the external dataset, and uses the retrieved knowledge samples to vote, and the sample with the highest number of votes is the predicted label of the sample.

[0034] In the existing related technology method, when constructing the memory memory, only the downstream dataset is used to construct the memory memory, and the rich knowledge in the large-scale dataset is not utilized. The value (Value) stored in the memory memory is the typical feature of each type, and the typical feature of each type is obtained by fusing the relevant downstream data. The fused features will cause confusion in the typical features.

[0035] The text tokens with high similarity are retrieved by enhanced branch retrieval, and then the text tokens are spliced into a total text token, which is input into the CLIP text encoder to obtain the response feature. However, the CLIP-based text encoder has a limit on the number of text tokens that can be processed, and this limit also limits the generation of effective response features. Secondly, the retrieved text tokens have many noise tokens, which also have a bad effect on the model performance. Finally, only the sample relationship between the downstream data sample (query) and the memory key value is used for retrieval, and this coarse-grained retrieval method cannot retrieve effective and useful knowledge.

[0036] For the method of artificially increasing the explainability of the model, the features in the external data set are first classified, and then the downstream data set is tested on the external data set. This way requires reorganizing the external data set for each downstream data set, and reorganizing the data each time the external data set is replaced.

[0037] To solve the above technical problems, the embodiments of the present application provide a knowledge relation retrieval enhancement method, which includes the following steps: first, constructing a memory; the memory includes an external memory and an enhanced memory; then, using a downstream data set as a query to retrieve knowledge from the external memory; using the retrieved knowledge as a retrieval query to retrieve semantic text features from the enhanced memory; next, based on the semantic text features and the original image features, obtaining the final enhanced features of image classification; finally, based on the final enhanced features, enhancing the original image features. The embodiments of the present application provide a knowledge relation retrieval enhancement method, model, device and storage medium, through a knowledge relation retrieval enhancement foundation model based on knowledge relation, using a large-scale external data set as a medium to retrieve more effective knowledge from the memory to enhance the original features, so as to improve the classification performance of the model.

[0038] The embodiments of the present application will be described in detail below with reference to the accompanying drawings. However, those skilled in the art can understand that in the embodiments of the present application, many technical details are proposed in order to make the reader better understand the present application. However, the technical solutions claimed by the present application can be realized even without these technical details and various changes and modifications based on the following embodiments.

[0039] Referring to Figure 1 The embodiments of the present application provide a knowledge relation retrieval enhancement method, which includes the following steps:

[0040] Step S1, constructing memory; the memory includes an external memory and an enhanced memory.

[0041] Step S2, retrieving knowledge from the external memory using the downstream data set as a query; retrieving semantic text features from the enhanced memory using the retrieved knowledge as a retrieval query.

[0042] Step S3, obtaining final enhanced features of image classification based on the semantic text features and the original image features.

[0043] Step S4, enhancing the original image features based on the final enhanced features.

[0044] The embodiment of the application provides a knowledge relationship retrieval enhancement method, which first constructs two memory memories, including an external memory and an enhanced memory. The external memory extracts rich knowledge from a large-scale data set. The enhanced memory is used to retrieve related knowledge from the external memory using a downstream training data set, and the knowledge is stored in the enhanced memory as a key (Key). The corresponding label (Label) is used to generate a text feature as a value (Value) stored in the enhanced memory. Then, the downstream data set is used as a query (Query) to retrieve related knowledge from the external memory, and the knowledge is used as a retrieval query (Query) to retrieve effective semantic text features from the enhanced memory. Finally, the original image features and the semantic text features are combined to obtain the final features of image classification. The knowledge relationship retrieval enhancement base model proposed in the application significantly improves the classification performance of the model on four image classification data sets of the West.

[0045] Figure 2 A knowledge relationship retrieval example is shown, wherein, Figure 2 The arc-shaped arrow represents direct retrieval of knowledge using the similarity between samples, and the horizontal straight arrow retrieves effective values using the similarity of sample-related knowledge. The external memory repository is composed of a large-scale data set, providing rich knowledge to construct the relationship between samples.

[0046] As Figure 2As shown, the existing memory retrieval enhancement method is directly used to retrieve knowledge by comparing the similarity between the query and the key (arc-shaped arrow), and the similar and effective knowledge does not have special similarity at the feature level. Such retrieval method cannot retrieve effective knowledge. The present application proposes that the external memory memory serves as a medium for the internal memory memory (straight arrow), which can provide the possibility for the retrieval enhancement module to retrieve more useful knowledge. The external memory memory is generated by an external large-scale dataset, and the internal memory memory is generated by a downstream dataset. The external memory memory provides a medium for the internal memory memory to establish knowledge relationships, and the external memory memory can be replaced by other larger datasets without the need for two training.

[0047] Based on the prior knowledge that the feature gap between samples of the same category is relatively large, but the knowledge retrieved from the memory memory by the same category sample is relatively large. The present application proposes a knowledge relationship retrieval enhancement method and model, which converts the determination of the similarity degree of the sample features into the determination of the similarity degree by comparing the knowledge features retrieved by the sample. A large-scale external dataset can establish a bridge between samples of the same category.

[0048] Specifically, the method of the present application based on the knowledge relationship retrieval enhancement basic model contains two stages. The first stage is to construct an external memory memory and an enhanced memory memory. The external memory memory is composed of features extracted by a large model from a large-scale dataset. The enhanced memory memory retrieves knowledge from the external memory memory by using each downstream data sample as a query (Query), and stores the retrieved knowledge as a key value (Key) in the enhanced memory memory. The value (Value) corresponding to each key value (Key) is generated by the label (Label) of the corresponding query (Query) text feature. The second stage uses the constructed external memory memory to retrieve useful content. The query (Query) of the downstream dataset retrieves corresponding knowledge from the external memory memory, and then compares the similarity between the retrieved knowledge and each key value (Key) in the enhanced memory. According to the obtained similarity, the effective value (Value) is retrieved, and then the high semantic text features are fused to obtain the final response feature. The original image feature is enhanced to obtain the final enhanced feature for image classification.

[0049] In some embodiments, the final enhanced feature of image classification is obtained based on the semantic text feature and the original image feature in step S3, comprising: fusing the semantic text feature by using the encoder of the transformer to obtain the final response feature; and obtaining the final enhanced feature of image classification by combining the original image feature and the final response feature. The encoder of the transformer is used to fuse and summarize the retrieved text feature to obtain the final response feature, so as to enhance the original image feature. This way does not limit the number of texts, can not waste the retrieved text features, and can better generate effective response features.

[0050] In some embodiments, the memory is constructed in step S1, comprising: constructing an external memory and an enhanced memory respectively; the external memory is composed of features extracted by a large model from a large-scale data set; the enhanced memory retrieves knowledge from the external memory by using each downstream data sample as a query, and stores the retrieved knowledge as a key value in the enhanced memory, and the value corresponding to each key value is generated by the label text feature of the corresponding query.

[0051] In some embodiments, the semantic text feature is retrieved from the enhanced memory by taking the retrieved knowledge as a retrieval query in step S2, comprising: taking the retrieved knowledge as a retrieval query, and comparing the similarity of each key value in the enhanced memory to obtain a similarity matrix; and obtaining the effective semantic text feature retrieved from the enhanced memory according to the similarity matrix.

[0052] In some embodiments, the similarity calculation formula is as follows:

[0053]

[0054] Wherein, i represents the sample index, and p and q represent the indexes of Q and K respectively.

[0055] Reference Figure 3The embodiment of the application further provides a knowledge relationship retrieval enhancement model, comprising a memory construction module 101, a knowledge relationship retrieval module 102 and an image feature enhancement module 103; the memory construction module 101 comprises an external memory construction module 1011 and an enhanced memory construction module 1012, and is used for constructing external memory and enhanced memory respectively; the knowledge relationship retrieval module 102 comprises a first retrieval module 1021 and a second retrieval module 1022, the first retrieval module 1021 is used for retrieving knowledge in the external memory and taking the retrieved knowledge as a retrieval query in the enhanced memory, and the second retrieval module 1022 is used for retrieving semantic text features in the enhanced memory; the image feature enhancement module 103 is used for obtaining final enhanced features of image classification according to semantic text features and original image features, and enhancing the original image features based on the final enhanced features.

[0056] In some embodiments, the external memory module 1011 extracts image features from a large-scale data set by using a visual encoder pre-trained on the large-scale data set.

[0057] In some embodiments, after extracting image features from the large-scale data set, the extracted image features are stored as knowledge in the external memory module 1011.

[0058] Figure 4 The architecture of the knowledge relationship retrieval enhancement base model is shown, wherein (a) is a construction process of the external memory, (b) is a construction process of the enhanced memory, and (c) is a model network flowchart, which uses the two memories to establish the knowledge relationship similarity between samples to retrieve effective response features for enhancing original visual features for image classification.

[0059] Figure 4 The entire flowchart of the knowledge relationship retrieval enhancement model of the application is shown. As shown in Figure 4 First, two memories, the external memory and the enhanced memory, are constructed. As shown in Figure 4 (a), the external memory is first created to serve as an intermediary for the downstream task data set. As shown in Figure 4 (b), the enhanced memory is then created, and the downstream task data set is used as a query to retrieve relevant knowledge from the external memory as a key value, and the corresponding label as a value. As shown in Figure 4(c) as shown, retrieval enhancement is performed using the constructed external memory and the augmented memory. Specifically, each training sample is used as a query to retrieve relevant knowledge from the external memory, and then the retrieved relevant knowledge is used as a query to compare with each key in the augmented memory to obtain a similarity matrix, and the retrieved value is obtained from the augmented memory using the obtained similarity matrix. Then, the retrieved classification features are summarized to obtain the final response features, and the original image features are augmented to obtain the final augmented features for image classification.

[0060] Given a set of samples Input to the model for training, where represents the height and width of the image, respectively, y n = {1, 2,..., L} represents the label, and L represents the total number of labels. The model is to correctly classify and predict the true label of the input sample.

[0061] As Figure 4 shown, the knowledge relationship-based retrieval enhancement model first needs to construct two memory memories: external memory and augmented memory. In Figure 4 , (a) is the construction process of the external memory; (b) is the construction process of the augmented memory; (c) is the model network flowchart. Figure 4 The present application shows the process of establishing the knowledge relationship similarity between samples using two memory memories to retrieve effective response features to strengthen the original visual features for image classification.

[0062] Specifically, the technical details of constructing the memory memory are as follows:

[0063] The specific process of constructing the external memory is as follows: as Figure 4 shown in (a), the external memory extracts image features from the YFCC15M large-scale dataset by using the ResNet50 visual encoder pre-trained on the YFFC15M large-scale dataset. YFCC15M is a subset of YFCC100M, and YFCC100M is the largest public multimedia collection consisting of 100 million media objects. The extracted image features are stored as knowledge in the external memory, and the external memory can be represented as a matrix where M represents the number of knowledge stored in the external memory, d v represents the dimension of the extracted image features. The rich knowledge extracted from the external dataset by the pre-trained large model is stored in the external memory. By comparing the knowledge retrieved from the external memory between different samples, a bridge can be built between these samples.

[0064] The specific process of building enhanced memory is as follows: Figure 4 As shown in (b), the enhanced memory consists of keys and values. Each key retrieves relevant knowledge from the external memory for each sample in the downstream data, represented as follows: Where m is the amount of retrieved knowledge stored in each slot of the enhanced memory, and d v This refers to the dimension of image features. Specifically, the process utilizes each sample x from the downstream dataset. i As a query Similarity calculation with external memory is shown below:

[0065]

[0066] Next, based on the similarity matrix Retrieve the top with the highest similarity r Index. This index is used to retrieve relevant knowledge from external memory, forming the key value K = EM[top]. r ] As for the value of enhanced memory, it first utilizes the label y of the corresponding sample. i Input into a pre-fixed template (prompt) (e.g.) Figure 4 (b) The resulting string is then input into a YFCC15M pre-trained text encoder to obtain text features. Where d t Let V be the dimension of the text feature, which corresponds to the slot value (V). The retrieval process of the augmented memory can be summarized as AM[K] = V. After obtaining two memory locations, this application obtains the final augmented features used for image classification.

[0067] By comparing the similarity of knowledge retrieved from external memory to determine the similarity between samples, more effective corresponding features can be obtained. Based on this, this application proposes a knowledge relationship retrieval module, which includes two stages: a first retrieval template (retrieval in external memory) and a second retrieval module (retrieval in enhanced memory).

[0068] Retrieval in external memory: First, each batch of images from the downstream training dataset is fed into a YFCC15M-based ResNet50 visual encoder E(·) to extract image features. Where b is the batch size, d v For image feature dimensions, the similarity between features in the training set as query features and features in external memory is calculated as follows:

[0069]

[0070] According to the similarity matrix The top s index is retrieved. The index is used to retrieve the corresponding knowledge from the external memory to form the query Q = EM[top s ]

[0071] After each round of training, the query features obtained above are used to update the augmented memory.

[0072] Retrieval in augmented memory: as shown in (c), the knowledge query Q obtained in the previous step is used to calculate the similarity with each key value K in the augmented memory. The strategy for similarity calculation is to calculate the highest score of each element matrix as the similarity score of the element. The specific similarity calculation method is as follows: Figure 4

[0073] Where i is the sample index, and p and q represent the indices of Q and K.

[0074] Using the knowledge relationship similarity matrix Sim, the top k text features T = AM[top k ]

[0075] Then, similar to the token ([class]) of Bert, a learnable encoding feature is added before the retrieved text sequence feature to obtain the final text feature which is input into the encoder of the Transformer to summarize these text features to generate response features. That is, T0 is the response feature summarized from the retrieved text features. The specific calculation process is as follows: Encoder(Q,K,V) = LN(X+Y),

[0076]

[0077]

[0078]

[0079] Where Q, K, and V are input features of the Transformer encoder, W Q ,W K ,W V ​is the projection matrix, LN (Layer Normalization) represents layer normalization, MHA (Multi-Head Attention) is a multi-head attention module with 4 heads, and FFN (Feed Forward Network) is a feed forward network.

[0080] Query features are generated using projected text features Key-value features Sum features They are input together into the encoder of the Transformer, the self-attention feature F t self = Encoder (Q, K, V). The final response feature is F t = F t self [0].

[0081] The final augmented feature F is obtained by combining the original visual feature F v and the response text feature F t . Cross-entropy is applied to calculate the final loss. The specific formula is as follows:

[0082]

[0083]

[0084] where L is the number of label categories, where is an indicator function ( and ), and FC is a classifier.

[0085] Based on this, the application proposes a novel knowledge relationship retrieval augmented base model for image classification. By displaying an addressable external memory, rich knowledge extracted by a large model on a large-scale dataset is stored. By using the external memory as a medium, the similarity of the downstream task dataset retrieving knowledge from the external memory is indirectly compared to establish the relationship between each sample of the downstream task dataset. The retrieved label text features are summarized to obtain response features to enhance the original image features for the final image classification. At the same time, the application compares the similarity of the sample retrieval knowledge to determine the similarity between the samples, which can better establish the relationship between the query and the key value, and then retrieve more effective content to enhance the original image features. In addition, the application uses the downstream task related labels to generate text features, which are used to generate response features with stronger semantics, which is beneficial to enhance the original image features. A sentence-level summary encoder is proposed, which eliminates the need to limit the number of knowledge summaries and reduces the waste of retrieved knowledge.

[0086] Compared with the prior art, the external memory is used as a medium for connecting downstream task data set samples in the application. Compared with the best method at present, the knowledge retrieved from the memory is a more semantic response feature, which more effectively enhances the original feature and improves the model performance.

[0087] In addition, the application uses a large-scale data set to construct the memory, not just the downstream data set, which can learn the rich knowledge in the large-scale data set. The typical feature obtained by fusing the feature is easy to cause feature confusion, while the discrete knowledge sample relationship comparison can improve the model interpretability, improve the model retrieval performance, and further improve the model performance. In addition, the application adopts a sentence-level knowledge summary method, which does not limit the summarized knowledge, can not waste the retrieved text features, and better generates effective response features to enhance the original image features and improve the model performance. In addition, the application uses the external memory as a medium for establishing knowledge relationships, which does not need to be reorganized multiple times for different downstream data sets, and can achieve plug and play.

[0088] The knowledge relationship retrieval enhancement method and model provided by the application have been tested and simulated, and the specific experimental process is as follows:

[0089] The application embodiments involve four data sets, including Food101, CIFAR10, CIFAR100 and Oxford102Flower. In this method, the external data set YFCC15M is used to construct the external memory. As shown in Table 1, the same training / test division method as other works is used. Food101 has 102 kinds, the training set size is 75750, the test set size is 25250, CIFAR10 has 10 categories, the training set size is 50000, the test set size is 10000, CIFAR1000 has 100 categories, the training set size is 50000, the test set size is 10000, Oxford 102Flower has 102 categories, the training set is 2040, and the test set is 6149.

[0090] Table 1 Statistics of four image classification benchmarks

[0091]

[0092] The application is implemented on Pytorch, and all models are trained on a single A100 GPU.

[0093] The present application adopts ResnNet50 pre-trained on YFCC15M. The present application uses Adam as the optimizer, weight decay 5e-4 and a small batch size of 50 for training. The present application uses a learning rate of 5e-5 and trains the network for 10 rounds. The entire network is trained in an end-to-end manner supervised by image labels.

[0094] The present application is compared with the baseline and existing methods. The baseline is fine-tuned according to the ResNet50 pre-training model pre-trained on YFCC15M. Deng et al. proposed a CMN model using an attention mechanism to obtain a similarity score as a weight to obtain a response feature from the memory memory, which is combined with the original visual feature to produce the final classification feature. RAC proposed by Long et al. uses an original attention mechanism to retrieve text information to enhance the original image features. Table 2 shows the quantitative comparison of the present application with different methods on four image classification benchmarks, as follows:

[0095] Table 2 Quantitative comparison of the present application with different methods on four image classification benchmarks

[0096]

[0097] As shown in Table 2, the accuracy of the method of the present application on Food101, CIFAR10, CIFAR100, Oxford 102Flower is 83.21%, 95.59%, 80.98% and 84.68% respectively. It is +1.25%, +0.09%, +1.29% and +0.5% higher than the baseline respectively. All quantitative results consistently demonstrate the advantages of the present application.

[0098] Ablation experiment of knowledge relation: The present application compares the two schemes of traditional sample representation relation and sample knowledge relation to prove that using knowledge relation to retrieve memory memory is more useful and effective than directly retrieving according to representation relation. The so-called representation relation (Representation Relation) is to compare the similarity between the downstream data sample and the internal memory memory, and then retrieve the top k knowledge to generate the response feature to enhance the original image feature to obtain the enhanced feature for image classification. The internal memory memory refers to the use of the downstream training dataset to extract features. The retrieval enhancement module based on knowledge relation (Knowledge Relation) proposed by the present application uses a large-scale external memory as a medium to connect the relevant knowledge of the internal memory. As shown in Table 3, the experimental results can evaluate that the knowledge relation can establish a closer relationship and obtain better performance, which shows that the knowledge relation strategy can make more use of the external memory and retrieve more useful content to enhance the original feature and improve the performance.

[0099] Table 3. Impact of knowledge relationship strategy on performance on CIFAR100

[0100]

[0101] Figure 5 Two challenging qualitative comparison results are shown. The top (first row) represents the representation relationship setting, and the bottom (second row) represents the knowledge relationship setting. The query image of the left example is a "Road". The most relevant knowledge retrieved from the external memory by the representation relationship strategy (upper left corner) is the picture labeled "Forest". The lower left corner demonstrates that the Query and Key establish the correct relationship and retrieve the valid information "Road" to achieve better performance. The query image of the right example is a "Train". Due to the influence of the background, the most relevant knowledge retrieved by the representation relationship strategy is the image labeled with "Mountain". However, the knowledge relationship strategy can establish a closer relationship between Query and Key to obtain more useful "Train" information for the final image enhancement.

[0102] As Figure 5 shown, the left example shows a picture containing "Road" content. Due to the influence of the forest background, the closest image retrieved from the memory using the representation relationship is a picture that can be described as a river in the forest, and secondly, the image is labeled as "Forest". While through the knowledge relationship, an accurate relationship can be established through the memory, and better performance can be achieved. In this example, the second image in the Query has the highest similarity with the third image in the Key from the augmented memory, and the corresponding value (Value) text feature "Road" can be retrieved to enhance the original image feature. The right example is a train winding on a mountain road. Due to the influence of the distant mountain background, the most relevant knowledge retrieved directly from the memory is about "Mountain". However, the knowledge relationship can provide more possibilities to establish relationships between samples and obtain effective classification text information "Train" from the augmented memory.

[0103] Knowledge relationship generation mode ablation study: As Figure 4 shown in (c), knowledge relationships are obtained through the similarity matrix between Query and each Key. There are at least three ways to calculate these knowledge relationships to obtain the final knowledge relationship similarity matrix. Including taking the maximum value, taking the average value and weighted average value.

[0104] Taking the average value is to take the average value of each element matrix. The specific calculation process is as follows:

[0105]

[0106] In the weighted average calculation process, the application uses the similarity matrix D retrieved from the external memory to calculate the attention weight score W, and then uses the weight score to perform weighted average on each element matrix in the knowledge relationship matrix. The specific calculation process is as follows:

[0107]

[0108]

[0109] Where i is the sample index, and p and q represent the indices of Q and K.

[0110] As shown in Table 4, the maximum method can obtain the best performance. Compared with other strategies, the most similar knowledge pair in all relationships can better represent the knowledge relationship. Regardless of which generation method is used to obtain the knowledge relationship, the application can see that all these results are better than the baseline, which can prove the effectiveness of the application.

[0111] Table 4 Ablation study of knowledge relationship generation methods on CIFAR100

[0112]

[0113] Ablation study of external memory: the application uses external memory to bridge between samples. A large-scale image-text dataset is used to construct the external memory. Text is also a form of knowledge, and it can also be used as a medium to connect samples. The ablation experiment of multi-modal knowledge relationship is also explored in the application. As shown in Table 5, “Caption Knowledge” represents retrieving related text features from the external memory as a knowledge storage form and storing them in the enhanced memory. “Caption Related Image Knowledge” represents calculating the similarity between the query and the text features in the external memory to retrieve the corresponding image features as a knowledge storage form and store them in the enhanced memory. “Image Knowledge” represents retrieving image features from the external memory as a knowledge storage form and storing them in the enhanced memory. From the experimental results, it can be seen that both image features and text features can establish knowledge relationship and obtain better performance than the baseline. “Image Knowledge” obtains the best performance in the three settings.

[0114] Table 5 Ablation study of external memory on CIFAR100

[0115]

[0116] Knowledge aggregation strategy: Figure 4 As shown in (c), the top k memory slots are retrieved from the enhanced memory. How to process these retrieved contents to obtain the final response features to enhance the original image features is worthy of investigation. Long obtains the final response features by combining all these text tokens and inputting them into a CLIP-based text encoder. It is essentially a word-level self-attention module for summarizing retrieved knowledge. However, the 76-token limit of the CLIP text encoder restricts the model's performance. Therefore, in this application, a Transformer encoder with learnable embeddings T0 is used to summarize retrieved knowledge, without being limited by the number of tokens. The proposed method is essentially a sentence-level self-attention module. As shown in Table 6, ablation experiments were conducted on the knowledge summarization strategy. The "Word-Level Encoder" performs word-level attention computation on text knowledge, and the "Sentence-Level Encoder" performs sentence-level attention computation on sentence knowledge. From the experimental results, the effectiveness of the proposed sentence-level knowledge summarization module can be observed.

[0117] Table 6 summarizes ablation studies of knowledge strategies at CIFAR100.

[0118]

[0119] The impact of r and s on the model: r and s represent the number of samples per K for the augmented memory query Q and the augmented memory, respectively. To investigate the impact of these two parameters on model performance, Table 7 shows the impact of hyperparameters r and s on model performance on CIFAR100. All results in the table are competitive with the baseline performance (79.69%).

[0120] Table 7 shows the impact of hyperparameters r and s on model performance on CIFAR100.

[0121]

[0122] The impact of k on the model: such as Figure 6 As shown, this application investigated the impact of parameter k on model performance, and achieved optimal performance when k = 500, which is equal to the number of samples of each class in the training dataset (n). c Unlike other works that set k as large as possible, the reason is that the augmented memory uses each training sample as a query and retrieves knowledge from the external memory as a key during its construction. The data distribution of the augmented memory is the same as that of the downstream training data, so k is set to n. c Therefore, if k is greater than n cMuch larger, so many lower-ranked noisy retrievals will impede the performance of the final aggregated response features. If k is much smaller than n c , then not all of the effectively retrieved knowledge can be used to summarize the knowledge. Therefore, it is reasonable to set k to be close to n c .

[0123] In the process of constructing the external memory in the present application, different pre-trained models can be used to extract features on different large model datasets as knowledge of the memory, and the form of knowledge in the memory is currently image features extracted by a pre-trained image encoder. In fact, different modalities of knowledge can still be applied, for example, knowledge can also use text features extracted by a pre-trained text encoder as knowledge, and fusion features obtained by fusing image features and text features can be used as knowledge.

[0124] In the present application, the relationship between the query and the enhanced memory key value is calculated in a discrete manner when establishing the knowledge relationship similarity. Other alternative methods can also be used to establish the knowledge relationship, such as fusing the r samples in the query to obtain abstract knowledge features, and correspondingly, the s samples in each key value in the enhanced memory can also obtain abstract knowledge features through fusion. In this way, the comparison of the knowledge relationship similarity becomes the comparison of the abstract knowledge. The specific fusion method can use addition, attention-based weighted average, or use a graph neural network or a Transformer to fuse the samples to obtain the fused abstract knowledge features.

[0125] Reference Figure 7 , another embodiment of the present application provides an electronic device, comprising: at least one processor 110; and a memory 111 in communication connection with the at least one processor; wherein the memory 111 stores instructions executable by the at least one processor 110, and the instructions are executed by the at least one processor 110 to enable the at least one processor 110 to execute any of the above method embodiments.

[0126] The memory 111 and the processor 110 are connected in a bus manner, the bus can include any number of interconnected buses and bridges, the bus connects one or more processors 110 and various circuits of the memory 111 together. The bus can also connect various other circuits such as peripheral devices, voltage stabilizers and power management circuits together, which are well known in the art, therefore, they will not be further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements such as multiple receivers and transmitters, which provide units for communicating with various other devices on the transmission medium. The data processed by the processor 110 is transmitted on the wireless medium through the antenna, further, the antenna also receives data and transmits the data to the processor 110.

[0127] The processor 110 is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interface, voltage regulation, power management and other control functions. The memory 111 can be used to store data used by the processor 110 in performing operations.

[0128] Another embodiment of the present application relates to a computer readable storage medium storing a computer program. The computer program is executed by the processor to implement the method embodiments described above.

[0129] That is, those skilled in the art can understand that all or part of the steps of the above-mentioned embodiment methods can be completed by programs instructing related hardware, the programs are stored in a storage medium, and include a plurality of instructions for causing a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the above-mentioned method of each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various storage medium capable of storing program codes.

[0130] According to the above technical solution, the embodiment of the present application provides a knowledge relationship retrieval enhancement method, model, device and storage medium, the method comprising the following steps: first, constructing a memory; the memory includes an external memory and an enhanced memory; then, using a downstream data set as a query to retrieve knowledge from the external memory; the retrieved knowledge is used as a retrieval query to retrieve semantic text features from the enhanced memory; next, based on the semantic text features and the original image features, obtaining the final enhanced features of image classification; finally, based on the final enhanced features, enhancing the original image features.

[0131] The knowledge relationship retrieval enhancement method provided by the application, on the one hand, the application constructs a memory memory using a large-scale data set, and uses a large model to extract features therefrom and stores them in the memory memory, so as to utilize the rich knowledge in the large-scale data set through the memory memory. The features retrieved from the memory memory by the application for generating a response feature are text features generated by using the labels associated with downstream samples. Compared with typical features fused by using downstream image data, the method used by the application has stronger semanticity and reduces confusion, thereby improving the performance of the model.

[0132] In addition, the application uses a transformer encoder to summarize the retrieved text features to obtain final response features, so as to enhance the original image features. This way does not limit the number of texts, can not waste the retrieved text features, and better generates effective response features. At the same time, the application regards the external data set as a bridge connecting the downstream data set, compares the similarity between the samples to obtain a more effective memory memory slot. In addition, the external memory is used as a medium for establishing knowledge relationships, which can realize plug and play without the need for reorganization when facing different data sets.

[0133] Those skilled in the art can understand that the above embodiments are specific examples for implementing the application, and in actual application, various changes can be made in form and details without departing from the spirit and scope of the application. Any person skilled in the art can make changes and modifications without departing from the spirit and scope of the application, therefore, the protection scope of the application should be limited by the scope defined in the claims.

Claims

1. A knowledge relationship-based enhanced retrieval method, characterized in that, include: Build memory; The memory includes external memory and enhanced memory; The downstream dataset is used as a query to retrieve knowledge from the external memory; the retrieved knowledge is used as a retrieval query to retrieve semantic text features from the enhanced memory. Based on the semantic text features and the original image features, the final enhanced features for image classification are obtained; The original image features are enhanced based on the final enhanced features.

2. The knowledge relationship-based retrieval enhancement method according to claim 1, characterized in that, The process of obtaining the final enhanced features for image classification based on the semantic text features and the original image features includes: The semantic text features are fused using a transformer encoder to obtain the final response features; By combining the original image features and the final response features, the final enhanced features for image classification are obtained.

3. The knowledge relationship-based retrieval enhancement method according to claim 1, characterized in that, The construction of memory memory includes: constructing external memory memory and enhancing memory memory respectively; The external memory consists of features extracted from a large dataset using a large model; The enhanced memory retrieves knowledge from the external memory by using various downstream data samples as queries, and stores the retrieved knowledge as key values ​​in the enhanced memory. The value corresponding to each key value is generated as a text feature by the corresponding query label.

4. The knowledge relationship-based retrieval enhancement method according to claim 1, characterized in that, The step of using the retrieved knowledge as a retrieval query to retrieve semantic text features from the enhanced memory includes: The retrieved knowledge is used as a search query, and its similarity is compared with each key value in the enhanced memory to obtain a similarity matrix; Based on the similarity matrix, the retrieved effective semantic text features are obtained from the enhanced memory.

5. The knowledge relationship-based retrieval enhancement method according to claim 4, characterized in that, The formula for calculating the similarity is as follows: Where i represents the sample index; Q is the augmented memory query; K is each key value in the augmented memory; r and s represent the number of samples for each K in the augmented memory query Q and the augmented memory query, respectively; p and q represent the indices of Q and K, respectively; and T is the text feature.

6. A knowledge relationship-based enhanced retrieval model, characterized in that, include: The system includes a memory construction module, a knowledge relationship retrieval module, and an image feature enhancement module. The memory construction module includes an external memory construction module and an enhanced memory construction module, which are used to construct external memory and enhanced memory respectively. The knowledge relationship retrieval module includes a first retrieval module and a second retrieval module. The first retrieval module is used to retrieve knowledge from the external memory and use the retrieved knowledge as a retrieval query in the enhanced memory. The second retrieval module is used to retrieve semantic text features from the enhanced memory. The image feature enhancement module is used to obtain the final enhanced features for image classification based on the semantic text features and the original image features; And based on the final enhanced features, the original image features are enhanced.

7. The knowledge relationship-based retrieval enhancement model according to claim 6, characterized in that, The external memory memory building module extracts image features from large-scale datasets by utilizing a visual encoder pre-trained on a large-scale dataset.

8. The knowledge relationship-based retrieval enhancement model according to claim 7, characterized in that, After extracting image features from a large dataset, the extracted image features are stored as knowledge in the external memory memory building module.

9. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the knowledge relationship-based retrieval enhancement method as described in any one of claims 1 to 5.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the knowledge relationship-based retrieval enhancement method as described in any one of claims 1 to 5.

Images