Knowledge-guided small sample image recognition system

Abstract

The present invention belongs to the field of computer technology, and specifically relates to a small sample image recognition system based on knowledge guidance. The system of the present invention includes a parameter initialization module based on knowledge guidance and a small sample classifier training module. The present invention first constructs a visual and semantic knowledge graph, and then uses the knowledge graph to transfer knowledge from a pre-trained classifier to a small sample data set through a parameter initialization module, and finally generates the initialization parameters of the small sample classifier. The small sample classifier training module then uses the obtained initialization parameters to initialize the parameters of the small sample classifier, and trains the small sample image classifier on the small sample data set for recognizing small sample images. The introduction of a dual-stream knowledge graph in the present invention can provide stronger connections between categories, provide better initialization parameters for the small sample classifier, and enable the classifier to converge quickly when there are fewer samples, thereby improving the classification performance of the classifier.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a small sample image recognition system. Background Art

[0002] In recent years, deep learning technology has made breakthrough progress. Deep network models are now widely used in real-life applications such as face recognition and autonomous driving. However, the success of deep models often relies on large amounts of manually labeled data. For example, training an image classification model typically requires collecting thousands of samples for each category and conducting long-term learning[1]. In contrast, humans can quickly learn new concepts in a short period of time. Therefore, how to enable machines to learn effectively from a small number of samples like humans has become an important research direction.

[0003] Currently, there are some related research works targeting small-sample learning tasks, and some progress has been made. Based on the different focuses of the methods, existing work can be roughly divided into three categories: small-sample recognition technology based on meta-learning, which mainly studies how to transfer the model's learning experience on a large number of learning tasks to achieve small-sample recognition; small-sample recognition technology based on data enhancement, which mainly studies how to design methods to expand limited data sets to improve the performance of the constructed model; small-sample recognition technology that introduces semantic relationships, which mainly uses the relationship between high-level semantic concepts to help establish connections between visual concepts for better small-sample recognition. In small-sample learning, prior knowledge can help the model effectively utilize existing learning experience and achieve rapid learning on a small number of samples. Therefore, the introduction of prior knowledge is very important for small-sample learning.

[0004] Of the three approaches mentioned above, only the last one leverages prior knowledge. However, current research is limited to utilizing textual semantic concepts. Because semantic textual relationships cannot fully reflect visual relationships, their effectiveness in small-sample visual recognition tasks is often limited. Therefore, fully tapping into visual prior knowledge and exploring small-sample recognition methods based on multimodal knowledge, thereby developing knowledge-driven small-sample learning theory and technology, is of great research significance and scientific value.

[0005] Based on the above understanding, this project is inspired by the learning process of the biological brain and studies the theory and technology of small-sample visual recognition based on knowledge-driven. Biological research shows [2] that the learning process of the biological brain does not start from scratch, but has important prior knowledge at the beginning of learning, including what species have learned in the process of evolution (called phylogeny in biology) and key knowledge about the real world learned by individuals in the process of life. This knowledge plays a very important role in the learning process of the biological brain. This also constitutes the theoretical basis of the knowledge-driven small-sample visual recognition task. However, how to construct, represent and utilize prior knowledge in visual recognition tasks so that the model can effectively learn with a small number of samples has become a very meaningful scientific problem. Summary of the Invention

[0006] The purpose of the present invention is to provide a small sample image recognition system based on knowledge guidance.

[0007] The knowledge-guided small-sample image recognition system proposed in this paper uses deep learning technology and mainly includes the following two modules: a knowledge-guided classifier parameter initialization module (hereinafter referred to as the parameter initialization module) and a small-sample classifier training module. First, the parameter initialization module uses the knowledge graph to transfer knowledge from the pre-trained classifier to the small-sample database, generating initialization parameters for the small-sample classifier. Then, the small-sample classifier training module uses the obtained initialization parameters to initialize the parameters of the small-sample classifier and trains it on the small-sample dataset to obtain a small-sample image classifier for recognizing small-sample images.

[0008] In the present invention, the small sample image refers to an image to which the category has very few training samples (generally, the number of samples is less than or equal to 5).

[0009] In the present invention, the classifier parameter initialization module first constructs visual relationship and semantic relationship knowledge graphs respectively to establish the connection between the categories in the pre-trained model and the small sample categories to be classified.

[0010] The specific steps for constructing a visual relationship knowledge graph are as follows:

[0011] (1) Taking categories as units, average the small sample categories and all training samples in each category in the source database, and extract the visual feature center of each category;

[0012] (2) Calculate the cosine similarity between each category;

[0013] (3) Use the adjacency matrix to represent the knowledge graph and set the threshold T v, two categories whose cosine similarity is greater than the threshold are considered to be connected on the visual knowledge graph, which is indicated by marking 1 at the corresponding position on the adjacency matrix.

[0014] The specific steps for constructing a semantic relationship knowledge graph are as follows:

[0015] (1) Calculate the distance between two categories in the wordnet hierarchy. The distance is expressed as the number of hops. For example, the distance from a category to its father or son category node is 1, and the distance to its grandfather or grandson node is 2.

[0016] (2) Use the adjacency matrix to represent the knowledge graph and set the threshold T s , two categories whose distance is less than the threshold are considered to be connected on the semantic knowledge graph, which is indicated by marking 1 at the corresponding position on the adjacency matrix.

[0017] After obtaining the visual and semantic knowledge graph, the present invention adopts the visual and semantic dual-stream graph convolutional network model (specifically the GCN module, see Figure 1 ), using visual and semantic knowledge graphs to transfer knowledge from the pre-trained two-stream graph convolutional network model to a small sample dataset, and generate initialization parameters for the small sample classifier. The specific steps and the two-stream graph convolutional network model proposed in this invention are as follows:

[0018] (1) Taking categories as units, average the small sample categories and all training samples of each category in the source database to extract the visual feature center of each category; use the word vector of each category name as the semantic feature center of the category;

[0019] (2) The visual feature centers and semantic features of all categories are input into the visual and semantic dual-stream graph convolutional network, and the classifier of the source data set category is used (the present invention uses a fully connected layer with the same dimension as the visual feature dimension as the classifier) ​​to supervise the output of the source data category node. The loss function used for supervision adopts the mean square loss function L.

[0020] The dual-stream graph convolutional neural network used in the present invention has the following specific calculation formula:

[0021]

[0022] Among them, σ is the leakyReLU activation function, l is the number of layers of the graph neural network, and H (l+1) is the output of the lth layer of the graph neural network, A s is the adjacency matrix of the semantic relationship knowledge graph, A v is the adjacency matrix of the visual relationship knowledge graph, H (l) is the input of the two-stream graph convolutional neural network when l is not equal to 0, θ (l,s)is the parameter of the semantic relationship flow graph convolutional neural network in the l-th layer of the dual-flow graph convolutional neural network, θ (l,v) is the parameter of the visual relationship flow graph convolutional neural network in the l-th layer of the dual-flow graph convolutional neural network, F s is the semantic input of the 0th layer of the graph convolution, which is specifically the word vector of the category name in this invention, F V It is the visual input of the 0th layer of the graph convolution, which is specifically the average value of the CLIP visual features of all training samples in the category in the present invention.

[0023] The loss function L uses the L2 distance between the pre-trained classifier parameters and the output of the last layer of the graph convolutional neural network. The specific mathematical formula for calculation is as follows:

[0024]

[0025] Where m is the total number of categories in the source dataset, d is the dimension of the classifier, and W base are the pre-trained classifier parameters, is the output of the last layer of the graph convolutional network.

[0026] In the present invention, the small sample classifier training module uses the initialization parameters generated by the parameter initialization module to fine-tune the small sample data set, ultimately obtaining the small sample classifier parameters. This module uses a sigmoid activation layer to allow the trained classifier to be directly superimposed on the pretrained classifier, resulting in good classification accuracy for both the pretrained and small sample training sets. The specific steps are as follows:

[0027] (1) Sample N images and obtain pre-extracted visual features and labels;

[0028] (2) The visual features of the image are passed through the Softmax activation function to obtain the predicted classification probability of each category;

[0029] (3) Use cross-entropy to calculate the loss and backpropagate to train the classifier;

[0030] (4) Repeat steps (1)-(3) until the number of cycles reaches the set value.

[0031] The specific steps of the testing process of the present invention are as follows:

[0032] (1) Sample N images and obtain pre-extracted visual features and labels;

[0033] (2) The visual features of the image are activated through the Softmax function to obtain the predicted classification probability of each category;

[0034] (3) The category with the highest classification probability is taken as the predicted category.

[0035] The main innovations of the present invention are:

[0036] (1) This paper proposes for the first time to apply semantic knowledge graph and visual knowledge graph to small sample image classification tasks;

[0037] (2) The present invention proposes a knowledge-based large-scale small-sample learning framework that utilizes a two-stream relationship knowledge graph between the categories of the source dataset and the categories of the small-sample dataset to assist in generating the initialization parameters of the small-sample classifier. Since the introduction of the two-stream knowledge graph can provide stronger connections between categories, the present invention can provide better initialization parameters for the small-sample classifier. This enables the classifier to converge quickly with fewer samples, thereby improving the classification performance of the classifier. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Flowchart of training and testing of the present invention. DETAILED DESCRIPTION

[0039] The present invention uses ImageNet as the source data set to pre-train the model and uses the CLIP features of the image as the visual features of the image.

[0040] Step 1. Pre-train a classifier on ImageNet The specific process is as follows:

[0041] (1) Extract and store the CLIP features of all images. The CLIP features use the ViT-L / 14 model provided in the CLIP paper, with an output dimension of 768.

[0042] (2) Select 32 images and obtain their CLIP features and corresponding labels;

[0043] (3) Use the fully connected layer as the classifier. The dimension of the fully connected layer is the same as the visual feature dimension, which is 768. The classification result is passed through a layer of Sigmoid activation function.

[0044] (4) Cross-entropy loss back propagation is used for training, and the back propagation learning rate is set to 0.01;

[0045] (5) Repeat steps (2)-(5) until the cycle reaches 5000 times.

[0046] Step 2. Construct a visual relationship knowledge graph and a semantic relationship knowledge graph, where:

[0047] The specific process of constructing a visual relationship knowledge graph is as follows:

[0048] (1) Taking categories as units, the visual features of the small sample and all training samples in each category in the source dataset are averaged and extracted to the feature center of each category;

[0049] (2) Calculate the cosine similarity between each category;

[0050] (3) Using the adjacency matrix A T Represents the knowledge graph and sets the threshold T v The cosine similarity is 0.9. Two categories with a cosine similarity greater than this threshold are considered to be connected in the visual knowledge graph, which is indicated by marking 1 at the corresponding position in the adjacency matrix.

[0051] The specific process of constructing a semantic relationship knowledge graph is as follows:

[0052] (1) Calculate the distance between two categories in the wordnet hierarchy. The distance is expressed as the number of hops between the two categories. For example, the distance from a category to its father or son category node is 1, and the distance to its grandfather or grandson node is 2.

[0053] (2) Using the adjacency matrix A s Represents the knowledge graph and sets the threshold T s The distance is 8 hops. Two categories with a distance less than this threshold are considered to be connected on the semantic knowledge graph, which is indicated by marking 1 at the corresponding position on the adjacency matrix.

[0054] Step 3. Visual and semantic dual-stream graph convolutional network uses visual and semantic knowledge graphs to transfer the knowledge in the pre-trained model to the small sample dataset and generate the initialization parameters of the small sample classifier; the specific process is as follows:

[0055] (1) Taking categories as units, average the small sample categories with all training samples in each category in the source database to extract the visual feature center of each category. The word vector of each category name is used as the semantic feature center of the category;

[0056] (2) The visual feature centers and semantic features of all categories are input into the visual and semantic dual-stream graph convolutional network. The classifier of the source dataset category is used to supervise the output of the source data category node. The loss function used for supervision is the mean square loss function. Its mathematical formula is as follows:

[0057]

[0058] Among them, σ is the leakyRelu activation function, l is the number of layers of the graph neural network, specifically 6 layers, H (l+1) is the output of the lth layer of the graph neural network, A s is the adjacency matrix of the semantic relationship knowledge graph, A vis the adjacency matrix of the visual relationship knowledge graph, H (l) is the input of the two-stream graph convolutional neural network when l is not equal to 0, θ (l,s) is the parameter of the semantic relationship flow graph convolutional neural network in the l-th layer of the dual-flow graph convolutional neural network, θ (l,v) is the parameter F of the visual relationship flow graph convolutional neural network in the lth layer of the dual-flow graph convolutional neural network s is the semantic input of the 0th layer of the graph convolution, which is specifically the word vector of the category name in this invention, F V It is the visual input of the 0th layer of the graph convolution, which is specifically the average value of the CLIP visual features of all training samples in the category in the present invention.

[0059] The mathematical formula for calculating the loss L is as follows:

[0060]

[0061] Where m is the total number of categories in the source dataset, d is the dimension of the classifier, and W base are the pre-trained classifier parameters, is the output of the last layer of the graph convolutional network.

[0062] Step 4. Use the initialization parameters generated by the parameter initialization module to fine-tune the small sample data set to obtain the parameters of the small sample classifier. This module uses a sigmoid activation layer so that the trained classifier can be directly superimposed on the pre-trained classifier, so that the final classifier can achieve good classification accuracy on both the pre-trained training set and the small sample training set. The specific process is as follows:

[0063] (1) Sample 32 images and obtain pre-extracted visual features and labels;

[0064] (2) The visual features of the image are activated by the Sigmoid function to obtain the predicted classification probability of each category;

[0065] (3) Use cross-entropy to calculate the loss and back-propagate to train the classifier. The back-propagation learning rate is set to 0.001.

[0066] (4) Repeat (1)-(3) until the number of cycles reaches the set value.

[0067] Step 5. The testing process of the present invention is as follows:

[0068] (1) Sampling 64 images and obtaining pre-extracted visual features and labels;

[0069] (2) The visual features of the image are activated by the Sigmoid function to obtain the predicted classification probability of each category;

[0070] (3) The category with the highest classification probability is taken as the predicted category.

[0071] The present invention uses accuracy as the evaluation index of the method, specifically dividing the number of correctly classified samples by the total number of test samples.

[0072] The experimental results of the present invention and the comparison results with other methods are as follows:

[0073] Methods / Training 1 2 3 4 5 S-init 64.94 78.13 83.63 85.72 86.75 V-init 66.18 74.81 82.09 85.35 86.78 PEME-BMS[4] 67.47 77.34 82.18 85.40 85.77 PMF[5] 59.53 73.90 80.31 82.78 84.36 The present invention 67.53 78.19 84.01 85.91 86.95

[0074] In this table, the V-init method averages all visual features extracted using the CLIP visual encoder for each category and initializes the classifier with the average visual features. The S-init method uses the CLIP semantic encoder to extract the semantic features of the category name for each category. PEME-BMS and PMF are the methods proposed in papers [4] and [5] respectively.

[0075] Initialization parameters for the classification. If the category has multiple names, this will average the semantic features of all the names extracted and initialize the classifier with the average visual features.

[0076] The data presented in this table provide the following observations: First, the performance of the feature extraction network using CLIP as a feature is excellent, indicating that CLIP learns stronger features. Second, compared to other comparison methods and baseline methods, the proposed method achieved the best performance for the 1-5 shot small sample classification tasks on both datasets. For the 1 shot classification task, the proposed method achieved an accuracy of 67.53% on ImageNet1000 / 360, validating its effectiveness. This also demonstrates that the proposed method also works well on higher-performing feature extraction networks. Third, compared to the PEME-BMS algorithm and the PMF algorithm, the proposed method performed better on the used datasets, especially for the 1- and 2-shot small sample classification tasks. This is primarily due to the introduction of knowledge, which enables the relationship between the base class and the new class to be exploited, leading to better performance. The results also demonstrate that even when extracting features from the large-scale pre-trained CLIP model, the introduction of knowledge is important for addressing data insufficiency.

[0077] References

[0078] [1]Krizhevsky,A.,I.Sutskever and G.E.Hinton.ImageNet Classificationwith Deep Convolutional Neural Networks[C].Proceedings of the Advances inNeural Information Processing Systems.2012.

[0079] [2] Danko.Practopoiesis:Or how life fosters a mind[J].Journalof Theoretical Biology 373(2015):40-61.

[0080] [3]Radford,Alec,et al."Learning transferable visual models fromnatural language supervision."International Conference on MachineLearning.PMLR,2021.

[0081] [4]HU Y,PATEUX S,GRIPON V.Squeezing backbone feature distributions tothe max for efficient few-shot learning[J].Algorithms,2022,15(5):147.

[0082] [5]HU S X,LI D, J,et al.Pushing the limits of simplepipelines for few-shot learning:External data and fine-tuning make adifference[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision andPattern Recognition.2022:9068-9077。

Claims

1. A knowledge-guided small sample image recognition system, characterized by: It adopts deep learning technology and includes two modules: a knowledge-guided classifier parameter initialization module, referred to as the parameter initialization module, and a small sample classifier training module; The parameter initialization module uses the knowledge graph to transfer knowledge from the pre-trained classifier to the small sample database, generating the initialization parameters of the small sample classifier. The small sample classifier training module uses the obtained initialization parameters to initialize the parameters of the small sample classifier, trains on the small sample dataset, and obtains a small sample image classifier for recognizing small sample images. In the parameter initialization module: First, we construct visual relationship and semantic relationship knowledge graphs to establish the connection between the categories in the pre-trained model and the small sample categories to be classified. The specific steps for constructing a visual relationship knowledge graph are as follows: (1) Taking categories as units, average the small sample categories and all training samples in each category in the source database, and extract the visual feature center of each category; (2) Calculate the cosine similarity between each category; (3) Use the adjacency matrix to represent the knowledge graph and set the threshold T v , two categories whose cosine similarity is greater than the threshold are considered to be connected on the visual knowledge graph, and the corresponding positions on the adjacency matrix are marked with 1; The specific steps for constructing a semantic relationship knowledge graph are as follows: (1) Calculate the distance between two categories in the wordnet hierarchy. The distance is expressed as the number of hops, that is, the distance from a category to its father or son category node is 1, and the distance to its grandfather or grandson node is 2; (2) Use the adjacency matrix to represent the knowledge graph and set the threshold T s , two categories whose distance is less than the threshold are considered to be connected on the semantic knowledge graph, and the corresponding positions on the adjacency matrix are marked with 1; Then, we use the visual and semantic dual-stream graph convolutional network model to transfer the knowledge in the pre-trained dual-stream graph convolutional network model to the small sample dataset using the visual and semantic knowledge graph, and generate the initialization parameters of the small sample classifier. The specific steps are as follows: (1) Taking categories as units, average the small sample categories and all training samples of each category in the source database to extract the visual feature center of each category; use the word vector of each category name as the semantic feature center of the category; (2) Input the visual feature centers and semantic features of all categories into the visual and semantic dual-stream graph convolutional network, and use the classifier of the source dataset category to supervise the output of the source data category node. The loss function used for supervision adopts the mean square loss function. L ; The dual-stream graph convolutional neural network is specifically expressed as follows: （1） Among them, σ is the leakyReLU activation function, l is the number of layers of the graph neural network, H （l+1） For graph neural network l The output of the layer, A S is the adjacency matrix of the semantic relationship knowledge graph, A V is the adjacency matrix of the visual relationship knowledge graph, H （1） For the time l The input of the two-stream graph convolutional neural network when it is not equal to 0, q （l,s） For the l Parameters of the semantic relation flow graph convolutional neural network in the layer two-stream graph convolutional neural network, q （l,v） For the first l Parameters of the visual relationship flow graph convolutional neural network in the layer two-stream graph convolutional neural network, F S It is the semantic input of the 0th layer of the graph convolution, specifically the word vector of the category name, F V is the visual input of the 0th layer of the graph convolution, specifically the average value of the CLIP visual features of all training samples in the category; Loss Function L The L2 distance between the pre-trained classifier parameters and the output of the last layer of the graph convolutional neural network is calculated using the following mathematical formula: （2） Where m is the total number of categories in the source dataset, d is the dimension of the classifier, are the pre-trained classifier parameters, is the output of the last layer of the graph convolutional network.

2. The knowledge-guided small sample image recognition system according to claim 1, characterized in that: In the small sample classifier training module, the initialization parameters generated by the parameter initialization module are used to perform fine-tuning on the small sample data set to ultimately obtain the small sample classifier parameters; This module uses a sigmoid activation layer so that the trained classifier can be directly superimposed on the pre-trained classifier, so that the final classifier can achieve good classification accuracy on both the pre-trained training set and the small sample training set. The specific steps are as follows: (1) Sample N images and obtain pre-extracted visual features and labels; (2) The visual features of the image are passed through the Softmax activation function to obtain the predicted classification probability of each category; (3) Use cross-entropy to calculate the loss and backpropagate to train the classifier; (4) Repeat steps (1)-(3) until the number of cycles reaches the set value.

3. The knowledge-guided small sample image recognition system according to claim 2, characterized in that: The specific steps of the test process are as follows: (1) Sample N images and obtain pre-extracted visual features and labels; (2) The visual features of the image are activated by the Softmax function to obtain the predicted classification probability of each category; (3) The category with the highest classification probability is taken as the predicted category.

Images