A small sample learning method based on class label hierarchical tree guidance and application

Abstract

The application discloses a kind of based on class label hierarchical tree guiding small sample learning method and application, the steps of method described in the application include:1, construct and vectorization class label hierarchical tree;2, extract semantic feature from tree and fuse into semantic attention vector;3, use semantic attention vector to guide visual feature learning.The application constructs tree-shaped class label hierarchical structure by class label name and noun hypernym concept, extracts semantic priori knowledge to guide small sample learning, uses a feature extractor for each level of each sample to extract, then uses fusion function to combine linearly by different level semantic features, and then carries out image classification, solves the problems, such as inaccurate and insufficient use of text semantics, large amount of process calculation and text semantics still required in test stage, in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image classification methods, more particularly, to a few-shot learning method based on class label hierarchical tree guidance and application. BACKGROUND

[0002] In recent years, deep learning models have achieved very outstanding results in many visual recognition tasks due to their large number of parameters and training on large-scale datasets. For example, ResNet and Inception have achieved superior results in the ILSVRC (ImageNet Large Scale Visual Recognition Challenge) classification competition, which is a kind of super fitting ability of the model under the condition of large parameters and large datasets. However, in real life, some sample data is difficult to obtain due to privacy, security, ethics, and legal reasons. When facing a small number of learnable samples, deep learning models are prone to overfitting, resulting in poor model performance.

[0003] In machine learning, learning from a small number of labeled sample data and then identifying and classifying new, unseen class samples is called few-shot learning (FSL). By analogy, humans can fully utilize past knowledge to quickly learn and reason in the case of insufficient samples. The goal of few-shot learning is also to transfer knowledge to learn new class samples like humans. CN202210775075.1 discloses a few-shot image classification method based on multi-directional knowledge transfer, which extracts visual and text features, calculates semantic relationships between classes, performs one-way self-transfer of text knowledge, one-way self-transfer of visual knowledge, bidirectional mutual transfer of knowledge between modalities, and model training and testing. The patent uses one-way self-transfer on the text and visual ends to improve the description of new classes from the base classes in a semantic reasoning and visual imagination manner, and exchanges knowledge from different modalities through cross-modal mutual transfer, so that different modalities learn from each other and are trained together, thereby reducing the influence of the difference in sample quantity between the base class and the new class, alleviating the semantic gap between different modalities, and improving the accuracy of few-shot image classification to some extent. However, the patent has the following shortcomings:

[0004] (1) Only the word vector distance between class label names is used as text knowledge, which is inaccurate and not rich;

[0005] (2) In terms of text knowledge transfer calculation, it needs to calculate the semantic relationship matrix between different class label names, and also needs to calculate the attention score in the one-way transfer process of text knowledge, resulting in a very large calculation amount;

[0006] (3) still need class label name assistance in the test phase.

[0007] Therefore, the small sample image classification method in the above patent publication has problems of inaccurate and insufficient text semantics, large process calculation amount, and still needing text semantics in the test phase, which need to be solved urgently. SUMMARY

[0008] The main technical problem to be solved by the present application is to provide a small sample learning method based on class label hierarchical tree guidance and the application of the learning method, aiming at the problems of inaccurate and insufficient text semantics, large process calculation amount, and still needing text semantics in the test phase in the prior art.

[0009] The object of the present application is achieved by the following technical solutions:

[0010] A small sample learning method based on class label hierarchical tree guidance, comprising the following steps:

[0011] S1. defining a class label hierarchical tree, and constructing and vectorizing a class label hierarchical tree structure file;

[0012] S2. extracting and fusing semantic attention vectors of the class label hierarchical tree;

[0013] S21. constructing a semantic feature extraction unit (SFEU) using a fully connected neural network, passing the label word vector through the semantic feature extraction unit to obtain semantic features;

[0014] S22. using a semantic feature fusion function to fuse semantic features of different levels to obtain a semantic attention vector fused with semantic prior knowledge;

[0015] S3. obtaining a visual feature vector from an image;

[0016] S4. performing Hadamard product on the semantic attention vector and the visual feature vector to guide visual feature learning and form a final classification weight vector.

[0017] Further, the height of the tree-shaped class label hierarchical structure in S1 is any height in 1-4. Preferably, the height of the tree-shaped class label hierarchical structure is 2 or 3.

[0018] Further, the class label hierarchical tree structure file in S1 is constructed using a WordNet tool, and the class label hierarchical tree structure file is vectorized using a GloVe tool.

[0019] Further, the process of semantic features in S21 includes using a semantic feature extraction unit to extract semantic prior knowledge at each layer of the class label of the sample.

[0020] Further, the semantic feature obtained in S21 is linearly combined by a fusion function to obtain a semantic attention vector:

[0021]

[0022] wherein, The calculation formula is as follows:

[0023]

[0024] x i The word embedding vector of the mth layer label, k m represents a linear combination hyperparameter and satisfies

[0025] Further, the step of obtaining a visual feature vector from the image in S3 includes:

[0026] S31. Taking the mean of the feature vectors of samples of the same class in each dimension to obtain a mean prototype:

[0027]

[0028] wherein, is a mean prototype of a task belonging to the cth category, is all samples belonging to the same class c in the support set of a task T i

[0029] S32. A Dynamic FSL class weight vector generator is used to obtain a visual vector feature, and a cosine similarity is used to obtain D train from D weighted transmission of visual prior knowledge to enhance the prototype to obtain a visual attention prototype:

[0030]

[0031] wherein φ q ∈R d×d is a learnable weight matrix, which is operated with the input sample feature to form Q in the attention mechanism; is a learnable key set, which is K in the attention mechanism; is a random clue V in the attention mechanism; and Att(·,·) is an attention kernel realized by using a cosine similarity function;

[0032] S33. The visual feature vector is linearly obtained from the mean prototype and the visual attention prototype:

[0033]

[0034] wherein a, b are learnable coefficients.

[0035] Further, the Hadamard product in S4 is expressed as:

[0036]

[0037] Further, the classification weight vector in S4 is classified by the cosine classifier, and the classification calculation formula of the classifier is:

[0038]

[0039] wherein temp is a temperature coefficient.

[0040] Any of the above small sample learning methods based on class label hierarchical semantic prior knowledge is applied to computer vision, recommendation system and medical diagnosis.

[0041] Compared with the prior art, the beneficial effects are:

[0042] The present application constructs a tree-shaped class label hierarchical structure through class label names and noun superordinate concept, extracts semantic prior knowledge from it to guide small sample learning, uses a feature extractor for each level of each sample for extraction, and then uses a fusion function to fuse the semantic features of different levels by linear combination, and then performs image classification. The present application extracts semantic features of different levels from the tree-shaped class label, which is more rich and accurate, and only needs simple linear combination to fuse into a semantic attention vector in terms of calculation amount, which is very small. And the present application guides visual feature learning through the semantic attention vector, and does not need any text semantics in the test stage, which is more convenient and fast. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 It is the system implementation graph of the present application in example 1;

[0044] Figure 2 It is the tree-shaped class label hierarchical structure construction and vectorization process graph in example 1;

[0045] Figure 3 It is the experimental data graph of the method model for different heights of class label hierarchical tree in example 2;

[0046] Figure 4 It is the experimental data graph of different linear combination coefficients in the fusion function when the height of the class label hierarchical tree is 2 in example 3.

[0047] Figure 5 It is the experimental data graph of different linear combination coefficients in the fusion function when the height of the class label hierarchical tree is 3 in example 3. DETAILED DESCRIPTION

[0048] The application will be further explained and illustrated with reference to the following examples, which do not limit the application in any way.

[0049] In the following examples, the tools mentioned are introduced as follows:

[0050] WordNet is a large lexical database of English, which groups words into sets of synonyms and marks the semantic relationships among these synonyms. The main relationship between nouns is synonymy, and the most common coding relationship between synonyms is hypernymy (also known as meronymy, hyponymy or ISA relationship). WordNet links general nouns to specific nouns, for example, WordNet indicates that the {furniture} category includes {bed}, and {bed} in turn includes {bunkbed}; conversely, concepts such as {bed} and {bunkbed} constitute the {furniture} category. Finally, all the noun hierarchies eventually rise to the root node {entity}.

[0051] The GloVe word vector tool uses an unsupervised learning algorithm to train the global word-word co-occurrence statistics from a large text corpus to obtain the vector representation of words. GloVe is different from both word2vec and BoW (Bag of Words). Word2vec is a neural network-based word vector representation method that can capture local semantic and syntactic relationships between words; BoW is a text representation method based on the bag-of-words model, which represents text as a frequency vector of all words in the vocabulary, and cannot capture the order and syntactic information between words. Although GloVe model is also a word vector representation method based on the distributed hypothesis, GloVe is the product of balancing between statistical language model and word2vec. GloVe learns word vectors by minimizing the weighted squared error of the co-occurrence matrix on the global corpus, and each element in this matrix represents the frequency of two words co-occurring in the context, so GloVe can capture the global semantic relationship between words, including lexical and syntactic relationships.

[0052] Example 1

[0053] This embodiment provides a small sample learning method based on class label hierarchical tree guidance, as shown in Figure 1 , comprising the following steps:

[0054] S1. Define a tree-shaped class label hierarchical structure, and construct and vectorize a tree-shaped class label hierarchical structure file;

[0055] Each sample is expanded to a one-to-many relationship with its corresponding label, and the labels have a hierarchical nature, so that all sample labels form a tree-like class label hierarchy. Then the WordNet tool is used to construct a tree-like class label hierarchy file based on hypernym relationships. Then the GloVe tool is used to vectorize the tree-like class label hierarchy file and convert it into corresponding word vectors, as shown in Figure 2

[0056] S2. Extract and fuse class label hierarchical semantic prior knowledge;

[0057] S21. Use a fully connected neural network to construct a semantic feature extraction unit, pass the label word vector through the semantic feature extraction unit to obtain semantic features. The parameter design of the semantic feature extraction unit is:

[0058] Layer name Embedding function Parameters Explanation Input layer Linear [300,300] Input dimension: :300, output dimension: 300 Activation function layer LeakReLU α＝0.1 - Deactivation layer Dropout 0.7 Neuron deactivation probability 70% Hidden layer Linear [300,640] Input dimension: 300, output dimension: 300 Output layer Sigmoid - Scale each dimension value of the output to (0, 1)

[0059] The input of the semantic feature extraction unit is 300 dimensions, and the output is 640 dimensions.

[0060] S22. Use a semantic feature fusion function to fuse semantic prior knowledge at different levels to obtain an attention vector that fuses semantic prior knowledge;

[0061] For a single task, I s For each layer of class labels between samples, a semantic feature extraction unit is used to extract semantic information, and finally a linear combination is used to form a final semantic prior knowledge fusion vector, represented as:

[0062]

[0063] wherein, The calculation formula is as follows:

[0064]

[0065] represents the sample x i The word embedding vector of the mth layer label, k m represents the linear combination hyperparameter and satisfies

[0066] S3. Obtain a visual feature vector from the image;

[0067] S31. Take the mean of the feature vectors of samples of the same class in each dimension to obtain a mean prototype:

[0068]

[0069] wherein, is the mean prototype of the task belonging to the c category,​ for a task T i Supporting all samples belonging to the same class c;

[0070] S32. Obtain the visual vector feature using the Dynamic FSL class weight vector generator, and generate the visual attention prototype based on the cosine similarity from D train Weighted transfer of visual prior knowledge to enhance the prototype, and obtain the visual attention prototype:

[0071]

[0072] where φ q ∈R d×d is a learnable weight matrix, which is operated with the input sample feature to form Q in the attention mechanism; is a learnable key set, which is K in the attention mechanism;

[0073] is a non-random clue V in the attention mechanism; Att(·,·) is an attention kernel implemented using a cosine similarity function;

[0074] In step S31, the mean prototype is more reliable when there are more samples of the same class, but in the case of small samples, the obtained prototype is inaccurate, unstable and unreliable. Therefore, in step S32, implicit visual-level class hierarchy attention guidance is introduced, while ensuring that the model can still retain the ability to recognize the base class in the test phase, i.e. transfer prior knowledge to facilitate the association of different categories. In step S32, through explicit semantic-level class hierarchy attention guidance, more accurate and reliable sample features are obtained under the interaction of visual and semantic attention.

[0075] S33. Obtain the visual feature vector by linearly combining the mean prototype and the visual attention prototype:

[0076]

[0077] where α,β∈R are learnable coefficients.

[0078] S4. Perform Hadamard product on the semantic prior knowledge attention vector and the visual feature vector, denoted as:

[0079]

[0080] refers to the class label hierarchy semantic prior knowledge attention vector in step four, which belongs to the same category c, represents the Hadamard product. The visual feature prototype vector obtained above is the abstract feature of the sample at different levels after the semantic prior knowledge fusion vector guides the visual attention, which not only pays attention to the similarity features of the upper levels of the tree-shaped class label hierarchical structure, but also retains the dissimilarity features of the lower levels of the tree-shaped class label hierarchical structure, thereby improving the overall performance of the model.

[0081] Finally, the visual feature learning is guided, and a cosine classifier is used to perform final classification to form a final classification weight vector. The classification calculation formula of the cosine classifier is

[0082]

[0083] where temp∈R is a temperature coefficient.

[0084] Embodiment 2

[0085] This embodiment simulates the experimental simulation according to the method of embodiment 1. The model constructed according to the method of embodiment 1 is trained and tested on the tieredImageNet dataset.

[0086] The height of the tree-shaped class label hierarchical structure is a hyperparameter. In order to give a recommended reference value of the height of the class label hierarchical tree, a class label hierarchical tree with a height of 1, 2, 3, and 4 is constructed in the experimental process, and the semantic prior knowledge is extracted from it for guidance. At the same time, for the tree with a height greater than 1, different linear combination coefficients k of the fusion function are set m Experiments are conducted.

[0087] First, for different heights of the class label hierarchical tree, the experimental results are as shown in Figure 3 The experimental data are the optimal linear combination coefficient result data under the corresponding height, where n-level indicates that the height of the class label hierarchical tree is n. Through the experimental data, it can be found that in different shot cases, the 2-level and 3-level experimental results are better. Therefore, the recommended reference value of the height of the class label hierarchical tree is 2 or 3.

[0088] Embodiment 3

[0089] In this embodiment, the experimental results of the different linear combination coefficients of the corresponding fusion function under the condition that the recommended value of the height of the class label hierarchical tree is 2 and 3 are as shown in Figure 4 and Figure 5 Through the corresponding radar chart, it can be clearly found that when the height is 2, the linear combination coefficient k1:k2=8:2 in the fusion function is the best, and when the height is 3, the linear combination coefficient k1:k2:k3=7:2:1 in the fusion function is the best. The above is the optimal linear combination coefficient reference value of the fusion function when the height of the class label hierarchical tree is 2 or 3.

[0090] Obviously, the above embodiments of the present application are merely exemplary but not intended to limit the embodiments of the present application. Based on the above description, any other variations or changes can be made by those skilled in the art without departing from the spirit and principles of the present application. It is not necessary to list all the embodiments here. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present application should be included in the protection scope of the claims of the present application.

Claims

1. A few-shot learning method based on class label hierarchical tree guidance, characterized in that, The method comprises the following steps: S1. Expanding each sample and its corresponding label into a one-to-many relationship with a hierarchical relationship between labels, so that all sample labels form a tree-like class label hierarchy, and a tree-like class label hierarchy file is constructed and vectorized; S2. Extract and fuse class label hierarchical semantic prior knowledge; S21. Construct a semantic feature extraction unit using a fully connected neural network, pass the label word vector through the semantic feature extraction unit, and obtain semantic features; S22. Use a semantic feature fusion function to fuse semantic features of different levels to obtain a semantic attention vector fused with semantic prior knowledge; S3. Obtain a visual feature vector from the image; S4. Perform Hadamard product on the semantic attention vector and the visual feature vector to guide visual feature learning and form a final classification weight vector.

2. The few-shot learning method based on class label hierarchical tree guidance according to claim 1, characterized in that, The height of the tree-like class label hierarchy in S1 is any height from 1 to 4.

3. The few-shot learning method based on class label hierarchical tree guidance according to claim 1, characterized in that, In S1, a class label hierarchy tree structure file is constructed using a WordNet tool.

4. The few-shot learning method based on class label hierarchical tree guidance according to claim 1, characterized in that, In S1, the class label hierarchy tree structure file is vectorized using a GloVe tool.

5. The few-shot learning method based on class label hierarchical tree guidance according to claim 1, characterized in that, In S21, semantic feature extraction includes using a semantic feature extraction unit at each level of the sample's class label hierarchy tree to extract features.

6. The few-shot learning method based on class label hierarchical tree guidance according to claim 5, characterized in that, In S21, the semantic attention vector output after linear combination of the semantic features is: wherein The calculation formula is as follows: representative samples x i first m word embedding vectors of layer labels, in which k m represent linear combination hyperparameters and satisfy .

7. The few-shot learning method based on class label hierarchical tree guidance according to claim 1, characterized in that, The step of obtaining a visual feature vector from the image in S3 comprises: S31. Taking the mean of feature vectors of samples of the same class in each dimension to obtain a mean prototype: in, In order to belong to the same task c The prototype of the category mean For a task T i The support is concentrated in the same category c All samples; S32. Obtain the class weight vector in the visual vector feature using the Dynamic FSL class weight vector generator, based on the cosine similarity from D train Weighted transfer visual prior knowledge to enhance the prototype, get visual attention prototype: wherein is a learnable weight matrix which, after operation with the input sample features, constitutes Q in the attention mechanism; is a learnable key set, K in the attention mechanism; is a non-random clue V in the attention mechanism; is an attention kernel implemented using a cosine similarity function; S33. Linearly combining the mean prototype and the visual attention prototype to obtain a visual feature vector: wherein are coefficients that can be learned.

8. The few-shot learning method based on class label hierarchical tree guidance according to claim 1, characterized in that, In S4, the Hadamard product is expressed as: 。 9. The method according to claim 1, wherein the class label hierarchy tree is guided by a small sample learning method. In S4, the classification weight vector is classified by a cosine classifier, and the classification calculation formula of the classifier is: wherein is the temperature coefficient.

10. The few-shot learning method based on class label hierarchical tree guidance according to any one of claims 1-9, characterized in that, The method is applied to computer vision, recommendation systems and medical diagnosis.

Images