A small sample image classification method based on manifold learning and high-order graph neural network

Abstract

The present invention discloses a small sample image classification method based on manifold learning and high-order graph neural network, which mainly solves the problem of poor generalization performance of current small sample learning methods in image classification tasks of unknown categories. Its main classification process is: first, a convolutional neural network is used to extract features of the input image, and then an embedding propagation network is constructed to map the image features obtained by feature extraction to a set of interpolated feature embeddings, and then the feature embedding is used to calculate high-order structural information, and then the feature propagation network processes the image features, including updating node features and edge features, normalization, layer-by-layer calculation of manifold loss, and finally outputs the labels of unlabeled samples, and constrains the model through back propagation. The method of the present invention provides a new idea for small sample image classification. By constructing an embedding propagation network and utilizing high-order information, the distribution offset problem is alleviated, and the generalization performance and classification performance of the model are improved.

Description

Technical Field

[0001] The present invention relates to the intersection of graph neural networks and image processing, and in particular to a small sample image classification method based on manifold learning and high-order graph neural networks. Background Art

[0002] In order to learn from limited samples, machine learning has developed a new subfield, namely Few-Shot Learning (FSL). The realization of Few-Shot Learning means the ability to learn by analogy. Once certain categories have been learned by the model, the useful information in the learning process will be abstracted, making the model more effective for learning other categories. FSL can learn new tasks from a small number of samples by combining prior information. Few-Shot Learning has attracted much attention because: first, the number of things in the real world conforms to the long-tail distribution, and the number of most things is small. In addition, when humans learn a concept, they do not need millions or tens of millions of data. Therefore, Few-Shot Learning has become a research hotspot in the field of neural networks and image processing.

[0003] Graph Neural Network (GNN) treats each sample as a node in the graph, and the relationship between samples as the edge in the graph. The model not only learns the embedding vector of each node, but also learns the embedding vector of each edge. First, all samples are embedded into the vector space using a convolutional neural network, and then the sample vector is connected with the label and input into the graph neural network to build the edge between each node. Then, the node vector is updated through graph convolution, and finally the edge vector is continuously updated through the node vector, which constitutes a deep graph neural network. GNN iteratively performs feature aggregation from neighbors through message passing, so it can represent complex interactions between data instances. Since small sample learning algorithms need to make full use of the relationship between the support set and the query, using GNN is naturally capable of solving small sample learning problems. Summary of the invention

[0004] Technical problem: A small sample image classification method based on manifold learning and high-order graph neural network is proposed to improve the accuracy of image classification and enhance the generalization ability of the model.

[0005] Technical solution: The present invention discloses a small sample image classification method based on manifold learning and high-order graph neural network, comprising the following steps:

[0006] Step 1: Obtain an image dataset as data for training and validating the graph neural network. Divide each type of sample in the image dataset into a training set and a test set. The ratio of the training set to the test set in each type of sample is 4:1. Label some images in the training set and divide the training set into labeled samples and unlabeled samples.

[0007] Step 2: Input the data in the training set into the feature extraction network to obtain the image feature vector. The feature extraction network adopts a convolutional neural network.

[0008] Step 3: Input the image feature vector obtained by the feature extraction network into the embedding propagation network to map and obtain a set of interpolated features, which are called embedded features.

[0009] Step 4: Use the embedded features as the baseline features, calculate the edge features and high-order features, and use the baseline features, edge features, and high-order features as the initial features of the feature propagation network;

[0010] Step 5: In each layer of the feature propagation network, node features and edge features are updated alternately. The node features refer to baseline features and high-order features. This iterative feature propagation method enables the graph neural network to update some parameters according to its own structural characteristics and propagate the information of labeled samples to unlabeled samples.

[0011] Step 6: The output layer is a fully connected layer using the softmax function, which maps the learned feature representation to the sample label space and outputs the model's predicted label for the input image.

[0012] Beneficial effects: The present invention provides a new graph neural network for small sample image classification, which is mainly divided into two parts: embedding propagation and feature propagation. Embedding propagation can obtain a regularized manifold for training and testing, and solve the distribution offset problem by expanding the decision boundary and reducing the noise of category representation. Feature propagation combines high-order structural information to improve the classification accuracy of the model. The popular smoothing method proposed in the present invention can optimize the model by calculating the manifold loss layer by layer to constrain and obtain a smoother decision boundary. At the same time, the model complexity is low and has good classification performance. The present invention has achieved good results on commonly used graph neural network algorithms and small sample benchmark data sets, and the classification accuracy under different settings has been improved to varying degrees. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 Flow chart of the present invention.

[0014] Figure 2 This is the network structure diagram of the feature extraction network.

[0015] Figure 3 An image of the Gaussian radial basis function used in the embedding propagation network.

[0016] Figure 4 Schematic diagram of feature vectors in feature propagation network.

[0017] Figure 5 The classification accuracy curve during training in a semi-supervised scenario. DETAILED DESCRIPTION

[0018] The present invention is further described below in conjunction with the accompanying drawings

[0019] Figure 1 This is the implementation process of the present invention. The present invention mainly includes five modules: feature extraction, embedding propagation, node feature update, edge feature update and loss calculation. The feature extraction network is a commonly used convolutional neural network, which can convert the input image into a feature vector. Embedding propagation network can make decision boundaries smoother and more robust to noise based on interpolation embedding, and output a set of interpolations by using the similarity of network output features in the graph, where the graph is constructed by radial basis functions using the pairwise similarity of features. Embedding propagation is non-parametric and can be applied to any feature extractor. It can be used as part of the network to obtain a regularized manifold for training and testing. Unlike manifold mixing methods, embedding propagation is unsupervised and can capture high-order interactions between embeddings.

[0020] The embedding propagation network calculates the distance between each pair of sample features obtained by feature extraction, which is expressed as:

[0021]

[0022] According to the Gaussian radial basis function, the adjacency matrix is ​​expressed as:

[0023]

[0024] Among them, σ 2 is the scale factor, and the model selects and

[0025] Then, calculate the Laplacian of the adjacency matrix:

[0026]

[0027] Afterwards, using the label propagation formula, the propagation matrix is ​​obtained as:

[0028] P=(I-αL) -1 # (4)

[0029] Where α is the scaling factor and I is the unit matrix. From this, we can get the embedded feature vector as:

[0030]

[0031] because is the weighted sum of its neighbors. Embedding propagation has the effect of removing noise from the feature vector. It is worth noting that the embedding propagation operation is easy to implement and compatible with feature extractors and classifiers. In addition, due to the small size of the episode, the computational complexity of formula (5) can be ignored. The obtained embedded feature vector is used as the baseline feature vector, that is, The difference vector is calculated according to formula (17) and formula (18) respectively and the edge eigenvector Use it as the initial value of the feature propagation network.

[0032] At each layer of the feature propagation network, node features and edge features need to be updated. Specifically, at layer l, first update the node features based on the features of the previous layer. To update

[0033]

[0034] Among them, || represents a connector. represents the node feature update network of the lth layer, Represents the network parameters of the lth layer. Then according to To calculate

[0035]

[0036] The edge feature update calculation method based on the high-order structure is as follows:

[0037]

[0038]

[0039]

[0040] Then the edge features are normalized:

[0041]

[0042] Among them, || represents a connector. represents the high-order metric network of node features at layer l, Represents the pairwise node feature metric network at layer l.

[0043] According to the principles of manifold learning and structural fusion, by minimizing the metric difference of the projection space, the structural information (similar relationship of samples) from the original space to the projection space can be preserved. Similarly, while the graph neural network is updating features, we can constrain model optimization by calculating the manifold loss layer by layer. Therefore, the loss function consists of two parts: cross entropy loss and manifold loss. The cross entropy loss function is as follows:

[0044]

[0045]

[0046] in Yes i At each layer, the estimated value, δ(y i =c) When y i = c is equal to 1, otherwise equal to 0. The loss of the manifold structure at different levels and measurement methods is calculated as follows:

[0047]

[0048] Therefore, the final loss function is:

[0049] L loss =L ep +λL ml # (15)

[0050] Where λ is a parameter that balances the impact of different losses.

[0051] The last layer of the model is the output layer, which uses the fully connected layer of the softmax function to predict the sample label. After feature propagation, the edge feature is The final predicted node label is:

[0052]

[0053] Where δ(y i =c) When y i =c is equal to 1, otherwise it is equal to 0.

[0054] After the feature extraction of the convolutional neural network and the label information propagation of the feature propagation network, the predicted label of the model is finally output, and the model fits the input data under the constraint of the loss function.

[0055] Figure 2 The network structure of the feature extraction network is shown. The network consists of four 3×3 convolution blocks, a linear unit, a batch normalization unit, and a leakyReLU unit. Each convolution block consists of a 3×3 convolution layer, a batch normalization unit, a 2×2 maximum pooling layer, and a leakyReLU unit.

[0056] Figure 3 The Gaussian radial basis function image used in the embedding propagation network. The radial basis function is a real-valued function whose value depends only on the distance to the origin, and it can solve the multivariate interpolation problem. Applying the Gaussian radial basis function in embedding propagation can map features to a set of interpolated features, increase the smoothness of the embedding manifold and remove noise from the feature vector, improving the effectiveness of generalization and semi-supervised learning methods.

[0057] Figure 4 It is a schematic diagram of feature vectors in feature propagation network. The baseline feature vector generated by the feature extraction network, It is generated by the baseline feature vectors of adjacent nodes and represents the difference between the two nodes, which is called the difference vector. As can be seen from the figure, the difference vector Has a certain manifold structure. In the initial layer of the network, the difference vector The calculation method is:

[0058]

[0059]

[0060] where f(·) represents the embedded propagation network, Represents the baseline feature vector of the node. Based on the difference vector, we continue to explore the higher-order structure of the relative metric graph. The difference vector can only represent the feature difference between two adjacent nodes, and we need to use higher-order graph information, so and The difference between The high-order metric vector can perceive the two-hop neighborhood of the current node and obtain richer structural information than the one-hop neighborhood. Intuitively, the high-order metric can be used to indicate how fast the sample features change.

[0061] After defining the high-order relationship measurement of nodes, we will continue to explore the high-order structure of edges. Consider the similarity and dissimilarity relationship between paired samples and the edge feature representation of the relative measurement of high-order structural information, that is, the edge feature is represented as a three-dimensional vector, e ij1 Represents high-order structural information, e ij2 represents the similarity between nodes, e ij3 represents the dissimilarity between nodes. Therefore, the initial edge features can be expressed in different measurement methods as follows:

[0062]

[0063] Among them, || represents a connector. The calculation method is as follows:

[0064]

[0065] Figure 5 The classification accuracy curve of the present invention in the training process in a semi-supervised scenario set in 5-way 5-shot is shown. In the support set, we mark some samples on all classes and test the robustness of the model. This situation is called semi-supervised small sample learning. The present invention uses the miniImageNet dataset. All images in the miniImageNet dataset are in RGB color, with a size of 84×84 pixels, from 100 different classes, and 600 samples for each class. The miniImageNet dataset is divided into two parts, labeled and unlabeled, using different proportions (20%, 40%, 60%, 80%, 100%). For example, in a 5-way 5-shot task, for a 20% labeled semi-supervised scenario, that is, in each training episode, there is only 1 labeled sample per class, and the remaining 4 samples are unlabeled. Figure 5 is the change in classification accuracy during training iterations under different semi-supervised settings, Figure 5 The corresponding labeling rates of the five classification accuracy curves from bottom to top are 20%, 40%, 60%, 80%, and 100%, respectively. It can be seen that at 30,000 iterations, the best result has almost been achieved, and as the labeling rate increases, the classification accuracy of the model gradually improves, which is consistent with our intuitive understanding. The higher the labeling rate, the more sample information is obtained during the training process, and the better the performance of the trained model. In the 100% labeled scenario, the classification accuracy reached 83.6%, which is a significant improvement in accuracy compared to the currently published results.

[0066] The experimental platform of the model established in the present invention is as follows: Ubuntu 16 system, GeForce GTX 1080Ti graphics card, and Pytorch framework are used for network training and testing. The present invention uses a learning rate of 5×10 -4 , weight decay is 10 -6 The Adam optimizer is used for training, the weight decay is set to 0.001, and L2 regularization is used to suppress overfitting. For the 5-way 1-shot and 5-way 5-shot tasks, the mini-batch size is set to 40 and 20, respectively.

Claims

1. A small sample image classification method based on manifold learning and high-order graph neural network, characterized in that: The steps include: Step 1: Obtain an image dataset as data for training and validating the graph neural network, divide each type of sample in the image dataset into a training set and a test set, label some samples in the training set, and divide the training set into labeled samples and unlabeled samples; Step 2: Input the data in the training set into the feature extraction network to obtain the image feature vector. The feature extraction network adopts a convolutional neural network. Step 3: Input the image feature vector obtained by the feature extraction network into the embedding propagation network to map and obtain a set of interpolated features, which are called embedded features. Step 4: Use the embedded features as the baseline features, calculate the edge features and high-order features, and use the baseline features, edge features, and high-order features as the initial features of the feature propagation network; Step 5: In each layer of the feature propagation network, node features and edge features are updated alternately. The node features refer to baseline features and high-order features. This iterative feature propagation method enables the graph neural network to update some parameters according to its own structural characteristics and propagate the information of labeled samples to unlabeled samples. Step 6: The output layer is a fully connected layer using the softmax function to map the learned feature representation to the sample’s label space. In step 4, the high-order features refer to the difference vector Obtained by subtracting adjacent reference features pairwise; Baseline eigenvector Difference Vector and the edge eigenvector Specifically expressed as: where λ(·) represents the embedded propagation network, e ij1 Represents high-order structural information, e ij2 represents the similarity between nodes, e ij3 represents the dissimilarity between nodes, ∥ represents a connector, The calculation method is as follows:

2. According to claim 1, a small sample image classification method based on manifold learning and high-order graph neural network is characterized in that: Embedding the propagation network in step 3 refers to using the similarity of the network output features in the graph to output a set of interpolations, where the graph is constructed by radial basis functions using the pairwise similarity of features.

3. According to claim 1, a small sample image classification method based on manifold learning and high-order graph neural network is characterized in that: In step 5, when updating node features and edge features alternately, the optimization is constrained by the loss function, which includes two parts: cross entropy loss and manifold loss. The cross entropy loss function is as follows: in Yes i At each layer, the estimated value, δ(y i =c) When y i = c is equal to 1, otherwise it is equal to 0; the loss of the manifold structure at different levels and measurement methods is calculated as follows: Therefore, the final loss function is: THE loss =L ep +λL ml Where λ is a parameter that balances the impact of different losses.

4. According to claim 1, a small sample image classification method based on manifold learning and high-order graph neural network is characterized in that: The network structure of the feature extraction network includes four 3×3 kernel convolution blocks, a linear unit, a batch normalization unit and a leakyReLU unit; each convolution block includes a 3×3 convolution layer, a batch normalization unit, a 2×2 maximum pooling layer and a leakyReLU unit.

5. According to claim 1, a small sample image classification method based on manifold learning and high-order graph neural network is characterized in that: The image dataset in step 1 uses the miniImageNet dataset.

6. According to claim 3, a small sample image classification method based on manifold learning and high-order graph neural network is characterized in that: In step 5, the alternating updating of node features and edge features refers to using the node features and edge features of the first layer of the feature propagation network to calculate the edge features and node features of the next layer, and then normalizing the edge features. The specific process is as follows: In the first layer, first according to the characteristics of the previous layer To update Among them, ∥ represents a connector. represents the node feature update network of the lth layer, Represents the network parameters of the lth layer; then according to To calculate The edge feature update calculation method based on the high-order structure is as follows: Then normalize the edge features: Among them, ∥ represents a connector. represents the high-order metric network of node features at layer l, represents the pairwise node feature metric network of the lth layer, Represents the high-order structural information of the lth layer, represents the similarity between nodes in the lth layer, represents the dissimilarity between nodes in the lth layer, Represents the edge feature vector of the lth layer.

Images