Small sample size image classification method based on multi-image guided neural network model

Abstract

The present invention discloses a small sample scene image classification method based on a multi-graph guided neural network model. Based on the features of the multi-graph guided neural network model at different levels (view), image feature information at different levels is modeled, and a graph convolutional neural network for relationship modeling of different samples is introduced at the same time. The sample features are pulled closer to the same type of samples and away from different types of samples through feature propagation between samples, so that the sample features are more discriminative. The present invention can effectively solve the problem of scene image classification with small samples and complexity. Its model architecture mainly includes four aspects: using a deep network to extract features of different levels of views of the scene image, constructing a graph based on the global feature representation of views at different levels, performing graph convolution representation of features of views at different levels respectively, and classifying query samples based on the features guided by multiple graphs.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a small sample scene image classification method based on a multi-graph guided neural network model. Background Art

[0002] In the field of machine learning, with the emergence of deep learning, more and more scenarios are facing insufficient data. Humans can identify new targets based on a small amount of sample data, but current machine recognition systems still need to rely on a large amount of labeled training data and have difficulty identifying new categories. In order to solve this problem, more and more researchers have proposed some methods to use a small number of samples to identify new category concepts. Therefore, small sample learning has become a very important research direction.

[0003] Currently, many researchers have proposed small sample research methods, which are mainly divided into meta-learning-based methods, metric learning-based methods and generation-based methods. Metric learning-based methods generally include feature extraction, feature representation and feature measurement. Their goal is to learn feature representations with better generalization performance. However, after extracting image features, metric learning-based methods directly measure and classify global features without considering the relationship between features at different levels and between support set samples and query set samples; although some methods can realize the relationship between the characteristics of different regions by sharing CNN modules, this does not take into account the relationship between the similarity and dissimilarity between support set samples and query set samples and the feature information between different levels of the same sample in small sample scenarios. It only relies on data to train the feature extraction network, and the global feature representation obtained is difficult to measure and classify well.

[0004] Yanbin L et al. proposed that the closed-form solution of label propagation can effectively predict the label of the entire query sample. By improving the Gaussian kernel, the deep network is used to learn the graph S corresponding to the specific small sample task σ; the query set samples are predicted by using the label propagation graph S, which also confirms that the prediction accuracy of small sample learning can be effectively improved by using the entire test set for transductive propagation. However, this method only relies on the global features of the last layer of the feature extraction network, which limits the feature expression of the entire scene image and cannot accurately classify complex scenes with small samples.

[0005] Thomas NK et al. proposed a graph convolutional network (GCN) that integrates graph structure and point features into a unified framework to learn the potential representation of graph nodes. Jiang et al. proposed multi-graph adversarial learning, which integrates the structural information of each graph and the relevant information across graphs, and learns the consistent representation of multiple graphs through a novel adversarial learning framework. In small sample learning tasks, graph convolutional networks can effectively narrow the distance between similar samples, and improving the discriminability of features is helpful for metric classification between samples.

[0006] However, the foreground and background of the emblem image are relatively complex, the image illumination changes, and the target morphology is diverse. The global features obtained only through the feature extraction network do not fully explore the commonalities of the same emblem, which will ignore the image details and is not easy to classify. Summary of the invention

[0007] Purpose of the invention: The purpose of the present invention is to solve the deficiencies in the prior art and to provide a small sample scene image classification method based on a multi-graph guided neural network model.

[0008] Technical solution: A small sample scene image classification method based on a multi-image guided neural network model of the present invention comprises the following steps:

[0009] Step S1: collect scene image samples, which are divided into a support set S and a query set T. During training, the scene image samples of the support set S and the query set T are respectively input into the feature extraction network.

[0010] Step S2: feature extraction, i.e., through feature extraction network Extract features of different levels of scene image samples;

[0011] It includes four layers of convolution modules. It is the learning parameter of the convolution kernel of the entire convolutional neural network. The output of each convolution module is the feature map of the corresponding hierarchical view of all image samples of the current task. Each convolution module obtains the feature map X corresponding to different levels of view of the scene image (v) ;

[0012]

[0013] h, w, c represent the height, width and number of channels of the corresponding feature map, respectively; i represents the i-th scene image sample; represents the corresponding hierarchical view, v = {1, 2, 3, 4};

[0014] Then, the graph constructs a network ψ to reduce the dimension and straighten the features of views at different levels to obtain the corresponding feature vector H (v) ;

[0015] and

[0016] N represents the category of the image, K represents the number of each category of image in the support set, |T| represents the number of image in the query set, and d represents the dimension of the feature vector corresponding to the v level;

[0017] Step S3: Multi-graph construction

[0018] Since existing small sample learning methods only rely on the features of the image itself, they are very limited by the training of the feature extraction network. Since the small sample learning algorithm requires full exploration of the relationship between the support set and the query sample, graph neural networks have shown great potential in small sample problems. The propagation aggregation of the graph can make full use of the information of all samples in each task to enrich its own features, thereby bringing similar samples closer and moving away from dissimilar samples. Therefore, the present invention explores the representation capabilities of the graph convolutional neural network as follows:

[0019] For each level of view, a Gaussian kernel similarity graph is selected as the adjacency matrix of the graph, and the feature vectors of the v-th level image sample i and sample j are and The similarity weight of is calculated as follows:

[0020]

[0021] in, represents the similarity weight of the v-th layer scene image samples i and j, d(.,.) is the distance metric function, the Euclidean distance function is used here, and δ is a scaling constant; and Represents the feature vectors of image sample i and image sample j in layer v;

[0022] Then we get the similarity matrix of the corresponding hierarchical view graph. From A (v) Take the k maximum values ​​in each row to construct a k-nearest neighbor graph;

[0023] Step S4: Multi-graph convolution representation

[0024] To apply GCN to multiple graphs, we will use graph A of each level view (v) and the eigenvector H (v) Use GCN to obtain features respectively After concatenation, the multi-image guided feature representation Z is obtained for classification;

[0025]

[0026] S (v) Representative in A (v)The normalized graph Laplacian is used on (v) =D (v)-1 / 2 A (v) D (v)-1 / 2 ;

[0027] where D is the degree matrix,

[0028] Among them, H (v) A represents the feature vector of the v-th layer image. (v) represents the similarity weight of the v-th layer graph, represents the output of the vth layer of graph convolutional network, and l is the number of graph convolutional network layers;

[0029] For graph convolution operations at different levels (views), the weight parameters of the hidden layer are θ = {θ0…θ L-1} is shared;

[0030] The present invention uses shared convolution parameters in multi-graph convolution, so that relevant information of different levels (views) can be propagated across graphs, ensuring the consistent representation of multi-graph learning features at different levels (views); by considering features at different levels (views), the final feature representation is more discriminative and the query samples are better classified;

[0031] Step S5: Similarity measurement and classification

[0032] Concatenate the graph convolution representations of different levels (views) to obtain feature Z. In order to make the query sample feature close to the class center of its own class and away from the class center of other classes, the contrastive learning function is introduced here

[0033]

[0034] The commonality of features of the same category is extracted through the contrastive learning function, and the similarity of samples of the same category is retained; based on the similarity measurement, the category with the highest similarity to the query sample is selected to complete the classification of the query sample;

[0035]

[0036] Among them, S n represents the sample of the nth class in the support set, c n represents the class center of the nth class; y i is the label of the support set sample;

[0037] Substitute the multi-image guided image features into the above contrast learning function In the process, the query samples are classified, the cross entropy classification loss is calculated, and the reverse gradient update is performed; each time the task samples under the specified settings are selected, the above steps are repeated for training.

[0038] Furthermore, the feature extraction network Each convolution module includes a 3×3 convolution layer, a normalization layer, a nonlinear activation function layer and a 2×2 pooling layer, and the Dropout layer is used between the nonlinear activation function layer and the pooling layer to prevent overfitting.

[0039] Furthermore, the graph construction network ψ includes a convolutional module and two fully connected layers.

[0040] The third part consists of a two-layer graph convolutional neural network with a nonlinear layer between the two layers. Each graph convolution operation includes two steps of propagation and aggregation.

[0041] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0042] (1) The present invention outputs the image features of the corresponding level (view) through each convolution module of the feature extraction network, and then composes the graph, inputs the graph structure and node features into the multi-graph convolutional neural network for cross-graph consistency learning, retains the feature information of each level (view), improves the ability to extract complex scene image features, and uses contrast learning functions to make the query sample features closer to the class center of its own class, thereby ensuring the similarity of samples of the same scene image class.

[0043] (2) Compared with direct feature measurement, the present invention utilizes a multi-graph convolutional neural network to fully mine the feature information of different levels (views) within the scene image, and can also better transduce and propagate the support set samples and the query set samples, integrating the structural relationship and node features between samples of different levels (views), and performing consistency learning of features of the same sample and features of different levels (views) of the same sample in a context-aware manner, which can effectively solve the problem of small sample scene image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is the overall flow chart of the present invention.

[0045] Figure 2 Schematic diagram of the network model structure in the embodiment. DETAILED DESCRIPTION

[0046] The technical solution of the present invention is described in detail below, but the protection scope of the present invention is not limited to the embodiments.

[0047] like Figure 1As shown, the small sample scene image classification method based on the multi-image guided neural network model of this embodiment includes the following steps:

[0048] Step 1: In this embodiment, two settings, 5-way 1-shot and 5-way 5-shot, are used for training. For the 5-way 1-shot setting, one image of each class is selected, and a total of 5 images of the image are used as the support set S. 15 images of each class are selected, and a total of 75 images of the image are selected as the query set T. The 80 images of the image are processed and scaled to 84×84 size and input into the feature extraction network. For the small sample classification task of the 5-way 5-shot setting, the same process is performed.

[0049] Step 2: Feature extraction network The four-layer convolutional network commonly used in small sample learning models is used. It contains four modules in total and has no skip connections.

[0050] Each convolution module contains a 3×3 convolution layer, a batch normalization layer, an activation function layer, and a 2×2 pooling layer. Compared with the first two layers, the last two convolution modules have an additional Dropout layer to prevent overfitting. In each round of training, a total of 80 samples of query set and support set scene images are input into the feature extraction network to extract features.

[0051] Compared with simple single-label images, the image content of the scene image is more complex, and the background and target forms are more diverse. Therefore, in order to better model the complex information inside the scene image, each convolution module will output the feature map of the corresponding level (view), which is then reduced and straightened through the graph construction network ψ and used as the feature vector of the input multi-graph convolutional network. In the 5-way 5-shot setting, the support set S and the query set T contain 100 scene images in total, and each convolution module will obtain 100 feature maps of the corresponding level (view).

[0052] Step 3: Get the output of each layer through the feature extraction network Represents the feature map corresponding to different levels (view), and then inputs it into the graph construction network of the corresponding module ψ to reduce the dimension and straighten it to obtain the feature vector of the corresponding level (view) of the entire task scene image Then the features are input into the Gaussian kernel function to construct the graph.

[0053]

[0054] in and Represents the feature vectors of sample i and sample j in layer v; the similarity matrix of the corresponding view graph v represents the corresponding level (view), v = {1, 2, 3, 4}, N represents the category of the image, K represents the number of each category of image in the support set, |T| represents the number of image in the query set;

[0055] Take A (v) The k-nearest neighbor graph is constructed with the k-maximum values ​​in each row, which can effectively aggregate the relevant information and discard the irrelevant information to avoid the problem of over-smoothing caused by all sample features in the multi-graph convolution representation process in step 4;

[0056] This embodiment uses the spectral domain graph convolution function, in A (v) The normalized graph Laplacian is used on (v) =D (v )-1 / 2 A (v) D (v)-1 / 2 , where D is the degree matrix,

[0057] Step 4: Graph A of the corresponding level (view) obtained based on the above steps (v) , learning convolutional representations of multiple graphs for graphs and feature vectors at different levels (views).

[0058] The purpose of the graph convolutional network GCN is to use the graph structure and node features to learn the potential representation of each graph node, that is, to integrate the graph structure and node features into a unified framework. Usually, given a graph, GCN can perform layer-by-layer propagation in the hidden layer, as shown below,

[0059]

[0060] Among them, l = 0, 1...L-1, v = 1, 2...4 and L is the number of layers. is the initial input feature matrix, represents the input features of the lth layer, It is the output feature representation after performing the graph convolution operation. θ={θ0…θ L-1} is the learnable weight matrix of each layer.

[0061] This embodiment uses the final feature output of GCN as the convolution representation of each graph; for the graph convolution operation, the spectral domain convolution function used in the present invention is,

[0062]

[0063] Here, σ represents an activation function, for example, ReLU(·)=max(0,·).

[0064] For the graph convolution operations of different levels (views) in the present invention, the weight parameters of the hidden layers of different graphs are θ={θ0…θ L-1} is shared, so some information of different levels (views) can be propagated across graphs, and a consistent representation of the multi-graph guided neural network model is obtained. By considering the features of different levels (views), the final feature representation is more discriminative, effectively selecting the important information in the scene image, and better measuring the classification of the query set scene images.

[0065] After two layers of GCN, Placed in a low-dimensional space, the features of different levels (views) are concatenated to obtain a multi-image guided consistent representation Z = [Z (1) ∥…∥Z (4) ].

[0066] Step 4: The purpose of multi-image guided consistency learning is to ensure that the features of different levels (views) are similar to the same type of features and not similar to the features of different types. Figure 2 As shown in , this embodiment obtains feature Z by splicing the output of the multi-graph convolutional representation module. In order to further make the query sample feature close to the class center of its own class and away from the class center of other classes, this embodiment introduces a contrastive learning function, which is implemented as follows:

[0067]

[0068] There are two types of small sample classification tasks here, one is the 5-way 1-shot small sample classification task, and the other is the 5-way 5-shot small sample classification task. For the 5-way 1-shot small sample classification task, each class has only one scene image, and the class center can be represented by the corresponding scene image. For the 5-way 5-shot small sample classification task, each class has 5 scene images, and the class center of the class is represented by the mean of the features of the 5 scene images.

[0069] Use S n represents the sample of the nth class in the support set, c n Represents the class center of the nth class, that is:

[0070]

[0071] where y i is the label of the support set samples, n∈{1,2,…,5}.

[0072] Here, the image features of the scene guided by multiple images are substituted into the contrastive learning function to complete the classification of the query sample, and the cross entropy classification loss is calculated to perform reverse gradient update. Each time a task sample under the specified setting is selected, the above steps are repeated for training.

[0073] This example uses the Adam optimizer with an initial learning rate of 10 -3 The small sample model guided by multiple images was trained. The learning rate was halved every 1,000 training tasks until the model converged and a small sample classification prediction model was obtained.

[0074] Step 5: This embodiment uses the above-mentioned small sample classification prediction model to classify the new scene image.

[0075] Compared with the existing method of directly performing global measurement, the technical solution of the present invention takes into account more detailed features by integrating the expression of scene image features at different levels, and further transduces and propagates the feature information between the support set and the query set, retaining the similarity of each category of images, and can better perform measurement classification.

[0076] The present invention effectively solves the problem of classifying small sample complex scene images. Its model architecture mainly includes four aspects: using deep network to extract features of different levels of views of scene images, constructing graphs based on global feature representations of views at different levels, performing convolutional representation of graphs for features of views at different levels, and classifying query samples based on features guided by multiple graphs.

[0077] like Figure 2 As shown, the model in this embodiment mainly includes three parts:

[0078] The first part is the feature extraction network pass To extract the features of different levels (view) of the scene image, It includes four convolution modules, each of which includes a convolution layer, a batch normalization layer, a nonlinear layer, and a pooling layer;

[0079] The second part is the graph construction network ψ, which is used to reduce the dimension and straighten the features of different levels (views) to obtain the feature vectors of the corresponding levels (views). The graph construction network ψ includes a convolution module and two fully connected layers.

[0080] The third part consists of a two-layer graph convolutional neural network with a nonlinear layer between the two layers. Each graph convolution operation includes two steps of propagation and aggregation.

[0081] Example:

[0082] This embodiment takes the images of the Huangshan scenery of the Welcoming Pine, Strange Rocks, Sea of ​​Clouds, Human-shaped Waterfall and Baizhang Spring as examples, and inputs the Huijing images into the small sample classification prediction model of the present invention with a 5-way1-shot small sample classification setting, and uses feature extraction, multi-image construction and multi-image convolution representation module processing to obtain a consistent representation of the Huijing image features guided by multiple images. Then it is input into the similarity measurement function to calculate the similarity between the features of the unknown label Huijing image and the center of the Welcoming Pine, Strange Rocks, Sea of ​​Clouds, Human-shaped Waterfall and Baizhang Spring category, and selects the category with the highest similarity to the Huijing image of the unknown label, so as to obtain the predicted label of the Huijing image of the corresponding query set.

Claims

1. A small sample scene image classification method based on a multi-image guided neural network model, characterized by: The steps include: Step S1: collect scene image samples, which are divided into a support set S and a query set T. During training, the scene image samples of the support set S and the query set T are respectively input into the feature extraction network. Step S2: feature extraction, i.e., through feature extraction network Extract features of different levels of scene image samples; It includes four layers of convolution modules. It is the learning parameter of the convolution kernel of the entire convolutional neural network. The output of each convolution module is the feature map of the corresponding hierarchical view of all image samples of the current task. Each convolution module obtains the feature map X corresponding to different levels of view of the scene image (v) ; h, w, c represent the height, width and number of channels of the corresponding feature map, respectively; i represents the i-th image sample; v represents the corresponding hierarchical view, v = {1, 2, 3, 4}; Then, the graph constructs a network ψ to reduce the dimension and straighten the features of views at different levels to obtain the corresponding feature vector H (v) ; and N represents the category of the image, K represents the number of each category of image in the support set, |T| represents the number of image in the query set, and d represents the dimension of the feature vector corresponding to the v level; Step S3: Multi-graph construction Each level of view selects a Gaussian kernel similarity graph as the adjacency matrix of the graph, and the feature vectors of the v-th level image sample i and sample j are and The similarity weight of is calculated as follows: in, represents the similarity weight of the v-th layer scene image samples i and j, d() is the distance metric function, and δ is the scaling constant; and Represents the feature vectors of image sample i and image sample j in layer v; Then we get the similarity matrix of the corresponding hierarchical view graph. From A (v) Take the k largest values ​​in each row to construct a k-nearest neighbor graph; Step S4: Multi-graph convolutional representation Each level view image A (v) and the eigenvector H (v) Use GCN to obtain features respectively After concatenation, the multi-image guided feature representation Z is obtained for classification; Using spectral domain graph convolution function S (v) Representative in A (v) The normalized graph Laplacian is used on where D is the degree matrix, Among them, H (v) A represents the feature vector of the v-th layer image. (v) represents the similarity weight of the v-th layer graph, represents the output of the vth layer of graph convolutional network, and l is the number of graph convolutional network layers; For graph convolution operations at different levels (views), the weight parameters of the hidden layer are θ = {θ0…θ L-1 } is shared; Step S5: Similarity measurement and classification The convolutional representations of graphs at different levels of view are concatenated to obtain feature Z. In order to make the query sample feature close to the class center of its own class and away from the class centers of other classes, the contrastive learning function is introduced here. The commonalities of features of the same category are extracted through this contrastive learning function, and the similarities of samples of the same category are retained; Among them, S n represents the sample of the nth class in the support set, c n represents the class center of the nth class; y i is the label of the support set sample; Substitute the multi-image guided image features into the above contrast learning function In the process, the query samples are classified, the cross entropy classification loss is calculated, and the reverse gradient update is performed; each time the task samples under the specified settings are selected, the above steps are repeated for training.

2. The small sample size scene image classification method based on the multi-graph guided neural network model according to claim 1 is characterized in that: The feature extraction network Each convolution module includes a 3×3 convolution layer, a normalization layer, a nonlinear activation function layer and a 2×2 pooling layer, and the Dropout layer is used between the nonlinear activation function layer and the pooling layer to prevent overfitting.

3. The small sample size scene image classification method based on the multi-graph guided neural network model according to claim 1 is characterized in that: The graph construction network ψ includes a convolutional module and two fully connected layers; The third part consists of a two-layer graph convolutional neural network with a nonlinear layer between the two layers. Each graph convolution operation includes two steps of propagation and aggregation.

Images