A few-shot image classification method based on global and local feature correlation analysis

Abstract

The present invention discloses a few-shot image classification method based on global and local feature association analysis, wherein the method comprises: step 1, inputting a training task into a convolutional network to obtain an original global feature and an original local feature set; step 2, calculating the correlation coefficient between the original global feature and the original local feature set; step 3, dividing the local features into background-related local features and class-related local features, adjusting the correlation coefficient and assigning weights to obtain a local feature set containing a small amount of background information; step 4, performing multi-scale pooling and splicing on the local feature set containing a small amount of background information to obtain a final query image local feature set and a final support image local feature set; step 5, calculating the similarity between the final query image local feature and the final support image local feature set, and updating the network parameters; step 6, repeating steps 1 to 5 using multiple tasks to obtain a few-shot classification model; and step 7, using the model for classification.

Description

Technical Field

[0001] This document relates to the field of image classification, and in particular to a few-shot image classification method based on global and local feature correlation analysis. Background Art

[0002] In recent years, deep learning has played an important role in the field of computer vision. However, deep learning models usually require a large amount of data training, while humans only need a small number of samples to recognize objects. Therefore, researchers have begun to study few-sample learning technology, that is, training a model with good generalization ability so that it can handle classification tasks with very few samples.

[0003] In the related technology, the existing few-shot learning technology is based on the metric learning method of local features, which realizes the semantic feature alignment in few-shot classification by measuring the local features between the support image samples and the query image samples. However, the above technical solution has the following defects: local features may be contained in background-related local features, and these background-related local features contain certain noise interference information, which will have a negative impact on the classification results.

[0004] Based on the above analysis of the development status of this technical field, the existing technology lacks a solution to eliminate the negative impact caused by background-related local features. Summary of the Invention

[0005] The purpose of the present invention is to provide a small sample image classification method based on global and local feature association analysis, aiming to solve the above problems in the prior art.

[0006] According to a first aspect of an embodiment of the present disclosure, a method for establishing a few-sample image model is provided, comprising:

[0007] Step 1: Initialize the convolutional neural network model parameters and input the few-shot training task containing the query set image and the support set image into the convolutional neural network model to obtain the original global features and original local feature sets of each training image in the query set and the support set;

[0008] Step 2: Calculate the correlation coefficients of the original global features and original local feature sets of various training images in the query set and support set respectively;

[0009] Step 3: Use the adaptive threshold method to divide the local features in the original local feature set into background-related local features and class-related local features, and use a custom scaling function to adjust the correlation coefficients of background-related local features and class-related local features; adjust the correlation coefficients using the attention formula to obtain the attention weight matrix and assign weights to the original local feature set according to the attention weight matrix to obtain a local feature set containing a small amount of background information;

[0010] Step 4: Perform multi-scale average pooling and concatenation on the local feature sets containing a small amount of background information corresponding to each type of image to obtain the final query image local feature set and the final support image local feature set;

[0011] Step 5: Calculate the similarity scores between the final query image local feature set and the final support image local feature set, input each similarity score into the multi-layer perceptron to obtain the similarity probability score, calculate the loss using the softmax function and the cross entropy loss function in turn, and update the convolutional neural network model parameters;

[0012] Step 6: Repeat steps 1 to 5 using multiple few-shot training tasks, retain the parameters of the convolutional neural network model with the best classification effect, and obtain the few-shot image classification model;

[0013] Step 7: Use the few-shot image classification model to classify the new query set images.

[0014] The technical solution provided by the embodiment of the present invention includes the following beneficial effects: in the process of establishing a few-sample image classification model, the features contained in the original local features are divided into background-related features and class-related features, and the importance of local features of different classes is adjusted and calculated, thereby reducing the negative impact of background-related local features on the classification results. Multiple training makes the establishment of the few-sample image classification model more accurate.

[0015] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 is a flowchart of a method for establishing a few-sample image classification model according to an embodiment of the present invention;

[0018] Figure 2 is a schematic diagram of a few-sample image classification model framework according to an embodiment of the present invention;

[0019] Figure 3 Schematic diagram of feature division and custom scaling according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below in conjunction with the drawings in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this document.

[0021] Method Example

[0022] According to an embodiment of the present invention, a method for establishing a few-sample image classification model is provided. Figure 1 Flowchart of the method for establishing a few-sample image classification model according to an embodiment of the present invention. Figure 1 As shown, the method for establishing a few-sample image classification model according to an embodiment of the present invention specifically includes:

[0023] In step S110, the convolutional neural network model parameters are initialized, and the few-shot training task containing the query set image and the support set image is input into the convolutional neural network model to obtain the corresponding original global feature and original local feature sets. Specifically, it includes:

[0024] The query set image and the support set image are input into the convolutional neural network, and the convolutional neural network is used to extract the feature tensor of the image. The feature tensor is processed in parallel through the convolution layer, activation function and maximum pooling layer to obtain the original global features and original local feature sets corresponding to each image, where the original local feature set includes multiple local features.

[0025] In step S120, the correlation coefficients of the original global features and the original local feature sets of various training images in the query set and the support set are calculated respectively. Specifically, the following steps are performed:

[0026] Calculate the correlation coefficients of the original global features and original local feature sets of various training images in the query set and support set respectively. Use formula 1 to calculate the correlation coefficient:

[0027]

[0028] Where, Represents the correlation coefficient matrix formed by calculating each correlation coefficient, Represents the embedding space where the feature tensor is located, h represents the height of the feature tensor, w represents the width of the feature tensor; cos() represents the cosine function, A represents the average pooling operation, M represents the maximum pooling operation, Represents a local feature set.

[0029] In step S130, the local features in the original local feature set are divided into background-related local features and class-related local features using an adaptive threshold method, and the correlation coefficients of the background-related local features and the class-related local features are adjusted using a custom scaling function; the correlation coefficients are adjusted using an attention formula to obtain an attention weight matrix, and the original local feature set is weighted according to the attention weight matrix to obtain a local feature set containing a small amount of background information. Specifically, the following steps are involved:

[0030] Use Formula 2 to define the threshold expression, use Formula 3 to adjust the importance of background-related features and class-related features, use the attention formula of Formula 4 to obtain the attention weight matrix, and assign weights to the local feature set of the corresponding training image according to the attention weight matrix to obtain a local feature set containing a small amount of background information. The sample features of the local feature set containing a small amount of background information are shown in Formula 5:

[0031]

[0032]

[0033]

[0034]

[0035] Where ∈ represents the defined threshold, Indicates that Through the local features after 5×5 pooling, it is determined whether the correlation coefficient corresponding to the sub-local feature is greater than the preset threshold; if the correlation coefficient is less than the threshold, the sub-local feature is classified as a background-related feature; otherwise, the sub-local feature is classified as a class-related feature. Represents the updated correlation coefficient matrix, s represents the custom scaling function, σ represents the sigmoid function, k represents the adjustable parameter of the scaling function, which controls the scaling degree of the scaling function, Atten() represents the attention formula representing the weight coefficients of different local features, c represents the number of channels, Represents the correlation coefficient of the local feature in the i-th row and j-th column; Represents a local feature set that contains a small amount of background information.

[0036] In step S140, multi-scale average pooling and concatenation are performed on the local feature sets containing a small amount of background information corresponding to each type of image to obtain the final query image local feature set and the final support image local feature set. Specifically, the following steps are performed:

[0037] The local feature sets containing a small amount of background information corresponding to the query image and the support image are respectively input into multiple average pooling layers of different dimensions, and average pooling processing is performed in the average pooling layers of different dimensions to obtain pooled feature sets of different dimensions; the pooled feature sets of different dimensions and the local feature sets containing a small amount of background information are combined to obtain the corresponding final query image local feature set and final support image local feature set, and the pooled feature set is obtained using Formula 6:

[0038]

[0039] In the formula, the local feature set containing a small amount of background information Input into the 1×1 average pooling layer to obtain a 1D pooled feature set Will Input into the average pooling layer of 2×2, 3×3, 5×5 and 7×7 respectively to obtain feature vectors of different dimensions cat() means and Combine to form the final local feature set

[0040] In step S150, the similarity score between the final query image local feature set and the final support image local feature set is calculated, the similarity score of the query image local feature set is input into the multi-layer perceptron to obtain the final similarity score, and the similarity probability is obtained using the softmax function; the network parameters of the convolutional neural network model are updated using the cross entropy loss and gradient descent algorithm. Specifically, the following steps are involved:

[0041] Calculate the similarity score between the final query image local feature set and the final support image local feature set, input each similarity score into the multi-layer perceptron to adaptively learn the importance of the similarity score, and obtain the final similarity score. Use formula 7, formula 8, and formula 9 to calculate the similarity score, and use formula 10 to obtain the similarity probability:

[0042]

[0043]

[0044]

[0045] P = softmax(MLP(R)) Formula 10;

[0046] In the formula, cos() represents the cosine function, topk() represents selection, sum() represents the summed cosine similarity score; R represents the similarity score, represents the final query image local feature set, represents the local feature set of similar support images, Ctrain Represents the training set; P represents the similarity probability, softmax() represents mapping the obtained score MLP(R) between (0,1), and MLP() is a multi-layer perceptron network.

[0047] In this embodiment, the similarity score between the local feature set of the final query image and the local feature set of the final supporting image is calculated; for each local feature of the local feature set of the final query set image, p local features with the highest similarity scores are selected from the local feature set of the final supporting image, and the selected scores are added together as the similarity score of the local feature of the query image.

[0048] In step S160, steps S110 to S150 are repeatedly executed using multiple training tasks, and the parameters of the convolutional neural network model with the best classification effect are retained to obtain a well-established few-sample image classification model.

[0049] In step S170 , the new query set images are classified using the few-shot image classification model.

[0050] To sum up, in response to the current problems, this paper invented a few-sample image classification method based on global and local feature correlation analysis. In the process of establishing the few-sample image classification model, the local features contained in the original local feature set are divided into background-related local features and class-related local features, the importance of each local feature is adjusted, and the correlation coefficient matrix is updated to obtain a local feature set containing a small amount of background information, thereby reducing the negative impact of background-related features on the classification results; the local feature set containing a small amount of background information is input into the average pooling layer of different dimensions to obtain multiple pooled local feature sets of different scales, and then the local feature set containing a small amount of background information in the pooled local feature set is spliced to obtain the final local feature set, thereby improving the feature expression ability and obtaining a rich local feature set; multiple groups of training tasks are used to train the convolutional neural network model to make the establishment of the few-sample image classification model more accurate.

[0051] The above technical solutions of the embodiments of the present invention are illustrated with reference to the following drawings.

[0052] Figure 2 Schematic diagram of the few-sample image classification model framework of an embodiment of the present invention. Figure 2 As shown in the figure, the process of establishing a few-shot image classification model is demonstrated, including completing a convolutional neural network training through the query set image and its corresponding set of support set images, distinguishing the background-related local features and class-related local features corresponding to each image and adjusting the importance of each calculation, updating the calculation weights to obtain the attention distinction weight matrix, calculating the final query image local feature set and the final support image similarity score, and obtaining the similarity probability. Figure 3Schematic diagram of feature division and custom scaling according to an embodiment of the present invention. Figure 3 As shown in the figure, the specific details of using the adaptive threshold method to distinguish between background-related local feature sets and class-related local feature sets, as well as using a custom scaling function to adjust the correlation coefficient are demonstrated.

[0053] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A few-sample image classification method based on global and local feature correlation analysis, characterized in that: include: Step 1: Initialize the parameters of the convolutional neural network model, input the few-shot training task containing the query set image and the support set image into the convolutional neural network model, and obtain the original global feature and original local feature set of each image in the query set and the support set; Step 2, respectively calculating the correlation coefficients of the original global features and the original local feature sets of the training images in the query set and the support set; Step 3: Using an adaptive threshold method to divide the local features in the original local feature set into background-related local features and class-related local features, and using a custom scaling function to adjust the correlation coefficients of the background-related local features and the class-related local features; adjusting the correlation coefficients using an attention formula to obtain an attention weight matrix, and assigning weights to the original local feature set according to the attention weight matrix to obtain a local feature set containing a small amount of background information; Step 4: Perform multi-scale average pooling and concatenation on the local feature sets containing a small amount of background information corresponding to each type of image to obtain the final query image local feature set and the final support image local feature set; Step 5: Calculate the similarity score between the local feature set of the final query image and the local feature set of the final support image, input each similarity score into a multi-layer perceptron to obtain a similarity probability score, calculate the loss using the softmax function and the cross entropy loss function in sequence, and update the convolutional neural network model parameters; Step 6: Repeat steps 1 to 5 using multiple few-shot training tasks, retain the parameters of the convolutional neural network model with the best classification effect, and obtain a few-shot image classification model; Step 7: Classify the new query set images using the few-shot image classification model.

2. The method according to claim 1, characterized in that The inputting of the few-shot training task including the query set image and the support set image into the convolutional neural network model to obtain the original global features and original local feature sets of each image in the query set and the support set specifically includes: The query set image and the support set image are input into a convolutional neural network, the convolutional neural network model is used to extract the feature tensor of the image, and the feature tensor is processed in parallel through a convolution layer, an activation function, and a maximum pooling layer to obtain the original global feature and the original local feature set corresponding to each image.

3. The method according to claim 1, characterized in that The step of respectively calculating the correlation coefficients of the original global features and the original local feature sets of the training images in the query set and the support set specifically includes: The cosine function is used to calculate the correlation coefficients of the original global features and original local feature sets of various training images in the query set and support set respectively.

4. The method according to claim 1, wherein The method of using the adaptive threshold method to divide the local features in the original local feature set into background-related local features and class-related local features specifically includes: Determine whether the correlation coefficient corresponding to the local feature is greater than a preset threshold; if the correlation coefficient is less than the threshold, classify the local feature as a background-related local feature; otherwise, classify the local feature as a class-related local feature.

5. The method according to claim 1, wherein The method of assigning weights to the original local feature set according to the attention weight matrix to obtain a local feature set containing a small amount of background information specifically includes: assigning weights to the local feature set of the corresponding training image according to the attention weight matrix to obtain a local feature set containing a small amount of background information.

6. The method according to claim 1, characterized in that The multi-scale average pooling and splicing of the local feature sets containing a small amount of background information corresponding to each type of image to obtain the final query image local feature set and the final support image local feature set specifically includes: The local feature sets containing a small amount of background information corresponding to the query image and the support image are respectively input into multiple average pooling layers of different dimensions, and average pooling processing is performed in the average pooling layers of different dimensions to obtain pooled feature sets of different dimensions; the pooled feature sets of different dimensions and the local feature set containing a small amount of background information are combined to obtain the corresponding final query image local feature set and final support image local feature set.

Images