An image classification method based on small sample learning cross-correlation network
By using a cross-correlation network based on few-shot learning and leveraging the cross-correlation module with embedded networks and self-attention mechanisms to generate robust features, the problems of overfitting and insufficient robustness in traditional methods are solved, thus improving the accuracy of image classification.
Patent Information
- Application Number
- CN202310287429.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-23
- Publication Date
- 2025-12-12
- Estimated Expiration
- 2043-03-23
AI Technical Summary
Traditional deep learning methods are prone to overfitting in small sample learning, which leads to spurious features affecting the model's generalization ability. Existing technologies such as RENet are not robust enough when dealing with severe deformations.
We employ a cross-correlation network based on few-shot learning, extract multi-level features through network embedding, and utilize a cross-correlation module with a self-attention mechanism to mine global consensus, remove useless information, and generate more robust image features.
It improves the accuracy of small sample classification, captures semantic information at different levels, and generates more robust image features.
Smart Images
Figure CN116403029B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of small sample classification, and particularly relates to an image classification method based on a small sample learning cross-correlation network. BACKGROUND
[0002] Humans can quickly learn from a small amount of training data. For example, a human child can distinguish which picture is an elephant from a large number of pictures after seeing a few elephant pictures. However, traditional deep learning methods will encounter great difficulty in the same task. With only a few training data, a deep learning model will produce a disastrous performance on new data, i.e. overfitting. In order to enable artificial intelligence to have the ability of small sample learning, these intelligent agents must obtain prior knowledge learned from previous tasks and transfer the learned prior knowledge to new tasks. In fact, humans also learn by using prior knowledge, extracting prior knowledge useful for the current task when doing a new task, and reconstructing new knowledge. This learning process is called "meta-learning". Meta-learning, also known as "learning to learn", is to learn some learning skills to enable the model to quickly and effectively learn on new tasks. Most methods in small sample learning use the meta-learning framework to train their models and have achieved good performance. In recent years, the academic community has proposed many meta-learning methods to solve the small sample learning problem, including methods based on fine-tuning and methods based on metrics. The methods based on metrics try to learn the features of images in a metric space and calculate the similarity (e.g. Euclidean distance or cosine similarity) between two image features to identify their classes. However, there are some disadvantages in directly using image features, because the embedding model often overfits irrelevant features, called "spurious features", which cause the model to focus on useless information and fail to transfer useful knowledge to new tasks. It is a great challenge to learn features with strong generalization ability without being affected by spurious features. Although there is a method in the prior art that uses RENet to solve this problem, the main idea is to learn the relationship pattern between two images to protect the model from spurious features, which represents how two images are related to each other and guides the model to generate image features by combining the information between a pair of images, so that the features are more robust. Such features are called relationship features. Specifically, in RENet, a cross-correlation attention module based on 4D convolution is used to update the initial embedding, however, limited by the limited receptive field of CNN-based methods, RENet may lack robustness to severe deformations. SUMMARY
[0003] The present application aims to provide an image classification method based on a small sample learning cross-correlation network to generate more robust image features and improve the accuracy of small sample classification.
[0004] The technical solution of the present application to solve the above technical problems is as follows:
[0005] The application provides an image classification method based on a small sample learning cross-correlation network, which comprises the following steps:
[0006] S1: extracting first multi-level features of support set images and second multi-level features of query set images by using an embedding network;
[0007] S2: processing the first multi-level features and the second multi-level features by using a correlation unit to obtain initial cross-correlation information;
[0008] S3: performing information extraction on the initial cross-correlation information by using a cross-correlation module to obtain extracted cross-correlation information;
[0009] S4: performing mean value operation on the extracted cross-correlation information to obtain correlation information of the support set images and the query set images;
[0010] S5: calculating a first tensor of the support set images and a second tensor of the query set images according to the correlation information;
[0011] S6: obtaining first relationship features of the support set images and second relationship features of the query set images by using the first tensor and the second tensor;
[0012] S7: assigning the query set images to the class of the support set image features closest to the query set images according to the first relationship features of all images in the support set and the second relationship features of all images in the query set.
[0013] Optionally, in S1, the embedding network is a ResNet18 embedding network.
[0014] Optionally, in S1, the first multi-level features E s are:
[0015]
[0016] The second multi-level features E q are:
[0017]
[0018] wherein, is the lth level feature of the first multi-level features, R is a real number set, L represents the number of levels of the multi-level features, h is the dimension of the first dimension of the first multi-level features, and w is the dimension of the second dimension of the first multi-level features, is the channel dimension of the first multi-level features, and l is a number between 1 and L, h' is a dimension of a first dimension of the second multi-level feature, w' is a dimension of a second dimension of the second multi-level feature, and respectively represent a dimension of a support set image base feature and a dimension of a query set image base feature.
[0019] Optionally, in the S2, the initial cross-correlation information corr is:
[0020]
[0021] wherein corr l represents an l-th level of the initial cross-correlation information, and reshape() represents a dimension change operation, which can convert a tensor with a dimension of h x w x c into a tensor with a dimension of hw x c, is a support set image base feature, R is a real set, L represents a number of levels of multi-level features, h is a dimension of a first dimension of a first multi-level feature, w is a dimension of a second dimension of the first multi-level feature, is a query set image base feature, l is a number between 1 and L, and hw x h'w' represents a dimension of corr l .
[0022] Optionally, in the S3, the cross-correlation module comprises LN units, an Intra-Correlation unit, a first LN&MLP unit, an Inter-Correlation unit and a second LN&MLP unit arranged in sequence.
[0023] The LN unit is configured to make outputs of intermediate layers of the network have the same mean and variance.
[0024] The Intra-Correlation unit is configured to explore correlation within each level of multi-level cross-correlation information.
[0025] The Inter-Correlation unit is configured to explore correlation between each level of multi-level cross-correlation information.
[0026] The first LN&MLP unit and the second LN&MLP unit are configured to alleviate a problem of reduced network expression capability after stacking of a multi-layer self-attention structure.
[0027] Optionally, the cross-correlation module adopts a self-attention mechanism and is two, an output result of a former cross-correlation module is taken as an input of a latter cross-correlation module.
[0028] The two cross-correlation modules are constructed in the same structure.
[0029] Optionally, the calculation manner of each of the cross-correlation modules is:
[0030] corr' = MLP(Intra(LN(corr + E pos )))
[0031] corr map = MLP(Intra(LN(corr' + E pos )))
[0032] wherein corr' represents an intermediate output after the first LN&MLP unit, MLP represents a multi-layer perception, Intra represents an Intra-Correlation unit, LN represents a layer normalization, corr represents initial cross-correlation information, E pos represents a position embedding vector, corr map represents refined cross-correlation information, and Intra represents.
[0033] Optionally, the objective function of the image classification method based on the small sample learning cross-correlation network is:
[0034] L = L1 + aL2
[0035] wherein L1 represents a loss value generated by a global classifier and w1 and b are weights and biases of a fully connected layer, represents a transpose of w1, E q represents a second multi-level feature, c represents an index corresponding to a correct category to which a query set picture to be classified belongs, c e {1, 2, 3, …, |C train |}, c' represents a number between 1 and |C train |, and C train represents a category set of a training set; a is a hyperparameter for balancing L1 and L2, L2 represents a loss generated by a nearest neighbor classifier, and cos() is a cosine similarity, g is a scalar and represents a temperature factor, n represents an index corresponding to a correct category to which a query set picture to be classified belongs, n' represents a number between 1 and N, represents a support set feature, represents a query set feature, and N represents a support set category number.
[0036] Optionally, in S6, the first relationship feature s is:
[0037] s = pool(E' s )
[0038] wherein E' s represents each level feature in E s and A s e Rh×w the intermediate output after Hadamard product and splicing, and is the lth level feature of the first multi-level feature, A s represents the first tensor, and A s ∈R h×w L represents the number of levels of the multi-level feature, and h x w represents the dimension of the first tensor;
[0039] The second relationship feature q is:
[0040] q = pool (E' q )
[0041] wherein, pool() represents average pooling, E' q represents the multi-level feature of the query set image, and is the lth level feature of the second multi-level feature, A q represents the second tensor, and A q ∈R h'×w' L represents the number of levels of the multi-level feature, Concat() represents splicing along the channel dimension, * represents Hadamard product, and h' x w' represents the dimension of the second tensor.
[0042] Alternatively, in the S7, the query set image is assigned to the category of the support set image feature closest to it by calculating the cosine similarity between the first relationship of all images in the support set and the second relationship of all images in the query set.
[0043] The cosine similarity is calculated by:
[0044]
[0045] wherein, sim (n) represents the similarity of the query set image and the nth support set image feature, cos() is the cosine similarity, γ is a scalar and represents a temperature factor, n represents a number between 1 and N, represents the support set feature, represents the query set feature, and N represents the number of support set images.
[0046] The present application has the following beneficial effects:
[0047] The image classification method based on the small sample learning cross-correlation network can generate multi-level features for images through an embedding network, thereby capturing semantic information at different levels; and can mine global consensus in initial cross-correlation information, eliminate useless information in the initial cross-correlation information, and generate more robust image features through the cross-correlation module based on the self-attention mechanism, thereby improving the accuracy of small sample classification. BRIEF DESCRIPTION OF DRAWINGS
[0048] Figure 1 A flowchart of the image classification method based on the small sample learning cross-correlation network of the present application;
[0049] Figure 2 A structural schematic diagram of the small sample learning cross-correlation network of the present application. DETAILED DESCRIPTION
[0050] The principles and characteristics of the present application are described below in combination with the drawings, and the examples are only used to explain the present application and are not used to limit the scope of the present application.
[0051] The present application provides an image classification method based on a small sample learning cross-correlation network, as shown in Figure 1 and Figure 2 , the image classification method comprises the following steps.
[0052] The small sample learning cross-correlation network of the present application comprises a multi-level embedding part and a cross-correlation part, the multi-level embedding part comprises an embedding network, the embedding network is used to generate multi-level features for images to capture semantic information at different levels, and the cross-correlation part comprises a correlation unit and a cross-correlation module arranged in sequence, the cross-correlation module is based on a self-attention mechanism to mine global consensus in initial cross-correlation information.
[0053] The image classification method based on the small sample learning cross-correlation network comprises:
[0054] S1: extracting first multi-level features of support set images and second multi-level features of query set images by using an embedding network;
[0055] It should be noted that the support set images and the query set images are adjusted to 84x84 in size before being input into the embedding network.
[0056] Given a support set image I s and a query set image I q , the multi-level features of the support set image and the query set image are extracted by using the embedding network. The embedding network of the present application is a ResNet18 embedding network.
[0057] The ResNet18 has an initial convolutional layer and 8 basic blocks, each block outputs a tensor of different dimensions, denoted as wherein t∈(0,1,2,3,4,5,6,7,8) represents the number of layers, as shown in Figure 2 The present application will be described as shown on the left side is adjusted to and L output is selected from these tensors to constitute a multi-level feature. That is, for the support set image and the query set image, the first multi-level feature E s and the second multi-level feature E q are respectively constituted.
[0058] The first multi-level feature E s is:
[0059]
[0060] The second multi-level feature E q is:
[0061]
[0062] wherein, is the l-th level feature of the first multi-level feature, R is the set of real numbers, L represents the number of layers of the multi-level feature, h is the dimension of the first dimension of the first multi-level feature, w is the dimension of the second dimension of the first multi-level feature, is the channel dimension of the first multi-level feature, and l is a number between 1 and L, is the l-th level feature of the second multi-level feature, h' is the dimension of the first dimension of the second multi-level feature, w' is the dimension of the second dimension of the second multi-level feature, and respectively represent the dimension of the support set image basic feature and the dimension of the query set image basic feature.
[0063] It should be noted that in the present application h=h' and w=w'.
[0064] S2: processing the first multi-level feature and the second multi-level feature by using a correlation unit to obtain initial cross-correlation information;
[0065] The correlation unit performs matrix multiplication on the features of each level in the first multi-level feature and the second multi-level feature, and further obtains the initial cross-correlation information, and the correlation unit is a correlation unit as shown in Figure 2
[0066] The initial cross-correlation information corr is:
[0067]
[0068] wherein, corr l represents the l-th level of the initial cross-correlation information, and reshape() represents a dimension change operation, which can convert a tensor with dimensions h x w x c into a tensor with dimensions hw x c, In order to support set image basic features, R is a real number set, L represents the number of layers of multi-level features, h is the dimension of the first dimension of the first multi-level feature, and w is the dimension of the second dimension of the first multi-level feature, In order to query set image basic features, l is a number between 1 and L, and hw x h'w' represents the dimensions of corr l .
[0069] The dimensions of corr l are hw x h'w', and the initial cross-correlation information corr l of all levels is stacked layer by layer to form corr∈R hw×h'w'×L . The cross-correlation information corr contains the correlation information between the support set image I s and the query set image I q , and can be used to generate relationship features for the image pair.
[0070] However, the initial cross-correlation information corr may have some inconsistent correlations, so it needs to be refined in the cross-correlation module, and the useless information is removed, so it enters S3.
[0071] S3: refining the initial cross-correlation information by using a cross-correlation module to obtain refined cross-correlation information;
[0072] Here, the value in the corr tensor represents the degree of cross-correlation between each value in the multi-level feature E s and E q . If E s and E q are regarded as images (in fact, E s and E q are high-level representations of images), the value in corr represents the degree of association between each pixel in the two images, and the larger the value, the stronger the association. After modification by the cross-correlation module, the degree of association between the pixel points contained in the target object in the image will be enhanced, and the degree of association between the pixel points contained in some useless background information will be weakened, so the cross-correlation module plays a role in removing useless cross-information.
[0073] The cross-correlation module comprises LN units, an Intra-Correlation unit, a first LN&MLP unit, an Inter-Correlation unit and a second LN&MLP unit arranged in sequence;
[0074] The LN unit is used to make the output of the intermediate layer of the network have the same mean and variance;
[0075] The Intra-Correlation unit is used to explore the correlation within each level of multi-level cross-correlation information;
[0076] The Inter-Correlation unit is used to explore the correlation between each level of multi-level cross-correlation information;
[0077] The first LN&MLP unit and the second LN&MLP unit are used to alleviate the problem of reduced network expressive power after stacking multiple self-attention structures.
[0078] Optionally, refer to Figure 2 As shown, to enhance the effect of the cross-correlation module, the cross-correlation module adopts a self-attention mechanism and consists of two modules, with the output of the first cross-correlation module serving as the input of the second cross-correlation module; the two cross-correlation modules are constructed with the same structure.
[0079] The calculation method for each of the aforementioned cross-correlation modules is as follows:
[0080] corr'=MLP(Intra(LN(corr+E pos )))
[0081] corr map =MLP(Intra(LN(corr'+E) pos )))
[0082] Where corr' represents the intermediate output after the first LN&MLP unit, MLP represents a multilayer perceptron, Intra represents an Intra-Correlation unit, LN represents layer regularization, corr represents the initial cross-correlation information, and E pos Represents the position embedding vector, corr map Intra represents the refined interrelationships.
[0083] In this invention, the calculation rules for Intra-Correlation units and Inter-Correlation units are as follows:
[0084] MultiHead(M)=Concat(head1,...,head h W 0
[0085] head i =Attention(MW) i Q MW i K MW i V )
[0086] Attention(Q, K, V) = softmax(QK T )V
[0087] where h is the number of heads, M is projected by three different matrices MW i Q , MW i K , MW i V , the output of MultiHead() is projected by matrix W 0 to a matrix of the same size as M.
[0088] S4: performing mean operation on the extracted cross-correlation information to obtain correlation information of the support set image and the query set image;
[0089] L is reduced to 1 by mean operation, and the correlation information corr s between images I q and I map is obtained. hw×h'w' ∈R map , which contains consistent correlation information between images I s and I q .
[0090] S5: calculating a first tensor A s ∈R h×w of the support set image and a second tensor A q ∈R h'×w' of the query set image according to the correlation information.
[0091] Specifically Figure 2 , the present application performs mean operation on different directions of the correlation information to obtain two tensors.
[0092] The two tensors are transformed into the same shape as the image features.
[0093] That is, for the support set image, the present application uses h and w without superscripts to represent the dimensions of the features , so mean operation is performed on corr map ∈R hw×h'w' along a fixed direction to obtain a tensor ((R 1 ×h×w ).
[0094] For the query set image, the present application uses h' and w' with superscripts to represent the dimensions of the features , so mean operation is performed on corr map ∈R hw×h'w'The mean operation is performed along another fixed direction to obtain another tensor (R 1 ×h'×w' ).
[0095] Co-attentions help the model to distinguish the relevant and irrelevant information in the two basic features of the image.
[0096] S6: Obtain the first relationship feature of the support set image and the second relationship feature of the query set image by using the first tensor and the second tensor;
[0097] The first relationship feature s is:
[0098] s = pool(E s ')
[0099] Wherein, E s ' represents the intermediate output after the Hadamard product and splicing of each level feature in E s and A s ∈R h×w , and is the lth level feature of the first multi-level feature, A s represents the first tensor, and A s ∈R h×w , L represents the number of levels of the multi-level feature, and h×w represents the dimension of the first tensor;
[0100] The second relationship feature q is:
[0101] q = pool(E' q )
[0102] Wherein, pool() represents the average pooling, E' q represents the multi-level feature of the query set image, and is the lth level feature of the second multi-level feature, A q represents the second tensor, and A q ∈R h'×w' , L represents the number of levels of the multi-level feature, Concat() represents splicing along the channel dimension, * represents Hadamard product, and h'×w' represents the dimension of the second tensor.
[0103] S7: According to the first relationship features of all images in the support set and the second relationship features of all images in the query set, the query set image is assigned to the category of the nearest support set image feature.
[0104] Specifically, the query set images are assigned to the class of the nearest support set image feature by calculating the cosine similarity between the first relationship of all images in the support set and the second relationship of all images in the query set;
[0105] The cosine similarity is calculated by:
[0106]
[0107] wherein, sim (n) represents the similarity of the query set image and the nth support set image feature, cos() is the cosine similarity, γ is a scalar and represents a temperature factor, n represents a number between 1 and N, represents the support set feature, represents the query set feature, and N represents the number of support set images.
[0108] Alternatively, the objective function of the image classification method based on the small sample learning cross-correlation network is:
[0109] L = L1 + αL2
[0110] wherein, L1 represents the loss value generated by the global classifier and w1 and b are the weights and bias of the full connection layer, represents the transpose of w1, E q represents the second multi-level feature, c represents the index corresponding to the correct class to which the query set image to be classified belongs, c ∈ {1, 2, 3, …, |C train |}, c' represents a number between 1 and |C train |, and C train represents the class set of the training set; α is a hyperparameter for balancing L1 and L2, L2 represents the loss generated by the nearest neighbor classifier and cos() is the cosine similarity, γ is a scalar and represents a temperature factor, n represents the index corresponding to the correct class to which the query set image to be classified belongs, and n' represents a number between 1 and N, represents the support set feature, represents the query set feature, and N represents the number of support set classes.
[0111] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. An image classification method based on a few-shot learned cross-correlation network, characterized in that, The image classification method based on a few-shot cross-correlation network includes: S1: Extract first-level multi-features of the support set image and second-level multi-features of the query set image using an embedding network; S2: The first multi-level features and the second multi-level features are processed using the correlation unit to obtain the initial cross-correlation information; S3: Use the cross-correlation module to extract information from the initial cross-correlation information to obtain the refined cross-correlation information; S4: Perform an averaging operation on the refined mutual information to obtain the relevant information of the support set image and the query set image; S5: Calculate the first tensor of the support set image and the second tensor of the query set image based on the relevant information; S6: Using the first tensor and the second tensor, obtain the first relational feature of the support set image and the second relational feature of the query set image; S7: Based on the first relational features of all images in the support set and the second relational features of all images in the query set, assign the query set images to the category of the features of the support set images closest to them; In S1, the first multi-level feature for: Second multi-level features for: in, The first multi-level feature l Level features, R For the set of real numbers, L The number of layers representing multi-level features. h The first dimension of the first multi-level feature. w The second dimension is the dimension of the first multi-level feature. For the channel dimension of the first multi-level feature, l From 1 to L The number between The second multi-level feature l Level features, The first dimension of the second multi-level feature. The second dimension is the second dimension of the second multi-level feature. and These represent the dimensions of the basic features of the supported image set and the dimensions of the basic features of the query image set, respectively. In S3, the cross-correlation module includes an LN unit, an Intra-Correlation unit, a first LN&MLP unit, an Inter-Correlation unit, and a second LN&MLP unit arranged sequentially. The LN unit is used to ensure that the outputs of the intermediate layers of the network have the same mean and variance; The Intra-Correlation unit is used to explore the correlation within each level of multi-level cross-correlation information; The Inter-Correlation unit is used to explore the correlation between each level of multi-level cross-correlation information; The first LN&MLP unit and the second LN&MLP unit are used to alleviate the problem of reduced network expressive power after stacking multiple self-attention structures; The cross-correlation module employs a self-attention mechanism and consists of two modules, with the output of the first cross-correlation module serving as the input of the second cross-correlation module. The two cross-correlation modules are constructed with the same structure.
2. The image classification method based on a few-shot learned cross-correlation network according to claim 1, characterized in that, In step S1, the embedded network is a ResNet18 embedded network.
3. The image classification method based on a few-shot learning cross-correlation network according to claim 1, characterized in that, In S2, the initial mutual information for: in, The first mutual correlation information is represented by the first... l Level, and , This represents a dimension change operation, which can transform a dimension into a variable. The tensor is transformed into a dimension of tensor, To support the collection of basic image features, R For the set of real numbers, L The number of layers representing multi-level features. h The first dimension of the first multi-level feature. w The second dimension is the dimension of the first multi-level feature. To query the basic features of the image set, l From 1 to L The number between express Dimensions.
4. The image classification method based on a few-shot learning cross-correlation network according to claim 1, characterized in that, The calculation method for each of the aforementioned cross-correlation modules is as follows: in, This represents the intermediate output after passing through the first LN&MLP unit. This represents a multilayer perceptron. Indicates Intra-Correlation unit, Representation layer regularization, Indicates initial mutual related information, Represents the position embedding vector. This represents the refined, interconnected information. express.
5. The image classification method based on a few-shot learned cross-correlation network according to claim 1, characterized in that, The objective function of the image classification method based on a few-sample learning cross-correlation network is: in, Represents the loss value generated by the global classifier and , w 1 and b These are the weights and biases of the fully connected layer. express w The transpose of 1 This indicates the second level of features. c This represents the index corresponding to the correct category to which the images in the query set belong. , Indicates 1 to The number between Represents the set of categories in the training set; To balance and hyperparameters, The loss generated by the nearest neighbor classifier is... , For cosine similarity, It is a scalar and represents the temperature factor. n This indicates the index corresponding to the correct category to which the images in the query set belong. Indicates 1 to N The number between Indicates support set features, Indicates the characteristics of the query set. N Indicates the number of supported set categories.
6. The image classification method based on a few-shot learned cross-correlation network according to claim 1, characterized in that, In S6, the first relation feature for: in, express Characteristics of each level and The intermediate output after performing Hadamard product and concatenation , The first multi-level feature l Level features, Indicates the first tensor and , L The number of layers representing multi-level features. Indicates the dimension of the first tensor; Second relation feature q for: in, Indicates average pooling. Represents the multi-level features of the query set image and , The second multi-level feature l Level features, Indicates the second tensor and , L The number of layers representing multi-level features. This indicates splicing along the channel dimension. This represents the Hadamard product. This represents the dimension of the second tensor.
7. The image classification method based on a few-shot learned cross-correlation network according to any one of claims 1-6, characterized in that, In step S7, the query set image is assigned to the category of the feature of the nearest support set image by calculating the cosine similarity between the first relationship of all images in the support set and the second relationship of all images in the query set. The cosine similarity is calculated in the following way: in, This represents the similarity of features between the query set image and the nth support set image. For cosine similarity, It is a scalar and represents the temperature factor. n Indicates 1 to N The number between Indicates support set features, Indicates the characteristics of the query set. N Indicates the number of images supported in the set.