Transductional small sample image classification method combining feature fusion and feature matching

Abstract

The application discloses a kind of transduction formula small sample image classification methods of combining feature fusion and feature matching, comprising the following steps: an image dataset is divided into training set, test set and verification set;The training set is split into support set and query set, the image of former is with class label, the image of latter is without class label, there is no overlapping image between the two, to form a small sample image classification task based on the two sets;An initial model of transduction formula small sample image classification method TICFFM combining feature fusion and feature matching is constructed using the training set;After the initial model of TICFFM is adjusted using the training set and verification set Parameter, the optimized model of TICFFM is obtained, and the classification accuracy of the optimized model is evaluated using the test set;The image in the image dataset or real world is classified using the obtained optimized model of TICFFM.The method provided by the application has high classification accuracy and wide application prospect in many fields.

Description

Technical Field

[0001] This invention belongs to the field of few-shot image classification technology, and relates to a few-shot image classification method based on deep learning, especially a transductive few-shot image classification method that combines feature fusion and feature matching. Background Technology

[0002] Few-shot learning (FSL), also known as low-shot learning, is a machine learning approach designed for small datasets. It aims to learn a machine learning model capable of solving real-world problems using a limited number of training samples (see: Zhao Kailin, Jin Xiaolong, Wang Yuanzhuo. A review of few-shot learning research. Journal of Software, 2021, 32(2): 349-369. http: / / www.jos.org.cn / 1000-9825 / 6138.htm). Image classification is a classic machine learning task with a wide range of applications. In many scenarios, it is difficult to collect enough training samples to train a classification model. Therefore, using few-shot learning for image classification can address the challenge of limited training data. Image classification based on few-shot learning—also known as few-shot image classification (see: Liu Ying, Lei Yanbo, Fan Jiulun, et al. A review of image classification techniques based on few-shot learning. Acta Automatica Sinica, 2021, 47(02): 297-315. http: / / www.aas.net.cn / cn / article / doi / 10.16383 / j.aas.c190720)—is a task that uses image classification as a few-shot learning method, aiming to learn new visual concepts from a limited set of labeled images.

[0003] Deep learning, based on deep neural networks, is a next-generation machine learning technique. Due to its powerful learning and representation capabilities, deep neural networks have been widely applied in fields such as speech recognition and computer vision, achieving great success. Deep neural networks are becoming increasingly deeper, with increasingly stronger representation capabilities for image features, and they can also automatically classify images. Deep neural networks have achieved great success in the field of few-shot learning. For example, residual networks (ResNet) are a typical deep neural network (see: Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun: Deep Residual Learning for Image Recognition. CVPR 2016: 770-778. https: / / doi.org / 10.1109 / CVPR.2016.90). ResNet has many bypass branches that directly connect the input to subsequent layers, allowing those layers to directly learn the residuals. This, to some extent, solves the problems of information loss and degradation during information transmission in traditional convolutional or fully connected layers. Wide Residual Networks (WRNs) are a variant of ResNet that improves upon ResNet by increasing network width, thus enhancing network performance and training speed. The set of features extracted from an image using WRNs is called a feature map, which is used for subsequent image classification. WRN feature maps exhibit significantly better robustness than ResNet features, but the features generated by WRNs also lack content adaptability. Therefore, the core idea of ​​this invention is to first extract image features using WRNs, then enhance the feature maps by fusing the content adaptability of the features, and finally use the enhanced feature maps to calculate the similarity and classification probability between features, thereby achieving small-sample image classification and improving the accuracy of image classification.

[0004] Few-sample image classification methods or algorithms, also known as classification models, mainly involve two problems: how to better represent image features and how to learn good classification parameters. With the increasing depth of convolutional neural networks (CNNs), their ability to represent image features has become stronger, and they can also automatically classify images. CNNs have great potential in few-sample image classification tasks. The few-sample image classification process includes preparing the dataset, building a backbone network for image feature extraction, and designing a classifier. There are two ways to process few-sample image datasets: one is to perform data augmentation, which increases the size and patterns of the dataset; the other is not to process the few-sample dataset, but to let the model adapt to the data and model according to the characteristics of the data when there are only a few samples. The feature extraction process is to build a feature extraction model to adapt to the data distribution. For a model, the extracted image features can effectively represent the image, ultimately achieving better classification results. The design of a classifier depends on the effectiveness of image features and the adaptability between the classifier and image features. That is, under the premise that the image features are effective, the classifier can distinguish different categories of image features to the greatest extent (see: Liu Ying, Lei Yanbo, Fan Jiulun, et al. A review of image classification technology based on few-shot learning. Acta Automatica Sinica, 2021, 47(02): 297-315. http: / / www.aas.net.cn / cn / article / doi / 10.16383 / j.aas.c190720).

[0005] The data processing for a few-shot image classification task consists of three stages: model training, validation, and testing. An image dataset contains a certain number of images and their corresponding class labels as ground truth. Images with class labels are called labeled images, and those without are called unlabeled images. Few-shot image classification involves a training set D. train and test set D test D train and D test There is no overlap in the label space. To train and test the classifier, D... trainOr D test Split into support set D S and query set D Q ,and A few-shot image classification task is often called an N-way K-shot classification task, where N-way represents N image categories (or simply classes), and K-shot represents K training samples (images) belonging to each category. This task is based on D... S The N×K labeled training samples (images) in D will be used to... Q The task is to classify a test example (image) into one of N unique categories. More specifically, the classification task aims to classify data from a support set D. S Learning to correctly classify D Q Image classifier f:x test →y test , where x test D represents test The image set in the image set, and y test D represents test The tag set in.

[0006] Few-shot learning has two typical learning paradigms (reasoning methods): inductive and transductive. Correspondingly, few-shot image classification also has two typical methods: inductive and transductive. Because transductive few-shot image classification methods (models) can use unlabeled data (i.e., images without class labels) from the query set for evaluation, their performance is generally better than inductive few-shot image classification methods (models). In transductive few-shot image classification methods, the model can learn from some unlabeled data, thus alleviating the problem of sample scarcity to some extent.

[0007] While transductive few-shot image classification methods (models) can utilize a portion of unlabeled data to improve classification performance, most existing transductive few-shot image classification methods (models) have some shortcomings (defects or limitations) in feature extraction and feature enhancement. Below are some state-of-the-art (SOTA) and representative examples of classification methods (models):

[0008] The EPNet model (see: Pau Rodríguez, Issam H. Laradji, Alexandre Drouin, Alexandre Lacoste: Embedding Propagation: Smoother Manifold for Few-Shot Classification. ECCV(26)2020:121-138. https: / / doi.org / 10.1007 / 978-3-030-58574-7_8) uses a wide residual network (WRN) to advance image features and uses embedding propagation as an unsupervised non-parametric regularizer for manifold smoothing of few-shot classification. However, this model only uses a single embedding of the image for manifold smoothing without further improving the representativeness of the features. This results in insufficient representativeness of the generated feature embeddings and the low-dimensional features cannot have sufficient smoothness, thereby reducing the generalization performance of the model and ultimately affecting the accuracy of image classification.

[0009] The iLPC model (see: Michalis Lazarou, Tania Stathaki, Yannis Avrithis: Iterative label cleaning for transductive and semi-supervised few-shot learning. ICCV 2021: 8731-8740. https: / / doi.org / 10.1109 / ICCV48922.2021.00863) uses a Wide Residual Network (WRN) to pre-train image features and leverages the manifold structure of labeled and unlabeled data distributions to predict pseudo-labels. It balances between classes and uses the loss value distribution of a finite-capacity classifier to select the cleanest label, iteratively improving the quality of pseudo-labels. However, this model uses a pre-trained network and does not utilize local features to further improve the feature representation capabilities of the support and test sets. This results in the graph network being unable to effectively propagate labels, thus failing to improve the similarity between images of the same class and ultimately affecting the accuracy of image classification.

[0010] The PEME model (see: Yuqing Hu, Stéphane Pateux, Vincent Gripon: Squeezing Backbone Feature Distributions to the Max for Efficient Few-Shot Learning. Algorithms 15(5): 147(2022). https: / / doi.org / 10.3390 / a15050147) uses a Wide Residual Network (WRN) to pre-process image features and make the feature vectors closer to a Gaussian distribution. Then, it uses the optimal transfer algorithm to calculate the similarity between query features and support features. It only uses the classic CNN model for feature extraction, ignoring the content adaptability of features and the similarity between local features. This makes the model unable to effectively model the dependencies between features, and it is easily interfered with by redundant features when calculating the similarity between query features and support features, thus affecting the improvement of image classification accuracy.

[0011] The EASE model (see: Hao Zhu, Piotr Koniusz: EASE: Unsupervised Discriminant Subspace Learning for Transductive Few-Shot Learning. CVPR 2022: 9068-9078. https: / / doi.org / 10.1109 / CVPR52688.2022.00887) uses a Wide Residual Network (WRN) to pre-define image features and learns a linear projection on a subspace constructed from features of the support set and the unlabeled query set at test time to improve the performance of transductive few-shot learning. This model indirectly uses local features (i.e., features extracted from local regions of the image) and also ignores the content adaptability of features and the similarity between local features, limiting the role of the discriminant subspace and thus affecting the improvement of the model's classification accuracy.

[0012] In summary, the common shortcomings of the aforementioned state-of-the-art (SOTA) and representative transductive few-sample image classification models / methods (i.e., existing technologies) are as follows: the image features extracted using the backbone network (e.g., WRN) lack content adaptability, and they do not make reasonable use of the similarity between local features to calculate the similarity between features and the classification probability, thus limiting the improvement of image classification performance.

[0013] To overcome the shortcomings of existing technologies in this field, this invention proposes to use deep neural network technology to construct a transductive few-sample image classification method (model) that combines feature fusion and feature matching. After extracting image features using a wide residual network (WRN), this method further utilizes feature fusion technology to enhance the feature map by combining the content adaptability of image features, and uses feature matching technology to optimize the calculation of similarity between features and classification probability, thereby improving the accuracy of few-sample image classification.

[0014] The following is a brief description of several prior art technologies used in the technical solution of the present invention. These prior art technologies are only a few individual technical features of the technical solution of the present invention. They constitute the technical solution of the present invention only after being organically combined with other technical features of the present invention in a non-obvious manner.

[0015] Pixel Adaptive Convolutional Neural Networks (PAC) (see: Hang Su, Vamn Jampani, Deqing Sun, Orazio Gallo, Erik G. Learned-Miller, Jan Kautz: Pixel-Adaptive Convolutional Neural Networks. CVPR 2019: 11166-11175. https: / / doi.org / 10.1109 / CVPR.2019.01142) is a simple and effective improvement to standard convolution, in which filter weights are multiplied by a spatially varying kernel, and the varying kernel depends on learnable, local pixel features. Essentially, PAC extends the standard convolutional operation in CNNs by employing the idea of ​​bilateral filtering (see: Carlo Tomasi, Roberto Manduchi: Bilateral Filtering for Gray and Color Images. ICCV 1998: 839-846. https: / / doi.org / 10.1109 / ICCV.1998.710815), making features content-adaptive and improving the interpretability of feature extraction.

[0016] Neighbourhood Consensus Networks (NCNet) (see: Ignacio Rocco, Mircea Cimpoi, Relja Arandjelovic, Akihiko Torii, Tomás Pajdla, Josef Sivic: Neighbourhood Consensus Networks. NeurIPS 2018: 1658-1669. http: / / papers.nips.cc / paper / 7437-neighbourhood-consensus-networks) is a method that learns neighborhood consistency constraints directly from training data. This method generates a matching score tensor to evaluate the similarity between local features of two images. NCNet aims to solve the image matching problem, which refers to identifying corresponding points between two or more images using a specific matching algorithm. NCNet belongs to the feature matching method within image matching. Feature matching solves matching problems based on feature points. The points to be matched are generally represented by coordinates in the image's pixel space. Thus, the image matching problem is transformed into extracting corresponding features from two images and pairing them. Specifically, NCNet proposes an end-to-end trainable convolutional neural network structure that obtains matching pairs by analyzing the neighborhood consistency of all possible corresponding points between a pair of images in four-dimensional space, without requiring global geometric model matching. Sparse Neighbourhood Consensus Networks (Sparse-NCNet) (see: Ignacio Rocco, Relja Arandjelovic, Josef Sivic: Efficient Neighbourhood Consensus Networks via Submanifold Sparse Convolutions. ECCV(9)2020:605-621. https: / / doi.org / 10.1007 / 978-3-030-58545-7_35) overcomes the performance limitations of the original NCNet. This network shows better results on challenging feature matching problems, making the model more practical and widely applicable.

[0017] PEME (Power, Euclidean normalization, Mean subtraction, Euclidean normalization) is a module for feature preprocessing (see: Yuqing Hu, Stéphane Pateux, Vincent Gripon: Squeezing Backbone Feature Distributions to the Max for Efficient Few-Shot Learning. Algorithms 15(5):147(2022). https: / / doi.org / 10.3390 / a15050147). This module uses power transform, Euclidean normalization, mean subtraction, and another Euclidean normalization in sequence to better align the image feature distribution with the Gaussian distribution.

[0018] Mean-square error (MSE) is a measure of the difference between the estimator and the estimated quantity. Let t be an estimator of the population parameter θ determined from the sample, and (θ-t) 2 The mathematical expectation of the estimator t is called the mean square error, which is equal to σ. 2 +b 2 , where σ 2 b and t represent the variance and bias of t, respectively.

[0019] Binary cross-entropy loss is a loss function that measures the accuracy of a model's predictions by calculating the difference between the true and predicted labels. In binary classification problems, the average cross-entropy loss for all samples is typically calculated using the following formula:

[0020]

[0021] In the formula, N is the sample size, and t j p represents the true class value of the j-th sample. j This represents the predicted probability of the corresponding sample.

[0022] Cosine similarity, also known as cosine similarity, measures the similarity between two vectors by calculating the cosine of the angle between them.

[0023] The Adam optimizer (see: Diederik P. Kingma, Jimmy Ba: Adam: A Method for Stochastic Optimization. 3rd International Conference on Learning Representations, ICLR 2015, May 7-9, 2015, Conference Track Proceedings. https: / / arxiv.org / abs / 1412.6980) is used to optimize the parameters of few-shot image classification models.

[0024] Global average pooling (GAP) is a pooling operation used to replace the traditional fully connected layers in CNNs. GAP performs an average operation on each channel separately, resulting in a single average value. Therefore, the number of output channels remains the same, but each channel contains only one element.

[0025] Prototype probability calculation is a prototype-based probability calculation method in prototypical networks (see: Jake Snell, Kevin Swersky, Richard S. Zemel: Prototypical Networks for Few-shot Learning. NIPS 2017: 4077-4087. http: / / papers.nips.cc / paper / 6996-prototypical-networks-for-few-shot-learning).

[0026] Softmax is an activation function, or normalized exponential function, that can "compress" a K-dimensional vector containing arbitrary real numbers into another K-dimensional real vector, such that the value of each component (element) is in the open interval (0, 1), and the sum of all components (elements) is 1.

[0027] Back propagation (BP) is an algorithm for gradient descent in multi-layer artificial neural networks. It uses the chain rule to calculate the gradient of the loss function with the weights of each layer of the network as variables, and then updates the weights to minimize the loss function. It is a common method for training artificial neural networks.

[0028] Stochastic Gradient Descent (SGD), also known as incremental gradient descent, is an iterative method for optimizing differentiable objective functions. This method iteratively updates the weights and biases by computing the gradient of the loss function on mini-batch data.

[0029] Grid search is a model hyperparameter optimization technique, essentially an exhaustive search method, commonly used to optimize various types of hyperparameters (hyperparameters refer to parameters that need to be pre-optimized rather than obtained through training). First, a small finite set is set for each hyperparameter to explore. Then, several sets of hyperparameter values ​​are obtained through the Cartesian product of these finite sets. Grid search uses each set of hyperparameter values ​​to train the model, selecting the hyperparameter value with the smallest error on the validation set as the best hyperparameter value.

[0030] When evaluating the classification accuracy of the few-shot image classification method (model) in this invention, the performance evaluation method commonly used in the relevant technical field is usually called N-way K-shot, where N∈{5, 10, 15, 20} and K∈{1, 5}, where N represents the number of categories and K represents the number of samples in each category. In practice, 5-way 1-shot and 5-way 5-shot are commonly used for evaluation. The predicted category is determined based on the prediction results, and the accuracy of the predicted category matching the actual category is the evaluation index value (see: Zhao Kailin, Jin Xiaolong, Wang Yuanzhuo. A review of few-shot learning research. Journal of Software, 2021, 32(02): 349-369. http: / / www.jos.org.cn / 1000-9825 / 6138.htm).

[0031] Similar to traditional classification tasks, precision is used for evaluating a single class, and this metric is defined by the following formula:

[0032]

[0033] In the formula, TP is called the ground truth, which represents the number of true values ​​(ground truth) in the classification result; FP is called the false positive, which represents the number of false values ​​in the classification result.

[0034] In summary, to overcome the shortcomings of existing technologies—namely, deep neural network-based transductive few-shot image classification methods / models that use backbone networks (e.g., WRN) to extract image features, resulting in features lacking content adaptability and failing to effectively utilize local features to calculate feature similarity and classification probabilities, thus limiting image classification performance—this invention creatively utilizes feature fusion technology to enhance feature maps and employs a sparse neighborhood consensus network for feature matching, optimizing the calculation of feature similarity and classification probabilities, thereby improving the accuracy of few-shot image classification. In short, this invention provides a transductive few-shot image classification method (TICFFM) that combines feature fusion and feature matching. Summary of the Invention

[0035] The technical problem to be solved by the present invention is to provide a transductive few-sample image classification method that combines feature fusion and feature matching and has high classification accuracy. It can use feature fusion technology to enhance feature maps and use sparse neighborhood consensus networks for feature matching to optimize the calculation of similarity between features and classification probability.

[0036] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0037] This invention provides a transductive few-sample image classification method that combines feature fusion and feature matching, comprising the following steps:

[0038] Step S1: Divide an image dataset into a training set, a test set, and a validation set;

[0039] Step S2: Split the training set into a support set and a query set. Images in the support set have class labels, while images in the query set do not. There are no overlapping images between the two sets. A small sample image classification task is formed based on these two sets.

[0040] Step S3: Construct an initial model of the transductive few-shot image classification method TICFFM, which combines feature fusion and feature matching, using the training set;

[0041] Step S4: After adjusting the parameters of the initial TICFFM model using the training set and validation set, the optimized TICFFM model is obtained, and the classification accuracy of the optimized model is evaluated using the test set.

[0042] Step S5: Use the obtained optimized TICFFM model to classify images in the image dataset or in the real world.

[0043] In this method, step S3 further includes:

[0044] Step S31: Construct the feature extraction layer in the initial model of the few-sample image classification method TICFFM, and use this layer to generate the initial support feature map and the initial query feature map;

[0045] Step S32: Construct the feature fusion layer in the initial model of the few-sample image classification method TICFFM, and use this layer to generate prototype feature maps and adaptive feature maps;

[0046] Step S33: Construct the feature matching layer in the initial model of the few-sample image classification method TICFFM, and use this layer to generate the matching score tensor and class prototype;

[0047] Step S34: Construct the probability prediction layer in the initial model of the few-sample image classification method TICFFM, and use this layer to generate a classification probability vector for all images in the image dataset, where each component represents the probability that an image belongs to the corresponding true category.

[0048] In this method, step S31 further includes:

[0049] Step S311: Train a wide residual network using the training set. This network extracts initial support feature maps and initial query feature maps from the images in the support set and the query set, respectively.

[0050] In this method, step S32 further includes:

[0051] Step S321: Use an H-layer pixel adaptive convolutional neural network to convert the initial support feature map and the initial query feature map into corresponding initial adaptive feature maps respectively;

[0052] Step S322: Use the PEME module to convert the initial support feature map and the initial query feature map into corresponding prototype feature maps, and convert the initial adaptive feature map into an adaptive feature map;

[0053] Step S323: Use global average pooling to convert the adaptive feature map into an adaptive prototype, and convert the prototype feature map into a feature prototype;

[0054] Step S324: Calculate the mean squared error of the adaptive prototype and the feature prototype to generate a similarity loss;

[0055] Step S325: The prototype feature map and the adaptive feature map are concatenated according to channels to form the final support feature map and the final query feature map.

[0056] In this method, step S33 further includes:

[0057] Step S331: Utilize a sparse neighborhood consensus network to transform the final support feature map and the final query feature map into corresponding matching score tensors, which are derived from the matching score tensors of the final support feature map. The matching score tensor of the final query feature map The structure consists of the following components: For an N-way K-shot classification task, N represents the number of image categories in the support set, K represents the number of training samples belonging to each image category, m = 1, 2, ..., K, K+1, ..., N×K, i and j are the x and y coordinates of the final support feature map, i = 1, 2, ..., h, j = 1, 2, ..., w, where h and w represent the height and width of the final support feature map, respectively; k and l are the x and y coordinates of the final query feature map, k = 1, 2, ..., h, l = 1, 2, ..., w, where h and w represent the height and width of the final query feature map, respectively.

[0058] Step S332: Use global average pooling to convert the final support feature map and the final query feature map into corresponding class prototypes.

[0059] In this method, step S34 further includes:

[0060] Step S341: Calculate the matching score tensor using the matching probability prediction module to generate a matching probability vector p for all images in the query set. match The calculation process is defined by the following formula:

[0061]

[0062]

[0063]

[0064] In the formula, y = c|query indicates that the category of the image in the query is c, and the meanings of the other variable symbols m, N, K, i, j, k, l, h, w are the same as those in step S331;

[0065] Step S342: Calculate the prototype probability of the class prototype using the prototype probability prediction module to generate a prototype probability vector p for all images in the query set. prototype ;

[0066] Step S343: Convert the matching probability vector p match and the prototype probability vector p prototype A weighted summation is performed, and the result is further normalized using a softmax function to generate a classification probability vector for all images in the query set. Its calculation process is defined by the following formula:

[0067]

[0068] In the formula, α is a weighting parameter used to balance the importance of the two probabilities, and its value is determined through parameter learning. Each component represents the classification probability of the corresponding image in the query set;

[0069] Step S344: Calculate the classification probability vector using the binary cross-entropy loss function to generate a classification loss. Then, add this classification loss to the similarity loss generated in step S324 to form the final loss. Use this to train the wide residual network through backpropagation.

[0070] In this method, step S4 further includes:

[0071] Step S41: Use the Adam optimizer on the training set to learn the parameters of the initial or tuned TICFFM model, and calculate the classification accuracy of the model on the validation set, specifically including:

[0072] First, the following two hyperparameters are set for the model using the grid search method: learning rate η and number of layers H in the pixel adaptive convolutional neural network. The search range of these hyperparameters is as follows: the range of η is the set {0.001, 0.005, 0.01, 0.1}, and the range of H is the set {1, 2, 3, 4, 5, 6}.

[0073] Then, the Adam optimizer is used on the training set to learn the parameter values ​​of the model, resulting in the TICFFM hyperparameter-tuned model.

[0074] Finally, the classification accuracy of the model is calculated on the validation set;

[0075] Step S42: Determine whether the classification accuracy of the TICFFM model after parameter tuning has reached a predetermined threshold. If the classification accuracy of the model has not yet reached the predetermined threshold, continue to adjust the parameters by repeating steps S41 and S42 in sequence. Otherwise, end the parameter adjustment process and obtain the optimized TICFFM model.

[0076] Step S43: Use the test set and accuracy performance evaluation metrics to evaluate the classification accuracy of the obtained TICFFM optimized model.

[0077] The beneficial technical effects of this invention mainly include three aspects: (1) This invention provides a small sample image classification method with high classification accuracy, which has certain scientific significance for the development of this technical field; (2) The technical solution of this invention has higher classification accuracy than the most advanced and representative transductive small sample image classification method or model. The reason is that the method of this invention uses feature fusion technology to enhance the feature map by combining the content adaptability of image features, and uses feature matching technology to optimize the calculation of similarity and classification probability between features; (3) The method of this invention has broad application prospects in many fields such as medicine, face recognition, video, transportation, and image retrieval.

[0078] The specific embodiments of the present invention will now be further described with reference to the accompanying drawings. Additional aspects and advantages of the invention will be set forth in part in the description which follows, and will be obvious from the description or may be learned by practice of the invention. Attached Figure Description

[0079] Figure 1 A flowchart illustrating the transductive few-sample image classification method combining feature fusion and feature matching according to the technical solution of the present invention is shown.

[0080] Figure 2 A model framework diagram of a transductive few-sample image classification method combining feature fusion and feature matching according to the technical solution of the present invention is shown.

[0081] Figure 3 A flowchart illustrating the construction of the initial model in the transductive few-sample image classification method combining feature fusion and feature matching according to the present invention is shown. Detailed Implementation

[0082] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar concepts, objects, elements, etc., or concepts, objects, elements, etc., having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0083] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains and in related fields. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0084] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0085] This invention provides a transductive few-sample image classification method that combines feature fusion and feature matching, such as... Figure 1 As shown, it includes the following steps S1 to S5:

[0086] Step S1: Divide an image dataset into a training set, a test set, and a validation set.

[0087] Step S2: Split the training set into a support set and a query set. The images in the support set have class labels, while the images in the query set do not. There are no overlapping images between the two sets. A small sample image classification task is formed based on these two sets.

[0088] Step S3: Construct an initial model of the transductive few-shot image classification method TICFFM, which combines feature fusion and feature matching, using the training set; as shown below. Figure 2 and Figure 3 As shown, step S3 further includes:

[0089] Step S31: Construct the feature extraction layer in the initial model of the few-sample image classification method TICFFM, and use this layer to generate the initial support feature map and the initial query feature map; Step S31 further includes:

[0090] Step S311: Train a wide residual network using the training set. This network extracts initial support feature maps and initial query feature maps from the images in the support set and the query set, respectively.

[0091] Step S32: Construct the feature fusion layer in the initial model of the few-sample image classification method TICFFM, and use this layer to generate prototype feature maps and adaptive feature maps; Step S32 further includes:

[0092] Step S321: Use an H-layer pixel adaptive convolutional neural network to convert the initial support feature map and the initial query feature map into corresponding initial adaptive feature maps respectively;

[0093] Step S322: Use the PEME module to convert the initial support feature map and the initial query feature map into corresponding prototype feature maps, and convert the initial adaptive feature map into an adaptive feature map;

[0094] Step S323: Use global average pooling to convert the adaptive feature map into an adaptive prototype, and convert the prototype feature map into a feature prototype;

[0095] Step S324: Calculate the mean squared error of the adaptive prototype and the feature prototype to generate a similarity loss;

[0096] Step S325: The prototype feature map and the adaptive feature map are concatenated according to channels to form the final support feature map and the final query feature map.

[0097] Step S33: Construct the feature matching layer in the initial model of the few-sample image classification method TICFFM, and use this layer to generate a matching score tensor and class prototypes; Step S33 further includes:

[0098] Step S331: Utilize a sparse neighborhood consensus network to transform the final support feature map and the final query feature map into corresponding matching score tensors, which are derived from the matching score tensors of the final support feature map. The matching score tensor of the final query feature map The structure consists of the following components: For an N-way K-shot classification task, N represents the number of image categories in the support set, K represents the number of training samples belonging to each image category, m = 1, 2, ..., K, K+1, ..., N×K, i and j are the x and y coordinates of the final support feature map, i = 1, 2, ..., h, j = 1, 2, ..., w, where h and w represent the height and width of the final support feature map, respectively; k and l are the x and y coordinates of the final query feature map, k = 1, 2, ..., h, l = 1, 2, ..., w, where h and w represent the height and width of the final query feature map, respectively.

[0099] Step S332: Use global average pooling to convert the final support feature map and the final query feature map into corresponding class prototypes.

[0100] Step S34: Construct the probability prediction layer in the initial model of the few-sample image classification method TICFFM, and use this layer to generate a classification probability vector for all images in the image dataset, where each component represents the probability that an image belongs to the corresponding true category; Step S34 further includes:

[0101] Step S341: Calculate the matching score tensor using the matching probability prediction module to generate a matching probability vector p for all images in the query set. match The calculation process is defined by the following formula:

[0102]

[0103]

[0104]

[0105] In the formula, y = c|query indicates that the category of the image in the query is c, and the meanings of the other variable symbols m, N, K, i, j, k, l, h, w are the same as those in step S331;

[0106] Step S342: Calculate the prototype probability of the class prototype using the prototype probability prediction module to generate a prototype probability vector p for all images in the query set. prototype ;

[0107] Step S343: Convert the matching probability vector p match and the prototype probability vector p prototype A weighted summation is performed, and the result is further normalized using a softmax function to generate a classification probability vector for all images in the query set. Its calculation process is defined by the following formula:

[0108]

[0109] In the formula, α is a weighting parameter used to balance the importance of the two probabilities, and its value is determined through parameter learning. Each component represents the classification probability of the corresponding image in the query set;

[0110] Step S344: Calculate the classification probability vector using the binary cross-entropy loss function to generate a classification loss. Then, add this classification loss to the similarity loss generated in step S324 to form the final loss. Use this to train the wide residual network through backpropagation.

[0111] Step S4: After tuning the parameters of the initial TICFFM model using the training and validation sets, an optimized TICFFM model is obtained, and the classification accuracy of the optimized model is evaluated using the test set; Step S4 further includes:

[0112] Step S41: Use the Adam optimizer on the training set to learn the parameters of the initial or tuned TICFFM model, and calculate the classification accuracy of the model on the validation set, specifically including:

[0113] First, the following two hyperparameters are set for the model using the grid search method: learning rate η and number of layers H in the pixel adaptive convolutional neural network. The search range of these hyperparameters is as follows: the range of η is the set {0.001, 0.005, 0.01, 0.1}, and the range of H is the set {1, 2, 3, 4, 5, 6}.

[0114] Then, the Adam optimizer is used on the training set to learn the parameter values ​​of the model, resulting in the TICFFM hyperparameter-tuned model.

[0115] Finally, the classification accuracy of the model is calculated on the validation set;

[0116] Step S42: Determine whether the classification accuracy of the TICFFM model after parameter tuning has reached a predetermined threshold. If the classification accuracy of the model has not yet reached the predetermined threshold, continue to adjust the parameters by repeating steps S41 and S42 in sequence. Otherwise, end the parameter adjustment process and obtain the optimized TICFFM model.

[0117] Step S43: Use the test set and accuracy performance evaluation metrics to evaluate the classification accuracy of the obtained TICFFM optimized model.

[0118] Step S5: Use the obtained optimized TICFFM model to classify images in the image dataset or in the real world.

[0119] The following specific embodiment further illustrates the implementation of the technical solution of the present invention, and the performance of the technical solution of the present invention (i.e., the TICFFM model) is compared with that of four typical transduction-based few-shot image classification models (i.e., EPNet model, iLPC model, PEME model, and EASE model) in terms of classification accuracy through experiments. Specifically, two performance evaluation methods are used: 5-way 1-shot and 5-way 5-shot.

[0120] EPNet model, see: Pau Rodriguez, Issam H.Laradji, Alexandre Drouin, Alexandre Lacoste: Embedding Propagation: Smoother Manifold for Few-Shot Classification. ECCV (26) 2020: 121-138. https: / / doi.org / 10.1007 / 978-3-030-58574-7_8;

[0121] iLPC model, see: Michalis Lazarou, Tania Stathaki, Yannis Avrithis: Iterative label cleaning for transductive and semi-supervised few-shot learning. ICCV 2021: 8731-8740. https: / / doi.org / 10.1109 / ICCV48922.2021.00863;

[0122] PEME model, see: Yuqing Hu, Stéphane Pateux, Vincent Gripon: Squeezing Backbone Feature Distributions to the Max for Efficient Few-Shot Learning. Algorithms 15(5): 147 (2022). https: / / doi.org / 10.3390 / a15050147;

[0123] EASE model, see: Hao Zhu, Piotr Koniusz: EASE: Unsupervised Discriminant Subspace Learning for Transductive Few-Shot Learning. CVPR 2022: 9068-9078. https: / / doi.org / 10.1109 / CVPR52688.2022.00887.

[0124] The specific implementation uses the miniImageNet dataset, a benchmark dataset in the field of few-shot learning, which contains images of nature (see: Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, Daan Wierstra: Matching Networks for One Shot Learning. NIPS 2016: 3630-3638. http: / / papers.nips.cc / paper / 6385-matching-networksfor-one-shot-leaming). The miniImageNet dataset is excerpted from the ImageNet dataset, which is a very well-known large-scale visual dataset. miniImageNet consists of 60,000 images evenly distributed across 100 classes. These classes are divided into 64, 16, and 20 classes for training, validation, and testing, respectively. The input samples are resized to 84×84 pixels and used as the backbone of the WRN network. First, we construct a set of N-way K-shot classification tasks. Taking 5-way 1-shot as an example, for each task, we first sample 5 classes in the training set, and then sample 1 and 15 images from each class to form the support set and query set, respectively. Each training iteration uses 64 batches of tasks. Table 1 shows a small portion of the data from the miniImageNet dataset.

[0125] Table 1. A small portion of data from the miniImageNet dataset

[0126] n0153282900000005 bird 27.97kB 500×367 n0153282900000006 bird 80.88kB 500×388 n0153282900000007 bird 74.16kB 500×375 ... ... ... ... n0208986700000003 dog 30.80kB 250×280 n0208986700000005 dog 192.48kB 1024×768 n0208986700000009 dog 40.04kB 333×500 n0208986700000011 dog 70.02kB 333×500 ... ... ... ... n0191074700000003 jellyfish 23.53kB 400×300 n0191074700000004 jellyfish 16.59kB 500×375 n0191074700000005 jellyfish 42.97kB 500×333

[0127] Furthermore, in order to compare the classification accuracy of the technical solution of this invention (i.e., the TICFFM model) with four typical transduction-based few-shot image classification models (i.e., the EPNet model, the iLPC model, the PEME model, and the EASE model) using the aforementioned miniImageNet dataset, the inventors implemented the above method / model in Python using PyCharm as the software development platform. The specific implementation technology is as follows:

[0128] The TICFFM model is implemented using Python 3.7.0 and PyTorch 1.9.0. The network structure of the TICFFM model is implemented as follows: For the feature fusion layer, code was downloaded from GitHub (see: https: / / github.com / NVlabs / pacnet) and modified as necessary; for the feature matching layer, code was downloaded from GitHub (see: https: / / github.com / ignacio-rocco / sparse-ncnet) and slightly modified; for the probability prediction layer, the `nn` class and the `dense` class of the `layer` module in the PyTorch deep learning framework (see: https: / / hithub.com / pytorch / pytorch) are used.

[0129] The EPNet model is implemented using Python 3.7.0 and PyTorch 1.6.0. The EPNet code is available in the GitHub repository embedding-propagation (see: https: / / github.com / ServiceNow / embedding-propagation).

[0130] The iLPC model is implemented using Python 3.7.0 and PyTorch 1.6.0. The iLPC code is available in the GitHub repository ilpc (see: https: / / github.com / michalislazarou / ilpc).

[0131] The PEME model is implemented using Python 3.7.0 and PyTorch 1.6.0. The PEME code is published in the GitHub repository BMS (see: https: / / hithub.com / yhu01 / BMS).

[0132] The EASE model is implemented using Python 3.7.0 and PyTorch 1.9.0. The EASE model code is published in the GitHub repository EASE (see: https: / / github.com / allenhaozhu / EASE).

[0133] The software operating environment of the experiment: (1) Hardware platform: Intel(R)Xeon(R)Silver 4116 CPU@2.10GHz×48, 2 processors, each processor has 12 cores, memory 96GB, GPU model is GEFORCE RTX 2080Ti; (2) Operating system: Ubuntu 18.04.5 LTS; (3) Software platform: PyCharm.

[0134] The experimental procedure is as follows:

[0135] Experimental process of TICFFM model: (1) Divide the miniImageNet dataset into training set, test set and validation set according to category; (2) Split the training set into support set and query set. The images in the former have class labels and the images in the latter do not have class labels. There are no overlapping images between the two sets. A small sample image classification task is formed based on these two sets; (3) Use the training set to build an initial model of TICFFM, a transductive small sample image classification method that combines feature fusion and feature matching; (4) Use the training set and validation set to adjust the parameters of the initial model of TICFFM to obtain the optimized model of TICFFM. The classification accuracy of the optimized model is evaluated using the test set. Here, the classification precision of the model is calculated.

[0136] Experimental procedure of EPNet model: (1) Set the hyperparameter values ​​according to the suggestions given in the original publication of EPNet model (see: Pau Rodríguez, Issam H. Laradji, Alexandre Drouin, Alexandre Lacoste: Embedding Propagation: Smoother Manifold for Few-Shot Classification. ECCV(26)2020:121-138. https: / / doi.org / 10.1007 / 978-3-030-58574-7_8); (2) Calculate the classification accuracy of EPNet model on the test set as the classification accuracy of the model.

[0137] Experimental procedure for the iLPC model: (1) Set the hyperparameter values ​​according to the recommendations given in the original publication of the iLPC model (see: Michalis Lazarou, Tania Stathaki, Yannis Avrithis: Iterative label cleaning for transductive and semi-supervised few-shot learning. ICCV 2021: 8731-8740. https: / / doi.org / 10.1109 / ICCV48922.2021.00863); (2) Calculate the classification accuracy of the iLPC model on the test set as the classification accuracy of the model.

[0138] Experimental procedure of PEME model: (1) Set the hyperparameter values ​​according to the suggestions given in the original publication of PEME model (see: Yuqing Hu, Stéphane Pateux, Vincent Gripon: Squeezing Backbone Feature Distributions to the Max for Efficient Few-Shot Learning. Algorithms 15(5):147(2022). https: / / doi.org / 10.3390 / a15050147); (2) Calculate the classification accuracy of PEME model on the test set as the classification accuracy of the model.

[0139] Experimental procedure for the EASE model: (1) Set the hyperparameter values ​​according to the recommendations given in the original publication of the EASE model (see: Hao Zhu, Piotr Koniusz: EASE: Unsupervised Discriminant Subspace Learning for Transductive Few-Shot Learning. CVPR 2022: 9068-9078. https: / / doi.org / 10.1109 / CVPR52688.2022.00887); (2) Calculate the classification accuracy of the EASE model on the test set as the classification accuracy of the model.

[0140] The specific implementation of the technical solution of the present invention, for the specific embodiments described above, is as follows:

[0141] like Figure 1 As shown, a transductive few-sample image classification method combining feature fusion and feature matching includes the following steps S1 to S5:

[0142] Step S1: Divide an image dataset into a training set, a test set, and a validation set; Step S2: Divide the training set into a support set and a query set. The images in the former have class labels, while the images in the latter do not. There are no overlapping images between the two sets. A small sample image classification task is formed based on these two sets; In this embodiment: The image dataset is miniImageNet; The dataset consists of 60,000 images evenly distributed in 100 categories; According to the common practice in this technical field, the image dataset is first divided into a training set and a test set in a ratio of 8:2, and then 25% of the training set is used as the validation set; Two points need to be further explained: (1) The dataset is divided according to image categories. The number of categories in the training set, test set, and validation set are 64, 16, and 20, respectively. The number of categories in each category is not the same. When constructing the small sample task, the images in each category are randomly obtained; (2) According to the common practice in this technical field, an image dataset can also be divided into a training set, a test set, and a validation set in other proportions. Data augmentation is performed in three ways: (1) the given image is randomly cropped into different sizes and aspect ratios, and then the cropped image is scaled to the specified size, which is 84×84 pixels; (2) the given image is randomly rotated horizontally with a given probability, which is 0.5 by default; (3) the image is standardized channel by channel (mean becomes 0 and standard deviation becomes 1), which can speed up the convergence of the model training iteration process.

[0143] Step S3: Construct an initial model of the transductive few-shot image classification method TICFFM, which combines feature fusion and feature matching, using the training set; as shown below. Figure 2 and Figure 3 As shown, step S3 further includes:

[0144] Step S31: Construct the feature extraction layer in the initial model of the few-sample image classification method TICFFM, and use this layer to generate the initial support feature map and the initial query feature map; Step S31 further includes:

[0145] Step S311: Train a wide residual network using the training set. This network extracts initial support feature maps and initial query feature maps from the images in the support set and the query set, respectively.

[0146] Step S32: Construct the feature fusion layer in the initial model of the few-sample image classification method TICFFM, and use this layer to generate prototype feature maps and adaptive feature maps; Step S32 further includes:

[0147] Step S321: Use an H-layer pixel adaptive convolutional neural network to convert the initial support feature map and the initial query feature map into corresponding initial adaptive feature maps; in this embodiment: the value of H is 3;

[0148] Step S322: Use the PEME module to convert the initial support feature map and the initial query feature map into corresponding prototype feature maps, and convert the initial adaptive feature map into an adaptive feature map;

[0149] Step S323: Use global average pooling to convert the adaptive feature map into an adaptive prototype, and convert the prototype feature map into a feature prototype;

[0150] Step S324: Calculate the mean squared error of the adaptive prototype and the feature prototype to generate a similarity loss;

[0151] Step S325: The prototype feature map and the adaptive feature map are concatenated according to channels to form the final support feature map and the final query feature map.

[0152] Step S33: Construct the feature matching layer in the initial model of the few-sample image classification method TICFFM, and use this layer to generate a matching score tensor and class prototypes; Step S33 further includes:

[0153] Step S331: Utilize a sparse neighborhood consensus network to transform the final support feature map and the final query feature map into corresponding matching score tensors, which are derived from the matching score tensors of the final support feature map. The matching score tensor of the final query feature map The structure consists of the following components: For an N-way K-shot classification task, N represents the number of image categories in the support set, K represents the number of training samples belonging to each image category, m = 1, 2, ..., K, K+1, ..., N×K, i and j are the x and y coordinates of the final support feature map, i = 1, 2, ..., h, j = 1, 2, ..., w, where h and w represent the height and width of the final support feature map, respectively; k and l are the x and y coordinates of the final query feature map, k = 1, 2, ..., h, l = 1, 2, ..., w, where h and w represent the height and width of the final query feature map, respectively.

[0154] Step S332: Use global average pooling to convert the final supporting feature map and the final query feature map into corresponding class prototypes;

[0155] Step S34: Construct the probability prediction layer in the initial model of the few-sample image classification method TICFFM, and use this layer to generate a classification probability vector for all images in the image dataset, where each component represents the probability that an image belongs to the corresponding true category; Step S34 further includes:

[0156] Step S341: Calculate the matching score tensor using the matching probability prediction module to generate a matching probability vector p for all images in the query set. match The calculation process is defined by the following formula:

[0157]

[0158]

[0159]

[0160] In the formula, y = c|query indicates that the category of the image in the query is c, and the meanings of the other variable symbols m, N, K, i, j, k, l, h, w are the same as those in step S331;

[0161] Step S342: Calculate the prototype probability of the class prototype using the prototype probability prediction module to generate a prototype probability vector p for all images in the query set. prototype ;

[0162] Step S343: Convert the matching probability vector p match and the prototype probability vector p prototype A weighted summation is performed, and the result is further normalized using a softmax function to generate a classification probability vector for all images in the query set. Its calculation process is defined by the following formula:

[0163]

[0164] In the formula, α is a weighting parameter used to balance the importance of the two probabilities, and its value is determined through parameter learning. Each component represents the classification probability of the corresponding image in the query set;

[0165] Step S344: Calculate the classification probability vector using the binary cross-entropy loss function to generate a classification loss. Then, add this classification loss to the similarity loss generated in step S324 to form the final loss. Use this to train the wide residual network through backpropagation.

[0166] Step S4: After tuning the parameters of the initial TICFFM model using the training and validation sets, an optimized TICFFM model is obtained, and the classification accuracy of the optimized model is evaluated using the test set; Step S4 further includes:

[0167] Step S41: Use the Adam optimizer on the training set to learn the parameters of the initial or tuned TICFFM model, and calculate the classification accuracy of the model on the validation set, specifically including:

[0168] First, the following two hyperparameters are set for the model using a grid search method: the learning rate η and the number of layers H in the pixel-adaptive convolutional neural network. The search range of these hyperparameters is as follows: the range of η is the set {0.001, 0.005, 0.01, 0.1}, and the range of H is the set {1, 2, 3, 4, 5, 6}. In this embodiment, the value of η is 0.01, and the value of H is 3.

[0169] Then, the Adam optimizer is used on the training set to learn the parameter values ​​of the model, resulting in the TICFFM hyperparameter-tuned model.

[0170] Finally, the classification accuracy of the model is calculated on the validation set;

[0171] Step S42: Determine whether the classification accuracy of the TICFFM model after parameter tuning has reached a predetermined threshold. If the classification accuracy of the model has not yet reached the predetermined threshold, continue parameter tuning by repeating steps S41 and S42 in sequence. Otherwise, end the parameter tuning process and obtain the optimized TICFFM model. In this embodiment, the classification accuracy is specifically calculated as the classification precision, and the predetermined threshold is 0.8.

[0172] Step S43: Use the test set and accuracy performance evaluation metrics to evaluate the classification accuracy of the obtained TICFFM optimized model; In this embodiment: The classification accuracy results of the TICFFM optimized model under two performance evaluation methods, 5-way 1-shot and 5-way 5-shot, are shown in Table 2. The table also lists the classification accuracy results of the EPNet model, iLPC model, PEME model and EASE model; The larger the index value in the table, the better the accuracy performance.

[0173] Table 2. Comparison of experimental results (%) of five models in small sample image classification on the miniImageNet dataset

[0174] TICFFM 85.81±0.17 91.87±0.34 EPNet 79.22±0.92 88.05±0.51 iLPC 83.05±0.79 88.82±0.42 PEME 83.35±0.25 89.53±0.13 EASE 83.00±0.21 88.92±0.13

[0175] Step S5: Classify the images in the image dataset or in the real world using the obtained TICFFM optimized model; In this embodiment: Classify the images in the test set using the obtained TICFFM optimized model.

[0176] The experimental results above show that the transductive few-shot image classification method TICFFM, which combines feature fusion and feature matching, provided by this invention has higher image classification accuracy than the current state-of-the-art and representative transductive few-shot image classification methods (EPNet model, iLPC model, PEME model, EASE model).

[0177] In general, as can be understood from the above technical solutions and specific implementations of the present invention, the beneficial technical effects of the present invention mainly include three aspects: (1) The present invention provides a small sample image classification method with high classification accuracy, which has certain scientific significance for the development of this technical field; (2) The technical solution of the present invention has higher classification accuracy than the most advanced and representative transductive small sample image classification method or model at present. The reason is that the method of the present invention uses feature fusion technology to enhance the feature map by combining the content adaptability of image features, and uses feature matching technology to optimize the calculation of similarity between features and classification probability; (3) The method of the present invention has broad application prospects in many fields such as medicine, face recognition, video, transportation, and image retrieval.

[0178] The above description is only a partial embodiment of the present invention. It should be noted that, except for the technical features described in the specification, all other technologies are known to those skilled in the art. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A transductive few-sample image classification method combining feature fusion and feature matching, comprising the following steps: Step S1: Divide an image dataset into a training set, a test set, and a validation set; Step S2: Split the training set into a support set and a query set. Images in the support set have class labels, while images in the query set do not. There are no overlapping images between the two sets. A small sample image classification task is formed based on these two sets. Step S3: Construct an initial model of the transductive few-shot image classification method TICFFM, which combines feature fusion and feature matching, using the training set; Step S3 further includes the following steps: Step S31: Construct the feature extraction layer in the initial model of the few-sample image classification method TICFFM, and use this layer to generate the initial support feature map and the initial query feature map; Step S32: Construct the feature fusion layer in the initial model of the few-sample image classification method TICFFM, and use this layer to generate prototype feature maps and adaptive feature maps; Step S33: Construct the feature matching layer in the initial model of the few-sample image classification method TICFFM, and use this layer to generate the matching score tensor and class prototype; Step S34: Construct the probability prediction layer in the initial model of the few-sample image classification method TICFFM, and use this layer to generate a classification probability vector for all images in the image dataset, where each component represents the probability that an image belongs to the corresponding true category; Step S4: After adjusting the parameters of the initial TICFFM model using the training set and validation set, the optimized TICFFM model is obtained, and the classification accuracy of the optimized model is evaluated using the test set. Step S5: Use the obtained optimized TICFFM model to classify images in the image dataset or in the real world.

2. The method according to claim 1, characterized in that, Step S31 further includes the following steps: Step S311: Train a wide residual network using the training set. This network extracts initial support feature maps and initial query feature maps from the images in the support set and the query set, respectively.

3. The method according to claim 1 or 2, characterized in that, Step S32 further includes the following steps: Step S321: Use an H-layer pixel adaptive convolutional neural network to convert the initial support feature map and the initial query feature map into corresponding initial adaptive feature maps respectively; Step S322: Use the PEME module to convert the initial support feature map and the initial query feature map into corresponding prototype feature maps, and convert the initial adaptive feature map into an adaptive feature map; Step S323: Use global average pooling to convert the adaptive feature map into an adaptive prototype, and convert the prototype feature map into a feature prototype; Step S324: Calculate the mean square error of the adaptive prototype and the feature prototype to generate a similarity loss; Step S325: The prototype feature map and the adaptive feature map are concatenated according to channels to form the final support feature map and the final query feature map.

4. The method according to claim 3, characterized in that, Step S33 further includes the following steps: Step S331: Utilize a sparse neighborhood consensus network to transform the final support feature map and the final query feature map into corresponding matching score tensors, which are derived from the matching score tensors of the final support feature map. The matching score tensor of the final query feature map The structure consists of the following components: For an N-way K-shot classification task, N represents the number of image categories in the support set, K represents the number of training samples belonging to each image category, m = 1, 2, ..., K, K+1, ..., N×K, i and j are the x and y coordinates of the final support feature map, i = 1, 2, ..., h, j = 1, 2, ..., w, where h and w represent the height and width of the final support feature map, respectively; k and l are the x and y coordinates of the final query feature map, k = 1, 2, ..., h, l = 1, 2, ..., w, where h and w represent the height and width of the final query feature map, respectively. Step S332: Use global average pooling to convert the final support feature map and the final query feature map into corresponding class prototypes.

5. The method according to claim 4, characterized in that, Step S34 further includes the following steps: Step S341: Calculate the matching score tensor using the matching probability prediction module to generate a matching probability vector p for all images in the query set. match The calculation process is defined by the following formula: In the formula, y = c|query indicates that the category of the image in the query is c, and the meanings of the other variable symbols m, N, K, i, j, k, l, h, w are the same as those in step S331; Step S342: Calculate the prototype probability of the class prototype using the prototype probability prediction module to generate a prototype probability vector for all images in the query set. Pprototype ; Step S343: Convert the matching probability vector pmatch and the prototype probability vector pprototype A weighted summation is performed, and the result is further normalized using a softmax function to generate a classification probability vector for all images in the query set. Its calculation process is defined by the following formula: In the formula, α is a weighting parameter used to balance the importance of the two probabilities, and its value is determined through parameter learning. Each component represents the classification probability of the corresponding image in the query set; Step S344: Calculate the classification probability vector using the binary cross-entropy loss function to generate a classification loss. Then, add this classification loss to the similarity loss generated in step S324 to form the final loss. Use this to train the wide residual network through backpropagation.

6. The method according to claim 5, characterized in that, Step S4 further includes the following steps: Step S41: Use the Adam optimizer on the training set to learn the parameters of the initial or tuned TICFFM model, and calculate the classification accuracy of the model on the validation set, specifically including: First, the following two hyperparameters are set for the model using the grid search method: learning rate η and number of layers H in the pixel adaptive convolutional neural network. The search range of these hyperparameters is as follows: the range of η is the set {0.001, 0.005, 0.01, 0.1}, and the range of H is the set {1, 2, 3, 4, 5, 6}. Then, the Adam optimizer is used on the training set to learn the parameter values ​​of the model, resulting in the TICFFM hyperparameter-tuned model. Finally, the classification accuracy of the model is calculated on the validation set; Step S42: Determine whether the classification accuracy of the TICFFM model after parameter tuning has reached a predetermined threshold. If the classification accuracy of the model has not yet reached the predetermined threshold, continue to adjust the parameters by repeating steps S41 and S42 in sequence. Otherwise, end the parameter adjustment process and obtain the optimized TICFFM model. Step S43: Use the test set and accuracy performance evaluation metrics to evaluate the classification accuracy of the obtained TICFFM optimized model.

Images