Few-shot image classification method, training method, apparatus, device, and storage medium

Abstract

The present application relates to the technical field of images, and discloses a few-shot image classification method, an apparatus, a device, and a computer storage medium. The method comprises: weighting, by means of a first neural network, an encoded feature map corresponding to each image in an image set to obtain a plurality of meta-prompt feature maps, and on the basis of the plurality of meta-prompt feature maps, obtaining an enhanced feature map corresponding to a query image and an enhanced feature map corresponding to each support image; then, acquiring a similarity score between each support image and the query image by means of a second neural network; and classifying the query image on the basis of the similarity score.

Description

Small sample image classification method, training method, device, equipment and storage medium Technical Field

[0001] The present application relates to the field of image technology, and in particular to a small sample image classification method, training method, device, equipment and storage medium. Background Art

[0002] Small-sample image classification methods utilize small-sample learning to classify images. Small-sample learning refers to machine learning methods that can use a small number of samples within a category, typically one or a few, to learn and classify or identify new, unseen samples. Image classification has a wide range of applications, and in many cases, it's difficult to collect sufficient data to train neural networks. Therefore, small-sample learning can be used to classify images.

[0003] A small sample image classification method first obtains a data set and divides the data set into a query set and a support set, wherein the query set includes multiple unclassified query images and the support set includes multiple classified support images. The query images and support images in the image set are then encoded to obtain an encoded feature map, and features are extracted from the encoded feature map to classify the query images according to the extracted features.

[0004] In the above methods, the accuracy of classifying query images is low.

[0005] Summary of the Invention

[0006] The present application provides a small sample image classification method, training method, apparatus, device, and storage medium. The technical solution is as follows:

[0007] According to one aspect of the present application, a small sample image classification method is provided, the method comprising:

[0008] Acquire an image set, the image set comprising a query image and a plurality of supporting images;

[0009] Inputting the image set into an image coding network to obtain a plurality of coding feature maps;

[0010] Performing weighted processing on the plurality of encoding feature maps through a first neural network to obtain a plurality of meta-prompt feature maps;

[0011] Adding and fusing the encoding feature map and the meta-prompt feature map to obtain an enhanced feature map;

[0012] Inputting the enhanced feature map into a second neural network to obtain similarity scores of the plurality of enhanced feature maps;

[0013] The query image is classified according to the similarity score.

[0014] Optionally, the image set is input into an image coding network to obtain a plurality of coding feature maps, including:

[0015] Obtaining a class code for each image in the image set, where the class code is a preset priori value;

[0016] Segmenting each image in the image set to obtain a plurality of block images corresponding to each image;

[0017] Inputting the plurality of block images into a position encoder to obtain a position code corresponding to each block image, wherein the position code is used to identify a position of the block image in a corresponding image;

[0018] The multiple block images are input into a feature encoder to obtain a block code corresponding to each block image, where the block code is a vector representing the block image, and the class code, the multiple position codes, and the multiple block codes are the coding feature map.

[0019] Optionally, the first neural network includes a first multi-layer perceptual network, a local multi-layer perceptual network, and a global multi-layer perceptual network, and the first multi-layer perceptual network is connected to the local multi-layer perceptual network and the global multi-layer perceptual network respectively;

[0020] The weighted processing of the plurality of encoding feature maps by the first neural network to obtain a plurality of meta-prompt feature maps includes:

[0021] Inputting multiple block codes corresponding to each of the encoded feature maps into the first multi-layer perception network to obtain an initial feature map;

[0022] Inputting the initial feature map into the local multi-layer perception network to obtain a local feature map;

[0023] Inputting the initial feature map into the global multi-layer perception network to obtain a global feature map;

[0024] Adding and fusing the local feature map and the global feature map to obtain a fused feature map;

[0025] Processing the fused feature map through an activation function to obtain a weight matrix;

[0026] The weight matrix and the initial feature map are multiplied and fused to obtain the meta-prompt feature map.

[0027] Optionally, inputting the plurality of block codes corresponding to each of the encoded feature maps into the first multi-layer perception network to obtain an initial feature map includes:

[0028] Performing three linear transformations on each block code in the coding feature map to obtain a query matrix, a key matrix, and a value matrix for each block code;

[0029] Multiplying and fusing the query matrix and the key matrix corresponding to each block code to obtain a first attention feature map of each block code;

[0030] The first attention feature map corresponding to each of the block codes is multiplied and fused with the value matrix to obtain a sub-initial feature map of each of the block codes, where the initial feature map includes multiple sub-initial feature maps corresponding to multiple block codes.

[0031] Optionally, the local multi-layer perception network includes:

[0032] The first convolutional layer, batch normalization layer, linear rectification function, second convolutional layer, second batch normalization layer and random activation layer are connected in sequence.

[0033] Optionally, the global multi-layer perception network includes:

[0034] The adaptive pooling layer, the first convolutional layer, the batch normalization layer, the linear rectification function, the second convolutional layer, the second batch normalization layer and the random dropout layer are connected in sequence.

[0035] Optionally, each image in the image set includes an object image and a background image, each supporting image corresponds to a category, and the weighting process is used to increase the weight of a feature representing the target object image in the coded feature map in the coded feature map;

[0036] The plurality of encoding feature maps include a first encoding feature map corresponding to the query image, and a second encoding feature map corresponding to each of the support images;

[0037] The plurality of meta-prompt feature maps include a first meta-prompt feature map corresponding to the first encoding feature map, and a second meta-prompt feature map corresponding to each of the second encoding feature maps.

[0038] Optionally, the adding and fusing the encoding feature map and the meta-prompt feature map to obtain an enhanced feature map, and inputting the enhanced feature map into a second neural network to obtain similarity scores of multiple enhanced feature maps, includes:

[0039] Adding and fusing the first encoded feature map and the first meta-hint feature map to obtain a first enhanced feature map corresponding to the query image, and adding and fusing the second encoded feature map and the second meta-hint feature map to obtain a second enhanced feature map corresponding to each of the support images;

[0040] The first enhanced feature map and each of the second enhanced feature maps in the plurality of second enhanced feature maps are input into a second neural network in pairs to obtain a similarity score between each of the plurality of second enhanced feature maps and the first enhanced feature map, wherein the number of the similarity scores is the same as the number of the plurality of supporting images, and the second neural network includes a plurality of neurons, the degree of influence of the features in the first enhanced feature map on the neurons is positively correlated with the weight of the features, and the degree of influence of the features in the second enhanced feature map on the neurons is positively correlated with the weight of the features.

[0041] Optionally, the second neural network includes a second multi-layer perception network and a feature scaling network;

[0042] The step of inputting the first enhanced feature map and each of the plurality of second enhanced feature maps into a second neural network in pairs to obtain a similarity score between each of the plurality of second enhanced feature maps and the first enhanced feature map comprises:

[0043] Inputting the first enhanced feature map and the second enhanced feature map into the second multi-layer perception network in pairs, to obtain a first feature map corresponding to the first enhanced feature map and a second feature map obtained from the first enhanced feature map;

[0044] Inputting the first feature map and the second feature map into a feature scaling network in pairs, to obtain a third feature map corresponding to the first feature map and a fourth feature map corresponding to the second feature map;

[0045] Processing the third feature map through an attention function to obtain a first attention map value, and processing the fourth feature map through the attention function to obtain a second attention map value;

[0046] Multiplying and fusing the first attention map value with the first feature map to obtain a second attention feature map, and multiplying and fusing the second attention map value with the second feature map to obtain a third attention feature map;

[0047] Obtain a cosine similarity between the second attention feature map and the third attention feature map, where the cosine similarity is the similarity score.

[0048] Optionally, classifying the query image according to the similarity score includes:

[0049] The category of the supporting image with the highest similarity score to the query image among the at least two supporting images is used as the category of the query image.

[0050] Optionally, the image coding network is pre-trained by mask image modeling, the pre-trained network framework includes a teacher network and a student network, and the pre-training method includes:

[0051] Acquire a panoramic sample image and a mask sample image, wherein the mask sample image is an image obtained by performing mask processing on the panoramic sample image;

[0052] Inputting the mask sample image into the student network to obtain a first encoded feature map;

[0053] Inputting the panoramic image into the teacher network to obtain a second encoded feature map;

[0054] Comparing the first coding feature map and the second coding feature map to obtain a comparison difference;

[0055] If the contrast difference is greater than a preset result, adjusting the student network based on the contrast difference, and performing the step of inputting the mask sample image into the student network;

[0056] If the contrast difference is less than or equal to the preset result, the student network is determined as the image coding network.

[0057] Optionally, the pre-trained first loss function includes:

[0058] Among them, L pre is the first loss function value, Dtrain is the training set, (x, y)∈D train Indicates that the domain to which the sample image belongs is the training set, m i is a random mask, m i is 0 or 1. x is a panoramic sample image, which can be divided into n block images, and the n block images are represented as The variable j represents the jth block image in the n block image, is the teacher network output corresponding to the jth block image in the n block images, is the output of the student network corresponding to the jth block image in the n block images.

[0059] According to another aspect of the present application, a small sample image classification training method is provided, the method comprising:

[0060] Acquire a training sample set from a sample set, wherein the training sample set includes a query sample image and a plurality of support sample images;

[0061] Inputting the training sample set into the image coding network to be trained to obtain multiple coding feature maps;

[0062] Performing weighted processing on the plurality of encoding feature maps through a first neural network to be trained to obtain a plurality of meta-prompt feature maps;

[0063] Adding and fusing the encoding feature map and the meta-prompt feature map to obtain an enhanced feature map;

[0064] Inputting the strong feature map into a second neural network to be trained to obtain similarity scores of the plurality of enhanced feature maps;

[0065] Classifying the query sample image according to the similarity score to obtain a predicted category of the query sample image;

[0066] Comparing the predicted category of the query sample image with the category corresponding to the query sample image to obtain a second comparison difference;

[0067] If the second contrast difference is greater than a preset result, adjusting the image coding network to be trained, the first neural network to be trained, and the second neural network to be trained based on the second contrast difference, and inputting the training sample set into the image coding network to obtain a plurality of coding feature maps;

[0068] If the second comparison difference is less than or equal to the preset result, the image coding network to be trained, the first neural network to be trained and the second neural network to be trained are determined as the image coding network, the first neural network and the second neural network.

[0069] Optionally, the query sample image corresponds to a category, each of the support sample images corresponds to a category, the query sample image and the support sample image both include an object image and a background image, and the weighting process is used to increase the weight of a feature representing the target object image in the encoded feature map in the encoded feature map;

[0070] The plurality of encoding feature maps include a first encoding feature map corresponding to the query sample image, and a second encoding feature map corresponding to each of the support sample images;

[0071] The plurality of meta-prompt feature maps include a first meta-prompt feature map corresponding to the first encoding feature map, and a second meta-prompt feature map corresponding to each of the second encoding feature maps.

[0072] Optionally, the adding and fusing the encoding feature map and the meta-prompt feature map to obtain an enhanced feature map, and inputting the enhanced feature map into the second neural network to be trained to obtain similarity scores of multiple enhanced feature maps, includes:

[0073] Adding and fusing the first encoded feature map and the first meta-hint feature map to obtain a first enhanced feature map corresponding to the query sample image, and adding and fusing the second encoded feature map and the second meta-hint feature map to obtain a second enhanced feature map corresponding to each of the support sample images;

[0074] The first enhanced feature map and each of the multiple second enhanced feature maps are input into the second neural network to be trained in pairs to obtain a similarity score between each of the multiple second enhanced feature maps and the first enhanced feature map, where the number of similarity scores is the same as the number of the multiple supporting sample images. The second neural network includes multiple neurons, and the degree of influence of the features in the first enhanced feature map on the neurons is positively correlated with the weight of the features, and the degree of influence of the features in the second enhanced feature map on the neurons is positively correlated with the weight of the features.

[0075] Optionally, the image coding network, the first neural network, and the second neural network are trained using a second loss function, where the second loss function is:

[0076] Among them, L metric is the second loss function value, cos(·,·) represents the cosine similarity, τ is the preset temperature factor, is the second enhanced feature map corresponding to the support sample image of category i, is the first enhanced feature map corresponding to the query sample image of category i, where i is an arbitrary natural number. A second attention feature map generated when the second neural network to be trained processes the first enhanced feature map, is the third attention feature map generated during the process of the second neural network to be trained processing the second enhanced feature map, E S,Q is the mathematical expectation of the query sample and the support sample, is the mathematical expectation of the query sample.

[0077] According to another aspect of the present application, a small sample image classification device is provided, the device comprising:

[0078] an acquisition module, configured to acquire an image set, wherein the image set includes a query image and a plurality of supporting images;

[0079] An encoding module, configured to input the image set into an image encoding network to obtain a plurality of encoding feature maps;

[0080] a prompt module, configured to perform weighted processing on the plurality of encoding feature maps through a first neural network to obtain a plurality of meta-prompt feature maps;

[0081] An enhancement module, configured to perform addition and fusion on the encoding feature map and the meta-prompt feature map to obtain an enhanced feature map;

[0082] a scoring module, configured to input the enhanced feature map into a second neural network to obtain similarity scores of the plurality of enhanced feature maps;

[0083] A classification module is used to classify the query image according to the similarity score.

[0084] According to another aspect of the present application, a small sample image classification device is provided, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the small sample image classification method as described above, or the small sample image classification training method as described above.

[0085] According to another aspect of the present application, a computer storage medium is provided, in which at least one instruction, at least one program, a code set or an instruction set is stored. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the small sample image classification method as described above, or the small sample image classification training method as described above.

[0086] According to another aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the small sample image classification method of any of the above embodiments.

[0087] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0088] An image acquisition method is provided, wherein a first neural network performs weighted processing on the encoded feature maps corresponding to each image in an image set to obtain multiple meta-prompt feature maps. Based on the multiple meta-prompt feature maps, an enhanced feature map corresponding to a query image and an enhanced feature map corresponding to each support image are obtained. A second neural network is then used to obtain a similarity score between each support image and the query image, and the query image is classified based on the similarity score. In this way, the first neural network can enhance key features in the query image and the support images, highlighting image features that are beneficial to image classification and suppressing interfering features such as image background. This can improve the accuracy of the second neural network's similarity scoring of the query image and the support images, thereby improving the accuracy of query image classification. This can address the low accuracy of query image classification in related technologies and achieve the effect of improving query image classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0090] FIG1 is a schematic diagram of an image set provided in an embodiment of the present application;

[0091] FIG2 is a flow chart of a small sample image classification method provided in an embodiment of the present application;

[0092] FIG3 is a flowchart of another small sample image classification method provided in an embodiment of the present application;

[0093] FIG4 is a flow chart of obtaining a coding feature map provided by an embodiment of the present application;

[0094] FIG5 is a schematic diagram of the structure of a first neural network provided by the present application;

[0095] FIG6 is a flow chart of obtaining an initial feature map according to an embodiment of the present application;

[0096] FIG7 is a schematic diagram of a local MLP and a global MLP structure provided in an embodiment of the present application;

[0097] FIG8 is a schematic diagram of a process for obtaining a first enhanced feature map or a second enhanced feature map provided by the present application;

[0098] FIG9 is a schematic diagram of the structure of a second neural network provided by the present application;

[0099] FIG10 is a flow chart of a pre-training training process provided by the present application;

[0100] FIG11 is a flow chart of a training process of an image coding network, a first neural network, and a second neural network provided by the present application;

[0101] FIG12 is a diagram of a network architecture provided in an embodiment of the present application;

[0102] FIG13 is a schematic diagram showing comparison results between a small sample image classification method provided by an embodiment of the present application and other image classification methods;

[0103] FIG14 is a schematic diagram showing comparison results between another small sample image classification method provided by an embodiment of the present application and other image classification methods;

[0104] FIG15 is a visualization of a small sample learning method provided in an embodiment of the present application;

[0105] FIG16 is a structural block diagram of a small sample image classification device provided in an embodiment of the present application.

[0106] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0107] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0108] First, the application scenarios involved in the embodiments of the present application are introduced.

[0109] With the development of computer vision technology, the problem of image classification has attracted increasing attention. Image classification refers to the process of labeling images of unknown categories. Few-shot image classification methods classify images through few-shot learning. The goal of few-shot learning is to enable a trained neural network to identify the category of a query sample using only a small number of support samples, which can be less than or equal to 5.

[0110] Referring to Figure 1 , for example, a small-sample image classification network can be trained using multiple images in image set A, and the trained small-sample image classification network can be used to classify multiple images in image set B. The image categories in image set A can be different from the image categories in image set B. Image set A includes multiple query images a1 of known categories and support images a2 of known categories. The small-sample image classification network is trained using image set A, enabling it to accurately classify query images a1 based on similarity scores between query images a1 and support images a2. Image set B includes multiple query images b1 of unknown categories and multiple support images b2 of known categories. The trained small-sample image classification network can be used to obtain similarity scores between query images b1 and support images b2 to classify query images b1. It should be understood that Figure 1 only illustrates a small number of query images and support images in the image set. The number of query images and support images can be 100, 500, or more, and the image categories included in the image set can be 20, 30, or more, but this embodiment of the present application is not limited to this.

[0111] It should be noted that the application scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field can know that the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0112] The implementation environment may include an input terminal, a server, and a display terminal. The server includes a processor and can establish a wired or wireless connection with the input terminal and the display terminal to classify images based on images input by the input terminal and display the image classification results on the display terminal.

[0113] Referring to FIG2 , a small sample image classification method provided in an embodiment of the present application can be applied to the server in the above implementation environment. The method may include the following steps:

[0114] Step 101: Acquire an image set, where the image set includes a query image and multiple supporting images.

[0115] Each image in the image set includes an object image and a background image, and each supporting image corresponds to a category.

[0116] Step 102: Input the image set into the image coding network to obtain multiple coding feature maps.

[0117] The multiple coding feature maps include a first coding feature map corresponding to the query image and a second coding feature map corresponding to each supporting image.

[0118] Step 103: Perform weighted processing on the multiple encoding feature maps through the first neural network to obtain multiple meta-prompt feature maps.

[0119] The plurality of meta-prompt feature maps include a first meta-prompt feature map corresponding to the first coding feature map and a second meta-prompt feature map corresponding to each second coding feature map. The weighting process is used to increase the weight of the feature representing the target object image in the coding feature map.

[0120] Step 104: Add and fuse the encoded feature map and the meta-prompt feature map to obtain an enhanced feature map.

[0121] The first encoded feature map and the first meta-hint feature map are added and fused to obtain a first enhanced feature map corresponding to the query image, and the second encoded feature map and the second meta-hint feature map are added and fused to obtain a second enhanced feature map corresponding to each support image.

[0122] Step 105: Input the enhanced feature map into the second neural network to obtain similarity scores of the multiple enhanced feature maps.

[0123] The first enhanced feature map and each of the plurality of second enhanced feature maps are input into a second neural network in pairs to obtain a similarity score between each of the plurality of second enhanced feature maps and the first enhanced feature map. The number of similarity scores is the same as the number of the plurality of supporting images. The second neural network includes a plurality of neurons. The degree of influence of a feature in the first enhanced feature map on a neuron is positively correlated with the weight of the feature, and the degree of influence of a feature in the second enhanced feature map on a neuron is positively correlated with the weight of the feature.

[0124] Step 106: Classify the query image based on the similarity score.

[0125] In summary, the embodiment of the present application provides an image acquisition method, which performs weighted processing on the coded feature map corresponding to each image in the image set through a first neural network to obtain multiple meta-prompt feature maps, and based on the multiple meta-prompt feature maps, obtains the enhanced feature map corresponding to the query image and the enhanced feature map corresponding to each supporting image, and then obtains the similarity score between each supporting image and the query image through a second neural network, and classifies the query image based on the similarity score. In this way, the key features in the query image and the supporting image can be enhanced through the first neural network, highlighting the image features that are beneficial to image classification and suppressing interference features such as the image background. This can improve the scoring accuracy of the second neural network when performing similarity scoring on the query image and the supporting image, thereby improving the accuracy of query image classification. This can solve the low accuracy of query image classification in related technologies and achieve the effect of improving query image classification accuracy.

[0126] Referring to FIG3 , another small sample image classification method provided in an embodiment of the present application can be applied to the server in the above implementation environment. The method may include the following steps:

[0127] Step 201: Acquire an image set, where the image set includes a query image and multiple supporting images.

[0128] Each image in the image set includes an object image and a background image, and each supporting image corresponds to a category. The target object can be an animal, a building, a machine, or a vehicle.

[0129] An image set may include multiple images, which may include graphics, photos, graphics and photos converted from video data streams, etc. Exemplarily, the multiple images are multiple photos captured by a camera of a terminal device; alternatively, the multiple images may be multiple photos obtained from within the terminal device, such as multiple photos stored in an album of the terminal device, or multiple photos obtained by the terminal device from the cloud.

[0130] In an optional embodiment, after acquiring the image set, the multiple images in the image set can be divided, and the small sample learning can obtain transferable knowledge by training the training data set (English: Ctrain) and use it on the test data set (English: Dtest). That is, in the process of training the network used for the small sample image classification method, the image set can also be divided into a training data set and a test data set. The training data set can be used to train the network used for the small sample image classification method, and the test data set can be used to test the training results of the network used for the small sample image classification method. Among them, the training data set does not overlap with the test data set, that is,

[0131] Exemplarily, multiple images in the training data set can be obtained in the following way. Small sample image classification can be called N-way K-shot classification task, where N represents the number of categories and K represents the number of images of each category. Each N-way K-shot classification task consists of a support set (S) and a query set (Q) set. The division process can include: first randomly selecting N different types of images from the image set, and extracting K images from each of the N different types of images as images of the support set (S), and then extracting T images from each of the N different types of images as images of the query set (Q). Wherein, N is an integer greater than or equal to 2, K is an integer greater than or equal to 1, and T is an integer greater than or equal to 1. Exemplarily, N is 5, K is 3, and T is 1. The support images in the support set (S) can have annotations. Exemplarily, the annotations can be in, represents the i-th support image, Indicates the category of the support image. The query images in the query set (Q) are not labeled. The goal of the few-shot image classification method is to accurately classify the query images in the query set (Q) into one of the N categories.

[0132] The method for acquiring multiple images in the test dataset can be similar to the method for acquiring multiple images in the training data. Each of the multiple support images in the support set of the test dataset corresponds to a category. That is, the categories of the multiple support images in the support set of the test dataset are known, while the category of the query image in the support set of the test dataset is unknown. The query image and support images in step 201 can be images in the test dataset, or the query image and support images in step 201 can be data of random categories acquired by the user. This embodiment of the present application is not limited to this.

[0133] Step 202: Input the image set into the image coding network to obtain multiple coding feature maps, where the multiple coding feature maps include a first coding feature map corresponding to the query image and a second coding feature map corresponding to each support image.

[0134] The query image can be processed by the image coding network to obtain a first coding feature map corresponding to the query image. At the same time, the support image can also be processed by the image coding network to obtain a second coding feature map corresponding to the support image.

[0135] Referring to FIG. 4 , step 202 may include the following sub-steps:

[0136] Sub-step 2021: Obtain the class code of each image in the image set, where the class code is a preset prior value.

[0137] Among them, since the category of the support image is known, the category code of the support image can be an accurate value (X i[cls] ), where i is less than or equal to N. For example, N is 5, which means that the support set includes 5 different types of images. The class codes of the 5 different images can be X 1[cls] 、X 2[cls] 、X 3[cls] 、X 4[cls] and X 5[cls] Since the category of the query image is unknown, the category code of the query image may be a preset character, which may include at least one of a number and an arbitrary symbol.

[0138] Sub-step 2022: Segment each image in the image set to obtain multiple block images corresponding to each image.

[0139] Each image in the image set can be divided into multiple block images and encoded, which can reduce image processing complexity and improve data processing efficiency. For example, for very large images, computational and storage limitations may be encountered. By dividing the image into multiple smaller block images, data processing complexity can be reduced and data processing efficiency can be improved.

[0140] Sub-step 2023: Input the multiple block images into the position encoder to obtain the position code corresponding to each block image.

[0141] The position code is used to identify the position of a block image within the corresponding image. When an image is divided into multiple block images for processing, the arrangement of the multiple block images will change. By obtaining the position code of each block image, the relative position relationship between the multiple block images can be clarified.

[0142] Sub-step 2024: Input the multiple block images into the feature encoder to obtain a block code corresponding to each block image.

[0143] Among them, the block code is a vector representing the block image. The class code, multiple position codes and multiple block codes are encoded feature maps.

[0144] By dividing the image into multiple patches and then converting each patch into a vector, the computational complexity can be reduced and the efficiency of feature extraction can be improved.

[0145] Step 203: Input multiple block codes corresponding to each coded feature map into the first multi-layer perception network to obtain an initial feature map.

[0146] Referring to FIG5 , the first neural network may include a first multi-layer perceptual network, a local multi-layer perceptual network, and a global multi-layer perceptual network, wherein the first multi-layer perceptual network is connected to the local multi-layer perceptual network and the global multi-layer perceptual network, respectively. An exemplary first neural network may be referred to as a meta-prompt learner (MPL). The first neural network performs weighted processing on multiple encoding feature maps to obtain multiple meta-prompt feature maps, including a first meta-prompt feature map corresponding to the first encoding feature map and a second meta-prompt feature map corresponding to each second encoding feature map.

[0147] Among them, the weighted processing is used to increase the weight of the features representing the target object image in the coding feature map.

[0148] A multilayer perceptron (MLP), also known as an artificial neural network (ANN), may include an input layer, at least one hidden layer, and an output layer.

[0149] Referring to FIG. 6 , step 203 may include the following sub-steps:

[0150] Sub-step 2031: Perform three linear transformations on each block code in the coding feature map to obtain the query matrix, key matrix and value matrix of each block code.

[0151] The query matrix (Q), key matrix (K), and value matrix (V) are collectively referred to as the QKV matrix. The QKV matrix is ​​used to calculate the attention weights in the self-attention mechanism. The query matrix is ​​used to generate query vectors. Each query vector represents a query for a block code in the attention mechanism and can be used to find information related to the block code. The key matrix can be used to generate key vectors. Each key vector represents the key of a block code in the attention mechanism and can be used to represent the relationship between the block code and other block codes. The value matrix is ​​used to generate value vectors. Each value vector represents the value of a block code in the attention mechanism and can be used to represent the feature information of the block code.

[0152] Sub-step 2032: Multiply and fuse the query matrix and key matrix corresponding to each block code to obtain the first attention feature map of each block code.

[0153] By multiplying the query matrix corresponding to each block code with the key matrix corresponding to each block code, we can obtain the attention weight matrix corresponding to each block code. The attention weight matrix is ​​the first attention feature map.

[0154] Sub-step 2033: multiply and fuse the first attention feature map corresponding to each block code and the value matrix to obtain the sub-initial feature map of each block code. The initial feature map includes multiple sub-initial feature maps corresponding to multiple block codes.

[0155] By multiplying the first attention feature map corresponding to each block code with the value matrix corresponding to each block code, a sub-initial feature map corresponding to each block code can be obtained. The dimension of the sub-initial feature map corresponding to each block code can be the same as the dimension of the block code.

[0156] Multiple block codes corresponding to multiple block images in an image can be input into the first multi-layer perception network respectively to obtain a sub-initial feature map corresponding to each block image. The multiple sub-initial feature maps corresponding to multiple image blocks can constitute the initial feature map of the image.

[0157] Step 204: Input the initial feature map into the local multi-layer perception network to obtain a local feature map.

[0158] Referring to Figure 7, a local multi-layer perception network may include a first convolutional layer [Conv1(1,1,0)], a batch normalization layer (BatchNorm2d), a linear rectification function (ReLU), a second convolutional layer [Conv2(1,1,0)], a second batch normalization layer (BatchNorm2d), and a random dropout layer (Dropout), which are connected in sequence.

[0159] Step 205: Input the initial feature map into the global multi-layer perception network to obtain a global feature map.

[0160] A global multi-layer perception network may include an adaptive pooling layer (AdaptiveAvgPool2d), a first convolutional layer, a batch normalization layer, a linear rectification function, a second convolutional layer, a second batch normalization layer, and a random dropout layer, which are connected in sequence.

[0161] The global feature map refers to the overall features of the image. For example, the global feature map may include color features, texture features, and shape features. The global feature map has the characteristics of good invariance and the ability to intuitively represent the image.

[0162] The local feature map is a feature extracted from a local area of ​​an image. For example, the local feature map may include features of edges, corners, lines, curves, and other areas.

[0163] Compared with the global feature map, the local feature map has the characteristic of small correlation between features. If some features in the local feature map disappear, when feature detection or feature matching is performed on other features in the local feature map except for the disappeared features, the disappeared features have little impact on other features.

[0164] Step 206: Add and fuse the local feature map and the global feature map to obtain a fused feature map.

[0165] Global context information is captured by performing global average pooling to obtain a global feature map. The local feature map is then added to the global feature map to obtain a fused feature map. This allows the global context information to be fused with the local feature map, highlighting the local features of the image while avoiding loss of overall image information.

[0166] Step 207: Process the fused feature map through an activation function to obtain a weight matrix.

[0167] The activation function may include a Sigmoid function, also known as an S-shaped growth curve. An activation function can act as a nonlinear mapping, limiting the output amplitude of a neural network to a certain range, typically between (-1, 1) or (0, 1). The Sigmoid function can map numbers from (-∞, +∞) to the range of (0, 1).

[0168] In an exemplary embodiment, a neural network includes multiple neurons, and the connection strength between the multiple neurons is represented by weights. In other words, the weights determine the degree of influence of the input signal during the propagation process in the neural network, that is, the weights determine the degree of influence of the input data on each neuron.

[0169] Step 208: Multiply and fuse the weight matrix and the initial feature map to obtain a meta-prompt feature map.

[0170] By multiplying and fusing the weight matrix with the initial feature map, the weight can be associated with each data in the initial feature map to indicate the importance of the data in the initial feature map. The resulting meta-prompt feature map can be used to prompt the second neural network in the subsequent step to enhance the perception intensity of features with larger weights.

[0171] Step 209: Add and fuse the first coded feature map and the first meta-prompt feature map to obtain a first enhanced feature map corresponding to the query image, and add and fuse the second coded feature map and the second meta-prompt feature map to obtain a second enhanced feature map corresponding to each support image.

[0172] Please refer to FIG8 . For each image in the image set (which may be a query image or a support image), the first (or second) enhanced feature map may be obtained according to the steps in FIG8 .

[0173] (1) Any image in the image set can be segmented. i ) can be a query image or a support image, and the segmented image can be expressed as: in, Represents the nth block image.

[0174] (2) You can get the image (x i ) class code X i[cls] , and obtain the position encoding of each block image through the position encoder (English: Position Embedding). The position encoding of multiple block images corresponding to an image can be expressed as: in, Indicates the position code of the nth block image.

[0175] (3) Input multiple block images into the feature encoder and output the block code corresponding to each block image. The multiple block codes corresponding to an image can be expressed as: in, Indicates the block code of the nth block image.

[0176] (4) Inputting multiple block codes into the first neural network, outputting a meta-prompt feature map, which may include multiple prompt vectors. Since the first neural network divides an image into multiple block images for processing, the resulting prompt feature map can be expressed as: in, Represents the hint vector corresponding to the nth block image in an image.

[0177] Since the class code, multiple position codes and multiple block codes of an image can constitute the coding feature map of the image, the coding feature map corresponding to an image and the meta-hint feature map corresponding to the image are added and fused to obtain the enhanced feature map corresponding to the image. Therefore, the enhanced feature map corresponding to a block image in the image can be expressed as: The enhanced feature map corresponding to the image can be expressed as: The enhanced feature map can be called a token.

[0178] If the image is a query image, the first enhanced feature map corresponding to the query image can be expressed as If the image is a support image, the second enhanced feature map corresponding to the support image can be expressed as,

[0179] Step 210: Input the first enhanced feature map and the second enhanced feature map into a second multi-layer perception network in pairs to obtain a first feature map corresponding to the first enhanced feature map and a second feature map obtained from the first enhanced feature map.

[0180] The second neural network includes a second multi-layer perception network and a feature scaling network; the second neural network can be called a Token-Awareness Enhancement Module (TAEM).

[0181] The first enhanced feature map and each of the multiple second enhanced feature maps are input into the second neural network in pairs to obtain a similarity score between each of the multiple second enhanced feature maps and the first enhanced feature map.

[0182] The number of similarity scores is the same as the number of the multiple supporting images, the second neural network includes multiple neurons, the degree of influence of the features in the first enhanced feature map on the neurons is positively correlated with the weight of the features, and the degree of influence of the features in the second enhanced feature map on the neurons is positively correlated with the weight of the features.

[0183] The structures of the second multi-layer perception network and the first enhanced perception network can be similar or identical. Therefore, in step 210, the process of inputting the first enhanced feature map into the second multi-layer perception network to output the first feature map, and inputting the second enhanced feature map into the second multi-layer perception network to output the second feature map can refer to step 203, and the embodiments of the present application will not be repeated here.

[0184] It can be understood that pairwise input means first forming a feature map pair with a first enhanced feature map (corresponding to the query image) and a second enhanced feature map from a plurality of second enhanced feature maps (corresponding to the support image), and then inputting the feature map pair into the second multi-layer perception network. A first enhanced feature map can form a feature map pair with each of the plurality of second enhanced feature maps, and the number of feature map pairs is the same as the number of the plurality of second enhanced feature maps. For example, if the number of first enhanced feature maps is 1 and the number of second enhanced feature maps is 5, then one first enhanced feature map and five second enhanced feature maps can form five feature map pairs.

[0185] Please refer to FIG9, the first enhanced feature map Z q and the second enhanced feature map Z s Input the second multi-layer perception network in pairs.

[0186] Step 211: Input the first feature map and the second feature map into a feature scaling network in pairs to obtain a third feature map corresponding to the first feature map and a fourth feature map corresponding to the second feature map.

[0187] The first feature map can be expressed as Z q1 ∈R H×W×C , the second feature map can be expressed as Z s1 ∈R H×W×C , the first feature map and the second feature map are input into the feature scaling network in pairs, and the first feature map and the second feature map are scaled by convolution. For example, a 4D tensor can be used to represent the first feature map and the second feature map, and the first feature map is represented by the 4D tensor as C q ∈R H×W×H×W , the second feature map is represented by this 4D tensor as C s ∈R H×W×H×W , the feature scaling network can transform the first feature map into C′ q ∈R H×W×U×V , and convert the second feature map into C′ s ∈R H×W×U×V Among them, H (English: Height; Chinese: Height) represents the number of pixels in the vertical dimension of the image, W (English: Width; Chinese: Width) represents the number of pixels in the horizontal dimension of the image, and C (English: Channel; Chinese: Channel number) represents the number of channels in an image; U represents the number of pixels in the vertical dimension of the image after scaling, and V represents the number of pixels in the horizontal dimension of the image after scaling.

[0188] Step 212: Process the third feature map through the attention function to obtain a first attention mapping value, and process the fourth feature map through the attention function to obtain a second attention mapping value.

[0189] The second neural network may further include a third multi-layer perceptual network. The third feature map and the fourth feature map may be first input into the third multi-layer perceptual network to output a QKV matrix corresponding to the third feature map and a QKV matrix corresponding to the fourth feature map. The attention function may map a QKV matrix into an output, which may be a vector representing the degree of correlation between the Q matrix and the K matrix. As shown in FIG9 , the first attention mapping value A may be obtained. q and the second attention map value A s .

[0190] Step 213: Multiply and fuse the first attention map value with the first feature map to obtain a second attention feature map, and multiply and fuse the second attention map value with the second feature map to obtain a third attention feature map.

[0191] The second attention feature map can be obtained by multiplying and fusing the first attention map value with the first feature map to retain the important image information in the first feature map and highlight the image features that are beneficial to classification. The third attention feature map can be obtained by multiplying and fusing the second attention map value with the second feature map to retain the important image information in the first feature map and highlight the image features that are beneficial to classification. As shown in Figure 9, the second attention feature map can be obtained. and the third attention feature map

[0192] Step 214: Obtain the cosine similarity between the second attention feature map and the third attention feature map, where the cosine similarity is a similarity score.

[0193] Cosine similarity refers to a measurement method that uses the cosine value of the angle between two vectors in the vector space to measure the size of the difference between two features. The closer the cosine value is to 1, the closer the angle is to 0 degrees, that is, the more similar the two vectors are. As shown in Figure 9, based on the cosine similarity of the second attention feature map and the third attention feature map, the similarity score of the second attention feature map and the third attention feature map can be obtained. Exemplarily, the similarity score and cosine similarity satisfy the following formula: Here, A(·,·) represents the similarity score.

[0194] Step 215: Classify the query image based on the similarity score.

[0195] The category of the supporting image with the highest similarity score to the query image among the at least two supporting images may be used as the category of the query image.

[0196] In summary, the embodiment of the present application provides an image acquisition method, which performs weighted processing on the coded feature map corresponding to each image in the image set through a first neural network to obtain multiple meta-prompt feature maps, and based on the multiple meta-prompt feature maps, obtains the enhanced feature map corresponding to the query image and the enhanced feature map corresponding to each supporting image, and then obtains the similarity score between each supporting image and the query image through a second neural network, and classifies the query image based on the similarity score. In this way, the key features in the query image and the supporting image can be enhanced through the first neural network, highlighting the image features that are beneficial to image classification and suppressing interference features such as the image background. This can improve the scoring accuracy of the second neural network when performing similarity scoring on the query image and the supporting image, thereby improving the accuracy of query image classification. This can solve the low accuracy of query image classification in related technologies and achieve the effect of improving query image classification accuracy.

[0197] In an optional implementation, the image coding network, the first neural network, and the second neural network in any of the above embodiments can be pre-trained fusion networks, or the image coding network can be trained in the implementation steps of the small sample classification method. It should be noted that the various networks used in the embodiments of the present application (such as the first multi-layer perception network, the local multi-layer perception network, the global multi-layer perception network, the second multi-layer perception network, and the feature scaling network, etc.) are all trained network structures. The embodiments of the present application do not limit the training methods of these networks.

[0198] Optionally, the image encoding network is pre-trained using masked image modeling. The pre-trained network framework includes a teacher network and a student network. The teacher network is typically a large, pre-trained deep learning model, while the student network is a smaller, more lightweight model. The goal of training the student network is to make the student network's performance as close to that of the teacher network as possible while reducing computing resource requirements and improving inference speed.

[0199] Pre-training can use panoramic sample images as the input of the teacher network, mask sample images as the input of the student network, and the output of the teacher network as the true value. In the process of pre-training, the first loss function (L pre ) is as follows:

[0200] Among them, L pre is the first loss function value, Dtrain is the training set, (x, y)∈D train Indicates that the domain to which the sample image belongs is the training set, m i is a random mask, m i is 0 or 1. x is a panoramic sample image, which can be divided into n block images, and the n block images are represented as The variable j represents the jth block image in the n block image, is the teacher network output corresponding to the jth block image in the n block images, is the output of the student network corresponding to the jth block image in the n block images.

[0201] As shown in FIG10 , in an embodiment of the present application, the pre-training method of the image coding network may include the following steps:

[0202] Step 301: Obtain a panoramic sample image and a mask sample image.

[0203] The masked sample image is an image obtained by performing mask processing on the panoramic sample image.

[0204] Step 302: Input the masked sample image into the student network to obtain a first encoded feature map.

[0205] Step 303: Input the panoramic image into the teacher network to obtain a second encoded feature map.

[0206] Step 304: Compare the first coding feature map and the second coding feature map to obtain a comparison difference.

[0207] Step 305: If the contrast difference is greater than the preset result, the student network is adjusted based on the contrast difference, and the step of inputting the mask sample image into the student network is executed.

[0208] If the output of the student network differs significantly from the output of the teacher network, it indicates that the parameters of the student network are not accurate enough, and the parameters of the student network can be further adjusted through multiple trainings.

[0209] Step 306: If the comparison difference is less than or equal to the preset result, the student network is determined as the image coding network.

[0210] If the output of the student network is slightly different from that of the teacher network, it means that the parameters of the student network are relatively accurate and the training of the student network can be terminated.

[0211] This embodiment of the present application also provides a training method for small sample image classification. Referring to FIG11 , in this embodiment of the present application, the training process of the image coding network, the first neural network, and the second neural network may include the following steps:

[0212] Step 401: Obtain a training sample set from a sample set, where the training sample set includes a query sample image and multiple support sample images.

[0213] The query sample image corresponds to a category, each support sample image corresponds to a category, and both the query sample image and the support sample image include object images and background images.

[0214] Step 402: Input the training sample set into the image coding network to be trained to obtain multiple coding feature maps.

[0215] The multiple coding feature maps include a first coding feature map corresponding to the query sample image and a second coding feature map corresponding to each supporting sample image.

[0216] Step 403: Perform weighted processing on the plurality of encoding feature maps through the first neural network to be trained to obtain a plurality of meta-prompt feature maps.

[0217] Among them, weighted processing is used to increase the weight of the features representing the target object image in the coding feature map. The multiple meta-prompt feature maps include a first meta-prompt feature map corresponding to the first coding feature map, and a second meta-prompt feature map corresponding to each second coding feature map.

[0218] In step 403, including the first multi-layer perceptual network, the local multi-layer perceptual network and the global multi-layer perceptual network, during the training process, the first multi-layer perceptual network, the local multi-layer perceptual network and the global multi-layer perceptual network can be trained synchronously.

[0219] Step 404: Add and fuse the encoded feature map and the meta-hint feature map to obtain an enhanced feature map.

[0220] Optionally, the first encoding feature map and the first meta-hint feature map are added and fused to obtain a first enhanced feature map corresponding to the query sample image, and the second encoding feature map and the second meta-hint feature map are added and fused to obtain a second enhanced feature map corresponding to each supporting sample image.

[0221] Step 405: Input the enhanced feature map into the second neural network to be trained to obtain similarity scores of the multiple enhanced feature maps.

[0222] Optionally, the first enhanced feature map and each second enhanced feature map in the multiple second enhanced feature maps are input into the second neural network to be trained in pairs to obtain a similarity score between each second enhanced feature map in the multiple second enhanced feature maps and the first enhanced feature map.

[0223] The number of similarity scores is the same as the number of multiple supporting sample images, the second neural network includes multiple neurons, the degree of influence of the features in the first enhanced feature map on the neurons is positively correlated with the weight of the features, and the degree of influence of the features in the second enhanced feature map on the neurons is positively correlated with the weight of the features.

[0224] In step 405, a second multi-layer perception network and a feature scaling network are included. During the training process, the second multi-layer perception network and the feature scaling network can be trained synchronously.

[0225] Step 406: Classify the query sample image based on the similarity score to obtain a predicted category of the query sample image.

[0226] Step 407: Compare the predicted category of the query sample image with the category corresponding to the query sample image to obtain a second comparison difference.

[0227] Step 408: If the second contrast difference is greater than the preset result, the image coding network to be trained, the first neural network to be trained, and the second neural network to be trained are adjusted based on the second contrast difference, and the training sample set is input into the image coding network to obtain multiple coding feature maps.

[0228] If the predicted category of the query sample image is significantly different from the category corresponding to the query sample image, it may indicate that the parameters of the image coding network, the first neural network, and the second neural network are not accurate enough. The parameters of the image coding network, the first neural network, and the second neural network may be further adjusted through multiple training sessions. In other words, step 401 or 402 is executed after step 408.

[0229] Step 409: If the second comparison difference is less than or equal to the preset result, the image coding network to be trained, the first neural network to be trained, and the second neural network to be trained are determined as the image coding network, the first neural network, and the second neural network.

[0230] If the predicted category of the obtained query sample image is slightly different from the category corresponding to the query sample image, it can be said that the parameters in the image coding network, the first neural network and the second neural network are relatively accurate, and the training of the image coding network, the first neural network and the second neural network can be terminated.

[0231] Optionally, the training of the image encoding network, the first neural network, and the second neural network can use the query image and the support image in the training set as input, and the category of the query image as the true value. During the training process, the formula of the second loss function (Loss) is as follows:

[0232] Among them, L metric is the second loss function value, cos(·,·) represents the cosine similarity, τ is the preset temperature factor, is the second enhanced feature map corresponding to the support sample image of category i, is the first enhanced feature map corresponding to the query sample image of category i, where i is an arbitrary natural number. is the second attention feature map generated when the second neural network to be trained processes the first enhanced feature map, is the third attention feature map generated when the second neural network to be trained processes the second enhanced feature map, E S,Q is the mathematical expectation of the query sample and the support sample, is the mathematical expectation of the query sample.

[0233] Optionally, as shown in FIG12 , the related art requires generating a prompt for each task, where the prompt is text or instruction that provides input to a neural network to guide it to generate a specific output. However, the present application uses a first neural network, also known as a meta-prompt learner (MPL), to generate an adaptive meta-prompt feature map based on the task content.

[0234] In an exemplary embodiment, the effectiveness of the present application can be further illustrated by the following simulation results. In the examples of the present application, the effectiveness of the present method was evaluated on seven datasets: the miniImageNet dataset, the tieredImageNet dataset, the FC100 dataset, the CIFAR-FS dataset, the CUB-200-211 dataset, the Stanford Dogs dataset, and the Stanford Cars dataset.

[0235] The embodiment of the present application adopts the FewTURE method to pre-train the image coding network. FewTURE is a method for obtaining representative coding using an architecture completely based on Transformer, while only training on training data of a small data set. The embodiment of the present application compares two different Transformer architectures: the first architecture is a small single-image block ViT architecture (ViT-Small), and the second architecture is a micro version (Swin-Tiny) of the multi-image block Swin architecture. Cosine similarity is used to calculate the similarity between the support sample and the query sample. Stochastic gradient descent (English: stochastic gradient descent; abbreviated: SGD) with a learning rate of 0.1 can be used as an optimizer.

[0236] Please refer to Table 1 and Table 2. The embodiments of the present application experimented with multiple image classification methods on four benchmark datasets. These four datasets include dataset 1 (miniImageNet), dataset 2 (tieredImageNet), dataset 1 (CIFAR-FS), and dataset 4 (FC100). Among them, 1shot means a 5way-1shot task, that is, there are 5 categories in the support set for multiple support images, each category has 1 support image, and each category has 1 query image. 5shot means a 5way-5shot task, that is, there are 5 categories in the support set for multiple support images, each category has 5 support images, and each category has 1 query image. As shown in Tables 1 and 2, the small sample image classification method (AMPL) in the embodiments of the present application is compared with the image classification methods containing additional semantic information (such as KTN, AM3, TRAML and DeepEMD). The classification accuracy of the small sample image classification in the embodiments of the present application under the 1-shot setting on the miniImageNet dataset and the tieredImageNet dataset is improved by 4.27%.

[0237] The present embodiment improves the robustness of the image coding network by pre-training it. When pre-training the image coding network using the ViT-Small architecture, the present embodiment achieves a classification accuracy of 55.34% in a 1-shot setting and 71.61% in a 1-shot setting on the FC100 dataset, a significant improvement over the other image classifications listed in Tables 1 and 2.

[0238] For example, Table 1 compares the 5-way classification accuracy (%) of other image classification methods on the test set of the miniImageNet dataset and the tieredImageNet dataset using an established evaluation protocol, with a 95% confidence interval. The middle row indicates additional semantic information in the class name or description. The symbol "-" indicates that the method was not reported.

[0239] Table 2 compares the 5-way classification accuracy (%) with previous image classification methods (95% confidence interval) on the test sets of the CIFAR FS dataset and the FC100 dataset using the established evaluation protocol.

[0240] Table 1

[0241] Table 2

[0242] Please refer to Table 3, which shows the results of the ablation experiment. In the 1-shot experiments of four benchmark datasets (miniImageNet dataset, tieredImageNet dataset, FC100 dataset, CIFAR-FS dataset), the average accuracy of the small sample classification method in the embodiment of the present application was improved by 2.3%. In order to verify the effectiveness of the small sample classification method in the embodiment of the present application, the embodiment of the present application conducted experiments on the trained small sample classification method using three different interaction mechanisms: pre-trained image coding network (PM), meta-prompt learner (MPL) and image block token perception module (TAEM). As shown in Table 3, MPL and TAEM improved the average accuracy by 2.3% and 2.5% respectively in the 1-shot experiments of the four benchmark datasets. In addition, all these mechanisms combined further enhance the 1-shot learning accuracy.

[0243] For example, Table 3 uses the established evaluation protocol to perform an ablation study of the 5-way 1-shot classification accuracy (%) on the test sets of four benchmark datasets with a 95% confidence interval.

[0244] Table 3

[0245] Please refer to Table 4. In the embodiments of the present application, the classification results of the small sample image classification method using adaptive meta-cues in the embodiments of the present application and the small sample classification method using fixed cues are compared on the CIFAR-FS dataset and the tieredImageNet dataset. The small sample classification method using fixed cues can be obtained by replacing the adaptive meta-cues generated by the meta-cue learner (MPL) in the small sample image classification method in the embodiments of the present application with fixed cues. One-shot and five-shot experiments were performed on vit-small and swin-tiny. Compared with the optimal results of the small sample classification method using fixed cues, the average accuracy of the small sample image classification method in the embodiments of the present application on vit-small for one shot and five shots was improved by 0.73% and 0.835%, respectively. The average accuracy of the small sample image classification method on swin-tiny for one shot and five shots was improved by 0.65% and 0.16%, respectively. In this way, the experimental results can verify the effectiveness of the small sample image classification method in the embodiments of the present application.

[0246] Table 4

[0247] Please refer to Table 5. The embodiment of the present application integrates the meta-prompt learner (MPL) into the VPT (English: Visual Prompt Tuning) method and compares it with the original VPT method on the animal (CUB) dataset, the car (Cars) dataset, and the dog (Dogs) dataset. "↑" indicates an improvement in accuracy. In the experiment, we used the sup-vit16 backbone network to be consistent with the VPT method. As shown in Table 5, in the comparison of the three datasets and various different prompts, the average accuracy of prompt = 50 and prompt = 100 increased by 3.03% and 2.67% respectively. In this way, the experimental results confirm that the meta-prompt learner (MPL) can be a plug-and-play neural network.

[0248] Table 5

[0249] Please refer to Figures 13 and 14, where the other method can be the FewTURE method, dataset 1 can be the miniImageNet dataset, and dataset 2 can be the FC100 dataset. Broken line 1 represents FewTURE vit-small, that is, the backbone network in the FewTURE method is vit-small; broken line 2 represents AMPL vit-small, that is, the backbone network in the AMPL method is vit-small; broken line 3 represents FewTURE swin-tiny, that is, the backbone network in the FewTURE method is swin-tiny; broken line 3 represents AMPL swin-tiny, that is, the backbone network in the AMPL method is swin-tiny. It can be seen from Figures 12 and 13 that the small sample classification method in the embodiment of the present application has a high accuracy.

[0250] [Corrected 08.04.2024 according to Rule 91] Please refer to Figure 15. The baseline is a reference in machine learning and can be used to compare the performance and effect of the algorithm. Figure 15 can be a visualization of the second attention feature map or the third attention feature map in step 213 in the above embodiment. It can be seen from Figure 15 that the perception of the object information by the baseline or the trained network of the present application can be represented by the picture in Figure 15. It can be seen that the object information and the background information are intertwined in the visualization of the baseline. In the visualization of the embodiment of the present application, the trained network can concentrate on processing object information, which can enhance the classification accuracy of the small sample image classification method. Dataset 1 in Figure 15 can be the miniImageNet dataset, dataset 2 can be the FC100 dataset, backbone network 1 is swin-tiny, and backbone network 2 is vit-small.

[0251] Referring to FIG. 16 , a small sample image classification apparatus 1400 according to an embodiment of the present application includes:

[0252] An acquisition module 1410 is configured to acquire an image set, the image set including a query image and multiple supporting images, each image in the image set including an object image and a background image, and each supporting image corresponding to a category;

[0253] An encoding module 1420 is configured to input the image set into an image encoding network to obtain a plurality of encoding feature maps, the plurality of encoding feature maps including a first encoding feature map corresponding to the query image and a second encoding feature map corresponding to each support image;

[0254] a prompting module 1430 configured to perform weighted processing on the plurality of coded feature maps using a first neural network to obtain a plurality of meta-prompt feature maps, wherein the weighted processing is configured to increase the weight of features representing the target object image in the coded feature maps, the plurality of meta-prompt feature maps including a first meta-prompt feature map corresponding to the first coded feature map and a second meta-prompt feature map corresponding to each of the second coded feature maps;

[0255] Enhancement module 1440, configured to perform additive fusion on the first coded feature map and the first meta-hint feature map to obtain a first enhanced feature map corresponding to the query image, and perform additive fusion on the second coded feature map and the second meta-hint feature map to obtain a second enhanced feature map corresponding to each support image;

[0256] Scoring module 1450 is configured to input the first enhanced feature map and each of the plurality of second enhanced feature maps into a second neural network in pairs to obtain a similarity score between each of the plurality of second enhanced feature maps and the first enhanced feature map, wherein the number of similarity scores is the same as the number of the plurality of supporting images, the second neural network comprising a plurality of neurons, the degree of influence of a feature in the first enhanced feature map on a neuron being positively correlated with a weight of the feature, and the degree of influence of a feature in the second enhanced feature map on a neuron being positively correlated with a weight of the feature;

[0257] The classification module 1460 is used to classify the query image based on the similarity score.

[0258] Optionally, the first neural network includes a first multi-layer perceptual network, a local multi-layer perceptual network, and a global multi-layer perceptual network, and the first multi-layer perceptual network is connected to the local multi-layer perceptual network and the global multi-layer perceptual network respectively;

[0259] Prompt module, including:

[0260] A perception unit, configured to input a plurality of block codes corresponding to each coded feature map into a first multi-layer perception network to obtain an initial feature map;

[0261] The local unit is used to input the initial feature map into the local multi-layer perception network to obtain a local feature map;

[0262] The global unit is used to input the initial feature map into the global multi-layer perception network to obtain the global feature map;

[0263] The first fusion unit is used to add and fuse the local feature map and the global feature map to obtain a fused feature map;

[0264] The weight unit is used to process the fused feature map through the activation function to obtain the weight matrix;

[0265] The second fusion unit is used to multiply and fuse the weight matrix and the initial feature map to obtain the meta-prompt feature map.

[0266] Optionally, the sensing unit is configured to:

[0267] Each block code in the coding feature map is subjected to three linear transformations to obtain the query matrix, key matrix and value matrix of each block code;

[0268] The query matrix and key matrix corresponding to each block encoding are multiplied and fused to obtain the first attention feature map of each block encoding;

[0269] The first attention feature map corresponding to each block code is multiplied and fused with the value matrix to obtain the sub-initial feature map of each block code. The initial feature map includes multiple sub-initial feature maps of the corresponding multiple block codes.

[0270] Optionally, the encoding module includes:

[0271] The class coding unit is used to obtain the class code of each image in the image set. The class code is a preset prior value.

[0272] The segmentation unit is used to segment each image in the image set to obtain multiple block images corresponding to each image.

[0273] The position coding unit is used to input multiple block images into a position coder to obtain a position code corresponding to each block image, where the position code is used to identify the position of the block image in the corresponding image.

[0274] The block coding unit is used to input multiple block images into the feature encoder to obtain a block code corresponding to each block image, where the block code is a vector representing the block image.

[0275] Optionally, the second neural network includes a second multi-layer perception network and a feature scaling network.

[0276] Scoring module, including:

[0277] The feature extraction unit is used to input the first enhanced feature map and the second enhanced feature map into the second multi-layer perception network in pairs to obtain a first feature map corresponding to the first enhanced feature map and a second feature map obtained from the first enhanced feature map.

[0278] The feature scaling unit is used to input the first feature map and the second feature map into the feature scaling network in pairs to obtain a third feature map corresponding to the first feature map and a fourth feature map corresponding to the second feature map.

[0279] An attention unit is used to process the third feature map through the attention function to obtain a first attention mapping value, and to process the fourth feature map through the attention function to obtain a second attention mapping value.

[0280] The third fusion unit is used to multiply and fuse the first attention map value with the first feature map to obtain a second attention feature map, and multiply and fuse the second attention map value with the second feature map to obtain a third attention feature map.

[0281] The similarity unit is used to obtain the cosine similarity between the second attention feature map and the third attention feature map, where the cosine similarity is a similarity score.

[0282] In summary, the embodiment of the present application provides an image acquisition device, which performs weighted processing on the coded feature map corresponding to each image in the image set through a first neural network to obtain multiple meta-prompt feature maps, and based on the multiple meta-prompt feature maps, obtains the enhanced feature map corresponding to the query image and the enhanced feature map corresponding to each support image, and then obtains the similarity score between each support image and the query image through a second neural network, and classifies the query image based on the similarity score. In this way, the key features in the query image and the support image can be enhanced through the first neural network, highlighting the image features that are beneficial to image classification and suppressing interference features such as the image background. This can improve the scoring accuracy of the second neural network when performing similarity scoring on the query image and the support image, thereby improving the accuracy of query image classification. This can solve the low accuracy of query image classification in related technologies and achieve the effect of improving query image classification accuracy.

[0283] In addition, an embodiment of the present application further provides an electronic device comprising one or more processors, a camera component, a memory, and a terminal. The memory may include a random access memory (RAM) and a read-only memory (ROM), and the camera component may be integrated with the terminal. The network training portion of the above-mentioned image acquisition method may be applied to a server, and the image classification portion other than the network training portion may be applied to both a server and a terminal.

[0284] In addition, an embodiment of the present application also provides a small sample image classification device, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a small sample image classification method as in any of the above embodiments.

[0285] In addition, an embodiment of the present application also provides a computer storage medium, which stores at least one instruction, at least one program, code set or instruction set. The at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement a small sample image classification method as in any of the above embodiments.

[0286] In addition, embodiments of the present application further provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the small sample image classification method of any of the above embodiments.

[0287] In this application, the terms "first", "second", "third" and "fourth" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The term "plurality" refers to two or more, unless otherwise expressly limited.

[0288] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0289] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0290] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0291] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A small sample image classification method, characterized in that: The method comprises: Acquire an image set, the image set comprising a query image and a plurality of supporting images; Inputting the image set into an image coding network to obtain a plurality of coding feature maps; Performing weighted processing on the plurality of encoding feature maps through a first neural network to obtain a plurality of meta-prompt feature maps; Adding and fusing the encoding feature map and the meta-prompt feature map to obtain an enhanced feature map; Inputting the enhanced feature map into a second neural network to obtain similarity scores of the plurality of enhanced feature maps; The query image is classified according to the similarity score.

2. The method according to claim 1, characterized in that The image set is input into an image coding network to obtain a plurality of coding feature maps, including: Obtaining a class code for each image in the image set, where the class code is a preset priori value; Segmenting each image in the image set to obtain a plurality of block images corresponding to each image; Inputting the plurality of block images into a position encoder to obtain a position code corresponding to each block image, wherein the position code is used to identify a position of the block image in a corresponding image; The multiple block images are input into a feature encoder to obtain a block code corresponding to each block image, wherein the block code is a vector representing the block image, and the class code, the multiple position codes and the multiple block codes are the coding feature map.

3. The method according to claim 2, characterized in that The first neural network includes a first multi-layer perceptual network, a local multi-layer perceptual network and a global multi-layer perceptual network, wherein the first multi-layer perceptual network is connected to the local multi-layer perceptual network and the global multi-layer perceptual network respectively; The weighted processing of the plurality of encoding feature maps by the first neural network to obtain a plurality of meta-prompt feature maps includes: Inputting multiple block codes corresponding to each of the encoded feature maps into the first multi-layer perception network to obtain an initial feature map; Inputting the initial feature map into the local multi-layer perception network to obtain a local feature map; Inputting the initial feature map into the global multi-layer perception network to obtain a global feature map; Adding and fusing the local feature map and the global feature map to obtain a fused feature map; Processing the fused feature map through an activation function to obtain a weight matrix; The weight matrix and the initial feature map are multiplied and fused to obtain the meta-prompt feature map.

4. The method according to claim 3, characterized in that The step of inputting the plurality of block codes corresponding to each of the encoded feature maps into the first multi-layer perception network to obtain an initial feature map comprises: Performing three linear transformations on each block code in the coding feature map to obtain a query matrix, a key matrix, and a value matrix for each block code; Multiplying and fusing the query matrix and the key matrix corresponding to each block code to obtain a first attention feature map of each block code; The first attention feature map corresponding to each of the block codes is multiplied and fused with the value matrix to obtain a sub-initial feature map of each of the block codes, where the initial feature map includes multiple sub-initial feature maps corresponding to multiple block codes.

5. The method according to claim 3, characterized in that The local multi-layer perception network includes: The first convolutional layer, batch normalization layer, linear rectification function, second convolutional layer, second batch normalization layer and random activation layer are connected in sequence.

6. The method according to claim 3, characterized in that The global multi-layer perception network includes: The adaptive pooling layer, the first convolutional layer, the batch normalization layer, the linear rectification function, the second convolutional layer, the second batch normalization layer and the random dropout layer are connected in sequence.

7. The method according to claim 1, characterized in that Each image in the image set includes an object image and a background image, each supporting image corresponds to a category, and the weighting process is used to increase the weight of the feature representing the target object image in the coded feature map in the coded feature map; The plurality of encoding feature maps include a first encoding feature map corresponding to the query image, and a second encoding feature map corresponding to each of the support images; The plurality of meta-prompt feature maps include a first meta-prompt feature map corresponding to the first encoding feature map, and a second meta-prompt feature map corresponding to each of the second encoding feature maps.

8. The method according to claim 7, characterized in that The step of adding and fusing the encoding feature map and the meta-prompt feature map to obtain an enhanced feature map, and inputting the enhanced feature map into a second neural network to obtain similarity scores of multiple enhanced feature maps includes: Adding and fusing the first encoded feature map and the first meta-hint feature map to obtain a first enhanced feature map corresponding to the query image, and adding and fusing the second encoded feature map and the second meta-hint feature map to obtain a second enhanced feature map corresponding to each of the support images; The first enhanced feature map and each of the second enhanced feature maps in the plurality of second enhanced feature maps are input into a second neural network in pairs to obtain a similarity score between each of the plurality of second enhanced feature maps and the first enhanced feature map, wherein the number of the similarity scores is the same as the number of the plurality of supporting images, and the second neural network includes a plurality of neurons, the degree of influence of the features in the first enhanced feature map on the neurons is positively correlated with the weight of the features, and the degree of influence of the features in the second enhanced feature map on the neurons is positively correlated with the weight of the features.

9. The method according to claim 8, characterized in that The second neural network includes a second multi-layer perception network and a feature scaling network; The step of inputting the first enhanced feature map and each of the plurality of second enhanced feature maps into a second neural network in pairs to obtain a similarity score between each of the plurality of second enhanced feature maps and the first enhanced feature map comprises: Inputting the first enhanced feature map and the second enhanced feature map into the second multi-layer perception network in pairs, to obtain a first feature map corresponding to the first enhanced feature map and a second feature map obtained from the first enhanced feature map; Inputting the first feature map and the second feature map into a feature scaling network in pairs, to obtain a third feature map corresponding to the first feature map and a fourth feature map corresponding to the second feature map; Processing the third feature map through an attention function to obtain a first attention map value, and processing the fourth feature map through the attention function to obtain a second attention map value; Multiplying and fusing the first attention map value with the first feature map to obtain a second attention feature map, and multiplying and fusing the second attention map value with the second feature map to obtain a third attention feature map; Obtain a cosine similarity between the second attention feature map and the third attention feature map, where the cosine similarity is the similarity score.

10. The method according to claim 1, characterized in that Classifying the query image according to the similarity score includes: The category of the supporting image with the highest similarity score to the query image among the at least two supporting images is used as the category of the query image.

11. The method according to claim 1, wherein The image coding network is pre-trained by mask image modeling. The pre-trained network framework includes a teacher network and a student network. The pre-training method includes: Acquire a panoramic sample image and a mask sample image, wherein the mask sample image is an image obtained by performing mask processing on the panoramic sample image; Inputting the mask sample image into the student network to obtain a first encoded feature map; Inputting the panoramic image into the teacher network to obtain a second encoded feature map; Comparing the first coding feature map and the second coding feature map to obtain a comparison difference; If the contrast difference is greater than a preset result, adjusting the student network based on the contrast difference, and performing the step of inputting the mask sample image into the student network; If the contrast difference is less than or equal to the preset result, the student network is determined as the image coding network.

12. The method according to claim 11, characterized in that The first loss function of the pre-training includes: Among them, L pre is the first loss function value, Dtrain is the training set, (x, y)∈D train Indicates that the domain to which the sample image belongs is the training set, m i is a random mask, m i is 0 or 1, x is a panoramic sample image, and the panoramic sample image can be divided into n block images, which are represented as The variable j represents the jth block image in the n block image, is the teacher network output corresponding to the jth block image in the n block images, is the output of the student network corresponding to the jth block image in the n block images.

13. A small sample image classification training method, characterized in that: The method comprises: Acquire a training sample set from a sample set, wherein the training sample set includes a query sample image and a plurality of support sample images; Inputting the training sample set into the image coding network to be trained to obtain multiple coding feature maps; Performing weighted processing on the plurality of encoding feature maps through a first neural network to be trained to obtain a plurality of meta-prompt feature maps; Adding and fusing the encoding feature map and the meta-prompt feature map to obtain an enhanced feature map; Inputting the strong feature map into a second neural network to be trained to obtain similarity scores of the plurality of enhanced feature maps; Classifying the query sample image according to the similarity score to obtain a predicted category of the query sample image; Comparing the predicted category of the query sample image with the category corresponding to the query sample image to obtain a second comparison difference; If the second contrast difference is greater than a preset result, adjusting the image coding network to be trained, the first neural network to be trained, and the second neural network to be trained based on the second contrast difference, and inputting the training sample set into the image coding network to obtain a plurality of coding feature maps; If the second comparison difference is less than or equal to the preset result, the image coding network to be trained, the first neural network to be trained and the second neural network to be trained are determined as the image coding network, the first neural network and the second neural network.

14. The method according to claim 13, characterized in that The query sample image corresponds to a category, each of the support sample images corresponds to a category, the query sample image and the support sample image both include an object image and a background image, and the weighting process is used to increase the weight of the feature representing the target object image in the coded feature map in the coded feature map; The plurality of encoding feature maps include a first encoding feature map corresponding to the query sample image, and a second encoding feature map corresponding to each of the support sample images; The plurality of meta-prompt feature maps include a first meta-prompt feature map corresponding to the first encoding feature map, and a second meta-prompt feature map corresponding to each of the second encoding feature maps.

15. The method according to claim 14, characterized in that The step of adding and fusing the encoding feature map and the meta-prompt feature map to obtain an enhanced feature map, and inputting the enhanced feature map into the second neural network to be trained to obtain similarity scores of the plurality of enhanced feature maps includes: Adding and fusing the first encoded feature map and the first meta-hint feature map to obtain a first enhanced feature map corresponding to the query sample image, and adding and fusing the second encoded feature map and the second meta-hint feature map to obtain a second enhanced feature map corresponding to each of the support sample images; The first enhanced feature map and each of the multiple second enhanced feature maps are input into the second neural network to be trained in pairs to obtain a similarity score between each of the multiple second enhanced feature maps and the first enhanced feature map, where the number of similarity scores is the same as the number of the multiple supporting sample images. The second neural network includes multiple neurons, and the degree of influence of the features in the first enhanced feature map on the neurons is positively correlated with the weight of the features, and the degree of influence of the features in the second enhanced feature map on the neurons is positively correlated with the weight of the features.

16. The method according to claim 13, characterized in that The image coding network, the first neural network, and the second neural network are trained using a second loss function, wherein the second loss function is: Among them, L metric is the second loss function value, cos(·,·) represents the cosine similarity, τ is the preset temperature factor, is the second enhanced feature map corresponding to the support sample image of category i, is the first enhanced feature map corresponding to the query sample image of category i, where i is an arbitrary natural number. A second attention feature map generated when the second neural network to be trained processes the first enhanced feature map, is the third attention feature map generated during the process of the second neural network to be trained processing the second enhanced feature map, E S,Q is the mathematical expectation of the query sample and the support sample, is the mathematical expectation of the query sample.

17. A small sample image classification device, characterized in that: The device comprises: An acquisition module, configured to acquire an image set, wherein the image set includes a query image and a plurality of supporting images; An encoding module, configured to input the image set into an image encoding network to obtain a plurality of encoding feature maps; a prompt module, configured to perform weighted processing on the plurality of encoding feature maps through a first neural network to obtain a plurality of meta-prompt feature maps; An enhancement module, configured to perform addition and fusion on the encoding feature map and the meta-prompt feature map to obtain an enhanced feature map; a scoring module, configured to input the enhanced feature map into a second neural network to obtain similarity scores of the plurality of enhanced feature maps; A classification module is used to classify the query image according to the similarity score.

18. A small sample image classification device, characterized in that The small sample image classification device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the small sample image classification method as described in any one of claims 1 to 12, or the small sample image classification training method as described in any one of claims 13 to 16.

19. A computer storage medium, characterized in that The computer storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by the processor to implement the small sample image classification method as described in any one of claims 1 to 12, or the small sample image classification training method as described in any one of claims 13 to 16.

Images