Few-shot image classification method, training method, apparatus, device, and storage medium

Abstract

The present application relates to the technical field of images, and discloses a few-shot image classification method, an apparatus, a device, and a computer storage medium. The method comprises: weighting, by means of a first neural network, an encoded feature map corresponding to each image in an image set to obtain a plurality of meta-prompt feature maps, and on the basis of the plurality of meta-prompt feature maps, obtaining an enhanced feature map corresponding to a query image and an enhanced feature map corresponding to each support image; then, acquiring a similarity score between each support image and the query image by means of a second neural network; and classifying the query image on the basis of the similarity score.

Description

Few-sample image classification methods, training methods, devices, equipment, and storage media Technical Field

[0001] This application relates to the field of image technology, and in particular to a small sample image classification method, training method, apparatus, device and storage medium. Background Technology

[0002] Few-shot image classification methods refer to image classification using few-shot learning. Few-shot learning is a machine learning method that can learn from a small number of samples in a category, i.e., one or a few samples, and classify or recognize new, unseen samples. Because image classification has a wide range of applications, it is often difficult to collect enough data to train neural networks; therefore, few-shot learning can be used to classify images.

[0003] A few-sample image classification method first obtains a dataset and divides it into a query set and a support set. The query set includes multiple unclassified query images, and the support set includes multiple classified support images. The query images and support images in the image set are then encoded to obtain encoded feature maps. Feature extraction is performed on the encoded feature maps to classify the query images based on the extracted features.

[0004] The above methods have low accuracy in classifying query images.

[0005] Summary of the Invention

[0006] This application provides a few-sample image classification method, training method, apparatus, device, and storage medium. The technical solution is as follows:

[0007] According to one aspect of this application, a few-sample image classification method is provided, the method comprising:

[0008] Obtain an image set, which includes a query image and multiple supporting images;

[0009] The image set is input into an image coding network to obtain multiple coded feature maps;

[0010] The multiple encoded feature maps are weighted by the first neural network to obtain multiple meta-cue feature maps;

[0011] The encoded feature map and the meta-cue feature map are added and fused to obtain an enhanced feature map;

[0012] The enhanced feature maps are input into a second neural network to obtain similarity scores for multiple enhanced feature maps;

[0013] The query image is classified based on the similarity score.

[0014] Optionally, inputting the image set into an image coding network to obtain multiple coded feature maps includes:

[0015] Obtain the class code of each image in the image set, where the class code is a preset prior value;

[0016] Each image in the image set is segmented to obtain multiple block images corresponding to each image;

[0017] The plurality of block images are input into a position encoder to obtain a position code corresponding to each block image. The position code is used to identify the position of the block image in the corresponding image.

[0018] The plurality of block images are input into a feature encoder to obtain a block code corresponding to each block image. The block code is a vector representing the block image. The class code, the plurality of position codes, and the plurality of block codes are the encoded feature map.

[0019] Optionally, the first neural network includes a first multilayer perceptron, a local multilayer perceptron, and a global multilayer perceptron, wherein the first multilayer perceptron is connected to the local multilayer perceptron and the global multilayer perceptron, respectively.

[0020] The weighted processing of the multiple encoded feature maps using a first neural network yields multiple meta-cue feature maps, including:

[0021] Each of the encoded feature maps is input into the first multilayer perceptron to obtain an initial feature map;

[0022] The initial feature map is input into the local multilayer perceptron to obtain a local feature map;

[0023] The initial feature map is input into the global multilayer perceptron to obtain the global feature map;

[0024] The local feature map and the global feature map are added and fused to obtain a fused feature map;

[0025] The fused feature map is processed by an activation function to obtain a weight matrix;

[0026] The weight matrix is ​​multiplied and fused with the initial feature map to obtain the meta-cue feature map.

[0027] Optionally, the step of inputting the multiple block encodings corresponding to each of the encoded feature maps into the first multilayer perceptron to obtain an initial feature map includes:

[0028] Each block of the encoded feature map is subjected to three linear transformations to obtain the query matrix, key matrix, and value matrix of each block.

[0029] The query matrix and the key matrix corresponding to each block code are multiplied and fused to obtain the first attention feature map of each block code;

[0030] The first attention feature map corresponding to each block encoding and the value matrix are multiplied and fused to obtain a sub-initial feature map for each block encoding. The initial feature map includes multiple sub-initial feature maps corresponding to multiple block encodings.

[0031] Optionally, the local multilayer sensing network includes:

[0032] The layers are connected in sequence: the first convolutional layer, the batch normalization layer, the linear rectified function, the second convolutional layer, the second batch normalization layer, and the random deactivation layer.

[0033] Optionally, the global multilayer sensing network includes:

[0034] The layers are connected in sequence: an adaptive pooling layer, a first convolutional layer, a batch normalization layer, a linear rectified function, a second convolutional layer, a second batch normalization layer, and a random deactivation layer.

[0035] Optionally, each image in the image set includes an object image and a background image, and each of the supporting images corresponds to a category. The weighting process is used to increase the weight of the features representing the target object image in the encoded feature map.

[0036] The multiple encoded feature maps include a first encoded feature map corresponding to the query image and a second encoded feature map corresponding to each of the supporting images;

[0037] The multiple meta-hint feature maps include a first meta-hint feature map corresponding to the first encoded feature map, and a second meta-hint feature map corresponding to each of the second encoded feature maps.

[0038] Optionally, the step of adding and fusing the encoded feature map and the meta-cue feature map to obtain an enhanced feature map, and inputting the enhanced feature map into a second neural network to obtain a similarity score of multiple enhanced feature maps, includes:

[0039] The first encoded feature map and the first meta-hint feature map are added and fused to obtain the first enhanced feature map corresponding to the query image, and the second encoded feature map and the second meta-hint feature map are added and fused to obtain the second enhanced feature map corresponding to each of the supporting images;

[0040] The first enhanced feature map is paired with each of the multiple second enhanced feature maps and input into a second neural network to obtain a similarity score between each of the multiple second enhanced feature maps and the first enhanced feature map. The number of similarity scores is the same as the number of the multiple support images. The second neural network includes multiple neurons. The influence of the features in the first enhanced feature map on the neurons is positively correlated with the weight of the features. The influence of the features in the second enhanced feature map on the neurons is also positively correlated with the weight of the features.

[0041] Optionally, the second neural network includes a second multilayer perceptron network and a feature scaling network;

[0042] The step of pairwise inputting the first enhanced feature map with each of the multiple second enhanced feature maps into the second neural network to obtain a similarity score between each of the multiple second enhanced feature maps and the first enhanced feature map includes:

[0043] The first enhanced feature map and the second enhanced feature map are input into the second multilayer perceptron in pairs to obtain the first feature map corresponding to the first enhanced feature map and the second feature map obtained from the first enhanced feature map.

[0044] The first feature map and the second feature map are input into a feature scaling network in pairs to obtain the third feature map corresponding to the first feature map and the fourth feature map corresponding to the second feature map;

[0045] The third feature map is processed by an attention function to obtain a first attention mapping value, and the fourth feature map is processed by the same attention function to obtain a second attention mapping value.

[0046] The first attention mapping value is multiplied and fused with the first feature map to obtain the second attention feature map, and the second attention mapping value is multiplied and fused with the second feature map to obtain the third attention feature map;

[0047] Obtain the cosine similarity between the second attention feature map and the third attention feature map, wherein the cosine similarity is the similarity score.

[0048] Optionally, classifying the query image based on the similarity score includes:

[0049] The category of the query image is determined by the supporting image that has the highest similarity score to the query image among the at least two supporting images.

[0050] Optionally, the image coding network is pre-trained using masked image modeling. The pre-trained network framework includes a teacher network and a student network, and the pre-training method includes:

[0051] Acquire panoramic sample images and mask sample images, wherein the mask sample images are obtained by masking the panoramic sample images;

[0052] The masked sample image is input into the student network to obtain the first encoded feature map;

[0053] The panoramic image is input into the teacher network to obtain the second encoded feature map;

[0054] The first encoded feature map and the second encoded feature map are compared to obtain the comparison difference;

[0055] If the comparison difference is greater than the preset result, the student network is adjusted based on the comparison difference, and the step of inputting the mask sample image into the student network is executed.

[0056] If the comparison difference is less than or equal to the preset result, then the student network is determined to be the image coding network.

[0057] Optionally, the first loss function of the pre-training includes:

[0058] Among them, L pre Let Dtrain be the first loss function value, and Dtrain be the training set, where (x,y)∈Dtrain. train The training set represents the neighborhood to which the sample images belong. i For a random mask, m i It can be 0 or 1. x is a panoramic sample image, which can be divided into n blocks, and the n blocks are represented as... Variable j represents the j-th image block out of n image blocks. This is the teacher network output corresponding to the j-th block image out of n block images. This is the output of the student network corresponding to the j-th block image out of n block images.

[0059] According to another aspect of this application, a few-sample image classification training method is provided, the method comprising:

[0060] Obtain a training sample set from the sample set, the training sample set including query sample images and multiple support sample images;

[0061] The training sample set is input into the image encoding network to be trained to obtain multiple encoded feature maps;

[0062] The multiple encoded feature maps are weighted by the first neural network to be trained to obtain multiple meta-cue feature maps;

[0063] The encoded feature map and the meta-cue feature map are added and fused to obtain an enhanced feature map;

[0064] The strong feature map is input into the second neural network to be trained to obtain similarity scores of multiple enhanced feature maps;

[0065] Based on the similarity score, the query sample image is classified to obtain the predicted category of the query sample image;

[0066] The predicted category of the query sample image is compared with the category corresponding to the query sample image to obtain a second comparison difference;

[0067] If the second comparison difference is greater than the preset result, the image coding network to be trained, the first neural network to be trained, and the second neural network to be trained are adjusted based on the second comparison difference, and the step of inputting the training sample set into the image coding network to obtain multiple coding feature maps is executed.

[0068] If the second comparison difference is less than or equal to the preset result, then the image coding network to be trained, the first neural network to be trained, and the second neural network to be trained are determined as the image coding network, the first neural network, and the second neural network shown.

[0069] Optionally, each query sample image corresponds to a category, and each support sample image corresponds to a category. Both the query sample image and the support sample image include object images and background images. The weighting process is used to increase the weight of the features representing the target object image in the encoded feature map.

[0070] The multiple encoded feature maps include a first encoded feature map corresponding to the query sample image and a second encoded feature map corresponding to each of the supporting sample images;

[0071] The multiple meta-hint feature maps include a first meta-hint feature map corresponding to the first encoded feature map, and a second meta-hint feature map corresponding to each of the second encoded feature maps.

[0072] Optionally, the step of adding and fusing the encoded feature map and the meta-cue feature map to obtain an enhanced feature map, and inputting the enhanced feature map into the second neural network to be trained to obtain similarity scores for multiple enhanced feature maps, includes:

[0073] The first encoded feature map and the first meta-hint feature map are added and fused to obtain the first enhanced feature map corresponding to the query sample image, and the second encoded feature map and the second meta-hint feature map are added and fused to obtain the second enhanced feature map corresponding to each of the support sample images;

[0074] The first enhanced feature map is paired with each of the multiple second enhanced feature maps and input into the second neural network to be trained, to obtain a similarity score between each of the multiple second enhanced feature maps and the first enhanced feature map. The number of similarity scores is the same as the number of multiple support sample images. The second neural network includes multiple neurons. The influence of the features in the first enhanced feature map on the neurons is positively correlated with the weight of the features. The influence of the features in the second enhanced feature map on the neurons is also positively correlated with the weight of the features.

[0075] Optionally, the image coding network, the first neural network, and the second neural network are trained using a second loss function, wherein the second loss function is:

[0076] Among them, L metric The second loss function value is represented by cos(·,·), where cos(·,·) represents the cosine similarity, and τ is a preset temperature factor. The second enhanced feature map is the support sample image corresponding to category i. This is the first enhanced feature map corresponding to the query sample image of category i, where i is any natural number. The second attention feature map is generated during the processing of the first enhanced feature map by the second neural network to be trained. E is the third attention feature map generated during the processing of the second enhanced feature map by the second neural network to be trained. S,Q To determine the mathematical expectation of the query sample and the supporting sample, To determine the mathematical expectation of the query sample.

[0077] According to another aspect of this application, a few-sample image classification apparatus is provided, the apparatus comprising:

[0078] The acquisition module is used to acquire an image set, which includes a query image and multiple supporting images;

[0079] The encoding module is used to input the image set into an image encoding network to obtain multiple encoded feature maps;

[0080] The prompting module is used to perform weighted processing on the multiple encoded feature maps through a first neural network to obtain multiple meta-prompt feature maps;

[0081] The enhancement module is used to add and fuse the encoded feature map and the meta-cue feature map to obtain an enhanced feature map;

[0082] A scoring module is used to input the enhanced feature map into a second neural network to obtain similarity scores for multiple enhanced feature maps;

[0083] The classification module is used to classify the query image based on the similarity score.

[0084] According to another aspect of this application, a few-shot image classification device is provided, the few-shot image classification device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the few-shot image classification method as described above, or the few-shot image classification training method as described above.

[0085] According to another aspect of this application, a computer storage medium is provided, wherein at least one instruction, at least one program, code set, or instruction set is stored therein, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the few-sample image classification method as described above, or the few-sample image classification training method as described above.

[0086] According to another aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the few-sample image classification method of any of the above embodiments.

[0087] The beneficial effects of the technical solutions provided in this application include at least the following:

[0088] A method for image acquisition is provided. A first neural network weights the encoded feature map corresponding to each image in an image set to obtain multiple meta-cue feature maps. Based on these meta-cue feature maps, an enhanced feature map corresponding to the query image and an enhanced feature map corresponding to each support image are obtained. A second neural network then obtains a similarity score between each support image and the query image, and classifies the query image based on the similarity score. In this way, the first neural network enhances key features in the query and support images, highlighting image features beneficial for image classification and suppressing interfering features such as background. This improves the accuracy of the second neural network's similarity scoring between the query and support images, thereby improving the accuracy of query image classification. This addresses the low accuracy of query image classification in related technologies and achieves the effect of improving query image classification accuracy. Attached Figure Description

[0089] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0090] Figure 1 is a schematic diagram of an image set provided in an embodiment of this application;

[0091] Figure 2 is a flowchart of a small sample image classification method provided in an embodiment of this application;

[0092] Figure 3 is a flowchart of another small sample image classification method provided in an embodiment of this application;

[0093] Figure 4 is a flowchart of an embodiment of this application for obtaining an encoded feature map;

[0094] Figure 5 is a schematic diagram of the structure of a first neural network provided in this application;

[0095] Figure 6 is a flowchart of obtaining an initial feature map according to an embodiment of this application;

[0096] Figure 7 is a schematic diagram of a local MLP and a global MLP structure provided in an embodiment of this application;

[0097] Figure 8 is a schematic diagram of a process for obtaining a first enhanced feature map or a second enhanced feature map provided in this application;

[0098] Figure 9 is a schematic diagram of the structure of a second neural network provided in this application;

[0099] Figure 10 is a flowchart illustrating a pre-training process provided in this application;

[0100] Figure 11 is a flowchart illustrating the training process of an image coding network, a first neural network, and a second neural network provided in this application.

[0101] Figure 12 is a network architecture diagram provided in an embodiment of this application;

[0102] Figure 13 is a schematic diagram showing the comparison results between a small sample image classification method provided in the embodiments of this application and other image classification methods;

[0103] Figure 14 is a schematic diagram showing the comparison results between another small-sample image classification method provided in the embodiments of this application and other image classification methods;

[0104] Figure 15 is a visualization of a few-shot learning method provided in an embodiment of this application;

[0105] Figure 16 is a structural block diagram of a small sample image classification device provided in an embodiment of this application.

[0106] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0107] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0108] First, the application scenarios involved in the embodiments of this application will be introduced.

[0109] With the development of computer vision technology, the problem of image classification has received increasing attention. Image classification refers to the process of labeling images of unknown categories. Few-shot image classification methods classify images through few-shot learning. The goal of few-shot learning is to enable a trained neural network to identify the category of a query sample using only a small number of support samples; this small number of samples can be less than or equal to 5.

[0110] Referring to Figure 1, for example, a few-shot image classification network can be trained using multiple images from image set A, and then used to classify multiple images from image set B. The image categories in image set A and image set B can be different. Image set A includes multiple query images a1 of known categories and support images a2 of known categories. Training the few-shot image classification network using image set A enables the network to learn to accurately classify query image a1 based on the similarity score between query image a1 and support images a2. Image set B includes multiple query images b1 of unknown categories and multiple support images b2 of known categories. The similarity score between query image b1 and support images b2 can be obtained using the trained few-shot image classification network to classify query image b1. It is understood that Figure 1 only shows a small number of query images and support images in the image set. The number of query images and support images can be 100, 500, or more, and the image categories included in the image set can be 20, 30, or more. This embodiment of the application does not limit this.

[0111] It should be noted that the application scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. Those skilled in the art will know that the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0112] The implementation environment may include an input terminal, a server, and a display terminal. The server includes a processor and can establish wired or wireless connections with the input terminal and the display terminal to classify images based on the images input by the input terminal and display the image classification results on the display terminal.

[0113] Please refer to Figure 2. The few-sample image classification method provided in this embodiment can be applied to the server in the above-described implementation environment. The method may include the following steps:

[0114] Step 101: Obtain the image set, which includes the query image and multiple supporting images.

[0115] Each image in the image set includes an object image and a background image, and each supporting image corresponds to a category.

[0116] Step 102: Input the image set into the image coding network to obtain multiple coded feature maps.

[0117] The multiple encoded feature maps include a first encoded feature map corresponding to the query image and a second encoded feature map corresponding to each supporting image.

[0118] Step 103: The first neural network is used to weight the multiple encoded feature maps to obtain multiple meta-cue feature maps.

[0119] The multiple meta-cue feature maps include a first-meta-cue feature map corresponding to the first encoded feature map, and a second-meta-cue feature map corresponding to each second encoded feature map. Weighting is used to increase the weight of features representing the target object image in the encoded feature maps.

[0120] Step 104: Add and fuse the encoded feature map and the meta-cue feature map to obtain the enhanced feature map.

[0121] The first encoded feature map and the first meta-cue feature map are added and fused to obtain the first enhanced feature map corresponding to the query image. The second encoded feature map and the second meta-cue feature map are added and fused to obtain the second enhanced feature map corresponding to each supporting image.

[0122] Step 105: Input the enhanced feature maps into the second neural network to obtain similarity scores for multiple enhanced feature maps.

[0123] In this process, the first enhanced feature map is paired with each of the multiple second enhanced feature maps and input into a second neural network to obtain a similarity score between each of the multiple second enhanced feature maps and the first enhanced feature map. The number of similarity scores is the same as the number of support images. The second neural network includes multiple neurons. The influence of features in the first enhanced feature map on the neurons is positively correlated with the feature weights, and the influence of features in the second enhanced feature map on the neurons is also positively correlated with the feature weights.

[0124] Step 106: Classify the query images based on similarity scores.

[0125] In summary, this application provides an image acquisition method. A first neural network weights the encoded feature map corresponding to each image in the image set to obtain multiple meta-cue feature maps. Based on these meta-cue feature maps, an enhanced feature map corresponding to the query image and an enhanced feature map corresponding to each support image are obtained. A second neural network then obtains a similarity score between each support image and the query image, and classifies the query image based on the similarity score. In this way, the first neural network enhances key features in the query and support images, highlighting image features beneficial for image classification and suppressing interfering features such as image background. This improves the accuracy of the second neural network's similarity scoring of the query and support images, thereby improving the accuracy of query image classification. This addresses the low accuracy of query image classification in related technologies and achieves the effect of improving the accuracy of query image classification.

[0126] Please refer to Figure 3. Another small-sample image classification method provided in this embodiment can be applied to the server in the above-described implementation environment. This method may include the following steps:

[0127] Step 201: Obtain the image set, which includes the query image and multiple supporting images.

[0128] Each image in the image set includes both an object image and a background image, and each supporting image corresponds to a category. The target object can be an animal, building, machine, or vehicle, etc.

[0129] The image set may include multiple images, which may include graphics, photographs, graphics or photographs converted from video data streams, etc. For example, the multiple images may be multiple photographs obtained by the terminal device through a camera; or, the multiple images may also be multiple photographs obtained from within the terminal device, such as multiple photographs stored in the terminal device's photo album, or multiple photographs obtained by the terminal device from the cloud.

[0130] In one alternative implementation, after acquiring the image set, multiple images in the image set can be divided. Few-shot learning can acquire transferable knowledge by training the training dataset (Ctrain) and then applying it to the test dataset (Dtest). That is, during the training of the network used for the few-shot image classification method, the image set can be divided into a training dataset and a test dataset. The training dataset can be used to train the network used for the few-shot image classification method, and the test dataset can be used to test the training results of the network used for the few-shot image classification method. The training dataset does not overlap with the test dataset.

[0131] For example, multiple images from the training dataset can be obtained in the following way. Few-shot image classification can be called an N-way K-shot classification task, where N represents the number of classes and K represents the number of images for each class. Each N-way K-shot classification task consists of a support set (S) and a query set (Q). The partitioning process may include: first, randomly selecting N images of different classes from the image set; then, extracting K images from each of the N different classes as the support set (S); and finally, extracting T images from each of the N different classes as the query set (Q). Here, N is an integer greater than or equal to 2, K is an integer greater than or equal to 1, and T is an integer equal to or equal to 1. For example, N is 5, K is 3, and T is 1. The support images in the support set (S) may have annotations. For example, the annotations can be... in, This represents the i-th supporting image. This indicates the category of the supporting image. The query image in the query set (Q) is unlabeled. The goal of the few-sample image classification method is to accurately classify the query image in the query set (Q) into one of the N classes.

[0132] The acquisition method for multiple images in the test dataset can be similar to that for multiple images in the training dataset. Each support image in the support set of the test dataset corresponds to a category; that is, the categories of the multiple support images in the support set of the test dataset are known, while the category of the query image in the support set of the test dataset is unknown. The query image and support image in step 201 can be images from the test dataset, or they can be data of random categories obtained by the user. This embodiment of the application does not limit this.

[0133] Step 202: Input the image set into the image coding network to obtain multiple coding feature maps. The multiple coding feature maps include the first coding feature map corresponding to the query image and the second coding feature map corresponding to each supporting image.

[0134] The query image can be processed using an image coding network to obtain the first coded feature map corresponding to the query image. At the same time, the supporting images can also be processed using an image coding network to obtain the second coded feature map corresponding to the supporting images.

[0135] Please refer to Figure 4. Step 202 may include the following sub-steps:

[0136] Sub-step 2021: Obtain the class code of each image in the image set. The class code is a preset prior value.

[0137] Since the category of the supported image is known, the class encoding of the supported image can be an exact value (X). i[cls] ), where i is less than or equal to N. For example, N is 5, meaning it supports a set of 5 different types of images. The class codes for the 5 different types of images can be X, X, and X, respectively. 1[cls] X 2[cls] X 3[cls] X 4[cls] and X 5[cls] Since the category of the query image is unknown, the category code of the query image can be a preset character, which can include at least one of numbers and any symbols.

[0138] Sub-step 2022: Segment each image in the image set to obtain multiple block images corresponding to each image.

[0139] Each image in an image set can be segmented into multiple blocks, and these blocks can be encoded to reduce the complexity of image processing and improve data processing efficiency. For example, for very large images, computational and storage limitations may be encountered; segmenting the image into multiple smaller blocks can reduce data processing complexity and improve data processing efficiency.

[0140] Sub-step 2023: Input multiple block images into the position encoder to obtain the position code corresponding to each block image.

[0141] The positional encoding is used to identify the location of a block image within its corresponding image. When an image is segmented into multiple blocks for processing, the arrangement of these blocks will change. By obtaining the positional encoding of each block image, the relative positional relationships between the multiple blocks can be determined.

[0142] Sub-step 2024: Input multiple block images into the feature encoder to obtain the block code corresponding to each block image.

[0143] Here, block encoding is a vector representing a block of image. Class encoding, multiple position encoding, and multiple block encoding are encoding feature maps.

[0144] By dividing an image into multiple patches and then converting each patch into a vector, computational complexity can be reduced and the efficiency of feature extraction can be improved.

[0145] Step 203: Input the multiple block codes corresponding to each encoded feature map into the first multilayer perceptron to obtain the initial feature map.

[0146] Referring to Figure 5, the first neural network may include a first multilayer perceptron, a local multilayer perceptron, and a global multilayer perceptron, with the first multilayer perceptron connected to both the local and global multilayer perceptrons. An exemplary first neural network may be called a Meta Prompt Learner (MPL). By weighting multiple encoded feature maps using the first neural network, multiple meta-prompt feature maps can be obtained. These meta-prompt feature maps include a first meta-prompt feature map corresponding to the first encoded feature map, and a second meta-prompt feature map corresponding to each second encoded feature map.

[0147] The weighting process is used to increase the weight of the features representing the target object image in the encoded feature map.

[0148] A multilayer perceptron (MLP), also known as an artificial neural network (ANN), can include an input layer, at least one hidden layer, and an output layer.

[0149] Please refer to Figure 6. Step 203 may include the following sub-steps:

[0150] Sub-step 2031: Perform three linear transformations on each block of the encoded feature map to obtain the query matrix, key matrix and value matrix of each block.

[0151] The query matrix (Q), key matrix (K), and value matrix (V) are collectively referred to as QKV matrices. These three matrices are used in self-attention mechanisms to calculate attention weights. The query matrix generates query vectors, each representing a query for a block of code within the attention mechanism, used to find information related to that block. The key matrix generates key vectors, each representing a key for a block of code within the attention mechanism, used to represent the relationship between that block and other blocks. The value matrix generates value vectors, each representing a value for a block of code within the attention mechanism, used to represent the feature information of that block.

[0152] Sub-step 2032: Multiply and fuse the query matrix and key matrix corresponding to each block encoding to obtain the first attention feature map of each block encoding.

[0153] The dot product of the query matrix and the key matrix corresponding to each block code yields the attention weight matrix for each block code, which is the first attention feature map.

[0154] Sub-step 2033: Multiply and fuse the first attention feature map and value matrix corresponding to each block encoding to obtain the sub-initial feature map of each block encoding. The initial feature map includes multiple sub-initial feature maps of the corresponding multiple block encodings.

[0155] By multiplying the first attention feature map corresponding to each block encoding with the value matrix corresponding to each block encoding, we can obtain the sub-initial feature map corresponding to each block encoding. The dimension of the sub-initial feature map corresponding to each block encoding can be the same as the dimension of the block encoding.

[0156] Multiple block codes corresponding to multiple block images in an image can be input into the first multilayer perceptron to obtain a sub-initial feature map corresponding to each block image. Multiple sub-initial feature maps corresponding to multiple image blocks can form the initial feature map of the image.

[0157] Step 204: Input the initial feature map into the local multilayer perceptron to obtain the local feature map.

[0158] Please refer to Figure 7. A local multilayer perceptron may include, in sequence, a first convolutional layer (Conv1(1,1,0)), a batch normalization layer (BatchNorm2d), a linear rectified function (ReLU), a second convolutional layer (Conv2(1,1,0)), a second batch normalization layer (BatchNorm2d), and a random deactivation layer (Dropout).

[0159] Step 205: Input the initial feature map into the global multilayer perceptron to obtain the global feature map.

[0160] A global multilayer perceptron may include, in sequence, an adaptive pooling layer, a first convolutional layer, a batch normalization layer, a linear rectified function, a second convolutional layer, a second batch normalization layer, and a random deactivation layer.

[0161] A global feature map refers to the overall features of an image. For example, a global feature map can include color features, texture features, and shape features. Global feature maps have the characteristics of good invariance and the ability to intuitively represent images.

[0162] Local feature maps are features extracted from local regions of an image. For example, local feature maps may include features of regions such as edges, corners, lines, and curves.

[0163] Compared with global feature maps, local feature maps have the characteristic of low correlation between features. If some features disappear in a local feature map, the disappearing features have a smaller impact on other features when performing feature detection or feature matching operations on the other features in the local feature map.

[0164] Step 206: Add and fuse the local feature map and the global feature map to obtain the fused feature map.

[0165] Global contextual information is captured by applying average pooling to obtain a global feature map. Then, the local and global feature maps are fused to obtain a fused feature map. This method combines global contextual information with local feature maps, highlighting local image features while avoiding the loss of overall image information.

[0166] Step 207: Process the fused feature map using an activation function to obtain the weight matrix.

[0167] The activation function can include the Sigmoid function, also known as the S-shaped growth curve. Activation functions act as non-linear mappings, limiting the output amplitude of a neural network to a certain range, typically between (-1, 1) or (0, 1). The Sigmoid function can map numbers from (-∞, +∞) to the range (0, 1).

[0168] In one exemplary implementation, a neural network includes multiple neurons, and the connection strength between these neurons is represented by weights. In other words, the weights determine the degree of influence of the input signal during its propagation in the neural network, that is, the weights determine the degree of influence of the input data on each neuron.

[0169] Step 208: Multiply and fuse the weight matrix with the initial feature map to obtain the meta-cue feature map.

[0170] Multiplying and fusing the weight matrix with the initial feature map allows us to associate the weights with each data point in the initial feature map, indicating the importance of that data point in the initial feature map. The resulting meta-cue feature map can be used to prompt the second neural network in subsequent steps to enhance its perception of features with larger weights.

[0171] Step 209: Add and fuse the first encoded feature map and the first meta-hint feature map to obtain the first enhanced feature map corresponding to the query image, and add and fuse the second encoded feature map and the second meta-hint feature map to obtain the second enhanced feature map corresponding to each supporting image.

[0172] Please refer to Figure 8. For each image in the image set (which can be a query image or a supporting image), the first (or second) enhanced feature map can be obtained by following the steps in Figure 8.

[0173] (1) It can segment any image in the image set, that image (x i The image can be either a query image or a supporting image. The segmented image can be represented as follows: in, This represents the nth image block.

[0174] (2) Image (x) can be obtained i Class encoding X i[cls] And the position code of each block image is obtained through position embedding. The position code of multiple block images corresponding to an image can be represented as: in, This represents the positional encoding of the nth image block.

[0175] (3) Input multiple block images into the feature encoder and output the block code corresponding to each block image. The multiple block codes corresponding to one image can be represented as follows: in, This represents the block code of the nth block image.

[0176] (4) Input multiple block codes into the first neural network and output a meta-cue feature map, which may include multiple cue vectors. Since the first neural network processes an image by segmenting it into multiple block images, the resulting cue feature map can be represented as: in, This represents the cue vector corresponding to the nth block of an image.

[0177] Since the class code, multiple position codes, and multiple block codes of an image can constitute the image's coding feature map, the augmented feature map corresponding to an image can be obtained by adding and fusing the coding feature map and the meta-cue feature map corresponding to the image. Therefore, the augmented feature map corresponding to a block image in the image can be represented as: The enhanced feature map corresponding to this image can be represented as: The enhanced feature map can be called a token.

[0178] If the image is the query image, then the first enhanced feature map corresponding to the query image can be represented as follows: If the image is a supporting image, then the second enhanced feature map corresponding to the supporting image can be represented as follows:

[0179] Step 210: Input the first enhanced feature map and the second enhanced feature map into the second multilayer perceptron in pairs to obtain the first feature map corresponding to the first enhanced feature map and the second feature map obtained from the first enhanced feature map.

[0180] The second neural network includes a second multilayer perceptual network and a feature scaling network; the second neural network can be called the Token-Awareness Enhancement Module (TAEM).

[0181] The first enhanced feature map is paired with each of the multiple second enhanced feature maps and input into the second neural network to obtain a similarity score between each of the multiple second enhanced feature maps and the first enhanced feature map.

[0182] The number of similarity scores is the same as the number of supporting images. The second neural network includes multiple neurons. The influence of features in the first enhanced feature map on neurons is positively correlated with the weight of the features. The influence of features in the second enhanced feature map on neurons is positively correlated with the weight of the features.

[0183] The structures of the second multilayer sensing network and the first enhanced sensing network can be similar or the same. Therefore, in step 210, the process of inputting the first enhanced feature map into the second multilayer sensing network to output the first feature map and inputting the second enhanced feature map into the second multilayer sensing network to output the second feature map can be referred to step 203. This application embodiment will not elaborate further on this.

[0184] Understandably, pairwise input means first combining one first augmented feature map (corresponding to the query image) with one of the multiple second augmented feature maps (corresponding to the support images) to form a feature map pair, and then inputting this feature map pair into the second multilayer perceptron. One first augmented feature map can form a feature map pair with each of the multiple second augmented feature maps, and the number of feature map pairs is the same as the number of second augmented feature maps. For example, if there is one first augmented feature map and five second augmented feature maps, then one first augmented feature map and five second augmented feature maps can form five feature map pairs.

[0185] Please refer to Figure 9, which shows the first enhanced feature map Z. q Second Enhanced Feature Map Z s The second layer of the perceptron is input in pairs.

[0186] Step 211: Input the first feature map and the second feature map into the feature scaling network in pairs to obtain the third feature map corresponding to the first feature map and the fourth feature map corresponding to the second feature map.

[0187] The first feature map can be represented as Z. q1 ∈R H×W×C The second feature map can be represented as Z. s1 ∈R H×W×C The first and second feature maps are input into a feature scaling network in pairs, and the first and second feature maps are scaled and transformed through convolution. For example, a 4D tensor can be used to represent the first and second feature maps, where the first feature map is represented by C in the 4D tensor. q ∈R H×W×H×W The second feature map is represented by the 4D tensor as C. s ∈R H×W×H×W The feature scaling network can convert the first feature map into C′ q ∈R H×W×U×V And convert the second feature map to C′ s ∈R H×W×U×V Where H (Height) represents the number of pixels in the vertical dimension of the image, W (Width) represents the number of pixels in the horizontal dimension of the image, C (Channel) represents the number of channels in an image, U represents the number of pixels in the vertical dimension of the image after scaling, and V represents the number of pixels in the horizontal dimension of the image after scaling.

[0188] Step 212: Process the third feature map using the attention function to obtain the first attention mapping value, and process the fourth feature map using the attention function to obtain the second attention mapping value.

[0189] The second neural network can also include a third multilayer perceptron. The third and fourth feature maps can be input into the third multilayer perceptron to output the QKV matrix corresponding to the third feature map and the QKV matrix corresponding to the fourth feature map. The attention function can map a QKV matrix into an output, which can be a vector representing the correlation between the Q matrix and the K matrix. As shown in Figure 9, the first attention mapping value A can be obtained. q Second attention mapping value A s .

[0190] Step 213: Multiply and fuse the first attention mapping value with the first feature map to obtain the second attention feature map, and multiply and fuse the second attention mapping value with the second feature map to obtain the third attention feature map.

[0191] A second attention feature map can be obtained by multiplying and fusing the first attention mapping value with the first feature map, thus preserving important image information from the first feature map and highlighting image features beneficial to classification. Similarly, a third attention feature map can be obtained by multiplying and fusing the second attention mapping value with the second feature map, thus preserving important image information from the first feature map and highlighting image features beneficial to classification. As shown in Figure 9, the second attention feature map can be obtained. and the third attention feature map

[0192] Step 214: Obtain the cosine similarity between the second attention feature map and the third attention feature map. The cosine similarity is a similarity score.

[0193] Cosine similarity is a measure of the difference between two features, calculated using the cosine of the angle between them in a vector space. The closer the cosine value is to 1, the closer the angle is to 0 degrees, meaning the two vectors are more similar. As shown in Figure 9, a similarity score can be obtained between the second and third attention feature maps based on their cosine similarity. For example, the similarity score and cosine similarity satisfy the following formula: Where A(·,·) represents the similarity score.

[0194] Step 215: Classify the query images based on similarity scores.

[0195] The category of the query image can be the supporting image with the highest similarity score among at least two supporting images.

[0196] In summary, this application provides an image acquisition method. A first neural network weights the encoded feature map corresponding to each image in the image set to obtain multiple meta-cue feature maps. Based on these meta-cue feature maps, an enhanced feature map corresponding to the query image and an enhanced feature map corresponding to each support image are obtained. A second neural network then obtains a similarity score between each support image and the query image, and classifies the query image based on the similarity score. In this way, the first neural network enhances key features in the query and support images, highlighting image features beneficial for image classification and suppressing interfering features such as image background. This improves the accuracy of the second neural network's similarity scoring of the query and support images, thereby improving the accuracy of query image classification. This addresses the low accuracy of query image classification in related technologies and achieves the effect of improving the accuracy of query image classification.

[0197] In an optional implementation, the image encoding network, the first neural network, and the second neural network in any of the above embodiments can be pre-trained fusion networks, or the image encoding network can be trained during the implementation steps of the few-shot classification method. It should be noted that the networks used in the embodiments of this application (e.g., the first multilayer perceptron, the local multilayer perceptron, the global multilayer perceptron, the second multilayer perceptron, and the feature scaling network, etc.) are all trained network structures. The embodiments of this application do not limit the training methods of these networks.

[0198] Optionally, the image coding network is pre-trained using masked image modeling. The pre-trained network framework includes a teacher network and a student network. The teacher network is typically a large, pre-trained deep learning model, while the student network is a smaller, lighter model. The purpose of training the student network is to reduce computational resource requirements and improve inference speed while making the student network's performance as close as possible to that of the teacher network.

[0199] Pre-training can use panoramic sample images as input to the teacher network and masked sample images as input to the student network, with the output of the teacher network used as the ground truth. During pre-training, the first loss function (L...) is used... pre The formula is as follows:

[0200] Among them, L pre Let Dtrain be the first loss function value, and Dtrain be the training set, where (x,y)∈Dtrain. train The training set represents the neighborhood to which the sample images belong. i For a random mask, m i It can be 0 or 1. x is a panoramic sample image, which can be divided into n blocks, and the n blocks are represented as... Variable j represents the j-th image block out of n image blocks. This is the teacher network output corresponding to the j-th block image out of n block images. This is the output of the student network corresponding to the j-th block image out of n block images.

[0201] As shown in Figure 10, in this embodiment of the application, the pre-training method for the image coding network may include the following steps:

[0202] Step 301: Obtain panoramic sample images and mask sample images.

[0203] Among them, the masked sample image is an image obtained by masking the panoramic sample image.

[0204] Step 302: Input the masked sample image into the student network to obtain the first encoded feature map.

[0205] Step 303: Input the panoramic image into the teacher network to obtain the second encoded feature map.

[0206] Step 304: Compare the first coding feature map and the second coding feature map to obtain the comparison difference.

[0207] Step 305: If the comparison difference is greater than the preset result, adjust the student network based on the comparison difference and execute the step of inputting the mask sample image into the student network.

[0208] If the output of the student network differs significantly from that of the teacher network, it indicates that the parameters in the student network are not accurate enough. Further adjustments to the parameters can be made through multiple training iterations. That is, step 302 is executed after step 305.

[0209] Step 306: If the comparison difference is less than or equal to the preset result, then the student network is determined as an image coding network.

[0210] If the output of the student network is similar to that of the teacher network, it indicates that the parameters in the student network are relatively accurate, and the training of the student network can be terminated.

[0211] This application embodiment also provides a training method for few-sample image classification. Referring to Figure 11, in this application embodiment, the training process of the image encoding network, the first neural network, and the second neural network may include the following steps:

[0212] Step 401: Obtain the training sample set from the sample set. The training sample set includes the query sample image and multiple support sample images.

[0213] Each query sample image corresponds to a category, and each support sample image corresponds to a category. Both query sample images and support sample images include object images and background images.

[0214] Step 402: Input the training sample set into the image encoding network to be trained to obtain multiple encoded feature maps.

[0215] The multiple encoded feature maps include a first encoded feature map corresponding to the query sample image and a second encoded feature map corresponding to each supporting sample image.

[0216] Step 403: The first neural network to be trained performs weighted processing on multiple encoded feature maps to obtain multiple meta-cue feature maps.

[0217] The weighting process is used to increase the weight of the features representing the target object image in the encoded feature map. The multiple meta-cue feature maps include the first meta-cue feature map corresponding to the first encoded feature map and the second meta-cue feature map corresponding to each second encoded feature map.

[0218] Step 403 includes a first multilayer sensing network, a local multilayer sensing network, and a global multilayer sensing network. During the training process, the first multilayer sensing network, the local multilayer sensing network, and the global multilayer sensing network can be trained simultaneously.

[0219] Step 404: Add and fuse the encoded feature map and the meta-cue feature map to obtain the enhanced feature map.

[0220] Optionally, the first encoded feature map and the first meta-cue feature map are added and fused to obtain the first enhanced feature map corresponding to the query sample image, and the second encoded feature map and the second meta-cue feature map are added and fused to obtain the second enhanced feature map corresponding to each support sample image.

[0221] Step 405: Input the enhanced feature maps into the second neural network to be trained to obtain similarity scores for multiple enhanced feature maps.

[0222] Optionally, the first enhanced feature map and each of the multiple second enhanced feature maps are paired and input into the second neural network to be trained to obtain a similarity score between each of the multiple second enhanced feature maps and the first enhanced feature map.

[0223] The number of similarity scores is the same as the number of supporting sample images. The second neural network includes multiple neurons. The influence of features in the first enhanced feature map on neurons is positively correlated with the weight of the features. The influence of features in the second enhanced feature map on neurons is also positively correlated with the weight of the features.

[0224] Step 405 includes a second multilayer perceptual network and a feature scaling network. During the training process, the second multilayer perceptual network and the feature scaling network can be trained simultaneously.

[0225] Step 406: Based on the similarity score, classify the query sample image to obtain the predicted category of the query sample image.

[0226] Step 407: Compare the predicted category of the query sample image with the category corresponding to the query sample image to obtain the second comparison difference.

[0227] Step 408: If the second comparison difference is greater than the preset result, then adjust the image encoding network to be trained, the first neural network to be trained, and the second neural network to be trained based on the second comparison difference, and execute the step of inputting the training sample set into the image encoding network to obtain multiple encoded feature maps.

[0228] If the predicted category of the obtained query sample image differs significantly from the corresponding category of the query sample image, it indicates that the parameters in the image encoding network, the first neural network, and the second neural network are not accurate enough. Further adjustments to the parameters in the image encoding network, the first neural network, and the second neural network can be made through multiple training iterations. That is, step 401 or 402 is executed after step 408.

[0229] Step 409: If the second comparison difference is less than or equal to the preset result, then the image coding network to be trained, the first neural network to be trained, and the second neural network to be trained are determined as the image coding network, the first neural network, and the second neural network shown.

[0230] If the predicted category of the obtained query sample image is similar to the category corresponding to the query sample image, it indicates that the parameters in the image encoding network, the first neural network, and the second neural network are relatively accurate, and the training of the image encoding network, the first neural network, and the second neural network can be terminated.

[0231] Optionally, the image coding network, the first neural network, and the second neural network can be trained using the query image and support images in the training set as input, with the category of the query image as the ground truth. During training, the formula for the second loss function (Loss) is as follows:

[0232] Among them, L metric The second loss function value is represented by cos(·,·), where cos(·,·) represents the cosine similarity, and τ is a preset temperature factor. The second enhanced feature map is the support sample image corresponding to category i. This is the first enhanced feature map corresponding to the query sample image of category i, where i is any natural number. The second attention feature map is generated during the processing of the first enhanced feature map by the second neural network to be trained. E represents the third attention feature map generated during the processing of the second enhanced feature map by the second neural network to be trained. S,Q To determine the mathematical expectation of the query sample and the supporting sample, To determine the mathematical expectation of the query sample.

[0233] Optionally, as shown in Figure 12, in related technologies, a prompt needs to be generated for each task. This prompt is text or instruction that provides input to the neural network to guide it in generating a specific output. This application, however, uses a first neural network, namely a meta-prompt learner (MPL), to generate adaptive meta-prompt feature maps based on the task content.

[0234] In one exemplary implementation, the effectiveness of this application can be further illustrated by the following simulation results. The effectiveness of this method was evaluated on seven datasets in this embodiment: miniImageNet dataset, tieredImageNet dataset, FC100 dataset, CIFAR-FS dataset, CUB-200-211 dataset, Stanford Dogs dataset, and Stanford Cars dataset.

[0235] This application employs the FewTURE method to pre-train an image coding network. FewTURE is a method that uses a fully Transformer-based architecture to obtain representative codes, while training only on a small dataset. This application compares two different Transformer architectures: the first is the small single-image-patch ViT architecture (ViT-Small), and the second is a micro-version of the multi-image-patch Swin architecture (Swin-Tiny). Cosine similarity is used to calculate the similarity between support samples and query samples. Stochastic gradient descent (SGD) with a learning rate of 0.1 can be used as the optimizer.

[0236] Please refer to Tables 1 and 2. This application embodiment conducted experiments on four benchmark datasets to test various image classification methods. These four datasets include: Dataset 1 (miniImageNet), Dataset 2 (tieredImageNet), Dataset 1 (CIFAR-FS), and Dataset 4 (FC100). Here, 1-shot refers to a 5-way-1-shot task, meaning that the dataset contains multiple support images across 5 categories, with one support image per category and one query image per category. 5-shot refers to a 5-way-5-shot task, meaning that the dataset contains multiple support images across 5 categories, with 5 support images per category and one query image per category. As shown in Tables 1 and 2, compared to image classification methods (such as KTN, AM3, TRAML, and DeepEMD) that include additional semantic information, the few-shot image classification method (AMPL) in this application embodiment improves the classification accuracy by 4.27% on the miniImageNet and tieredImageNet datasets under the 1-shot setting.

[0237] This application embodiment improves the robustness of the image coding network by pre-training it. After pre-training the image coding network using the ViT-Small architecture, this application embodiment achieves a classification accuracy of 55.34% on the FC100 dataset with a 1-shot setting and 71.61% with a 1-shot setting, which is a significant improvement compared to other image classifications in Tables 1 and 2.

[0238] For example, Table 1 compares the 5-way classification accuracy (%) with other image classification methods on the test sets of the miniimageNet and tieredImageNet datasets with a 95% confidence interval using established evaluation protocols. Middle rows indicate additional semantic information in the class names or descriptions. The symbol "-" indicates that the method was not reported.

[0239] Table 2 uses the established evaluation protocol to compare the 5-way classification accuracy (%) with other previous image classification methods (95% confidence interval) on the test sets of the CIFAR FS and FC100 datasets.

[0240] Table 1

[0241] Table 2

[0242] Please refer to Table 3, which shows the ablation experiment results. In 1-shot experiments on four benchmark datasets (miniImageNet, tieredImageNet, FC100, and CIFAR-FS), the average accuracy of the few-shot classification method in this embodiment improved by 2.3%. To verify the effectiveness of the few-shot classification method in this embodiment, experiments were conducted on the trained few-shot classification method using three different interaction mechanisms: pre-trained image coding network (PM), meta-cue learner (MPL), and image block token-aware module (TAEM). As shown in Table 3, MPL and TAEM improved the average accuracy by 2.3% and 2.5%, respectively, in 1-shot experiments on the four benchmark datasets. Furthermore, the combination of all these mechanisms further enhanced the 1-shot learning accuracy.

[0243] For example, Table 3 uses the established evaluation protocol to conduct an ablation study of 5-way 1-shot classification accuracy (%) on the test set of four benchmark datasets with a 95% confidence interval.

[0244] Table 3

[0245] Please refer to Table 4. This embodiment compares the classification results of the few-shot image classification method using adaptive meta-cues and the few-shot classification method using fixed cues on the CIFAR-FS and TieredImageNet datasets. The fixed-cue few-shot classification method can be obtained by replacing the adaptive meta-cues generated by the meta-cue learner (MPL) in the few-shot image classification method of this embodiment with fixed cues. One-shot and five-shot experiments were conducted on vit-small and swin-tiny datasets. Compared to the optimal results of the few-shot classification method using fixed cues, the few-shot image classification method of this embodiment improves the average accuracy by 0.73% and 0.835% on vit-small (1-shot and 5-shot), respectively. On swin-tiny, the average accuracy improves by 0.65% and 0.16% on swin-tiny (1-shot and 5-shot), respectively. Thus, these experimental results verify the effectiveness of the few-shot image classification method of this embodiment.

[0246] Table 4

[0247] Please refer to Table 5. This embodiment integrates the Meta-Prompt Learner (MPL) into the Visual Prompt Tuning (VPT) method and compares it with the original VPT method on the Animal (CUB), Cars, and Dogs datasets. "↑" indicates an improvement in accuracy. In the experiments, we used the sup-vit16 backbone network to maintain consistency with the VPT method. As shown in Table 5, in the comparison of the three datasets and various different prompts, the average accuracy improved by 3.03% and 2.67% for prompt=50 and prompt=100, respectively. Thus, the experimental results confirm that the Meta-Prompt Learner (MPL) can be a plug-and-play neural network.

[0248] Table 5

[0249] Please refer to Figures 13 and 14. The other method can be the FewTURE method, dataset 1 can be the miniImageNet dataset, and dataset 2 can be the FC100 dataset. Line 1 represents FewTURE vit-small, meaning the backbone network in the FewTURE method is vit-small; line 2 represents AMPL vit-small, meaning the backbone network in the AMPL method is vit-small; line 3 represents FewTURE swin-tiny, meaning the backbone network in the FewTURE method is swin-tiny; line 4 represents AMPL swin-tiny, meaning the backbone network in the AMPL method is swin-tiny. As can be seen from Figures 12 and 13, the small-sample classification method in this embodiment has high accuracy.

[0250] [Corrected according to Rule 91 08.04.2024] Please refer to Figure 15. The baseline is a reference point in machine learning, which can be used to compare the performance and effectiveness of algorithms. Figure 15 can be a visualization of the second or third attention feature map in step 213 of the above embodiment. As can be seen from Figure 15, the perception of object information by the baseline or the network trained in this application can be represented by the image in Figure 15. It can be seen that the object information and background information are intertwined in the visualization of the baseline. In the visualization of this embodiment, the trained network can focus on processing object information, which can enhance the classification accuracy of the few-sample image classification method. Dataset 1 in Figure 15 can be the miniImageNet dataset, dataset 2 can be the FC100 dataset, backbone network 1 is swin-tiny, and backbone network 2 is vit-small.

[0251] Please refer to Figure 16. The small sample image classification device 1400 in this embodiment includes:

[0252] The acquisition module 1410 is used to acquire an image set, which includes a query image and multiple supporting images. Each image in the image set includes an object image and a background image, and each supporting image corresponds to a category.

[0253] The encoding module 1420 is used to input the image set into the image encoding network to obtain multiple encoded feature maps, including a first encoded feature map corresponding to the query image and a second encoded feature map corresponding to each supporting image;

[0254] The prompting module 1430 is used to perform weighted processing on multiple encoded feature maps through a first neural network to obtain multiple meta-prompt feature maps. The weighted processing is used to increase the weight of the features representing the target object image in the encoded feature map. The multiple meta-prompt feature maps include a first meta-prompt feature map corresponding to the first encoded feature map and a second meta-prompt feature map corresponding to each second encoded feature map.

[0255] The enhancement module 1440 is used to add and fuse the first encoded feature map and the first meta-hint feature map to obtain the first enhanced feature map corresponding to the query image, and to add and fuse the second encoded feature map and the second meta-hint feature map to obtain the second enhanced feature map corresponding to each supporting image.

[0256] The scoring module 1450 is used to input the first enhanced feature map and each of the multiple second enhanced feature maps into the second neural network in pairs to obtain the similarity score between each of the multiple second enhanced feature maps and the first enhanced feature map. The number of similarity scores is the same as the number of multiple support images. The second neural network includes multiple neurons. The influence of the features in the first enhanced feature map on the neurons is positively correlated with the weight of the features. The influence of the features in the second enhanced feature map on the neurons is also positively correlated with the weight of the features.

[0257] The classification module 1460 is used to classify query images based on similarity scores.

[0258] Optionally, the first neural network includes a first multilayer perceptron, a local multilayer perceptron, and a global multilayer perceptron, wherein the first multilayer perceptron is connected to the local multilayer perceptron and the global multilayer perceptron, respectively.

[0259] The prompt module includes:

[0260] The perceptual unit is used to input the encoding of multiple blocks corresponding to each encoded feature map into the first multilayer perceptual network to obtain the initial feature map;

[0261] Local units are used to input the initial feature map into a local multilayer perceptron to obtain a local feature map.

[0262] The global unit is used to input the initial feature map into the global multilayer perceptron to obtain the global feature map;

[0263] The first fusion unit is used to add and fuse the local feature map and the global feature map to obtain a fused feature map;

[0264] The weight unit is used to process the fused feature map through an activation function to obtain the weight matrix;

[0265] The second fusion unit is used to multiply and fuse the weight matrix with the initial feature map to obtain the meta-cue feature map.

[0266] Optionally, the sensing unit is used for:

[0267] Each block of the encoded feature map is subjected to three linear transformations to obtain the query matrix, key matrix, and value matrix of each block.

[0268] The query matrix and key matrix corresponding to each block encoding are multiplied and fused to obtain the first attention feature map of each block encoding;

[0269] The first attention feature map and the value matrix corresponding to each block encoding are multiplied and fused to obtain the sub-initial feature map of each block encoding. The initial feature map includes multiple sub-initial feature maps of the corresponding multiple block encodings.

[0270] Optionally, the encoding module includes:

[0271] The class encoding unit is used to obtain the class encoding of each image in the image set. The class encoding is a preset prior value.

[0272] A segmentation unit is used to segment each image in the image set to obtain multiple block images corresponding to each image.

[0273] The position encoding unit is used to input multiple block images into the position encoder to obtain the position code corresponding to each block image. The position code is used to identify the position of the block image in the corresponding image.

[0274] The block coding unit is used to input multiple block images into the feature encoder to obtain the block code corresponding to each block image. The block code is a vector representing the block image.

[0275] Optionally, the second neural network includes a second multilayer perceptual network and a feature scaling network.

[0276] The rating module includes:

[0277] The feature extraction unit is used to input the first enhanced feature map and the second enhanced feature map into the second multilayer perceptron in pairs to obtain the first feature map corresponding to the first enhanced feature map and the second feature map obtained from the first enhanced feature map.

[0278] The feature scaling unit is used to input the first feature map and the second feature map into the feature scaling network in pairs to obtain the third feature map corresponding to the first feature map and the fourth feature map corresponding to the second feature map.

[0279] The attention unit is used to process the third feature map through the attention function to obtain the first attention mapping value, and to process the fourth feature map through the attention function to obtain the second attention mapping value.

[0280] The third fusion unit is used to multiply and fuse the first attention mapping value with the first feature map to obtain the second attention feature map, and multiply and fuse the second attention mapping value with the second feature map to obtain the third attention feature map.

[0281] The similarity unit is used to obtain the cosine similarity between the second attention feature map and the third attention feature map. The cosine similarity is the similarity score.

[0282] In summary, this application provides an image acquisition device that uses a first neural network to weight the encoded feature map corresponding to each image in an image set to obtain multiple meta-cue feature maps. Based on these multiple meta-cue feature maps, an enhanced feature map corresponding to the query image and an enhanced feature map corresponding to each support image are obtained. Then, a second neural network is used to obtain a similarity score between each support image and the query image, and the query image is classified according to the similarity score. In this way, the first neural network can enhance key features in the query image and support images, highlighting image features beneficial to image classification and suppressing interfering features such as image background. This improves the accuracy of the second neural network's similarity scoring of the query image and support images, thereby improving the accuracy of query image classification. This addresses the issue of low accuracy in query image classification in related technologies and achieves the effect of improving the accuracy of query image classification.

[0283] Furthermore, embodiments of this application also provide an electronic device, which includes one or more processors, a camera component, a memory, and a terminal. The memory may include random access memory (RAM) and read-only memory (ROM), and the camera component may be integrated with the terminal. The portion of the image acquisition method described above concerning network training can be applied to a server, while the other portions concerning image classification, excluding network training, can be applied to either a server or a terminal.

[0284] Furthermore, this application embodiment also provides a few-sample image classification device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, code set, or instruction set. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the few-sample image classification method as described in any of the above embodiments.

[0285] Furthermore, this application also provides a computer storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the small sample image classification method as described in any of the above embodiments.

[0286] Furthermore, embodiments of this application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the few-sample image classification method described in any of the above embodiments.

[0287] In this application, the terms "first," "second," "third," and "fourth" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The term "multiple" means two or more, unless otherwise expressly defined.

[0288] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0289] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0290] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0291] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A small sample image classification method, characterized in that, The method includes: Obtain an image set, which includes a query image and multiple supporting images; The image set is input into an image coding network to obtain multiple coded feature maps; The multiple encoded feature maps are weighted by the first neural network to obtain multiple meta-cue feature maps; The encoded feature map and the meta-cue feature map are added and fused to obtain an enhanced feature map; The enhanced feature maps are input into a second neural network to obtain similarity scores for multiple enhanced feature maps; The query image is classified based on the similarity score.

2. The method of claim 1, wherein, The step of inputting the image set into an image coding network to obtain multiple coded feature maps includes: Obtain the class code of each image in the image set, where the class code is a preset prior value; Each image in the image set is segmented to obtain multiple block images corresponding to each image; The plurality of block images are input into a position encoder to obtain a position code corresponding to each block image. The position code is used to identify the position of the block image in the corresponding image. The plurality of block images are input into a feature encoder to obtain a block code corresponding to each block image. The block code is a vector representing the block image. The class code, the plurality of position codes, and the plurality of block codes are the encoded feature map.

3. The method of claim 2, wherein, The first neural network includes a first multilayer perceptron, a local multilayer perceptron, and a global multilayer perceptron, wherein the first multilayer perceptron is connected to the local multilayer perceptron and the global multilayer perceptron, respectively. The weighted processing of the multiple encoded feature maps using a first neural network yields multiple meta-cue feature maps, including: Each of the encoded feature maps is input into the first multilayer perceptron to obtain an initial feature map; The initial feature map is input into the local multilayer perceptron to obtain a local feature map; The initial feature map is input into the global multilayer perceptron to obtain the global feature map; The local feature map and the global feature map are added and fused to obtain a fused feature map; The fused feature map is processed by an activation function to obtain a weight matrix; The weight matrix is ​​multiplied and fused with the initial feature map to obtain the meta-cue feature map.

4. The method of claim 3, wherein, The step of inputting the encoded blocks corresponding to each of the encoded feature maps into the first multilayer perceptron to obtain the initial feature map includes: Each block of the encoded feature map is subjected to three linear transformations to obtain the query matrix, key matrix, and value matrix of each block. The query matrix and the key matrix corresponding to each block code are multiplied and fused to obtain the first attention feature map of each block code; The first attention feature map corresponding to each block encoding and the value matrix are multiplied and fused to obtain a sub-initial feature map for each block encoding. The initial feature map includes multiple sub-initial feature maps corresponding to multiple block encodings.

5. The method of claim 3, wherein, The local multilayer sensing network includes: The layers are connected in sequence: the first convolutional layer, the batch normalization layer, the linear rectified function, the second convolutional layer, the second batch normalization layer, and the random deactivation layer.

6. The method of claim 3, wherein, The global multilayer sensing network includes: The layers are connected in sequence: an adaptive pooling layer, a first convolutional layer, a batch normalization layer, a linear rectified function, a second convolutional layer, a second batch normalization layer, and a random deactivation layer.

7. The method of claim 1, wherein, Each image in the image set includes an object image and a background image. Each of the supporting images corresponds to a category. The weighting process is used to increase the weight of the features representing the target object image in the encoded feature map. The multiple encoded feature maps include a first encoded feature map corresponding to the query image and a second encoded feature map corresponding to each of the supporting images; The multiple meta-cue feature maps include the first meta-cue feature map corresponding to the first encoded feature map. And the second-ary cue feature map corresponding to each of the second encoded feature maps.

8. The method of claim 7, wherein, The process of adding and fusing the encoded feature map and the meta-cue feature map to obtain an enhanced feature map, and then inputting the enhanced feature map into a second neural network to obtain a similarity score for multiple enhanced feature maps, including: The first encoded feature map and the first meta-hint feature map are added and fused to obtain the first enhanced feature map corresponding to the query image, and the second encoded feature map and the second meta-hint feature map are added and fused to obtain the second enhanced feature map corresponding to each of the supporting images; The first enhanced feature map is paired with each of the multiple second enhanced feature maps and input into a second neural network to obtain a similarity score between each of the multiple second enhanced feature maps and the first enhanced feature map. The number of similarity scores is the same as the number of the multiple support images. The second neural network includes multiple neurons. The influence of the features in the first enhanced feature map on the neurons is positively correlated with the weight of the features. The influence of the features in the second enhanced feature map on the neurons is also positively correlated with the weight of the features.

9. The method of claim 8, wherein, The second neural network includes a second multilayer perceptron and a feature scaling network; The step of pairwise inputting the first enhanced feature map with each of the multiple second enhanced feature maps into the second neural network to obtain a similarity score between each of the multiple second enhanced feature maps and the first enhanced feature map includes: The first enhanced feature map and the second enhanced feature map are input into the second multilayer perceptron in pairs to obtain the first feature map corresponding to the first enhanced feature map and the second feature map obtained from the first enhanced feature map. The first feature map and the second feature map are input into a feature scaling network in pairs to obtain the third feature map corresponding to the first feature map and the fourth feature map corresponding to the second feature map; The third feature map is processed by an attention function to obtain a first attention mapping value, and the fourth feature map is processed by the same attention function to obtain a second attention mapping value. The first attention mapping value is multiplied and fused with the first feature map to obtain the second attention feature map, and the second attention mapping value is multiplied and fused with the second feature map to obtain the third attention feature map; Obtain the cosine similarity between the second attention feature map and the third attention feature map, wherein the cosine similarity is the similarity score.

10. The method of claim 1, wherein, The classification of the query image based on the similarity score includes: The category of the query image is determined by the supporting image that has the highest similarity score to the query image among the at least two supporting images.

11. The method of claim 1, wherein, The image coding network is pre-trained using masked image modeling. The pre-trained network framework includes a teacher network and a student network. The pre-training method includes: Acquire panoramic sample images and mask sample images, wherein the mask sample images are obtained by masking the panoramic sample images; The masked sample image is input into the student network to obtain the first encoded feature map; The panoramic image is input into the teacher network to obtain the second encoded feature map; The first encoded feature map and the second encoded feature map are compared to obtain the comparison difference; If the comparison difference is greater than the preset result, the student network is adjusted based on the comparison difference, and the step of inputting the mask sample image into the student network is executed. If the comparison difference is less than or equal to the preset result, then the student network is determined to be the image coding network.

12. The method of claim 11, wherein, The pre-trained first loss function comprises: wherein L pre is a first loss function value, Dtrain is a training set, (x, y) ∈ D train represents a domain range to which a sample image belongs, Dtrain is a training set, m i is a random mask, m i is 0 or 1, x is a panoramic sample image, the panoramic sample image can be divided into n block images, and the n block images are represented as The variable j represents the jth block image in the n block images, for the jth block image in the n block images, This is the output of the student network corresponding to the j-th block image out of n block images.

13. A small sample image classification training method, characterized in that, The method includes: Obtain a training sample set from the sample set, the training sample set including query sample images and multiple support sample images; The training sample set is input into the image encoding network to be trained to obtain multiple encoded feature maps; The multiple encoded feature maps are weighted by the first neural network to be trained to obtain multiple meta-cue feature maps; The encoded feature map and the meta-cue feature map are added and fused to obtain an enhanced feature map; The strong feature map is input into the second neural network to be trained to obtain similarity scores of multiple enhanced feature maps; Based on the similarity score, the query sample image is classified to obtain the predicted category of the query sample image; The predicted category of the query sample image is compared with the category corresponding to the query sample image to obtain a second comparison difference; If the second comparison difference is greater than the preset result, the image coding network to be trained, the first neural network to be trained, and the second neural network to be trained are adjusted based on the second comparison difference, and the step of inputting the training sample set into the image coding network to obtain multiple coding feature maps is executed. If the second comparison difference is less than or equal to the preset result, then the image coding network to be trained, the first neural network to be trained, and the second neural network to be trained are determined as the image coding network, the first neural network, and the second neural network shown.

14. The method of claim 13, wherein, Each query sample image corresponds to a category, and each support sample image corresponds to a category. Both the query sample image and the support sample image include object images and background images. The weighting process is used to increase the weight of the features representing the target object image in the encoded feature map. The multiple encoded feature maps include a first encoded feature map corresponding to the query sample image and a second encoded feature map corresponding to each of the supporting sample images; The multiple meta-hint feature maps include a first meta-hint feature map corresponding to the first encoded feature map, and a second meta-hint feature map corresponding to each of the second encoded feature maps.

15. The method of claim 14, wherein, The process of adding and fusing the encoded feature map and the meta-cue feature map to obtain an enhanced feature map, and then inputting the enhanced feature map into the second neural network to be trained, in order to obtain similarity scores for multiple enhanced feature maps, includes: The first encoded feature map and the first meta-hint feature map are added and fused to obtain the first enhanced feature map corresponding to the query sample image, and the second encoded feature map and the second meta-hint feature map are added and fused to obtain the second enhanced feature map corresponding to each of the support sample images; The first enhanced feature map is paired with each of the multiple second enhanced feature maps and input into the second neural network to be trained, to obtain a similarity score between each of the multiple second enhanced feature maps and the first enhanced feature map. The number of similarity scores is the same as the number of multiple support sample images. The second neural network includes multiple neurons. The influence of the features in the first enhanced feature map on the neurons is positively correlated with the weight of the features. The influence of the features in the second enhanced feature map on the neurons is also positively correlated with the weight of the features.

16. The method of claim 13, wherein, training the image encoding network, the first neural network, and the second neural network by a second loss function, the second loss function being: wherein L metric is a second loss function value, cos(·,·) represents a cosine similarity, and τ is a preset temperature factor, a second enhanced feature map corresponding to a support sample image of the class i, a first enhanced feature map corresponding to a query sample image of a class i, i being an arbitrary natural number, a second attention feature map generated in a process in which the to-be-trained second neural network processes the first enhanced feature map, For the third attention feature map generated in the process of the to-be-trained second neural network processing the second enhanced feature map, E S,Q For the mathematical expectation of the query sample and the support sample, To determine the mathematical expectation of the query sample.

17. A small sample image classification apparatus characterized by comprising: The device includes: The acquisition module is used to acquire an image set, which includes a query image and multiple supporting images; The encoding module is used to input the image set into an image encoding network to obtain multiple encoded feature maps; The prompting module is used to perform weighted processing on the multiple encoded feature maps through a first neural network to obtain multiple meta-prompt feature maps; The enhancement module is used to add and fuse the encoded feature map and the meta-cue feature map to obtain an enhanced feature map; A scoring module is used to input the enhanced feature map into a second neural network to obtain similarity scores for multiple enhanced feature maps; The classification module is used to classify the query image based on the similarity score.

18. A small sample image classification device, characterized by, The few-shot image classification device includes a processor and a memory. The memory stores at least one instruction, at least one program, a code set, or an instruction set. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the few-shot image classification method as described in any one of claims 1 to 12, or the few-shot image classification training method as described in any one of claims 13 to 16.

19. A computer storage medium, comprising, The computer storage medium stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the small sample image classification method according to any one of claims 1 to 12, or the small sample image classification training method according to any one of claims 13 to 16.

Images