Small sample image classification method and device fused with saliency enhancement

Abstract

The invention discloses a small sample image classification method and device fused with saliency enhancement, and the method comprises the steps: firstly processing an original image, filtering out irrelevant background pixels, enhancing the pixels of an important region of the image, effectively reducing the overfitting problem caused by insufficient data, and improving the generalization capability of a model. Secondly, generating a saliency mask by calculating the gradient of each layer of the picture feature and then performing weighted average on the gradients, then directly weighting the saliency mask to the original picture, and adding the product result to the original picture; by calculating the saliency mask, errors or deviations existing in the model can be found, then parameters of the model are further adjusted to optimize the model, and by multiplying the original image and the saliency mask element by element, the attention of the model to the key area of the image is enhanced, and some features are prevented from being excessively amplified on the basis of keeping the overall image information, so that the image quality is improved. And over-fitting to accidental noise or background features is avoided, and the transparency and interpretability of the model are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and in particular to a small sample image classification method and device integrating significance enhancement. Background Art

[0002] In recent years, deep learning has made significant progress in many computer vision fields such as target detection, target tracking, and semantic segmentation. Traditional deep learning methods often use deep neural networks and require a large amount of labeled data to train the model. However, in the real world, it is difficult to obtain a large number of high-quality labeled samples due to high costs, data confidentiality, security, and other issues. For example, in the medical field, there are few rare cases and a large amount of relevant case data cannot be obtained. Training the model with a small amount of sample data is prone to problems such as overfitting and poor generalization performance. Therefore, how to train the model with a small number of samples so that it can quickly identify new categories has gradually become a hot topic in the current computer vision field.

[0003] Few-shot learning trains models with a small number of labeled samples, reducing the model's dependence on large-scale labeled data, thereby achieving rapid recognition of new types of samples. Few-shot image classification is one of the most valuable and promising tasks in few-shot learning. Therefore, researchers have proposed methods based on metric learning, meta-learning, and model optimization that are suitable for few-shot image classification. At present, methods based on metric learning have attracted much attention from researchers because of their simplicity and efficiency. Among them, the learning method that uses sample local features for measurement performs metric learning on local features between samples, making the model more accurate while increasing the interpretability of the classification results.

[0004] However, compared with large sample image classification, the performance of existing small sample image classification is still unsatisfactory, which greatly limits the practical application of small sample image classification technology. Small sample image classification faces the following problems that need to be solved urgently:

[0005] Highly recognizable feature learning. For large sample image classification, existing deep learning techniques can learn highly recognizable image features by increasing model elasticity and sample size. However, for small sample classification tasks with very few labeled samples, existing deep learning techniques are not applicable. Therefore, how to learn highly recognizable feature representations based on base class data and new class data with very few labeled samples is a problem worth exploring.

[0006] Identify background information or redundant information. When existing small sample image classification methods based on metric learning perform similarity measurement on the support sample and the query sample, there are too many interfering features or irrelevant features, which reduces the classification performance of the model. Therefore, how the model can identify the background information or redundant information of the image and then remove it is a very exploratory problem.

[0007] In addition, the traditional method of calculating the saliency mask of the original image directly multiplies the saliency mask by the sample features. However, this method mainly focuses on the sample feature level and ignores the direct impact of saliency in the image pixel space, resulting in information loss or over-emphasis on specific features. Summary of the invention

[0008] Aiming at the technical problems of high-recognition feature learning and redundant information in small sample image classification, the present invention proposes a small sample image classification method and device integrating saliency enhancement.

[0009] In order to achieve the above object, the present invention provides the following technical solutions:

[0010] The present invention first provides a small sample image classification method fused with significance enhancement, comprising the following steps:

[0011] S1, data preparation, image data includes training set and test set;

[0012] S2, use the base class data to train the model and obtain the optimal classification model; the classification model consists of three parts: embedding module f θ , saliency mask generation module and distance measurement module; where the embedding module f θ Used to extract features from input image data, θ is a learnable parameter; saliency mask generation module For generating saliency masks, it contains a learnable multi-layer perceptron MLP. is a learnable parameter of MLP; the distance measurement module is used to calculate the cosine distance between the supporting sample prototype and the query sample feature after the saliency mask is enhanced, and classify the query sample and optimize the model according to the distance of the cosine distance; step S2 includes the following steps:

[0013] S21. Using embedded module f θ Extract sample features and enhance pixels in important areas of the image;

[0014] S22. Generate module using saliency mask Calculate the gradient of each layer of sample features, and then weight the average of these gradients to generate a saliency mask. Then, weight the saliency mask directly onto the original image, and add the product result to the original image to enhance the saliency of the image pixels.

[0015] S23, using the distance measurement module to find the prototype of the support sample, calculate the cosine distance, and optimize the model to obtain the optimal classification model;

[0016] S3, classify the test image using the optimal classification model.

[0017] Further, step S1 includes:

[0018] Divide the data into training sets and test set Two parts, among which L train and L test are the categories of the training set and the test set respectively, and the category spaces of these two parts are mutually exclusive, that is, N train is the total number of samples in the training set, N test is the total number of samples in the test set, x i ∈R 3×H×W is the image sample, H is the height of the original image, W is the width of the original image, and y i is the category of the sample, i is the index of the sample;

[0019] From D train Randomly select C categories, randomly select M+K samples from each category, of which K samples are used as support samples S e , M samples are used as query samples Q e , S e and Q e Constitute a task T e , where e = 1, 2, ..., E is the index of the task, and E is the total number of tasks sampled on the base class data; for D test Have a mission in is the total number of tasks for sampling the new class data; for a task T of the base class data e ={S e ,Q e},in represents the kth support sample of the cth class, represents the jth query sample, c∈[1,2,…,C], k∈[1,2,…,K], j∈[1,2,…,C×M]; Represents the one-hot label of the k-th support sample of the c-th class. The one-hot label is a 1×C vector. If the sample belongs to the c-th class, the value of the c-th position is 1, and the rest of the positions are 0.

[0020] Further, step S21 includes:

[0021] The image sample xi Input to the embedding module f θ Extract features from d, h, and w are the dimension, length, and width of the feature respectively. A d×1×1 vector is called a local feature, and each sample has a total of h×w local features.

[0022] Furthermore, the embedded module f in step S21 θ It contains four layers of convolution blocks, each of which contains the following parts: a convolution layer with a 3×3 convolution kernel with an output dimension of 64, a stride of 1, and an input padding of 1; a 2×2 maximum pooling layer; a batch normalization layer and a nonlinear activation function ReLU.

[0023] Further, step S22 includes:

[0024] The sample features Flattening operation is performed to obtain i=1,2,…,C×(M+K); then, Input into the multi-layer perceptron MLP and output the predicted value

[0025] Assume that the true category of each image is y i , the loss function of each sample is:

[0026]

[0027] Then calculate the gradient of the image And average the obtained image gradients by channel to get the weights:

[0028]

[0029] Suppose the weighted image sample m i =weight i ·x i Then, use the ReLU function to weight i Set the negative values ​​to zero and keep only the positive values; and set the weight weight i Normalization is performed to scale the pixel value range of the image to [0, 1], and further obtain the value that is consistent with the input image x i Saliency masks of the same size:

[0030]

[0031] The saliency mask i Multiply element-wise to the original image x i Then add the multiplication result to the original image, and the image after saliency enhancement is: ⊙ represents element-wise product.

[0032] Furthermore, in step S22, the multi-layer perceptron MLP is composed of two identical modules, wherein each module is composed of FC, ReLU, and BatchNorm in sequence.

[0033] Further, step S23 includes:

[0034] The image after saliency enhancement obtained in step S22 For enhanced support images, The enhanced query image is then input into the embedding module f θ Extract the corresponding supporting features of the image and query features Compute the class prototype for each support example:

[0035]

[0036] Among them, c represents the cth class, there are C classes in total, k represents the kth sample of each class in a task, there are K samples in total;

[0037] The class prototype P of the supporting sample obtained in step S23 is calculated using the cosine distance formula c The cosine similarity between the query sample features after and significance enhancement:

[0038]

[0039] Represents the cosine similarity between the j-th query sample and the c-th class support prototype;

[0040] The probability is calculated using the following formula:

[0041]

[0042] represents the probability of assigning the j-th query sample to the c-th class sample;

[0043] Use the obtained probability and the true label of the query sample Calculate the cross entropy loss:

[0044]

[0045] Sum the loss function in step S22:

[0046]

[0047] Among them, CELoss is the cross entropy loss function, and the final loss function is: loss = loss 1 +loss 2 .

[0048] Further, step S23 uses the SGD optimization algorithm to optimize the embedding module f θ and saliency mask generation module The parameters θ and The specific calculation of the SGD optimization algorithm is:

[0049] Where η is the learning rate, which is initially set to 0.01. is θ and The gradient of is the result after optimization; then the number of tasks e=e+1, resample the task T in the base class data e , repeat steps S21-S23 until e=E; in the iteration of step S23, the model with the smallest loss is taken as the optimal classification model.

[0050] Further, step S3 includes:

[0051] S31. Construction of new task types

[0052] A task for new types of data in represents the kth support sample of the cth class, represents the jth query sample, c∈[1,2,…,C], k∈[1,2,…,K], j∈[1,2,…,C×M]; represents the one-hot label of the k-th support sample of the c-th class, If the sample belongs to the cth class, the cth position value is 1, and the remaining position values ​​are 0;

[0053] S32. Classify the query samples in the current new class task

[0054] The support sample in step S31 and query sample After steps S21-S23, the distance of the j-th query sample to the c-th class is obtained. The category corresponding to the maximum value of the distance between the j-th query sample and all C categories is taken as the predicted category of the query sample, that is, For all query samples j∈[1,2,…,C×M] All are classified;

[0055] S33. Classify the query samples in the next task

[0056] Next task number Repeat steps S31-S32 until

[0057] On the other hand, the present invention also provides a small sample image classification device integrating significance enhancement, which is used to implement the above method, and includes the following functional modules:

[0058] Embed module f θ , used to extract features from input image data, θ is a learnable parameter;

[0059] Saliency Mask Generation Module For generating saliency masks, it contains a learnable multi-layer perceptron MLP. are the learnable parameters of MLP;

[0060] The distance measurement module is used to calculate the cosine distance between the supporting sample prototype and the query sample feature after the saliency mask is enhanced, and classify the query sample and optimize the model according to the cosine distance.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] 1. The small sample image classification method and device with fusion significance enhancement of the present invention first processes the original image to filter out irrelevant background pixels and enhance pixels in important areas of the image, effectively reducing the overfitting problem caused by insufficient data and improving the generalization ability of the model.

[0063] 2. The present invention proposes a new saliency enhancement method, which generates a saliency mask by calculating the gradient of each layer of image features, and then weighting and averaging these gradients. The saliency mask is then directly weighted onto the original image, and the product is added to the original image. By calculating the saliency mask, errors or deviations in the model can be found, and the parameters of the model can be further adjusted to optimize the model. By multiplying the original image by the saliency mask element by element, the model's attention to the key areas of the image is enhanced, and on the basis of retaining the overall image information, over-amplification of certain features is avoided, and overfitting to accidental noise or background features is avoided.

[0064] 3. The present invention ensures that while retaining the global information of the original image, the key areas for model discrimination are highlighted, and the image information can be directly adjusted in the pixel space, thereby more intuitively reflecting the model's attention to important areas. This method acts directly on the original image and can be easily applied to different network architectures and tasks. It has high versatility and is easy to integrate into existing small sample image classification models. In addition, it is very useful for understanding and explaining the behavior of the model. By displaying the areas or important features that the model pays attention to, it helps explain the model's decision-making process and improves the transparency and interpretability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0066] Figure 1 A flow chart of a small sample image classification method integrating significance enhancement provided in an embodiment of the present invention.

[0067] Figure 2 A small sample image classification model with fusion significance enhancement is provided in an embodiment of the present invention.

[0068] Figure 3 A flow chart of a saliency mask generation module provided in an embodiment of the present invention.

[0069] Figure 4 A schematic diagram of the functional modules of a small sample image classification device fused with significance enhancement provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0070] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. The embodiments of the present invention and all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0071] The present invention also provides a small sample image classification method integrating significance enhancement, the process is as follows Figure 1 As shown, the following steps are included.

[0072] S1, data preparation, image data includes training set and test set;

[0073] S2, use the base class data to train the model and obtain the optimal classification model;

[0074] S3, classify the test image using the optimal classification model.

[0075] Wherein, step S1 comprises:

[0076] S11, data preparation.

[0077] Divide the data into training sets and test set Two parts, among which L train and L test are the categories of the training set and the test set respectively, and the category spaces of these two parts are mutually exclusive, that is, N train is the total number of samples in the training set, N test is the total number of samples in the test set, x i ∈R 3×H×W is the image sample, H is the height of the original image, W is the width of the original image, and y i is the category of the sample, i is the index of the sample; in this invention, we will train As the base class data training model, D test The small sample image classification performance of the new class data evaluation model is further organized into a C-way K-shot classification task.

[0078] Specifically, the present invention is from D train Randomly select C categories, randomly select M+K samples from each category, of which K samples are used as support samples S e , M samples are used as query samples Q e , S e and Q e Constitute a task T e , where e = 1, 2, ..., E is the index of the task, and E is the total number of tasks sampled on the base class data; for D test Have a mission in is the total number of tasks for sampling the new class data; for a task T of the base class data e ={S e ,Q e},in represents the kth support sample of the cth class, represents the jth query sample, c∈[1,2,…,C], k∈[1,2,…,K], j∈[1,2,…,C×M]; Represents the one-hot label of the k-th support sample of the c-th class. The one-hot label is a 1×C vector. If the sample belongs to the c-th class, the value of the c-th position is 1, and the rest of the positions are 0.

[0079] In step S2, the small sample image classification model with fusion saliency enhancement consists of three parts: the embedding module f θ , saliency mask generation module and distance measurement module; where the embedding module f θ Used to extract features from input image data, and θ is a learnable parameter.

[0080] Saliency Mask Generation Module For generating saliency masks, it contains a learnable multi-layer perceptron MLP. It is a learnable parameter of MLP; through forward propagation, gradient calculation, loss calculation, etc., the saliency mask of the image is generated. The operation of enhancing the saliency mask further helps the model to filter out unimportant features that are overly dependent in the classification process, thereby optimizing feature extraction. The saliency enhancement module performs corresponding operations on the saliency mask and the original image, which can guide the model to focus on features that have a greater impact on the model's classification performance and improve the interpretability of the model.

[0081] The distance measurement module is used to calculate the cosine distance between the supporting sample prototype and the query sample feature after the saliency mask is enhanced, and classify the query sample and optimize the model according to the cosine distance.

[0082] like Figure 2 As shown, step S2 includes the following steps:

[0083] S21. Using embedded module f θ Extract sample features and enhance pixels in important areas of the image;

[0084] S22. Generate module using saliency mask Calculate the gradient of each layer of sample features, and then weight the average of these gradients to generate a saliency mask. Then, weight the saliency mask directly onto the original image, and add the product result to the original image to enhance the saliency of the image pixels.

[0085] S23. Use the distance measurement module to find the prototype of the supporting samples, calculate the cosine distance, and optimize the model to obtain the optimal classification model.

[0086] Specifically, step S21 includes:

[0087] The image sample x i Input to the embedding module f θ Extract features from d, h, and w are the dimension, length, and width of the feature respectively. A d×1×1 vector is called a local feature, and each sample has a total of h×w local features.

[0088] Embed module f θ A commonly used convolutional neural network is used, which contains four layers of convolution blocks. Each convolution block contains the following parts: a convolution layer with a 3×3 convolution kernel with an output dimension of 64, a step size of 1, and an input padding of 1; a 2×2 maximum pooling layer; a batch normalization layer (BatchNorm) and a nonlinear activation function ReLU.

[0089] Step S22 includes:

[0090] The sample features Flattening operation is performed to obtain i=1,2,…,C×(M+K); then, Input into the multi-layer perceptron MLP, which consists of two identical modules, each of which is composed of FC, ReLU, and BatchNorm in turn, and outputs the predicted value

[0091] Assume that the true category of each image is y i , define the loss function of each sample as:

[0092]

[0093] Then, calculate the gradient of the image And average the obtained image gradients by channel to get the weights:

[0094]

[0095] Suppose the weighted image sample m i =weight i ·x i Then, use the ReLU function to weight i Set the negative values ​​to zero and keep only the positive values; and set the weight weight i Normalization is performed to scale the pixel value range of the image to [0, 1], and further obtain the value that is consistent with the input image x i Saliency masks of the same size:

[0096]

[0097] The saliency mask i Multiply element-wise to the original image x i Then add the multiplication result to the original image, and the image after saliency enhancement is: ⊙ represents the element-wise product (Hadamard product).

[0098] Traditional methods usually directly multiply the saliency mask with the feature map, while the saliency enhancement method proposed in this invention creatively multiplies the saliency mask directly with the original image, and then adds the product result to the original image, which not only screens out the key feature areas in the classification task, but also further enhances these key areas, and can directly adjust the image information in the pixel space, thereby more intuitively reflecting the model's attention to the important areas of the image. This method significantly improves the ability of the small sample image classification model to distinguish the target category, enhances the classification performance of the model, and improves the model interpretability.

[0099] The saliency mask generation module process is as follows Figure 3 shown.

[0100] Step S23 includes:

[0101] The image after saliency enhancement obtained in step S22 For enhanced support images, The enhanced query image is then input into the embedding module f θ Extract the corresponding supporting features of the image and query features Compute the class prototype for each support example:

[0102]

[0103] Among them, c represents the cth class, there are C classes in total, k represents the kth sample of each class in a task, there are K samples in total;

[0104] The class prototype P of the supporting sample obtained in step S23 is calculated using the cosine distance formula c The cosine similarity between the query sample features after and significance enhancement:

[0105]

[0106] Represents the cosine similarity between the j-th query sample and the c-th class support prototype;

[0107] The probability is calculated using the following formula:

[0108]

[0109] represents the probability of assigning the j-th query sample to the c-th class sample;

[0110] Use the obtained probability and the true label of the query sample Calculate the cross entropy loss:

[0111]

[0112] Sum the loss function in step S22:

[0113]

[0114] Among them, CELoss is the cross entropy loss function, and the final loss function is: loss = loss 1 +loss 2 .

[0115] Step S23 uses the SGD optimization algorithm to optimize the embedding module f θ and saliency mask generation module The parameters θ and The specific calculation of the SGD optimization algorithm is: Where η is the learning rate, which is initially set to 0.01. is θ and The gradient of This is the optimized result.

[0116] Then the number of tasks e=e+1, resample the task T in the base class data e , repeat steps S21-S23 until e=E.

[0117] In the iteration of step S23, the model with the smallest loss is taken as the optimal classification model.

[0118] Step S3 includes:

[0119] S31. Construction of new task types

[0120] A task for new types of data in represents the kth support sample of the cth class, represents the jth query sample, c∈[1,2,…,C], k∈[1,2,…,K], j∈[1,2,…,C×M]; represents the one-hot label of the k-th support sample of the c-th class, If the sample belongs to the cth class, the cth position value is 1, and the remaining position values ​​are 0;

[0121] S32. Classify the query samples in the current new class task

[0122] The support sample in step S31 and query sample After steps S21-S23, the distance of the j-th query sample to the c-th class is obtained. The category corresponding to the maximum value of the distance between the j-th query sample and all C categories is taken as the predicted category of the query sample, that is, For all query samples j∈[1,2,…,C×M] All are classified;

[0123] S33. Classify the query samples in the next task

[0124] Next task number Repeat steps S31-S32 until

[0125] On the other hand, the present invention also provides a small sample image classification device fused with significance enhancement, which is used to implement the above method, such as Figure 4 As shown, it includes the following functional modules:

[0126] Embed module f θ , used to extract features from input image data, θ is a learnable parameter;

[0127] Saliency Mask Generation Module For generating saliency masks, it contains a learnable multi-layer perceptron MLP. are the learnable parameters of MLP;

[0128] The distance measurement module is used to calculate the cosine distance between the supporting sample prototype and the query sample feature after the saliency mask is enhanced, and classify the query sample and optimize the model according to the cosine distance.

[0129] In summary, the present invention first filters the background information of the original image and strengthens the important areas of the image. This method makes the model more interpretable. The present invention directly acts on the original image, and can be easily applied to different network architectures and models. It has high versatility and can be integrated into visual tasks in different fields. In addition, the present invention is an end-to-end classification model that does not require any pre-training or multi-stage training. The training stage adopts a meta-training method that matches the new class of tasks, which effectively reduces the time and resource overhead required for training.

[0130] The above is an explanation of the proposed small sample image classification method and device for fusion significance enhancement and the specific implementation of the method in conjunction with the accompanying drawings. Through the description of the above implementation, those skilled in the art can clearly understand the implementation of the method and device.

[0131] The algorithms and displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general purpose systems may also be used together with the teachings based thereon. Based on the above description, it is apparent that the structure required for constructing such systems is not intended for any particular programming language. In addition, disclosed herein is not intended for any particular programming language. However, it should be understood that various programming languages ​​may be utilized to implement the disclosed herein described, and the description of the specific languages ​​above is intended to disclose the best implementation method disclosed herein.

[0132] Similarly, it should be understood that in order to make this document as concise as possible and to aid in understanding one or more of the various disclosed aspects, in the above description of the exemplary embodiments disclosed herein, the various features disclosed herein are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following schematic diagram: the claims disclosed herein that are claimed to be protected have more features than the features explicitly stated in each claim. More specifically, as reflected in the claims below, the disclosed aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment example disclosed in the present invention.

[0133] The above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the aforementioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the aforementioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical specialties therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A small sample image classification method integrating saliency enhancement, characterized in that: The following steps are involved: S1, data preparation, image data includes training set and test set; S2, use the base class data to train the model and obtain the optimal classification model; The classification model consists of three parts: embedding module f θ , saliency mask generation module and distance measurement module; where the embedding module f θ Used to extract features from input image data, θ is a learnable parameter; saliency mask generation module For generating saliency masks, it contains a learnable multi-layer perceptron MLP. is a learnable parameter of MLP; the distance measurement module is used to calculate the cosine distance between the supporting sample prototype and the query sample feature after the saliency mask is enhanced, and classify the query sample and optimize the model according to the distance of the cosine distance; step S2 includes the following steps: S21. Using embedded module f θ Extract sample features and enhance pixels in important areas of the image; S22. Generate module using saliency mask Calculate the gradient of each layer of sample features, and then weight the average of these gradients to generate a saliency mask. Then, weight the saliency mask directly onto the original image, and add the product result to the original image to enhance the saliency of the image pixels. S23, using the distance measurement module to find the prototype of the support sample, calculate the cosine distance, and optimize the model to obtain the optimal classification model; S3, classify the test image using the optimal classification model.

2. The small sample image classification method with fusion significance enhancement according to claim 1, characterized in that: Step S1 includes: Divide the data into training sets and test set Two parts, among which L train and L test are the categories of the training set and the test set respectively, and the category spaces of these two parts are mutually exclusive, that is, N train is the total number of samples in the training set, N test is the total number of samples in the test set, x i ∈R 3×H×W is the image sample, H is the height of the original image, W is the width of the original image, and y i is the category of the sample, i is the index of the sample; From D train Randomly select C categories, randomly select M+K samples from each category, of which K samples are used as support samples S e , M samples are used as query samples Q e , S e and Q e Constitute a task T e , where e = 1, 2, ..., E is the index of the task, and E is the total number of tasks sampled on the base class data; for D test Have a mission in is the total number of tasks for sampling the new class data; for a task T of the base class data e ={S e ,Q e },in represents the kth support sample of the cth class, represents the jth query sample, c∈[1,2,…,C], k∈[1,2,…,K], j∈[1,2,…,C×M]; Represents the one-hot label of the k-th support sample of the c-th class. The one-hot label is a 1×C vector. If the sample belongs to the c-th class, the value of the c-th position is 1, and the rest of the positions are 0.

3. The small sample image classification method with fusion significance enhancement according to claim 2, characterized in that: Step S21 includes: The image sample x i Input to the embedding module f θ Extract features from d, h, and w are the dimension, length, and width of the feature respectively. A d×1×1 vector is called a local feature, and each sample has a total of h×w local features.

4. The small sample image classification method with fusion significance enhancement according to claim 3 is characterized in that: The embedded module f in step S21 θ Contains four layers of convolutional blocks, each of which contains the following parts: a convolutional layer with a 3×3 convolution kernel with an output dimension of 64, a stride of 1, and an input padding of 1; a 2×2 maximum pooling layer; A batch normalization layer and a non-linear activation function ReLU.

5. The small sample image classification method with fusion significance enhancement according to claim 3 is characterized in that: Step S22 includes: The sample feature x i Flattening operation is performed to obtain i=1,2,…,C×(M+K); then, Input into the multi-layer perceptron MLP and output the predicted value Assume that the true category of each image is y i , the loss function of each sample is: Then calculate the gradient of the image And average the obtained image gradients by channel to get the weights: Suppose the weighted image sample m i =weight i ·x i Then, use the ReLU function to weight i Set the negative values ​​to zero and keep only the positive values; and set the weight weight i Normalization is performed to scale the pixel value range of the image to [0,1], and further obtain the value that is consistent with the input image x i Saliency masks of the same size: The saliency mask i Multiply element-wise to the original image x i Then add the multiplication result to the original image, and the image after saliency enhancement is: ⊙ represents element-wise product.

6. The small sample image classification method with fusion significance enhancement according to claim 5, characterized in that: In step S22, the multi-layer perceptron MLP is composed of two identical modules, each of which is composed of FC, ReLU, and BatchNorm in sequence.

7. The small sample image classification method with fusion significance enhancement according to claim 5, characterized in that: Step S23 includes: The image after saliency enhancement obtained in step S22 For enhanced support images, The enhanced query image is then input into the embedding module f θ Extract the corresponding supporting features of the image and query features Compute the class prototype for each support example: Among them, c represents the cth class, there are C classes in total, k represents the kth sample of each class in a task, there are K samples in total; The class prototype P of the supporting sample obtained in step S23 is calculated using the cosine distance formula c The cosine similarity between the query sample features after and significance enhancement: Represents the cosine similarity between the j-th query sample and the c-th class support prototype; The probability is calculated using the following formula: represents the probability of assigning the j-th query sample to the c-th class sample; Use the obtained probability and the true label of the query sample Calculate the cross entropy loss: Sum the loss function in step S22: Among them, CELoss is the cross entropy loss function, and the final loss function is: loss = loss 1 +loss 2 .

8. The small sample image classification method with fusion significance enhancement according to claim 7, characterized in that: Step S23 uses the SGD optimization algorithm to optimize the embedding module f θ and saliency mask generation module The parameters θ and The specific calculation of the SGD optimization algorithm is: * , Where η is the learning rate, which is initially set to 0.

01. is θ and The gradient of θ * , is the result after optimization; then the number of tasks e=e+1, resample the task T in the base class data e , repeat steps S21-S23 until e=E; in the iteration of step S23, the model with the smallest loss is taken as the optimal classification model.

9. The small sample image classification method with fusion significance enhancement according to claim 1, characterized in that: Step S3 includes: S31. Construction of new task types A task for new types of data in represents the kth support sample of the cth class, represents the jth query sample, c∈[1,2,…,C], k∈[1,2,…,K], j∈[1,2,…,C×M]; represents the one-hot label of the k-th support sample of the c-th class, If the sample belongs to the cth class, the cth position value is 1, and the remaining position values ​​are 0; S32. Classify the query samples in the current new class task The support sample in step S31 and query sample After steps S21-S23, the distance of the j-th query sample to the c-th class is obtained. The category corresponding to the maximum value of the distance between the j-th query sample and all C categories is taken as the predicted category of the query sample, that is, For all query samples j∈[1,2,…,C×M] All are classified; S33. Classify the query samples in the next task Next task number Repeat steps S31-S32 until 10. A small sample image classification device integrating saliency enhancement, characterized in that: The method for implementing any one of claims 1 to 9 comprises the following functional modules: Embed module f θ , used to extract features from input image data, θ is a learnable parameter; Saliency Mask Generation Module For generating saliency masks, it contains a learnable multi-layer perceptron MLP. are the learnable parameters of MLP; The distance measurement module is used to calculate the cosine distance between the supporting sample prototype and the query sample feature after the saliency mask is enhanced, and classify the query sample and optimize the model according to the cosine distance.

Images