Small sample image classification method, system and device based on coordinate attention and bdc metric and medium

Abstract

The application discloses a kind of based on coordinate attention and BDC metric small sample image classification method, system, device and medium, comprising: image acquisition and the image collected is divided, obtains training set, verification set and test set;Depth model framework is constructed;Based on training set, depth model is trained, when training frequency reaches maximum training frequency threshold, the performance of depth model on verification set reaches preset requirement or no longer significantly improves, stop training, obtain the optimized depth model;Based on the optimized depth model and the way of meta-learning N-way K-shot, test set is classified and predicted, and classification accuracy is evaluated.The application can help model to better complete small sample image classification task by introducing the coordinate attention of space information and position information, and can effectively improve classification precision.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and relates to a small sample image classification method, system and device based on coordinate attention and BDC measurement and a medium. BACKGROUND

[0002] Deep learning has made great progress in image classification tasks with large-scale data. However, the training of a deep learning model requires a large amount of labeled data, which is difficult to obtain in practice. In the case of a small number of samples, the model is prone to overfitting. However, in special fields such as medicine and military, image classification faces the problem of small samples due to the difficulty in obtaining data and labeling. Therefore, research on image classification technology for small samples has become an important research direction.

[0003] Current small sample image classification technology mainly uses meta-learning strategies to divide data, uses deep neural networks to extract features from input images, and finally uses metric learning to complete classification. For example, the SENet attention mechanism is introduced to establish the correlation between feature channels, and finally the similarity between images and categories is calculated to realize small sample image classification. However, the above method does not consider the position information that is crucial to the selective attention of the generated space, resulting in limited feature extraction capability of the model. Therefore, how to improve the feature expression capability of the model and thus improve the classification performance is an urgent technical problem to be solved in the field. SUMMARY

[0004] The purpose of the present application is to solve the problem that the position information crucial to the selective attention of the generated space is not considered in the prior art when performing small sample image classification, resulting in limited feature extraction capability of the model. A small sample image classification method, system, device and medium based on coordinate attention and BDC measurement are provided.

[0005] To achieve the above purpose, the following technical solutions are adopted:

[0006] The small sample image classification method based on coordinate attention and BDC measurement comprises:

[0007] Collecting images and dividing the collected images to obtain a training set, a validation set and a test set;

[0008] Constructing a deep model framework;

[0009] Training the deep model based on the training set, stopping training when the number of training reaches a maximum training threshold, the performance of the deep model on the validation set reaches a preset requirement or no longer improves significantly, and obtaining an optimized deep model;

[0010] The test set is classified and predicted in an optimization-based deep model and meta-learning N-way K-shot manner, and the classification accuracy is evaluated.

[0011] Further improvements of the present application are:

[0012] Further, images are collected and the collected images are divided, including: obtaining public small sample data sets miniImageNet and Omniglot and small sample fine-grained data set CUB-200, and tire pattern data set CIIP-TPID applied in the field of public security; the miniImageNet data set, the Omniglot data set, the CUB-200 data set and the CIIP-TPID data set are respectively divided into training set, test set and validation set.

[0013] Further, the deep model framework comprises: an embedding module, a coordinate attention module, a BDC metric module and a Softmax layer.

[0014] The embedding module extracts features from the input training set; the coordinate attention module performs global average pooling along the horizontal and vertical directions thereof respectively, decomposes the input feature map X into one-dimensional features in the horizontal and vertical directions, and obtains the horizontal and vertical position information of the input feature map; the horizontal and vertical position information of the input feature map is connected together, and a 1x1 convolution is performed to obtain the corresponding attention feature mapping through F1 transformation operation, and after nonlinear operation, the output result is decomposed into horizontal feature mapping f h ∈R C / r×H and vertical feature mapping f w ∈R C / r×W ; the horizontal feature mapping f h ∈R C / r×H and the vertical feature mapping f w ∈R C / r×W respectively pass through 1x1 convolution and F h and F w transformation operation, utilize the Sigmoid activation function to obtain the weight of attention, and finally multiply the horizontal and vertical attention weights with the input feature map data to obtain the final output feature;

[0015] The BDC metric module introduces a deep Brownian distance covariance method, first calculates the BDC matrix of the support set, and performs average operation on the obtained BDC matrix to obtain the prototype representation of each category, then calculates the BDC matrix of the query set and performs inner product with the category prototype respectively, to obtain the similarity between the query image and each category, thereby predicting the category to which the image belongs;

[0016] The Softmax layer outputs the result of the BDC metric module classification.

[0017] Further, the input feature map X is decomposed into one-dimensional features in horizontal and vertical directions, specifically:

[0018]

[0019]

[0020] wherein h represents the height of the feature vector, c represents the cth channel, W is the width of the input feature map of the current module, x is the decomposed feature vector, w represents the width of the feature vector, and H is the height of the input feature map of the current module;

[0021] The horizontal and vertical position information of the input feature map is connected together, and a corresponding attention feature map is obtained through F1 transformation operation by 1x1 convolution, specifically:

[0022] f = δ (F1 ([z h ,z w ])), f ∈ R C / r×(H+W) (3)

[0023] wherein f is the encoded intermediate feature map, δ is a nonlinear activation function, [·,·] is a concatenation operation, and r is a hyperparameter for controlling the size of the module;

[0024] The horizontal feature map f h ∈R C / r×H and the vertical feature map f w ∈R C / r×W are respectively transformed by 1x1 convolution and F h and F w transformation operations, and the weights of attention are obtained using a Sigmoid activation function, specifically:

[0025] g h = σ (F h (f h )) (4)

[0026] g w = σ (F w (f w )) (5)

[0027] wherein σ represents a Sigmoid activation function;

[0028] The horizontal and vertical attention weights are multiplied with the input feature map data to obtain the final output feature, specifically:

[0029]

[0030] Further, the BDC metric module introduces a deep Brownian distance covariance method to calculate the BDC matrix of the support set image, specifically:

[0031] Calculate the squared Euclidean distance matrix of the input feature map:

[0032] Square root the squared Euclidean distance matrix to obtain the Euclidean distance matrix

[0033] Subtract the mean of each row, the mean of each column, and the mean of all elements from the Euclidean distance matrix to obtain the final BDC matrix.

[0034] The squared Euclidean distance matrix of the input feature map is calculated, specifically:

[0035]

[0036] Wherein denotes the squared Euclidean distance between the kth and lth columns of matrix X; I represents the identity matrix, is the Hadamard product, defined as (U) sym = (U+U T ) / 2; 1∈R c×c denotes a matrix with all values being 1;

[0037] Square root the squared Euclidean distance matrix to obtain the Euclidean distance matrix Specifically:

[0038]

[0039] Subtract the mean of each row, the mean of each column, and the mean of all elements from the Euclidean distance matrix to obtain the final BDC matrix, specifically:

[0040]

[0041] Further, the deep model is trained based on the training set, specifically:

[0042] Input the training set data into the deep model, use the Adam optimization method to accelerate the training of the deep model, and use the ReLU activation function, the loss function selects the cross-entropy loss function, and the loss function is specifically:

[0043] L = cross_entropy(softmax(F), Label) (10)

[0044] ​​Wherein, cross_entropy is a cross-entropy loss function, F is the feature extracted by the feature extractor from the training set, and Label represents the real distribution corresponding to the feature extracted by the feature extractor from the training set.

[0045] Further, the test set is classified and predicted based on the optimized deep model and the meta-learning N-way K-shot mode, and the classification accuracy is evaluated, specifically:

[0046] The test set is input into the optimized deep model, and the meta-learning N-way K-shot mode is used to classify and predict the test set data, that is, N classes of images are randomly selected in the test set, K samples of each class are selected as the support set to train the model, and the remaining samples in the N classes are selected as the query set, and the query set is used for verification, and the classification accuracy is calculated by comparison to verify the superiority of each algorithm.

[0047] The small sample image classification system based on coordinate attention and BDC metric comprises:

[0048] The division module collects images and divides the collected images to obtain a training set, a validation set and a test set;

[0049] The construction module constructs a deep model framework;

[0050] The acquisition module trains the deep model based on the training set, stops training when the number of training reaches a maximum training number threshold, the performance of the deep model on the validation set reaches a preset requirement or no longer improves significantly, and obtains an optimized deep model;

[0051] The evaluation module classifies and predicts the test set based on the optimized deep model and the meta-learning N-way K-shot mode, and evaluates the classification accuracy.

[0052] A terminal device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the above method when executing the computer program.

[0053] A computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the above method.

[0054] Compared with the prior art, the present application has the following beneficial effects:

[0055] The application constructs a deep model framework including an embedding module, a coordinate attention module, a BDC measurement module and a Softmax layer; realizes mining of deep features of images based on the embedding module; combines spatial information and position information of an input feature map through the coordinate attention module; then calculates a BDC matrix of a support set image through the BDC measurement module; calculates a BDC matrix of a query set image, respectively performs inner product on the BDC matrix and a category prototype to obtain similarity between the query image and each category, and completes a classification task. Moreover, the deep model is trained based on a training set, when the number of training times reaches a maximum training times threshold, performance of the deep model on a verification set reaches a preset requirement or no longer improves significantly, the training is stopped, and an optimized deep model is obtained; the test set is classified and predicted based on the optimized deep model and a meta-learning N-way K-shot mode, and classification accuracy is evaluated. The application introduces coordinate attention of spatial information and position information, and the BDC measurement mode combining differences between image edge distribution and joint distribution can help the model to better complete a small sample image classification task, and can effectively improve classification precision. The application is not only suitable for small sample image classification, but also performs well on a small sample fine-grained dataset, and has excellent performance when facing tire pattern image data in different fields. BRIEF DESCRIPTION OF DRAWINGS

[0056] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some of the embodiments of the application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0057] Figure 1 A flowchart of a small sample image classification method based on coordinate attention and BDC measurement of the application;

[0058] Figure 2 A structural diagram of a small sample image classification system based on coordinate attention and BDC measurement of the application;

[0059] Figure 3 Another flowchart of a small sample image classification method based on coordinate attention and BDC measurement of the application;

[0060] Figure 4 A deep model framework diagram of the application;

[0061] Figure 5 A network structure diagram of ResNet-12;

[0062] Figure 6A schematic diagram of the principle of the coordinate attention mechanism;

[0063] Figure 7 A schematic diagram of the principle of the depth Brownian distance covariance method. DETAILED DESCRIPTION

[0064] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some but not all of the embodiments of the present application. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0065] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0066] It should be noted that: similar reference numbers and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0067] In the description of the embodiments of the present application, it should be noted that if the terms "upper", "lower", "horizontal", "inner" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship when the product of the present application is used, only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the indicated device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second" and the like are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0068] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly inclined. For example, "horizontal" only means that its direction is relatively more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly inclined.

[0069] In the description of the embodiments of the present application, it also needs to be explained that, unless explicitly specified and limited, if the terms "arrange", "install", "connect", "connect" appear, they should be understood in a broad sense, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, or it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0070] The present application will be further described in detail below in combination with the drawings:

[0071] Referring to Figure 1 The present application discloses a small sample image classification method based on coordinate attention and BDC metric, comprising:

[0072] S101, collecting images and dividing the collected images to obtain a training set, a validation set and a test set.

[0073] Obtain public small sample data sets miniImageNet and Omniglot and small sample fine-grained data set CUB-200, and tire pattern data set CIIP-TPID applied in the field of public security; respectively, the miniImageNet data set, the Omniglot data set, the CUB-200 data set and the CIIP-TPID data set are divided into training set, test set and validation set.

[0074] S102, constructing a deep model framework.

[0075] The deep model framework comprises an embedding module, a coordinate attention module, a BDC metric module and a Softmax layer.

[0076] The embedding module extracts features from the input training set; the coordinate attention module performs global average pooling along its horizontal and vertical directions respectively, decomposes the input feature map X into one-dimensional features in the horizontal and vertical directions, and obtains the horizontal and vertical position information of the input feature map; the horizontal and vertical position information of the input feature map is connected together, and the F1 transformation operation is performed through 1*1 convolution to obtain the corresponding attention feature mapping, after nonlinear operation, the output result is decomposed into horizontal direction feature mapping f h ∈R C / r×H and vertical direction feature mapping f w ∈R C / r×W ; horizontal direction feature mapping f h ∈R C / r×H and vertical direction feature mapping f w ∈R C / r×Wthrough 1x1 convolution and F h and F w transform operations, the weight of attention is obtained by using a sigmoid activation function, and finally the attention weights in the horizontal and vertical directions are multiplied with the input feature map data to obtain the final output feature;

[0077] The input feature map X is decomposed into one-dimensional features in the horizontal and vertical directions, specifically:

[0078]

[0079]

[0080] wherein h represents the height of the feature vector, c represents the cth channel, W is the width of the input feature map of the current module, x is the decomposed feature vector, w represents the width of the feature vector, and H is the height of the input feature map of the current module;

[0081] The horizontal and vertical position information of the input feature map is connected together, and a corresponding attention feature mapping is obtained through 1x1 convolution and F1 transform operation, specifically:

[0082] f=δ(F1([z h ,z w ])),f∈R C / r×(H+W) (3)

[0083] wherein f is the encoded intermediate feature mapping, δ is a nonlinear activation function, [·,·] is a concatenation operation, and r is a hyperparameter for controlling the size of the module;

[0084] The horizontal feature mapping f h ∈R C / r×H and the vertical feature mapping f w ∈R C / r×W are respectively obtained through 1x1 convolution and F h and F w transform operations, and the weight of attention is obtained by using a sigmoid activation function, specifically:

[0085] g h =σ(F h (f h )) (4)

[0086] g w =σ(F w (f w )) (5)

[0087] wherein σ represents a sigmoid activation function;

[0088] The horizontal and vertical attention weight is multiplied with the input feature map data to obtain the final output feature, specifically:

[0089]

[0090] The BDC metric module introduces a deep Brownian distance covariance method, first calculates the BDC matrix of the support set, and performs an average operation on the obtained BDC matrix to obtain the prototype representation of each category, then calculates the BDC matrix of the query set and performs inner product with the category prototype respectively, to obtain the similarity between the query image and each category, thereby predicting the category to which the image belongs;

[0091] The square Euclidean distance matrix of the input feature map is calculated:

[0092] The square Euclidean distance matrix is squared to obtain the Euclidean distance matrix

[0093] The Euclidean distance matrix Subtract the mean value of each row element, the mean value of each column element, and the mean value of all elements to obtain the final BDC matrix.

[0094] The square Euclidean distance matrix of the input feature map is calculated, specifically:

[0095]

[0096] Wherein The square Euclidean distance of the kth column and the lth column of the matrix X is represented; I represents the unit matrix, The Hadamard product is defined as (U) sym =(U+U T ) / 2;1∈R c×c Indicates a matrix with all values being 1;

[0097] The square Euclidean distance matrix is squared to obtain the Euclidean distance matrix Specifically:

[0098]

[0099] The Euclidean distance matrix Subtract the mean value of each row element, the mean value of each column element, and the mean value of all elements to obtain the final BDC matrix, specifically:

[0100]

[0101] The Softmax layer outputs the classification result of the BDC metric module.

[0102] S103, training the deep model based on the training set, when the number of training reaches the maximum training number threshold, the performance of the deep model on the validation set reaches the preset requirement or no longer improves significantly, stop training, and obtain the optimized deep model.

[0103] The training set data is input into the deep model, the Adam optimization method is used to accelerate the training of the deep model, and the ReLU activation function is used, and the loss function is selected as the cross entropy loss function, and the loss function is specifically:

[0104] L=cross_entropy(softmax(F),Label) (10)

[0105] Where, cross_entropy is a cross entropy loss function, F is the feature extracted by the feature extractor from the training set, and Label represents the real distribution corresponding to the feature extracted by the feature extractor from the training set.

[0106] S104, based on the optimized deep model and the meta-learning N-way K-shot method, the test set is classified and predicted, and the classification accuracy is evaluated.

[0107] The test set is input into the optimized deep model, and the meta-learning N-way K-shot method is used to classify and predict the test set data, that is, N classes of images are randomly selected in the test set, K samples are selected from each class as a support set to train the model, and the remaining samples in the N classes are selected as a query set, and the query set is used for verification, and the classification accuracy is calculated by comparison to verify the superiority of each algorithm.

[0108] Referring to Figure 2 The application discloses a small sample image classification system based on coordinate attention and BDC metric, comprising:

[0109] The division module collects images and divides the collected images to obtain a training set, a validation set and a test set;

[0110] The construction module constructs a deep model framework;

[0111] The acquisition module trains the deep model based on the training set, when the number of training reaches the maximum training number threshold, the performance of the deep model on the validation set reaches the preset requirement or no longer improves significantly, stop training, and obtain the optimized deep model;

[0112] The evaluation module classifies and predicts the test set based on the optimized deep model and the meta-learning N-way K-shot method, and evaluates the classification accuracy.

[0113] Embodiment: see Figure 3 The application discloses a small sample image classification method based on coordinate attention and BDC measurement, and particularly relates to the following:

[0114] Step S1: Obtain and divide the data set.

[0115] S11: Obtain the public small sample data sets miniImageNet and Omniglot and the small sample fine-grained data set CUB-200, and the tire pattern image data set CIIP-TPID built by the Institute of Image and Information Processing of Xi'an University of Posts and Telecommunications relying on a platform cooperating with the public security department.

[0116] S12: Divide the miniImageNet data set in the manner of 64 classes of training set, 16 classes of verification set and 20 classes of test set, and the image size is 84*84. 1200 samples in the Omniglot data set are trained, and the remaining 423 classes are used for testing, and the size of each image is 28*28. The CUB-200 data set has a total of 200 categories, of which 130 categories are training sets, 20 categories are verification sets and 50 categories are test sets, and the image size is 84*84. In this experiment, the CIIP-TPID data set is divided into three sub-data sets containing different images: a tire surface pattern data set, a tire indentation data set and a mixed data set, each of which has 69 categories of image data. 46 categories are used as the training set, 10 categories are used as the verification set, and 13 categories are used as the test set, and the size of each image is set to 48*48. Each category in the surface data set and the indentation data set contains different tire surface pattern images and tire indentation images, and each category has 80 image samples. The mixed data set contains 160 mixed samples of tire surface pattern images and tire indentation images per category.

[0117] Step S2: Construct a deep model framework.

[0118] As shown in Figure 4 , the present example is composed of an embedding module Resnet-12, a coordinate attention module, a BDC measurement module and a Softmax layer.

[0119] The present example adopts ResNet-12 as the embedding feature extraction network, and the network structure of ResNet-12 is as shown in Figure 5 . The embedding module extracts features from the input training set;

[0120] As shown in Figure 6As shown, the coordinate attention module performs global average pooling along its horizontal and vertical directions respectively, decomposes the input feature map X into one-dimensional features in the horizontal and vertical directions, and obtains the horizontal and vertical position information of the input feature map; the horizontal and vertical position information of the input feature map is connected together, and a 1x1 convolution is performed to obtain the corresponding attention feature map through F1 transformation operation, and after nonlinear operation, the output result is decomposed into horizontal feature map f h ∈R C / r×H and vertical feature map f w ∈R C / r×W ; the horizontal feature map f h ∈R C / r×H and the vertical feature map f w ∈R C / r×W are respectively transformed through 1x1 convolution and F h and F w transformation operation, the weight of attention is obtained by using Sigmoid activation function, and finally the horizontal and vertical attention weights are multiplied with the input feature map data to obtain the final output feature;

[0121] The coordinate attention module of the example performs global average pooling along its horizontal and vertical directions respectively, decomposes the input feature map X into two one-dimensional features, and obtains the horizontal and vertical position information of the input feature map. The one-dimensional features output in the horizontal and vertical directions are shown in formulas (11) and (12) respectively:

[0122]

[0123]

[0124] wherein h represents the height of the feature vector, c represents the cth channel, W is the width of the input feature map of the current module, and x is the decomposed feature vector; w represents the width of the feature vector, and H is the height of the input feature map of the current module.

[0125] The horizontal and vertical position information of the input feature map is connected together, and a 1x1 convolution is performed to obtain the corresponding attention feature map through F1 transformation operation, as shown in formula (13):

[0126] f=δ(F1([z h ,z w ])),f∈R C / r×(H+W) (13)

[0127] wherein f is the encoded intermediate feature map, δ is a nonlinear activation function, [·,·] is a splicing operation, and r is a hyperparameter for controlling the size of the module.

[0128] After the nonlinear operation, the output result is decomposed into horizontal feature map f h ∈R C / r×H and vertical feature map f w ∈R C / r×W respectively. The horizontal feature map f h ∈R C / r×H and the vertical feature map f w ∈R C / r×W are respectively obtained by 1x1 convolution and F h and F w transformation operation, and the weight of attention is obtained by using Sigmoid activation function, as shown in formula (14) and formula (15):

[0129] g h = σ (F h (f h )) (14)

[0130] g w = σ (F w (f w )) (15)

[0131] Wherein, σ represents Sigmoid activation function.

[0132] The horizontal and vertical attention weights are multiplied with the input feature map data to obtain the final output feature, as shown in formula (16):

[0133]

[0134] In the BDC measurement module, the BDC matrix of the support set image is first calculated, then the obtained BDC matrix is averaged by the prototype network idea to obtain the prototype representation of each category, then the BDC matrix of the query set image is calculated and the inner product is performed with the category prototype respectively, to obtain the similarity between the query image and each category, so as to predict the category to which the image belongs.

[0135] In order to fully tap the statistical characteristics between the distribution of image features, the deep Brownian distance covariance method is introduced into the BDC measurement module, as shown in formula (17). First, the squared Euclidean distance matrix of the input feature map is calculated according to formula (17): Figure 7

[0136]

[0137] Wherein represents the squared Euclidean distance between the kth column and the lth column of matrix X. I represents the unit matrix, is the Hadamard product, defined as (U) sym ​= (U + U T ) / 2;1∈R c×c is represented as a matrix with all values being 1. Then square root operation is performed on the squared Euclidean distance matrix to obtain the Euclidean distance matrix As shown in equation (18):

[0138]

[0139] The Euclidean distance matrix Subtract the mean value of each row element, the mean value of each column element and the mean value of all elements to obtain the final BDC matrix, as shown in equation (19):

[0140]

[0141] Step S3: network model training.

[0142] The 64 classes of the miniImageNet dataset are trained, 16 classes are used as the training set, and 20 classes are tested; 1200 classes of samples in the Omniglot dataset are trained, and the remaining 423 classes are used for testing; 130 classes of the CUB-200 dataset are used as the training set, 20 classes are used as the training set, and 50 classes are tested; 46 classes of the CIIP-TPID dataset are used as the training set, 10 classes are used as the validation set, and 13 classes are used as the test set for testing. The training set is input into the network for training in the following method:

[0143] The training set data is input into the network, the Adam optimization method is used to accelerate the model training, the ReLU activation function is used, the cross-entropy loss function is selected, and the batch size is 16. The experiment is performed for a total of 200 epochs, and each epoch is trained 100 times. The learning rate of the model is 1×10 -3 , which is reduced to half of the original every 20 epochs. When the specified number of training rounds is completed, the performance of the model on the validation set reaches the preset requirement or no longer improves significantly, the training is completed.

[0144] The loss function is specifically:

[0145] L = cross_entropy(softmax(F), Label) (20)

[0146] Where cross_entropy is the cross-entropy loss function, F is the feature extracted by the feature extractor from the training set, and Label represents the true distribution corresponding to the feature extracted by the feature extractor from the training set.

[0147] The ReLU activation function is shown in equation (21), and the gradient is 1 when x>0, thereby alleviating the gradient vanishing problem. However, when x<0, the weight cannot be updated, causing the "death problem" of ReLU.

[0148]

[0149] Step S4: classification prediction and model performance evaluation.

[0150] In this example, the test set is input into the model with the highest accuracy on the validation set obtained in step S3, and the meta-learning N-way K-shot method is used to classify and predict the test set data, that is, N classes of images are randomly selected from the data set, and K samples are selected from each class as a support set to train the model, generally N e {5, 10}, K e {1, 5}. Secondly, a small number of samples remaining in the N classes are selected as a query set, and finally the query set is used for verification. The superiority of each algorithm is verified by comparing the classification accuracy calculated. Specifically, 5-way 1-shot and 5-way 5-shot methods are used for testing and verification.

[0151] Complete the small sample image classification method based on coordinate attention.

[0152] In order to verify the beneficial effects of the present application, the inventors carried out simulation experiments using the method of Example 1 of the present application, and the experimental conditions are as follows:

[0153] 1. Simulation conditions

[0154] Hardware platform: NVIDIA TITAN XP GPUs,

[0155] Software platform: operating system Ubuntu 16.04,

[0156] Software environment: PyCharm 2019.2

[0157] CPU: Intel(R) Xeon(R) CPU E5-2620 v4@2.10GHz

[0158] 2. Simulation content and results

[0159] Under the simulation conditions, the experiment tests the classification accuracy of the model in a 5-way 1-shot and 5-way 5-shot manner on miniImageNet, Omniglot, CUB-200, CIIP-TPID data sets. The 1-shot classification result of the method on the Omniglot, miniImageNet, CUB-200 data sets is improved by 20.15%, 1.03%, and 33.82% compared with the prototype network of the 4-layer convolutional network, and the 1-shot classification result on the CIIP-TPID data set applied in the public security field is improved by at least 5.36% compared with the prototype network of the 4-layer convolutional network, so the method has strong generalization.

[0160] An embodiment of the present application provides a terminal device. The terminal device of the embodiment comprises a processor, a memory, and a computer program stored in the memory and executable on the processor. The processor implements the steps in each of the method embodiments when executing the computer program. Alternatively, the processor implements the functions of each module / unit in each of the device embodiments when executing the computer program.

[0161] The computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present application.

[0162] The terminal device can be a desktop computer, a notebook computer, a palm computer, a cloud server, and other computing devices. The terminal device can include, but is not limited to, a processor and a memory.

[0163] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0164] The memory can be used to store the computer program and / or modules. The processor realizes various functions of the terminal device by running or executing the computer program and / or modules stored in the memory, and calling the data stored in the memory.

[0165] The modules / units integrated in the terminal device, if realized in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. When the computer program is executed by a processor, the steps of the above-mentioned various method embodiments can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms, etc. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content included in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0166] The above only describes the preferred embodiments of the present application and is not used to limit the present application. The present application can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A small sample image classification method based on coordinate attention and BDC metrics, characterized in that, The application relates to a small sample classification method and device. Collect images and divide the collected images to obtain a training set, a verification set and a test set; A deep model framework is constructed; The deep model is trained based on the training set; when the number of training times reaches a maximum training time threshold, the performance of the deep model on the verification set reaches a preset requirement or can no longer be significantly improved, the training is stopped, and an optimized deep model is obtained; The test set is classified and predicted based on the optimized deep model and a meta-learning N-way K-shot mode, and the classification accuracy is evaluated; The collected images are divided to obtain a training set, a verification set and a test set, including: obtaining public small sample data sets miniImageNet and Omniglot and a small sample fine-grained data set CUB-200 and a tire pattern data set CIIP-TPID applied to the public security field; the miniImageNet data set, the Omniglot data set, the CUB-200 data set and the CIIP-TPID data set are divided into a training set, a test set and a verification set; the deep model framework comprises an embedding module, a coordinate attention module, a BDC metric module and a Softmax layer; The embedding module extracts features from the input training set; the coordinate attention module respectively performs global average pooling along the horizontal and vertical directions of the coordinate attention module, decomposes the input feature map into one-dimensional features in the horizontal and vertical directions to obtain horizontal and vertical position information of the input feature map; the horizontal and vertical position information of the input feature map is connected together, and the 1×1 convolution is performed to obtain the corresponding attention feature mapping, after nonlinear operation, the output result is decomposed into horizontal feature mapping and vertical feature mapping according to the spatial dimension ; the horizontal feature mapping and the vertical feature mapping are respectively transformed by 1×1 convolution and and transform operation, the weight of attention is obtained by using the Sigmoid activation function, and finally the horizontal and vertical attention weights are multiplied with the input feature map data to obtain the final output feature In the BDC metric module, a deep Brown distance covariance method is introduced; firstly, the BDC matrix of the support set is calculated, and the obtained BDC matrix is subjected to an average operation to obtain a prototype representation of each category; then, the BDC matrix of the query set is calculated and subjected to an inner product with the category prototype to obtain the similarity between the query image and each category, so that the category to which the image belongs is predicted; The Softmax layer outputs a result of classification of the BDC metric module; and the input feature map The one-dimensional features decomposed into horizontal and vertical directions are specifically: wherein, represents a height of the feature vector, represents a first channel, is a width of the input feature map of the current module, is a decomposed feature vector, represents a width of the feature vector, is a height of the input feature map of the current module; The horizontal and vertical position information of the input feature map is connected together by 1x1 convolution The transformation operation obtains the corresponding attention feature mapping, specifically: wherein, is the encoded intermediate feature map, is a non-linear activation function, is a concatenation operation, is a hyper-parameter controlling the size of the module; the horizontal direction feature map and the vertical direction feature map respectively through 1x1 convolution and and transform operation, the weight of attention is obtained by using Sigmoid activation function, specifically: wherein, denotes a Sigmoid activation function; The attention weight in the horizontal and vertical directions is multiplied with the input feature map data to obtain the final output feature, and the specific operation is as follows:

2. The small sample image classification method based on coordinate attention and BDC metric according to claim 1, characterized in that, In the BDC metric module, a deep Brown distance covariance method is introduced; firstly, the BDC matrix of the support set is calculated, and the obtained BDC matrix is subjected to an average operation to obtain a prototype representation of each category; then, the BDC matrix of the query set is calculated and subjected to an inner product with the category prototype to obtain the similarity between the query image and each category, so that the category to which the image belongs is predicted; The square Euclidean distance matrix of the input feature map is calculated, and the specific operation is as follows: Square the square Euclidean distance matrix to obtain the Euclidean distance matrix ; Euclidean distance matrix Subtracting the mean of each row, the mean of each column and the mean of all elements, the final BDC matrix is obtained. The square Euclidean distance matrix of the input feature map is calculated, and the specific operation is as follows: wherein , denotes the squared Euclidean distance between the th column and the th column of the matrix ; denotes the identity matrix, is the Hadamard product, defined as ; denotes a matrix of all values being one; square the square Euclidean distance matrix to obtain a Euclidean distance matrix , specifically: the pair-wise euclidean distance matrix Subtracting the mean of each row, the mean of each column and the mean of all elements, the final BDC matrix is obtained, specifically:

3. The small sample image classification method based on coordinate attention and BDC metric according to claim 2, characterized in that, The deep model is trained based on the training set, and the specific operation is as follows: The training set data is input into the deep model, the Adam optimization method is used to accelerate the training of the deep model, and the ReLU activation function is used; the loss function is a cross-entropy loss function, and the specific operation is as follows: wherein, is a cross-entropy loss function, F is the feature extracted by the feature extractor for the training set, and Label represents the true distribution corresponding to the feature extracted by the feature extractor for the training set.

4. The small sample image classification method based on coordinate attention and BDC metric according to claim 3, characterized in that, The test set is classified and predicted based on the optimized deep model and the meta-learning N-way K-shot mode, and the classification accuracy is evaluated, and the specific operation is as follows: The test set is input into the optimized deep model, and meta-learning is employed. N -way K The -shot method performs classification prediction on the test set data, that is, randomly selecting data from the test set. N Image class, select for each class K The model was trained using samples as the support set, and... N The remaining samples in the class are selected as the query set, and the query set is used for verification. The superiority of each algorithm is verified by comparing the calculated classification accuracy.

5. A small sample image classification system based on coordinate attention and BDC metric, characterized in that, The application relates to a small sample classification method and device. A division module is configured to collect images and divide the collected images to obtain a training set, a verification set and a test set; A construction module is configured to construct a deep model framework; An acquisition module is configured to train the deep model based on the training set; when the number of training times reaches a maximum training time threshold, the performance of the deep model on the verification set reaches a preset requirement or can no longer be significantly improved, the training is stopped, and an optimized deep model is obtained; An evaluation module is configured to classify and predict the test set based on the optimized deep model and a meta-learning N-way K-shot mode, and evaluate the classification accuracy. The collected images are divided, including: obtaining public small sample data sets miniImageNet and Omniglot and small sample fine-grained data set CUB-200, and tire pattern data set CIIP-TPID applied in the field of public security; the miniImageNet data set, the Omniglot data set, the CUB-200 data set and the CIIP-TPID data set are respectively divided into training set, test set and validation set; the deep model framework comprises: an embedding module, a coordinate attention module, a BDC metric module and a Softmax layer; The embedding module extracts features from the input training set; the coordinate attention module respectively performs global average pooling along the horizontal and vertical directions of the coordinate attention module, decomposes the input feature map into one-dimensional features in the horizontal and vertical directions to obtain horizontal and vertical position information of the input feature map; the horizontal and vertical position information of the input feature map is connected together, and a 1×1 convolution is performed on the horizontal and vertical position information to obtain corresponding attention feature mapping through a transformation operation After a nonlinear operation, the output result is decomposed into a horizontal feature mapping and a vertical feature mapping according to the spatial dimension ; the horizontal feature mapping and the vertical feature mapping respectively pass through a 1×1 convolution and a transformation operation and , and the weight of attention is obtained by using a Sigmoid activation function, and finally the horizontal and vertical attention weights are multiplied with the input feature map data to obtain the final output feature The BDC metric module introduces a deep Brown distance covariance method, first calculates the BDC matrix of the support set, and performs an average operation on the obtained BDC matrix to obtain the prototype representation of each category, then calculates the BDC matrix of the query set and performs inner product with the category prototype respectively, to obtain the similarity between the query image and each category, thereby predicting the category to which the image belongs; The Softmax layer outputs a result of classification of the BDC metric module; and the input feature map is decomposed into one-dimensional features in horizontal and vertical directions, specifically: wherein, represents a height of the feature vector, represents a first channel, is a width of the input feature map of the current module, is a decomposed feature vector, represents a width of the feature vector, is a height of the input feature map of the current module; The horizontal and vertical position information of the input feature map is connected together by 1x1 convolution The transformation operation obtains the corresponding attention feature mapping, specifically: wherein, is the encoded intermediate feature map, is a non-linear activation function, is a concatenation operation, is a hyperparameter that controls the size of the module. The horizontal direction feature map And the vertical direction feature map Respectively through 1x1 convolution and And The transformation operation obtains the weight of attention by using the Sigmoid activation function, specifically: wherein, denotes a Sigmoid activation function; The attention weights in the horizontal and vertical directions are multiplied with the input feature map data to obtain the final output feature, specifically:

6. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the steps of the method of any one of claims 1-4.

7. A computer-readable storage medium storing a computer program, wherein the computer program comprises the following steps of: The computer program is executed by the processor to realize the steps of the method of any one of claims 1-4.

Images