A cross-granularity few-shot learning method based on similarity regularization within-class mining

Abstract

This invention provides a cross-granularity few-shot learning method based on intra-class mining with similarity regularization, comprising: generating training image sample features using an image feature extraction network; implementing a first optimization loop to constrain the coarse-grained inter-class semantic discriminativeness of the deep learning model using weak supervision; implementing a second optimization loop to achieve coarse-grained intra-class similarity regularization through a symmetric feature embedder and an asymmetric feature predictor; adding the losses of the first and second optimization loops and performing end-to-end error backpropagation on the entire network structure to optimize the network parameters; and using scenario-based fine-tuning to adapt the obtained image feature extraction network to the few-shot classification task, ultimately obtaining a fine-grained few-shot classification model. This invention combines and improves the collaborative optimization of weak supervision and self-supervision learning paradigms, while considering both intra-class and inter-class discriminativeness of coarse-grained categories, resulting in a few-shot classification model with more fine-grained discriminative capabilities.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically, to a cross-granularity few-shot learning method based on intra-class mining using similarity regularization. Background Technology

[0002] Few-shot learning is a key task on the path to general artificial intelligence and a hot research topic in the field of deep learning in recent years. It is applicable to technologies such as edge computing and personalized recommendation. Few-shot learning can be defined as: given a small number of examples of categories that have not been seen during training, the model can quickly learn new concepts and perform new tasks. It is an automatic and rapid learning technique that can quickly adapt to new knowledge of interest in new environments, and is an important step in applications such as sustainable learning and the customization of deep learning products.

[0003] Existing few-shot learning methods, based on the idea that training and test data have the same granularity, typically still require a large amount of fine-grained annotation to estimate and simulate future data granularity. Therefore, insufficient training data annotation, such as coarse-grained category annotation, can negatively impact network performance, resulting in higher accuracy during training but poor performance during testing. The root cause of this phenomenon is the problem of cross-granularity information mining.

[0004] In recent years, some researchers at home and abroad have begun to turn to cross-granularity few-shot learning problems, using complete coarse-grained category information to learn fine-grained recognition capabilities.Existing methods fall into three categories. The first involves learning fine-grained re-representations through inter-sample classification, proposing a coarse-to-fine pseudo-labeling algorithm to refine the coarse-grained category distribution (see Yang, J., Yang, H., & Chen, L. (2021, October). Towards cross-granularity few-shot learning: coarse-to-fine pseudo-labeling with visual-semantic meta-embedding. In Proceedings of the 29th ACM International Conference on Multimediapages 3005-3014.). However, this method is too costly, and the accuracy of the labeling process is not high enough. The second approach improves performance by proposing angle normalization constraints and combining them with self-supervised learning to leverage the synergistic optimization effect of different supervisory signals (see Bukchin, G., Schwartz, E., Saenko, K., Shahar, O., Feris, R., Giryes, R., & Karlinsky, L. (2021). Fine-grained angular contrastive learning). With coarse labels. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. Pages 8730-8740., this method, while considering how to effectively fuse coarse-grained supervision information and self-supervised information that does not require external supervision, still does not fundamentally resolve the conflict between different supervision information, thus the performance improvement is very limited; the third method is to explicitly model the internal structure of coarse-grained categories and use a Gaussian mixture distribution to re-represent each fine-grained sub-category (see Ni, J., Cheng, W., Chen, Z., Asakura, T., Soma, T., Kato, S., & Chen, H. (2021, September). Superclass-Conditional Gaussian Mixture Model For Learning Fine-Grained Embeddings. In International Conference on Learning Representations.), this method is essentially similar to the first method, based on incompletely accurate fine-grained category reconstruction, and its performance is difficult to guarantee. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the purpose of this invention is to provide a cross-granularity few-sample learning method based on intra-class mining using similarity regularization.

[0006] According to one aspect of the present invention, a cross-granularity few-shot learning method based on similarity regularization intra-class mining is provided, comprising:

[0007] Use an image feature extraction network to generate features for training image samples;

[0008] Based on the image sample features, a first optimization loop is implemented, using weak supervision to constrain the coarse-grained inter-category semantic discriminative ability of the deep learning model.

[0009] Based on the image sample features, a second optimization loop is implemented, which uses a symmetric feature embedder and an asymmetric feature predictor to achieve coarse-grained similarity regularization constraints within categories.

[0010] The losses of the first optimization loop and the second optimization loop are added together, and the error is backpropagated end-to-end across the entire network structure to optimize the network parameters, resulting in an optimized image feature extraction network.

[0011] By using scenario-based fine-tuning, the optimized image feature extraction network is adapted to the few-shot classification task, ultimately resulting in a fine-grained few-shot classification model.

[0012] Preferably, the step of generating training image sample features using an image feature extraction network includes:

[0013] Construct the image feature extraction network;

[0014] Define an augmented set of data;

[0015] Image samples in the training set are used to generate hidden layer views based on the data augmentation set and the image feature extraction network.

[0016] Preferably, the image feature extraction network includes several supervised deep convolutional neural networks and a global average pooling layer whose output is data of a set dimension;

[0017] The input image is processed in the deep convolutional neural network to reduce the feature resolution while increasing the number of feature channels, thereby extracting high-dimensional features.

[0018] The high-dimensional features are processed by a global average pooling layer, which takes a global average of the features of each channel to obtain a feature vector of a set dimension.

[0019] Preferably, the data augmentation set includes at least five sets of image transformation techniques, namely:

[0020] Random deformation cropping: For any input image, a portion of it is randomly taken, the size of which is limited to 20%-100% of the whole image, and then resampled to the specified size required by the image feature extraction network;

[0021] Random color jitter: For any input image, perform random color transformation on it with a set probability, wherein the color transformation includes adjusting one or more of brightness, contrast, saturation and hue;

[0022] Random grayscale conversion: For any input image, perform random grayscale conversion with a set probability to convert the color image into a grayscale image with the same values ​​for each channel;

[0023] Random Gaussian blur: For any input image, perform random Gaussian blur with a set probability, wherein the standard deviation of the Gaussian kernel is between [0.1, 2.0].

[0024] Random horizontal flip: For any input image, randomly flip it horizontally with a set probability.

[0025] Preferably, the image samples generate a hidden layer view based on a data augmentation set and an image feature extraction network, including:

[0026] For each image sample in the training data, two augmentation functions are obtained by sampling through the data augmentation set in each iteration;

[0027] Based on the two augmentation functions, each image sample is projected as a positive sample pair;

[0028] For the two sample matrices in the positive sample pair, the image feature extraction network is used to project them from the sample space into a real vector space of a set dimension, resulting in two augmented training image sample features, called hidden layer views.

[0029] Preferably, the step of implementing a first optimization loop based on the graphic sample features, and using weak supervision to constrain the coarse-grained inter-category semantic discriminative ability of the deep learning model, includes:

[0030] A coarse-grained class classifier is formed based on a fully connected neural network and a Softmax activation layer;

[0031] The training image sample features are used as input, and the coarse-grained category classifier is used to obtain the coarse-grained category probability.

[0032] Based on coarse-grained category probabilities, a coarse-grained image classification task is performed, cross-entropy loss is calculated as coarse-grained inter-class semantic supervision, and the error is backpropagated.

[0033] Preferably, the set of coarse-grained categories is set as follows: Input hidden view v∈R d Then the classifier Perform a mapping from the feature space to the class label space. And obtain the forward propagation output q i :

[0034]

[0035] Remember q i The j-th dimension is q i,j And the true coarse-grained label of the image sample is y j The cross-entropy loss for coarse-grained classification is calculated as follows:

[0036]

[0037] Preferably, based on the image sample features, a second optimization loop is implemented, using a symmetric feature embedder and an asymmetric feature predictor to achieve coarse-grained similarity regularization constraints within categories, including:

[0038] Based on a multilayer perceptron, a three-layer feature embedder and a two-layer feature predictor are constructed.

[0039] The method for performing symmetric embedding prediction on the hidden layer view based on the feature embedder and feature predictor includes: projecting the hidden layer view into embedding vectors using the feature embedder; performing mutual prediction between the embedding vectors using the feature predictor to obtain prediction vectors; and calculating the cosine similarity between the prediction vector corresponding to one embedding vector and the other embedding vector itself.

[0040] Preferably, a symmetric embedding prediction process is performed on the hidden layer view, and the coarse-grained internal self-supervised loss is calculated as follows:

[0041]

[0042] Here, stopgrad is a gradient truncation operation to prevent feature collapse between samples on both sides. p1 and z1 are the prediction vector and embedding vector corresponding to one hidden view, respectively; p2 and z2 are the prediction vector and embedding vector corresponding to the other hidden view, respectively.

[0043] Preferably, the step of using scenario-based fine-tuning to adapt the optimized image feature extraction network to the few-shot classification task, ultimately obtaining a fine-grained few-shot classification model, includes:

[0044] Sample a small number of classification scenarios from the training data;

[0045] Image features are extracted from the several small sample classification scenarios using an image feature extractor optimized by the first and second optimization loops.

[0046] The extracted image features are combined with cosine distance and a parameterless differentiable nearest neighbor classifier to perform a few-shot classification task. Cross-entropy loss is calculated, and error backpropagation and gradient descent are performed to obtain the final fine-grained few-shot classification model.

[0047] According to a second aspect of the present invention, an image classification method is provided, comprising:

[0048] Obtain the image to be classified;

[0049] A fine-grained few-shot classification model is constructed for classification, wherein the fine-grained few-shot classification model adopts the cross-granularity few-shot learning method based on similarity regularization intra-class mining as described above.

[0050] The image to be classified is input into a fine-grained few-sample classification model to achieve image classification.

[0051] According to a third aspect of the present invention, a terminal for image processing is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to run the cross-granularity few-sample learning method based on similarity regularization intra-class mining, or to execute the image classification method.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] The cross-granularity few-shot learning method based on similarity regularization intra-class mining in this embodiment of the invention combines and improves the collaborative optimization of weak supervision and self-supervision learning paradigms, while considering the intra-class and inter-class discriminability of coarse-grained categories. Therefore, it can obtain a few-shot classification model with more fine-grained discriminative ability.

[0054] The cross-granularity small sample learning method based on intra-class mining with similarity regularization in this embodiment of the invention utilizes similarity regularization constraints to mine the internal structure of coarse-grained categories. Based on a given coarse-grained category distribution, it automatically learns the differences between samples within the coarse-grained category manifold, reducing the performance loss caused by the lack of fine-grained annotations. It does not require additional fine-grained annotation work, thus reducing annotation costs.

[0055] The cross-granularity few-sample learning method based on similarity regularization intra-class mining in this embodiment of the invention uses weak supervision to constrain the coarse-grained inter-class semantic discriminativeness of the deep learning model. Through external supervision of the inter-class structure, it increases the model's understanding of high-level image features, thereby improving the recognition accuracy. Attached Figure Description

[0056] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0057] Figure 1 This is a schematic diagram of the network structure and composition of a cross-granularity few-sample learning method based on intra-class mining with similarity regularization in an embodiment of the present invention.

[0058] Figure 2 This is a schematic diagram of a gradient flow mode in one embodiment of the present invention. Detailed Implementation

[0059] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0060] join Figure 1 This invention provides an embodiment of a cross-granularity few-shot learning method based on similarity regularization intra-class mining, comprising:

[0061] S100 utilizes an image feature extraction network to generate training image sample features;

[0062] S200, based on the image sample features obtained in S100, implements the first optimization loop, using weak supervision to constrain the coarse-grained inter-category semantic discriminativeness of the deep learning model;

[0063] S300, implements the second optimization loop, and achieves coarse-grained similarity regularization within categories through symmetric feature embedders and asymmetric feature predictors;

[0064] S400 adds the losses from the first optimization loop in S200 and the second optimization loop in S300, performs end-to-end error backpropagation on the entire network structure, and optimizes the network parameters.

[0065] S500 utilizes scenario-based fine-tuning to adapt the optimized image feature extraction network of S400 to the few-shot classification task, ultimately resulting in a fine-grained few-shot classification model.

[0066] This embodiment combines and improves the collaborative optimization of weakly supervised and self-supervised learning paradigms, forming two parallel optimization loops. The first optimization loop uses weak supervision to constrain the coarse-grained inter-class semantic discriminative ability of the deep learning model, increasing the model's understanding of high-level image features and improving recognition accuracy through external supervision of inter-class structures. Simultaneously, the second optimization loop utilizes similarity regularization constraints to perform extensive intra-class mining. Based on a given coarse-grained class distribution, it automatically explores subtle differences between samples within each coarse-grained class, thereby improving the model's fine-grained small-sample classification ability.

[0067] Thus, this embodiment considers both intra-class and inter-class discriminative power of coarse-grained categories, and jointly utilizes different supervision techniques to generate richer feature re-representations. The trained model has excellent cross-granularity learning capabilities, can better solve and adapt to performance loss caused by the lack of fine-grained information, has universal applicability, and while significantly reducing annotation costs, has considerable accuracy in fine-grained small sample classification.

[0068] In a preferred embodiment of the present invention, S100 is implemented. Specifically, it includes:

[0069] S101. Based on a supervised deep convolutional neural network, combined with a global average pooling layer with an output dimension of d, an image feature extraction network is formed. Image feature extraction network It contains several units composed of deep convolutional neural networks.

[0070] In this embodiment, for lower resolution images, such as the tieredImageNet database, four residual modules are used, each containing three convolutional modules. Each residual module is followed by a 2×2 max pooling layer and a skip connection. Each convolutional module contains a 3×3 convolutional layer, followed by a batch normalization layer and a leaky linear rectified unit as the activation function. For the input image, this network gradually reduces the feature resolution while correspondingly widening the number of feature channels. After extracting high-dimensional features, a global average pooling layer is used to take the global average of the features of each channel, finally obtaining a d-dimensional feature vector.

[0071] In this embodiment, for higher resolution images, such as the BREEDS database, the standard ResNet-50 structure is used (see He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778). After removing the terminal and fully connected layers, it serves as the image feature extraction network, and its feature extraction process is consistent with that of the low-resolution image feature extraction network described above.

[0072] S102. Define a data augmentation set consisting of various image transformation techniques such as random deformation cropping, random horizontal flipping, random grayscale conversion, and random Gaussian blur. Each time, a transformation is randomly sampled from this set, and the resulting random distribution is represented as...

[0073] The data augmentation set includes the following five groups of image transformation techniques:

[0074] - Random deformation cropping: For any input image, a portion of it is randomly taken, with the size range limited to 20%-100% of the full image, and resampled to the specified size required by the feature extractor;

[0075] - Random color dithering: For any input image, perform random color transformations with a probability of 0.8. The possible color transformations include adjustments to brightness, contrast, saturation, hue, etc.

[0076] - Random grayscale conversion: For any input image, perform random grayscale conversion with a probability of 0.2 to convert the color image into a grayscale image with the same values ​​in each channel;

[0077] - Random Gaussian Blur: For any input image, perform random Gaussian blur with a probability of 0.5, where the standard deviation of the Gaussian kernel may be between [0.1, 2.0], with left and right closing;

[0078] - Random horizontal flip: For any input image, randomly flip it horizontally with a probability of 0.5.

[0079] In this embodiment, the specified size for random deformation cropping of the BREEDS database is 224, and the specified size for random deformation cropping of the tieredImageNet database is 84.

[0080] S103. For each image sample in the training data In each iteration, it starts from a random distribution. Two augmented functions are obtained by mid-sampling. and in and Using the two augmentation functions obtained from sampling, each image sample is projected as a positive sample pair {x1, x2}, where x1 and x2 are respectively... and

[0081] S104. For the two sample matrices in the positive sample pair, use the feature extractor respectively. Project it from the sample space into a d-dimensional real vector space: The augmented training image features v1 and v2 are obtained, where and All of these are d-dimensional feature vectors, referred to as hidden layer views.

[0082] In this embodiment, a single data sample is used to obtain two samples through the augmented function obtained by sampling, and then paired training is performed.

[0083] In another preferred embodiment of the present invention, S200 is implemented based on the training image features obtained in S100, namely the two hidden layer views. Specifically, it includes:

[0084] S201. Based on a fully connected neural network and a Softmax activation layer, a coarse-grained classifier is formed.

[0085] S202. Using the hidden layer views v1 and v2 obtained in S104 as input, a coarse-grained category classifier is applied. We obtain coarse-grained class probabilities, perform a coarse-grained image classification task, calculate cross-entropy loss as coarse-grained inter-class semantic supervision, and backpropagate the error.

[0086] Let the set of coarse-grained categories be . Input hidden view v∈R d Then the classifier Perform a mapping from the feature space to the class label space. And obtain the forward propagation output q i :

[0087]

[0088] Remember q i The j-th dimension is q i,j And the true coarse-grained label of the image sample is y j The cross-entropy loss for coarse-grained classification is calculated as follows:

[0089]

[0090] In this embodiment, weak supervision is used to constrain the coarse-grained inter-category semantic discriminativeness of the deep learning model. Through external supervision of the inter-category structure, the model's understanding of high-level image features is enhanced, thereby improving the recognition accuracy.

[0091] See Figure 2 In another preferred embodiment of the present invention, S300 is implemented. Specifically, it includes:

[0092] S301. Based on a multilayer perceptron, construct a three-layer feature embedder ε and a two-layer feature predictor. Among them, the feature embedder ε and the feature predictor All are based on a multilayer perceptron architecture. The feature embedding unit ε has d-dimensional vectors as input and output for each layer, and a batch normalization layer is used after each linear layer for stable training. Except for the output layer, the other two layers use rectified linear units as activation functions and have three fully connected layers. Feature predictor It consists of two linear layers with an hourglass-shaped bottleneck structure. The input dimension of the middle hidden layer is d, the output is reduced to d / 4 dimensions, and then restored to d dimensions in the output layer. In a preferred embodiment, the feature dimension d of the tieredImageNet database is set to 512, and that of the BREEDS database is set to 2048.

[0093] S302, using the feature embedder ε, the hidden layer views v1 and v2 obtained in S104 are projected into embedding vectors z1 and z2, where z i =ε(v i ), i∈{1,2}; Feature predictor The feature predictor performs mutual prediction between embedded vectors. It is asymmetric, working only on one of the embedding vectors at a time. For example, for embedding vector z1, the feature predictor... Project it as a prediction vector Predict the embedding vector of another hidden view (e.g., z2) and perform similarity regularization, i.e., calculate the cosine similarity with the other embedding vector z2:

[0094]

[0095] Where ||·|| represents l2 normalization.

[0096] S303, to improve the efficiency of similarity regularization, a symmetrical embedding prediction process is implemented on the two hidden layer views, ultimately obtaining a coarse-grained internal self-supervised loss, calculated as follows:

[0097]

[0098] Stopgrad is a gradient truncation operation that prevents feature collapse from occurring between samples on both sides.

[0099] In this embodiment, similarity regularization constraints are used to mine the internal structure of coarse-grained categories. Based on the given coarse-grained category distribution, the differences between samples within the coarse-grained category manifold are automatically learned, reducing the performance loss caused by the lack of fine-grained annotations. No additional fine-grained annotation work is required, thus reducing annotation costs.

[0100] In a preferred embodiment of the present invention, based on the two parts of loss in S200 and S300, S400 is implemented to add the two parts together, and the error backpropagation is performed end-to-end on the entire network structure. The network parameters are then optimized using gradient descent techniques.

[0101] In a preferred embodiment of the present invention, a feature extractor based on the network parameters optimized in S400 after the first and second optimization loops is used. Implement the S500, use scenario-based fine-tuning, and further train the resulting feature extractor. The specific process is as follows:

[0102] Several small sample classification scenarios are sampled from the training set. Image features are extracted by a feature extractor. The small sample classification task is performed by combining cosine distance and a parameterless differentiable nearest neighbor classifier. Cross-entropy loss is calculated, and error backpropagation and gradient descent are performed.

[0103] Each time, N-way K-shot tasks are sampled from the training set. Support set It contains N categories, each with K samples. A feature extractor obtained using S100-S400 is used. First, use the support set. Calculate the template features for each category:

[0104]

[0105] in refer to The set of samples of category i.

[0106] For any test sample The probability of a class belonging to the i-th category in this small sample classification scenario is calculated using a non-parametric differentiable nearest neighbor classifier combined with cosine distance as follows:

[0107]

[0108] Here, τ is a learnable parameter. These probability values ​​are then used to further optimize the feature extractor through cross-entropy loss and gradient descent. Let y be the true fine-grained label of the test sample x. j The cross-entropy loss here is calculated as follows:

[0109]

[0110] In practical use, the model makes inferences in the same way as above, but without gradient backpropagation and optimization.

[0111] In this embodiment, N=5, K=1 or K=5 is used to verify the small sample classification performance.

[0112] This embodiment combines and improves the collaborative optimization of weakly supervised and self-supervised learning paradigms, while also considering the intra-class and inter-class discriminability of coarse-grained categories, thus obtaining a small-sample classification model with more fine-grained discriminative capabilities.

[0113] The cross-granularity small sample classification method proposed in the above embodiments of the present invention can be widely used in various image classification tasks, especially in the application stage where fine-grained expansion of the supported classification categories is required.

[0114] Specifically, in another embodiment of the present invention, an image classification method is also provided, comprising:

[0115] Acquire images to be classified; the images can be any images that need to be classified, and the images contain the target to be classified. The acquisition method is not limited, and can be obtained directly by image acquisition equipment or provided by other image websites, etc.

[0116] Construct a fine-grained few-shot classification model for classification. The fine-grained few-shot classification model adopts the cross-granularity few-shot learning method based on similarity regularization intra-class mining in any of the above embodiments.

[0117] The image to be classified is input into the fine-grained few-sample classification model constructed above to achieve image classification.

[0118] For example, taking an image-based species classification task as an example: during training, only coarse-grained annotations (such as felines, canines, etc.) are needed on the training data. During the testing phase, only a small number of samples are collected for newly added fine-grained categories (such as cheetahs, snow leopards), resulting in a fine-grained small-sample classification model for the corresponding image classification task. Using this fine-grained small-sample classification model, classification ability for the new categories can be achieved, thus realizing the final image-based species classification. This embodiment of the invention significantly reduces the requirement for high-precision annotation of training data, lowers the practical cost of the annotation process, and has high industrial application value.

[0119] In another embodiment of the present invention, a terminal for image processing is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to run the cross-granularity few-sample learning method based on similarity regularization intra-class mining in any of the above embodiments, or to execute the image classification method in the above embodiments.

[0120] Tables 1-3 below show the numerical comparison results of the final recognition accuracy obtained based on this embodiment, which are the results on the LIVING-17 database, NONLIVING-26 database, and tieredImageNet database, respectively. This embodiment is compared with the most advanced related methods internationally. Reference method one implements a regularization of inter-instance similarity but does not incorporate coarse-grained weak supervision loss (see Chen, X., & He, K. (2021). Exploring simple siameserepresentation learning. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758). Reference method two implements angle normalization to leverage the synergistic effect between coarse-grained supervision and self-supervision (see Bukchin, G., Schwartz, E., Saenko, K., Shahar, O., Feris, R., Giryes, R., & Karlinsky, L. (2021). Fine-grained angularcontrastive learning with coarse labels. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition). (Pages 8730-8740), referencing Method 3 to explicitly generate fine-grained categories from coarse-grained distributions (see Ni, J., Cheng, W., Chen, Z., Asakura, T., Soma, T., Kato, S., & Chen, H. (2021, September). Superclass-Conditional Gaussian Mixture Model For Learning Fine-Grained Embeddings. In International Conference on Learning Representations.)

[0121] Table 1

[0122]

[0123] Table 2

[0124]

[0125] Table 3

[0126]

[0127] Table 1-3 lists, from top to bottom, the numerical comparisons between other results used for comparison and the results of this embodiment. It can be seen that this embodiment demonstrates a significant performance lead in accuracy across different datasets. Compared to existing methods, this invention achieves better cross-granularity few-sample learning performance.

[0128] In summary, the steps of this invention combine and improve the collaborative optimization of weakly supervised and self-supervised learning paradigms, while considering the intra-class and inter-class discriminative power of coarse-grained categories, resulting in a small-sample classification model with more fine-grained discriminative capabilities. Based on a given coarse-grained category distribution, it automatically learns the inter-sample differences within the coarse-grained category manifold, reducing performance loss due to missing fine-grained annotations, eliminating the need for additional fine-grained annotation work, and lowering annotation costs. Simultaneously, through external supervision of the inter-category structure, it enhances the model's understanding of high-level image features, further improving recognition accuracy.

[0129] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention. The above preferred features can be used in any combination without conflict.

Claims

1. A cross-granularity few-sample learning method based on similarity regularization within-class mining, characterized in that, include: Use an image feature extraction network to generate features for training image samples; Based on the image sample features, a first optimization loop is implemented, using weak supervision to constrain the semantic discriminativeness between coarse-grained categories of the deep learning model; Based on the image sample features, a second optimization loop is implemented, which uses a symmetric feature embedder and an asymmetric feature predictor to achieve coarse-grained similarity regularization constraints within categories. The losses of the first optimization loop and the second optimization loop are added together, and the error is backpropagated end-to-end across the entire network structure to optimize the network parameters, resulting in an optimized image feature extraction network. By using scenario-based fine-tuning, the optimized image feature extraction network is adapted to the few-shot classification task, ultimately resulting in a fine-grained few-shot classification model. The process of generating training image sample features using an image feature extraction network includes: Construct the image feature extraction network; Define an augmented set of data; Image samples in the training set are used to generate hidden layer views based on the data augmentation set and the image feature extraction network; The image samples in the training set are used to generate hidden layer views based on the data augmentation set and the image feature extraction network, including: For each image sample in the training set, two augmentation functions are obtained by sampling through the data augmentation set in each iteration; Based on the two augmentation functions, each image sample is projected as a positive sample pair; For the two sample matrices in the positive sample pair, the image feature extraction network is used to project them from the sample space into a real vector space of a set dimension to obtain two augmented training image sample features, which are called hidden layer views. Based on the image sample features, a second optimization loop is implemented, using a symmetric feature embedder and an asymmetric feature predictor to achieve coarse-grained similarity regularization constraints within categories, including: Based on a multilayer perceptron, a three-layer feature embedder and a two-layer feature predictor are constructed. Symmetric embedding prediction is performed on the hidden layer view based on the feature embedder and feature predictor, wherein the hidden layer view is projected into an embedding vector using the feature embedder; mutual prediction between the embedding vectors is performed using the feature predictor to obtain a prediction vector; and the cosine similarity between the prediction vector corresponding to one embedding vector and the other embedding vector itself is calculated. Symmetric embedding prediction is performed on the hidden layer view, wherein the coarse-grained internal self-supervised loss is calculated as follows: ； in, This is a gradient truncation operation to prevent feature collapse between samples on both sides. and These are the prediction vector and embedding vector corresponding to a hidden view, respectively. and These are the prediction vector and embedding vector corresponding to another hidden view, respectively, and CS represents the cosine similarity.

2. The method for cross-granularity few-sample learning based on similarity regularization intra-class mining according to claim 1, characterized in that, It also includes one or more of the following options: - The image feature extraction network includes several supervised deep convolutional neural networks and a global average pooling layer that outputs data of a set dimension. The input image is processed in the deep convolutional neural network to reduce the feature resolution and increase the number of feature channels, thereby extracting high-dimensional features. The high-dimensional features are then processed by the global average pooling layer, which takes a global average of the features of each channel to finally obtain a feature vector of the set dimension. - The data augmentation set includes at least five sets of image transformation techniques, namely: Random deformation cropping: For any input image, a portion of it is randomly selected, the size of which is limited to 20%-100% of the entire image, and then resampled to the specified size required by the image feature extraction network; Random color jitter: For any input image, perform random color transformation on it with a set probability, wherein the color transformation includes adjusting one or more of brightness, contrast, saturation and hue; Random grayscale conversion: For any input image, perform random grayscale conversion with a set probability to convert the color image into a grayscale image with the same values ​​for each channel; Random Gaussian Blur: For any input image, perform random Gaussian blur with a set probability, and the standard deviation of the Gaussian kernel is between [0.1, 2.0]. Random horizontal flip: For any input image, randomly flip it horizontally with a set probability.

3. The method for cross-granularity few-sample learning based on similarity regularization intra-class mining according to claim 1, characterized in that, Based on the image sample features, a first optimization loop is implemented, which uses weak supervision to constrain the coarse-grained inter-category semantic discriminative ability of the deep learning model, including: A coarse-grained class classifier is formed based on a fully connected neural network and a Softmax activation layer; The training image sample features are used as input, and the coarse-grained category classifier is used to obtain the coarse-grained category probability. Based on coarse-grained category probabilities, a coarse-grained image classification task is performed, cross-entropy loss is calculated as coarse-grained inter-class semantic supervision, and the error is backpropagated.

4. The cross-granularity few-sample learning method based on similarity regularization intra-class mining according to claim 3, characterized in that, The process of performing coarse-grained image classification based on coarse-grained category probabilities and calculating cross-entropy loss includes: Let the set of coarse-grained categories be... The input hidden view Then the classifier Perform a mapping from the feature space to the class label space. And obtain the forward propagation output. : ； remember The j-th dimension is And the true coarse-grained labels of the image samples are The cross-entropy loss for coarse-grained classification is calculated as follows: 。 5. The cross-granularity few-sample learning method based on similarity regularization intra-class mining according to claim 1, characterized in that, The scenario-based fine-tuning adapts the optimized image feature extraction network to the few-shot classification task, ultimately resulting in a fine-grained few-shot classification model, including: Sampling a small number of classification scenarios from the training set; Image features are extracted from the several small sample classification scenarios using an image feature extractor optimized by the first and second optimization loops. The extracted image features are combined with cosine distance and a parameterless differentiable nearest neighbor classifier to perform a few-shot classification task. Cross-entropy loss is calculated, and error backpropagation and gradient descent are performed to obtain the final fine-grained few-shot classification model.

6. An image classification method, characterized in that, include: Obtain the image to be classified; Construct a fine-grained few-sample classification model for classification, wherein the fine-grained few-sample classification model adopts the cross-granularity few-sample learning method based on similarity regularization intra-class mining as described in any one of claims 1-5; The image to be classified is input into a fine-grained few-sample classification model to achieve image classification.

7. A terminal for image processing, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it can be used to run the cross-granularity few-sample learning method based on similarity regularization intra-class mining as described in any one of claims 1-5, or to execute the image classification method as described in claim 6.

Images