An image classification method based on contrastive learning pre-training and hyperbolic space

Abstract

The application discloses an image classification method based on contrast learning pre-training and hyperbolic space, comprising: obtaining a target image to be identified; using an auxiliary locator to generate a class activation map to obtain a foreground region; inputting the target image and the foreground region into a proposed pre-training framework for contrast learning between global-global, global-local and local-local; inputting the target image into a pre-trained deep neural network to obtain an image-level feature vector and a region-level feature vector of each region, and converting to a hyperbolic space; splicing the image-level and the most discriminative region-level feature vectors; inputting a fully connected layer defined in the hyperbolic space to obtain an identification result; the pre-training framework can relieve the attention error of the model after contrast learning, the classes in the hyperbolic space present a tree structure, which is more conducive to classifying classes, and the region information is added, so that the image classification ability can be finally improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent recognition technology, and in particular to an image classification method based on contrastive learning pre-training and hyperbolic space. Background Technology

[0002] Deep learning has seen tremendous development in various fields in recent years and is gradually being applied across different areas. Among the various branches of deep learning, image recognition is a relatively mature field, with deep learning models maintaining state-of-the-art (SOTA) performance in various image recognition tasks. However, in classic image recognition tasks, the target category is usually a coarse-grained broad category, such as dog, fruit, or bird. In fine-grained image recognition, however, the target category is usually different subcategories within a broad category, such as different breeds of dogs.

[0003] In summary, fine-grained recognition images are characterized by: 1) Significant intra-class differences and inter-class similarities among fine-grained categories. For example, in the CUB dataset, the postures and backgrounds of the same type of seagull vary greatly, while different species of seagulls show high similarity, with differences existing only in small areas such as beaks, wings, or feathers. 2) Fine-grained images contain complex background information. Complex background information not only fails to provide effective recognition information but also interferes with the model's learning of features at informative locations. For example, in images of birds in a forest, the birds are usually in the background of trees, making it difficult to distinguish them from the background. Sometimes, background occlusion can also cause only part of the object to appear in the image.

[0004] Based on a review of recent research on fine-grained localization, methods based on localization-classification subnetworks are among the most prominent solutions. Early localization-classification subnetworks relied heavily on annotation information during training, utilizing object-level and part-level annotations to obtain local part localization. These methods can be termed localization-classification subnetwork methods utilizing strongly supervised information. These methods not only use image-level category labels but also introduce object-level or part-level manually labeled information. Zhang et al. (Zhang N, Donahue J, Girshick R, et al. Part-based R-CNNs for Fine-grained Category Detection[J]. Springer International Publishing, 2014.) proposed a part-based region-convolutional neural network (PB R-CNN), which learns a part detector through bottom-up region recommendation and adds geometric prior knowledge to improve localization accuracy. Existing methods utilizing strongly supervised information depend on object-level and part-level annotation information, but such annotation information is often unavailable in practice. Moreover, the above models have not improved the ability to extract image features. Therefore, this invention proposes a classification model based on hyperbolic space.

[0005] Meanwhile, existing contrastive learning pre-training algorithms, such as Moco (He K, Fan H, Wu Y, et al. Momentum contrast for unsupervised visual representation learning[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition.2020:9729-9738.) and SimCLR (Chen T, Kornblith S, Norouzi M, et al. A simple framework for contrastive learning of visual representations[C] / / International conference on machine learning.PMLR,2020:1597-1607.), only consider image-level contrastive learning and ignore the influence of the background. This makes it easy for the model to learn background information during the pre-training stage. Therefore, this invention proposes a new contrastive learning algorithm that introduces region-level contrastive learning through an auxiliary locator to reduce the influence of the background. Summary of the Invention

[0006] The purpose of this invention is to provide an image classification method based on contrastive learning pre-training and hyperbolic space, so as to reduce the problem of poor recognition ability of deep neural network models when the background influence is large and the differences between target object classes are small.

[0007] The present invention is achieved by at least one of the following technical solutions.

[0008] An image classification method based on contrastive learning pre-training and hyperbolic space includes the following steps:

[0009] Acquire the target image to be identified;

[0010] The target image is input into the auxiliary locator to generate a class activation map, thus obtaining the foreground region;

[0011] The target image and the foreground region are input into the proposed pre-trained framework for global-global, global-local, and local-local comparative learning.

[0012] The target image is input into a pre-trained deep neural network to obtain image-level feature vectors and region-level feature vectors for each region, which are then transformed into hyperbolic space.

[0013] The image-level feature vector is concatenated with the most discriminative region-level feature vector;

[0014] Input a fully connected layer defined in hyperbolic space to obtain the recognition results.

[0015] Furthermore, the auxiliary locator is a classification model pre-trained on ImageNet from the timm library.

[0016] Furthermore, the auxiliary locator employs a Grad-CAM-like activation graph algorithm.

[0017] Furthermore, Grad-CAM is:

[0018]

[0019]

[0020] Where A represents a feature layer, k represents the k-th channel in A, and A k This represents the data of channel k in feature layer A. This represents the data at coordinate ij in channel k of feature layer A. Z equals the width × height of the feature layer. max The output value represents the category with the highest confidence in a multi-class classification. When the representative category is the one with the highest confidence level, it refers to A. k The weight, This represents the final class activation graph.

[0021] Furthermore, after obtaining the class activation map weights, post-processing such as threshold segmentation and contour extraction is performed to obtain the final class activation map bounding box. The foreground region is then cropped and scaled according to the coordinates.

[0022] Furthermore, the pre-trained framework includes a Siamese network on the left and a Siamese network on the right, both of which include two encoders with the same structure and a fully connected layer.

[0023] Further, after image enhancement, the target image is encoded into a feature vector in the Siamese network on the left, generating positive sample pairs G1 and G2. The first negative sample, N1, is stored in the first negative sample queue. The foreground region image obtained in step S2 is first shuffled using a mosaic technique to eliminate dependencies between object parts. It is then input into the Siamese network on the right, generating positive sample pairs P1 and P2. The second negative sample queue stores the second negative sample, N2. The process then proceeds to the loss function calculation stage, which includes global-global, global-local, and local-local calculations, as shown in the following formula:

[0024]

[0025]

[0026]

[0027] l represents the length of the negative sample queue;

[0028] The loss function for the pre-trained framework based on contrastive learning is:

[0029] Loss 总 =Loss 全局-全局 +αLoss 全局-局部 +βLoss 局部-局部

[0030] α and β are hyperparameter weights; after calculating the loss, negative sample queue one and negative sample queue two are updated respectively.

[0031] Furthermore, the target image is input into a pre-trained deep neural network to obtain image-level feature vectors and region-level feature vectors for each region, which are then transformed into hyperbolic space, specifically including:

[0032] The input target image is fed into the encoder to generate image-level feature vectors. The proposed local image is generated through an equidistant sliding window. The proposed local image is fed into the encoder to generate region-level feature vectors. All the obtained feature vectors are converted into feature vectors in hyperbolic space.

[0033] Furthermore, the model in hyperbolic space is a Poincaré disk model. Its curvature parameter is c:

[0034]

[0035] D n ={x∈R n :c||x|| 2 <1,c≥0}

[0036] in It is the conformal factor, g E =I n It is the Euclidean metric tensor; g D D is the Riemannian metric. n Let x be an n-dimensional sphere; let x be a base point in space; R n For European-style space; I n It is the identity matrix;

[0037] Any vector in hyperbolic space The addition is defined as:

[0038]

[0039] For a certain base point vector Transform vector v from Euclidean space to hyperbolic space. Exponential mapping Defined as:

[0040]

[0041] The base point x is set to 0. The conformal factor represents the curvature of c and the base point x.

[0042] Furthermore, pseudo-labels are first obtained by classifying image-level feature vectors, and then all region-level feature vectors are classified. The region with the highest confidence in the pseudo-label category is selected as the most discriminative region. After obtaining the required region feature vectors, the two information are merged by concatenation.

[0043] Compared with existing technologies, the beneficial effects of this invention are as follows: by incorporating contrastive learning of local regions during pre-training, the generalization ability of the model in subsequent classification tasks is improved; by transforming images from Euclidean space to a more hierarchical hyperbolic space, the classification ability of the model is improved; and by incorporating regional features that help the model make judgments, the classification ability of the model is further improved. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is an algorithm flowchart of an image classification method based on contrastive learning pre-training and hyperbolic space in an example of the present invention;

[0046] Figure 2 This is an example of how an auxiliary locator is used to calculate the class activation map in an embodiment of the present invention to obtain an instance map of the foreground region;

[0047] Figure 3 This is a flowchart of the model of the pre-training framework based on foreground regions proposed in the embodiments of the present invention;

[0048] Figure 4 This is a flowchart of a classification model based on hyperbolic space in an example of the present invention. Detailed Implementation

[0049] To enable those skilled in the art to better understand the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] In the field of image recognition, particularly fine-grained image recognition, the following challenges exist: (1) deep learning models are greatly affected by the background; (2) the differences between target object categories are small; and (3) in practical applications, only image-level labels are typically obtained, and more informative labels such as object-level labels are not available. Therefore, these challenges make image recognition tasks in such scenarios particularly challenging.

[0051] To overcome the above problems, this invention proposes an image classification method based on contrastive learning pre-training and hyperbolic space. This invention first performs a pre-training phase, generating class activation maps for the target image using an auxiliary locator, extracting foreground regions, and removing background interference to obtain object region images. The target image and region images are then input into the proposed pre-training framework for global-global, global-local, and local-local contrastive learning, yielding model initialization parameters superior to those obtained through random initialization and pre-training on the ImageNet dataset. This concludes the pre-training phase, leading to the classification phase. In this phase, the target image first enters a feature extractor to generate image-level feature vectors. Then, by applying a sliding window to the target image, images of different regions are generated and input into the feature extractor to generate region-level feature vectors. All obtained feature vectors are input into a hyperbolic space transformation layer to transform them into vector representations in hyperbolic space. The region-level feature vectors are then input into a fully connected layer defined in hyperbolic space for discriminative recognition, obtaining the region-level feature vector with the highest discriminative power. This region-level feature vector is concatenated with the image-level feature vector to generate the final feature vector, which is then used for classification to obtain the recognition result. By providing pre-trained parameters that are more focused on the object region and introducing a hyperbolic space that is more advantageous for classification and the most discriminative region information, the impact of the aforementioned difficulties can be reduced, thus improving the model's recognition ability. The following sections will describe each step in detail.

[0052] refer to Figure 1 The deep learning model in the proposed invention first undergoes contrastive learning pre-training to obtain better model initialization parameters. (Reference) Figure 3This invention proposes a contrastive learning pre-training method based on class activation graphs. It's important to note that since the subsequent classification stage uses a network model defined in hyperbolic space, consistency must be maintained during pre-training, and the model should also undergo contrastive learning in hyperbolic space. The network model used in the subsequent classification stage is referenced... Figure 4 .

[0053] An image classification method based on contrastive learning pre-training and hyperbolic space includes the following steps:

[0054] Step S1: Obtain the target image to be identified.

[0055] In this embodiment, the acquired target images only have image-level category labels and do not contain more specific label information such as object-level labels like bounding boxes in object detection tasks. Therefore, this task can be defined as a weakly supervised image classification task. The images only need to undergo conventional preprocessing methods such as size transformation and standardization before they can be input into the network.

[0056] Step S2: Input the auxiliary locator to calculate the class activation map and obtain the foreground region.

[0057] As a preferred embodiment, the auxiliary locator is a ResNet pre-trained on ImageNet from the timm open-source library. This model does not participate in updates throughout the process. During the pre-training phase, the input image is first processed by the auxiliary locator to generate a class activation map. In this invention, the class activation map algorithm used is Grad-CAM. Although the auxiliary locator is not trained on the target image, its initialization parameters, pre-trained on ImageNet, enable the model to distinguish between foreground and background; therefore, it can be used for foreground localization. The original Grad-CAM formula is:

[0058]

[0059]

[0060] Where A represents a feature layer, k represents the k-th channel in A, and A k This represents the data of channel k in feature layer A. This represents the data at coordinate ij in channel k of feature layer A, where c represents category c. Representative targeting A k The weights, Z equal to the width × height of the feature layer, y c This represents the output value for category c. This represents the class activation graph at this point.

[0061] As the formula shows, class activation graph calculation requires class information, but labels are not introduced in self-supervised pre-training. Therefore, the original Grad-CAM formula needs to be modified, and the modified formula is as follows:

[0062]

[0063]

[0064] Where y max The output value represents the category with the highest confidence in a multi-class classification. When the representative category is the one with the highest confidence level, it refers to A. k The weight, Represents the final class activation graph

[0065] As the formula shows, the gradient value used here changes from the original specified category to the gradient value calculated from the category with the highest classification confidence. After obtaining the class activation map weights, post-processing such as threshold segmentation and contour extraction is performed to obtain the final class activation map bounding box. The foreground region is then cropped according to its coordinates and scaled to the model input size. See the effect reference. Figure 2 .

[0066] Step S3: Input the target image and the foreground region into the pre-training framework to perform comparative learning between global-global, global-local, and local-local regions.

[0067] like Figure 3 As shown, in this embodiment, the pre-training framework consists of two branches, left and right. First, the target image, after image enhancement, is input into the Siamese network on the left and encoded as a feature vector, generating positive sample pairs G1 and G2. The first negative sample, N1, is stored in the negative sample queue one. The foreground region image obtained in step S2 is first shuffled using a jigsaw puzzle technique to eliminate dependencies between object parts, and then input into the Siamese network on the right for encoding, generating positive sample pairs P1 and P2. The second negative sample queue two stores the second negative sample, N2. The aforementioned Siamese networks all consist of two encoders with identical structures and fully connected layers. The loss function calculation stage then proceeds, consisting of three parts: global-global, global-local, and local-local, as shown in the following formula:

[0068]

[0069]

[0070]

[0071] l represents the length of the negative sample queue.

[0072] In contrastive learning, random cropping is a key aspect of data augmentation. However, the location of random cropping can lead to attention errors in the model after training, meaning the model easily learns background information. By introducing a global and local loss term, the model's attention scope can be constrained, focusing it on the object. Furthermore, since image augmentation using a mosaic technique on the region image breaks the connections between object parts, the model can better learn part features. Therefore, the loss function of the proposed contrastive learning framework is:

[0073] Loss 总 =Loss 全局-全局 +αLoss 全局-局部 +βLoss 局部-局部

[0074] α and β are hyperparameter weights.

[0075] After calculating the loss, update negative sample queue one and negative sample queue two respectively.

[0076] Step S4: Input the target image into the pre-trained deep neural network to obtain image-level feature vectors and region-level feature vectors of each region, and convert them to hyperbolic space.

[0077] In this embodiment, the encoder is initialized with the parameters obtained after pre-training in step S3, and the classification model structure based on hyperbolic space is as follows: Figure 4 As shown, the input target image is first fed into the encoder to generate image-level feature vectors, and then a local image to be suggested is generated through an equidistant sliding window. This local image is then fed into the encoder to generate region-level feature vectors, and all the obtained feature vectors are converted into feature vectors in hyperbolic space.

[0078] Traditional deep learning models essentially involve performing various geometric transformations in Euclidean space. However, when the data distribution exhibits non-Euclidean geometry, such as tree-like characteristics, the representational power of Euclidean space becomes limited. In contrast, most natural data in the open world is inherently hierarchical, making hyperbolic space highly suitable for such data. Furthermore, hyperbolic space possesses another desirable property: it allows for embedding using low-dimensional manifolds without sacrificing model accuracy or representational power.

[0079] As a preferred embodiment, a Poincaré disk model in hyperbolic space is used. Its curvature parameter is c (the actual curvature value is -c). 2 The Poincaré disk model was implemented as a model with Riemannian metric. n-dimensional sphere D n ={x∈R n :c||x|| 2 <1,c≥0}, where It is the conformal factor, gE =I n It is a Euclidean metric tensor, where x is a point in space, and R is a metric tensor. n For European-style space, I n It is the identity matrix. This means that the local distance near the sphere boundary will be factored by λ. c Infinite scaling produces the property of "space dilation" in hyperbolic space, which enables the network to make accurate predictions even with low-dimensional embeddings.

[0080] Hyperbolic space is not a vector space, and its operational formulas also need to be modified for any vector in hyperbolic space. Their sum is defined as:

[0081]

[0082] For a certain base point vector Transform vector v from Euclidean space to hyperbolic space. Exponential mapping Defined as:

[0083]

[0084] The base point x is typically set to 0. The conformal factor represents the curvature of c and the base point x.

[0085] Step S5: Concatenate the image-level feature vector with the most discriminative region-level feature vector.

[0086] In this embodiment, after step S4, both image-level and region-level feature vectors are represented in hyperbolic space. The most discriminative region-level feature vector needs to be selected. The criterion for evaluating discriminability in this step is classification confidence. Since there is no label information during prediction, pseudo-labels are first obtained by classifying the image-level feature vectors. Then, all region-level feature vectors are classified, and the region with the highest confidence in the pseudo-label category is selected as the most discriminative region. It should be noted that the classifiers used in steps S5 and S6 are fully connected layers defined in hyperbolic space. After obtaining the required region feature vectors, a concatenation operation is used to fuse the two information. If the dimensions of the image-level and region-level feature vectors are B×N×C, where B is the batch size, N is the feature length, and C is the number of channels, then the concatenated feature vector is B×2N×C.

[0087] Step S6: Input the fully connected layer defined in hyperbolic space to obtain the recognition result.

[0088] In this example, the feature vector obtained in step S5 is input into a fully connected layer defined in hyperbolic space to obtain the final recognition result.

[0089] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. An image classification method based on contrastive learning pre-training and hyperbolic space, characterized in that, Includes the following steps: Acquire the target image to be identified; The target image is input into the auxiliary locator to generate a class activation map, thus obtaining the foreground region; The target image and the foreground region are input into the proposed pre-trained framework for global-global, global-local, and local-local comparative learning. The target image is input into a pre-trained deep neural network to obtain image-level feature vectors and region-level feature vectors for each region, which are then transformed into hyperbolic space. The target image is input into a pre-trained deep neural network to obtain image-level feature vectors and region-level feature vectors for each region, which are then transformed into hyperbolic space. Specifically, this includes: The input target image is fed into the encoder to generate image-level feature vectors. The proposed local image is generated through an equidistant sliding window. The proposed local image is fed into the encoder to generate region-level feature vectors. All the obtained feature vectors are converted into feature vectors in hyperbolic space. The model in hyperbolic space is the Poincaré disk model. Its curvature parameter is c: ； in It is a conformal factor. It is the Euclidean metric tensor; Riemannian measurement; It is an n-dimensional sphere; It is a base point in space; It is a European-style space; It is the identity matrix; Any vector in hyperbolic space The addition is defined as: For a certain base point vector , will vector v Transition from European-style space to hyperbolic space Exponential mapping Defined as: Among the base points x Set to 0, The representative curvature is c, and the base point is x Conformal factor at time; The image-level feature vector is concatenated with the most discriminative region-level feature vector; Input a fully connected layer defined in hyperbolic space to obtain the recognition results.

2. The image classification method based on contrastive learning pre-training and hyperbolic space according to claim 1, characterized in that, The auxiliary locator is a classification model pre-trained on ImageNet from the timm library.

3. The image classification method based on contrastive learning pre-training and hyperbolic space according to claim 1, characterized in that, The auxiliary locator uses the Grad-CAM class activation graph algorithm.

4. The image classification method based on contrastive learning pre-training and hyperbolic space according to claim 3, characterized in that, Grad-CAM is: Where A represents a feature layer, and k represents the k-th channel in A. This represents the data of channel k in feature layer A. The feature layer A is located in channel k, with coordinates as follows: ij For the data at this location, Z equals the width × height of the feature layer. The output value represents the category with the highest confidence in a multi-class classification. When the representative category is the one with the highest confidence level, it is for The weight, This represents the final class activation graph.

5. The image classification method based on contrastive learning pre-training and hyperbolic space according to claim 1, characterized in that, After obtaining the class activation map weights, post-processing such as threshold segmentation and contour extraction is performed to obtain the final class activation map bounding box. The foreground region is then cropped and scaled according to the coordinates.

6. The image classification method based on contrastive learning pre-training and hyperbolic space according to claim 1, characterized in that, The pre-training framework includes a Siamese network on the left and a Siamese network on the right, both of which include two encoders with the same structure and a fully connected layer.

7. The image classification method based on contrastive learning pre-training and hyperbolic space according to claim 6, characterized in that, After image enhancement, the target image is encoded into a feature vector in the Siamese network on the left, generating positive sample pairs G1 and G2. The first negative sample N1 is stored in the negative sample queue one. The foreground region image obtained in step S2 is first shuffled by a jigsaw puzzle method. This step is to eliminate the dependencies between object parts. Then it is input into the Siamese network on the right for encoding, generating positive sample pairs P1 and P2. The second negative sample queue two stores the second negative sample N2. The loss function calculation stage is then entered, which includes: global-global, global-local, and local-local components, as shown in the following formula: This represents the length of the negative sample queue. The loss function for the pre-trained framework based on contrastive learning is: , The weights are hyperparameters; after calculating the loss, the negative sample queue one and the negative sample queue two are updated respectively.

8. The image classification method based on contrastive learning pre-training and hyperbolic space according to claim 1, characterized in that, First, pseudo-labels are obtained by classifying image-level feature vectors. Then, all region-level feature vectors are classified, and the region with the highest confidence in the pseudo-label category is selected as the most discriminative region. After obtaining the required region feature vectors, the two information are merged by concatenation.

Images