Image classification method based on multi-mode and hyperbolic space

Abstract

The invention discloses an image classification method based on multi-mode and hyperbolic space. The method comprises the steps of obtaining a to-be-recognized target image and a category text corresponding to the image; performing fine tuning on the large model by using a given target image and a category text to obtain a candidate category text suggestion device; constructing an image classification model comprising a deep neural network, a candidate category text suggestion device, a multi-modal fusion module, a feature mask module, an Euclidean space classifier and a hyperbolic space classifier; and training the image classification model by using a given target image and a category text, and classifying the target image to be identified by using the trained image classification model. According to the method, extra classification information is provided for the model, and the feature quality is improved by using the feature mask module and the hyperbolic space classification head, so that the image classification capability can be improved finally.

Description

Technical Field

[0001] The present invention relates to the field of intelligent recognition technology, and in particular to an image classification method based on multimodality and hyperbolic space. Background Art

[0002] At present, deep neural network models are crucial in image classification tasks such as fine-grained image recognition and industrial defect recognition. However, in tasks such as fine-grained image recognition, there are difficulties such as bottlenecks in image classification information, large influence of background content, and small differences between classes. These difficulties limit the recognition ability of deep neural network models in these fields and make it difficult to achieve a high recognition accuracy.

[0003] In summary, how to improve image classification information, reduce background influence, and improve recognition ability on images with small inter-class differences are technical problems that technical personnel in this field urgently need to solve.

[0004] In recent years, many excellent studies have emerged in the field of multimodality, demonstrating the great potential of multimodal models. In multimodal research, data from different perceptual modes are usually combined to obtain more comprehensive and accurate information and understanding. Among them, modal information similar to text contains descriptive information about the image, which can provide additional discriminant information and solve the problem that classification information is prone to bottlenecks in image classification models.

[0005] At present, there are some works that introduce other modal information similar to text into fine-grained image recognition, such as "A crack detection method, device, storage medium and electronic device" (CN118038281A). However, these methods require additional labels of the corresponding modalities, and such data are often not available in prediction. Summary of the invention

[0006] The purpose of the present invention is to provide an image classification method based on multimodal and hyperbolic space to reduce the problems of image classification information bottleneck, great background influence, and poor recognition ability when the difference between target object classes is small in deep neural network models.

[0007] The present invention proposes a solution of a candidate category text suggester and a multimodal fusion module. Only image input is required, and the possible category text of the image is suggested through a large model to obtain the required text information and perform multimodal fusion. At the same time, a feature mask module and a hyperbolic space classification head are introduced to obtain more discriminative fine-grained features under an end-to-end architecture.

[0008] The purpose of the present invention is achieved by at least one of the following technical solutions.

[0009] A method for image classification based on multimodality and hyperbolic space comprises the following steps:

[0010] S1, obtain the target image to be identified and the category text corresponding to the image;

[0011] S2. Fine-tune the large model using the given target image and category text to obtain a candidate category text suggester;

[0012] S3, build an image classification model including a deep neural network, a candidate category text suggester, a multimodal fusion module, a feature mask module, a Euclidean space classifier, and a hyperbolic space classifier;

[0013] S4. Train the image classification model using the given target image and category text, and use the trained image classification model to classify the target image to be identified.

[0014] Furthermore, in step S2, the existing known CLIP (Contrastive Language-Image Pre-Training) large model is used, and then an image adapter and a text adapter are respectively added after the image encoder and text encoder of the large model, and the residual connections with residual ratios α and β are respectively set to retain the original knowledge to complete the construction of the candidate category text suggester, and then given the target image and category text obtained in step S1, the candidate category text suggester is trained using the original CLIP loss function to obtain a trained candidate category text suggester; wherein the residual ratios α and β are hyperparameters and the image encoder and text encoder parameters are frozen.

[0015] Further, in step S3, in the image classification model, the target image is input into a deep neural network to obtain original image features;

[0016] The target image is input to the candidate category text suggester, which calculates the similarity between the input target image and all the category texts related to the task, and uses the text features of a set number of category names with the highest similarity as the candidate category text features;

[0017] The candidate category text features and the original image features are input into the multimodal fusion module and fused to obtain multimodal features;

[0018] The original image features are input into the feature mask module to calculate the mask degree, and the original image features are masked by the mask degree to obtain the image mask features;

[0019] The original image features are converted to the hyperbolic space through exponential mapping and input into the hyperbolic space classifier, which outputs the hyperbolic space prediction results;

[0020] The original image features, multimodal features and image mask features are concatenated and input into the Euclidean space classifier to output the Euclidean space prediction results.

[0021] Furthermore, the deep learning network includes a transformer model;

[0022] The deep learning network converts the input image into a high-dimensional feature vector, which contains the semantic features of the image. These feature vectors are usually generated at a deeper level of the network, and the original image feature vector obtained will be used for subsequent tasks.

[0023] Furthermore, the multimodal fusion module utilizes multi-head self-attention to form a multimodal fusion layer, and fuses the input original image features and the candidate category text features to obtain multimodal features.

[0024] Furthermore, the feature mask module is composed of fully connected layers.

[0025] Furthermore, the hyperbolic space classifier adopts a fully connected layer of the hyperbolic space.

[0026] Furthermore, the Euclidean space classifier adopts a fully connected layer of the Euclidean space.

[0027] Furthermore, in step S4, the image classification model is trained and optimized by using the overall loss function; the overall loss function Loss is specifically as follows:

[0028] Loss=Loss Concat +γLoss Hyp

[0029] Among them, Loss Concat Represents the classification loss of the concatenated vector obtained by concatenating the original image features, multimodal features, and image mask features. Hyp is the classification loss in hyperbolic space and γ is a hyperparameter.

[0030] Furthermore, in step S4, during the training process, the candidate category text suggester parameters are frozen throughout the training process.

[0031] Compared with the prior art, the advantages of the present invention are:

[0032] The image classification model proposed in the present invention can use text to increase the discriminative information in the final classification and use the feature mask module to reduce the influence of the background in the recognition task, thereby improving the adaptability of the model. At the same time, when facing recognition tasks such as fine-grained image recognition, which has the characteristic of small inter-class differences, the hyperbolic space is used to improve the accuracy of the model recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 It is an algorithm flow chart of an image classification method based on multimodality and hyperbolic space in an example of the present invention;

[0035] Figure 2 is a flow chart of a fine-tuning model of a candidate category text suggester in the present invention;

[0036] Figure 3 is a model flow chart of the feature mask module proposed in the example of the present invention;

[0037] Figure 4 It is a model flow chart of an image classification method based on multimodality and hyperbolic space in an example of the present invention. DETAILED DESCRIPTION

[0038] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0039] In the field of image recognition similar to fine-grained image recognition, there are the following difficulties:

[0040] (1) There is a bottleneck in image classification information;

[0041] (2) Deep learning models are greatly affected by background;

[0042] (3) There is little difference between the categories of target objects.

[0043] Therefore, based on the difficulties mentioned above, image recognition tasks in such scenarios are challenging.

[0044] In order to overcome the above problems, the present invention proposes an image classification method based on multimodality and hyperbolic space.

[0045] A multimodal and hyperbolic space-based image classification method, such as Figure 1 As shown, the following steps are included:

[0046] S1, obtain the target image to be identified and the category text corresponding to the image;

[0047] In this embodiment, the target image to be identified and the category text corresponding to the image are obtained, wherein the obtained target images only have image-level category labels and do not carry more specific label information such as the object-level label of the bounding box in the target detection task. Therefore, this task can be defined as a weakly supervised image classification task.

[0048] S2. Fine-tune the large model using the given target image and category text to obtain a candidate category text suggester;

[0049] like Figure 2 As shown, the existing known CLIP (Contrastive Language-Image Pre-Training) large model is used, and then the image adapter and text adapter are added after the image encoder and text encoder of the large model respectively, and the residual connection of the residual ratio α and β is set to retain the original knowledge respectively, and the candidate category text suggester is constructed. Then, given the target image and category text obtained in step S1, the candidate category text suggester is trained using the loss function consistent with the original CLIP model to obtain the trained candidate category text suggester. Among them, the residual ratios α and β are hyperparameters and the image encoder and text encoder are frozen parameters.

[0050] S3, such as Figure 4 As shown, an image classification model including a deep neural network, a candidate category text suggester, a multimodal fusion module, a feature mask module, a Euclidean space classifier, and a hyperbolic space classifier is constructed;

[0051] In the image classification model, the target image is input into the deep neural network to obtain the original image features;

[0052] The target image is input to the candidate category text suggester, which calculates the similarity between the input target image and all the category texts related to the task, and uses the text features of a set number of category names with the highest similarity as the candidate category text features;

[0053] The candidate category text features and the original image features are input into the multimodal fusion module and fused to obtain multimodal features. In fine-grained image recognition, since the area supporting discrimination in the target image is small, the classification information is prone to bottlenecks, making the image difficult to distinguish. Although fine-grained images are difficult to distinguish, their category texts may not be. In addition, when people name objects, they will associate the descriptive information of the object. For example, the name "Yellow Headed Blackbird" among birds indicates that the bird has the characteristics of a yellow head and a black body. Therefore, by introducing the category name of the image for multimodal fusion, the discriminative information of the model can be supplemented and the classification ability of the model can be improved.

[0054] In one embodiment, the multimodal fusion module uses multi-head self-attention to form a multimodal fusion layer to fuse the input original image features and the candidate category text features to obtain multimodal features.

[0055] The original image features are input into the feature mask module to calculate the mask degree, and the original image features are masked by the mask degree to obtain the image mask features;

[0056] In one embodiment, Figure 3 As shown, the feature mask module consists of fully connected layers;

[0057] Assume that the input is image features of size (B, N, D), where B is the batch size, N is the number of channels, and D is the feature length. The process of masking image features using a feature masker can be defined as: input all features (N, D) of each image into the feature masker, map them to (N, C), and then calculate the mask degree, which is defined as:

[0058] Score mask = -max(x)

[0059] Where x is a feature vector of dimension (N, C), and finally a mask degree vector of size (N, 1) is obtained, where C is equal to the number of categories. The mask ratio is set, and the features with higher mask degrees in N dimensions are masked to obtain the image mask features after removing irrelevant information. This helps to reduce the complexity of the feature space, highlight the key features related to the task, and thus improve the model's ability to understand and process the input image.

[0060] The original image features are converted to the hyperbolic space through exponential mapping and input into the hyperbolic space classifier, which outputs the hyperbolic space prediction results;

[0061] In one embodiment, the hyperbolic space classifier uses a fully connected layer of the hyperbolic space;

[0062] Generally, deep learning models are defined in the discriminant space of Euclidean distance, which results in close distances between different categories in the feature space. However, in hyperbolic space, the shortest path between points is no longer a straight line in Euclidean space, but appears in the form of a curve. This change enhances the separability between objects. Unlike Euclidean space, the volume of hyperbolic space grows exponentially with the growth of radius, which means that for a given embedding length, the model has more powerful representation capabilities and is more suitable for processing tree-structured data. Most natural data are hierarchical data with tree structures. For example, fine-grained images actually belong to different subcategories under the same category, that is, different child nodes of the same parent node. Therefore, hyperbolic space is suitable as a representation space for fine-grained images.

[0063] In one embodiment, the Poincare disk model in hyperbolic space is used. Its curvature parameter is c (the actual curvature value is -c 2 ). The model is implemented as a Riemannian metric The n-dimensional sphere D n = {x∈R n :c||x|| 2 <1,c≥0}, where is the conformal factor, g E =I n is the Euclidean metric tensor. This means that local distances close to the sphere boundary are increased by a factor of λ c Infinite scaling produces the property of "spatial expansion" in hyperbolic space, which enables the network prediction to be accurate even under low-dimensional embedding.

[0064] Hyperbolic space is not a vector space. It has its own operation formulas. Their sum is defined as:

[0065]

[0066] So for a certain base point Convert vector v from Euclidean space to hyperbolic space The exponential mapping of can be defined as:

[0067]

[0068] The base point x is usually set to 0.

[0069] Converting features into hyperbolic space can increase the separability between fine-grained features, thereby improving the model's ability to learn discriminative features and resist interference features, and helps to learn features related to the essence of the category. The final separability of the original image features before exponential mapping will be effectively improved.

[0070] The original image features, multimodal features and image mask features are concatenated and input into the Euclidean space classifier to output the Euclidean space prediction results.

[0071] In one embodiment, if the feature vector dimension is B×C, where B is the batch size and C is the feature length, the concatenated feature vector is B×3C.

[0072] In one embodiment, the Euclidean space classifier uses a fully connected layer in the Euclidean space;

[0073] S4. Use the given target image and category text to train the image classification model, and use the trained image classification model to classify the target image to be identified.

[0074] In one embodiment, the image classification model is trained and optimized by using an overall loss function; the overall loss function Loss is specifically as follows:

[0075] Loss=Loss Concat +γLoss Hyp

[0076] Among them, Loss Concat Represents the classification loss of the concatenated vector obtained by concatenating the original image features, multimodal features, and image mask features. Hyp is the classification loss in hyperbolic space and γ is a hyperparameter.

[0077] During the training process, the candidate category text suggester parameters are frozen throughout.

Claims

1. An image classification method based on multimodal and hyperbolic space, characterized in that: The following steps are involved: S1, obtain the target image to be identified and the category text corresponding to the image; S2. Fine-tune the large model using the given target image and category text to obtain a candidate category text suggester; S3, build an image classification model including a deep neural network, a candidate category text suggester, a multimodal fusion module, a feature mask module, a Euclidean space classifier, and a hyperbolic space classifier; S4. Train the image classification model using the given target image and category text, and use the trained image classification model to classify the target image to be identified.

2. The image classification method based on multimodal and hyperbolic space according to claim 1, characterized in that: In step S2, the existing known CLIP large model is used, and then an image adapter and a text adapter are added after the image encoder and text encoder of the large model respectively, and the residual connections with residual ratios α and β are set respectively to retain the original knowledge to complete the construction of the candidate category text suggester, and then given the target image and category text obtained in step S1, the candidate category text suggester is trained using the original CLIP loss function to obtain a trained candidate category text suggester; wherein the residual ratios α and β are hyperparameters and the image encoder and text encoder parameters are frozen.

3. The image classification method based on multimodal and hyperbolic space according to claim 1, characterized in that: In step S3, in the image classification model, the target image is input into the deep neural network to obtain the original image features; The target image is input to the candidate category text suggester, which calculates the similarity between the input target image and all the category texts related to the task, and uses the text features of a set number of category names with the highest similarity as the candidate category text features; The candidate category text features and the original image features are input into the multimodal fusion module and fused to obtain multimodal features; The original image features are input into the feature mask module to calculate the mask degree, and the original image features are masked by the mask degree to obtain the image mask features; The original image features are converted to the hyperbolic space through exponential mapping and input into the hyperbolic space classifier, which outputs the hyperbolic space prediction results; The original image features, multimodal features and image mask features are concatenated and input into the Euclidean space classifier to output the Euclidean space prediction results.

4. The image classification method based on multimodal and hyperbolic space according to claim 3, characterized in that: The deep learning network includes a transformer model; The deep learning network converts the input image into a high-dimensional feature vector, which contains the semantic features of the image. These feature vectors are usually generated at a deeper level of the network, and the original image feature vector obtained will be used for subsequent tasks.

5. The image classification method based on multimodal and hyperbolic space according to claim 3, characterized in that: The multimodal fusion module utilizes multi-head self-attention to form a multimodal fusion layer, which fuses the input original image features and the candidate category text features to obtain multimodal features.

6. The image classification method based on multimodal and hyperbolic space according to claim 3, characterized in that: The feature mask module consists of fully connected layers.

7. The image classification method based on multimodal and hyperbolic space according to claim 3, characterized in that: The hyperbolic space classifier uses a fully connected layer in the hyperbolic space.

8. The image classification method based on multimodal and hyperbolic space according to claim 3, characterized in that: The Euclidean space classifier uses a fully connected layer in the Euclidean space.

9. The image classification method based on multimodal and hyperbolic space according to claim 3, characterized in that: In step S4, the image classification model is trained and optimized by the overall loss function; the overall loss function Loss is as follows: Loss=Loss Concat +γLoss Hyp Among them, Loss Concat Represents the classification loss of the concatenated vector obtained by concatenating the original image features, multimodal features, and image mask features. Hyp is the classification loss in hyperbolic space and γ is a hyperparameter.

10. The image classification method based on multimodal and hyperbolic space according to claim 9, characterized in that: In step S4, during the training process, the candidate category text suggester parameters are frozen throughout the training process.

Images