Image classification method, apparatus, electronic device, medium, and computer program product

Abstract

The present disclosure provides an image classification method and device, electronic equipment, medium and computer program product, comprising obtaining similarity between prompt data and an image to be classified based on a visual language model, the prompt data comprising at least one instance image, the prompt data corresponding to at least one category; determining the category of the image to be classified based on the category of the prompt data corresponding to the maximum value in the similarity. The present disclosure uses prompt data containing instance images to classify the image to be classified through a pre-trained visual language model. The instance images are equivalent to telling the model what the image to be classified can refer to, which can improve the accuracy of the classification model and eliminate the ambiguity of words and phrases. The structure or parameters of the original model are not changed, and its generalization ability is not affected. Moreover, no training or fine-tuning is required, achieving low cost and flexibility. Moreover, the instance images can be dynamically updated or supplemented at model runtime, and are not dependent on a specific model or use environment.

Description

Technical Field

[0001] This disclosure relates to the field of image classification, and in particular to an image classification method, apparatus, electronic device, medium, and computer program product. Background Technology

[0002] Image classification is a technology that automatically categorizes images into specific classes. It is widely used in various fields such as industry, medicine, and security, and can improve image processing speed and the level of automation in decision-making.

[0003] In recent years, Visual Language Models (VLMs) have combined image and text information to learn the association between image content and text descriptions on a large number of general datasets. This enables the models to understand the semantic connections between images and text, achieving powerful zero-shot inference capabilities. That is, for image categories not seen during training, they can classify target samples using textual prompts. However, when there are significant domain differences between the target task dataset and these general datasets, directly applying these models for inference still struggles to achieve high accuracy.

[0004] Visual language models such as CLIP, SigLIP, and MobileCLIP perform well on zero-shot image classification tasks. Their ability to classify images into their untrained categories makes them suitable for a wide range of tasks.

[0005] Dual encoder VLMs, exemplified by CLIP, consist of two main components: a text encoder and an image encoder. The standard use of CLIP is for semantic classification. The text encoder and image encoder encode text cues, such as "a picture of a dog" or "a picture of a cat," and the input image into the same dimension within the same embedding space, respectively. Then, the cosine similarity between the text vector labels and the image vector labels is evaluated to determine which category the image belongs to, such as "dog" or "cat."

[0006] Although these VLMs have been pre-trained on large datasets, their generalization knowledge may be insufficient in specific scenarios, and there are many special cases that may lead to confusion with other items. Summary of the Invention

[0007] The technical problem to be solved by this disclosure is to overcome the deficiencies in the prior art and provide an image classification method, including the following steps:

[0008] The similarity between the prompt data and the image to be classified is obtained based on a visual language model. The prompt data includes at least one instance image and corresponds to at least one category.

[0009] The category of the image to be classified is determined based on the category of the prompt data corresponding to the maximum value in the similarity score.

[0010] Optionally, obtaining the similarity between the cue data and the image to be classified based on the visual language model specifically includes:

[0011] The encoder based on the visual language model obtains the first vector label and the image vector label corresponding to the prompt data and the image to be classified, respectively;

[0012] The similarity between the first vector label and the image vector label is obtained as the similarity between the corresponding prompt data and the image to be classified.

[0013] Optionally, determining the category of the image to be classified based on the category of the prompt data corresponding to the maximum value in the similarity score specifically includes:

[0014] The similarity scores of the corresponding categories are merged to obtain a similarity sequence;

[0015] The category of the image to be classified is determined based on the category of the prompt data corresponding to the maximum value in the similarity sequence.

[0016] Optionally, the prompt data may also include at least one text prompt.

[0017] The step of merging the similarities of corresponding categories in the similarity data to obtain a similarity sequence specifically includes:

[0018] The similarity scores of the instance image and the image to be classified are combined to obtain the first similarity sequence;

[0019] The similarity scores of the text prompt and the image to be classified are combined to obtain a second similarity sequence;

[0020] The first similarity sequence and the second similarity sequence are normalized to obtain the normalized first similarity sequence and the normalized second similarity sequence, respectively.

[0021] The similarity sequence is obtained by merging the similarity scores of corresponding categories in the normalized first similarity sequence and the normalized second similarity sequence;

[0022] or,

[0023] The first similarity sequence and the second similarity sequence are normalized together to obtain a normalized similarity sequence;

[0024] The similarity sequences are obtained by merging the similarity scores of corresponding categories in the normalized similarity sequences.

[0025] Optionally, the image to be classified is a stitched image, and the stitched image includes at least two stitched regions;

[0026] The similarity between the prompt data and the image to be classified, obtained based on the visual language model, specifically includes:

[0027] The similarity between the prompt data and the at least two spliced ​​regions is obtained based on the visual language model.

[0028] The step of determining the category of the image to be classified based on the category of the prompt data corresponding to the maximum value in the similarity score specifically includes:

[0029] The categories of the at least two spliced ​​regions are determined based on the category of the prompt data corresponding to the maximum value in the similarity.

[0030] Optionally, obtaining the similarity between the prompt data and the at least two spliced ​​regions based on the visual language model specifically includes:

[0031] The encoder based on the pre-trained visual language model converts the stitched image into an image embedding;

[0032] At least two region embeddings are extracted from the image embedding, and each region embedding corresponds to a splicing region;

[0033] The at least two region embeddings are converted to the size of the image embedding to obtain at least two region vector labels;

[0034] The similarity between the first vector marker and the at least two region vector markers is obtained as the similarity between the prompt data and the at least two spliced ​​regions.

[0035] Optionally, the image encoder of the pre-trained visual language model has independent, modular, and separable pooling layers, and an interest region alignment layer is added between the feature extraction layer and the pooling layer of the image encoder of the pre-trained visual language model. The interest region alignment layer is used to convert the region embedding into the size of the image embedding to obtain the region vector label.

[0036] Optionally, the at least two splicing regions are the same size.

[0037] This disclosure also provides an image classification apparatus, comprising:

[0038] A similarity acquisition module is used to obtain the similarity between prompt data and the image to be classified based on a visual language model. The prompt data includes at least one instance image and corresponds to at least one category.

[0039] The category determination module is used to determine the category of the image to be classified based on the category of the prompt data corresponding to the maximum value in the similarity.

[0040] In another aspect of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and for running on the processor, wherein the processor executes the computer program to implement the image classification method described in any one of the above descriptions.

[0041] In another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the image classification method described in any one of the above descriptions.

[0042] In another aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the image classification method as described in any of the preceding descriptions.

[0043] Based on common knowledge in the field, the above-mentioned preferred conditions can be combined arbitrarily to obtain various preferred embodiments of this disclosure.

[0044] The positive advancements of this disclosure are as follows: This disclosure uses cue data, which can be simply instance images, or instance images and text cues. A pre-trained visual language model is used to classify the image to be classified. The instance images essentially inform the model of the possible representations of the image to be classified, thus improving the accuracy of the classification model and eliminating ambiguity related to words and phrases. The instance images are encoded by the image encoder of the visual language model, without altering the structure or parameters of the original model or affecting its generalization ability. Furthermore, no training or fine-tuning is required, achieving low cost and flexibility. Moreover, the instance images can be dynamically updated or supplemented during model runtime, independent of any specific model or usage environment. Attached Figure Description

[0045] Figure 1 A schematic diagram illustrating the implementation environment of an image classification method provided in this embodiment of the disclosure;

[0046] Figure 2 A flowchart of an image classification method provided in Embodiment 1 of this disclosure;

[0047] Figure 3 This is a flowchart illustrating an image classification method provided in Embodiment 1 of this disclosure;

[0048] Figure 4A schematic flowchart illustrating another implementation of an image classification method provided in Embodiment 1 of this disclosure;

[0049] Figure 5 This is a schematic diagram of the process of merging similarity scores in an image classification method provided in Embodiment 2 of this disclosure;

[0050] Figure 6 This is a flowchart illustrating the merging of similarities in an image classification method provided in Embodiment 2 of this disclosure;

[0051] Figure 7 This is a schematic diagram of obtaining the region of interest provided in Embodiment 3 of this disclosure;

[0052] Figure 8 This is a schematic diagram of the stitched image provided in Embodiment 3 of this disclosure;

[0053] Figure 9 This is a flowchart illustrating the image classification method provided in Embodiment 3 of this disclosure;

[0054] Figure 10 This is a schematic diagram of the process for obtaining the similarity between the stitched region and the prompt data in the image classification method provided in Embodiment 3 of this disclosure;

[0055] Figure 11 This is a flowchart illustrating the image classification method provided in Embodiment 3 of this disclosure;

[0056] Figure 12 This is a schematic diagram of the frame of the image classification device provided in Embodiment 4 of this disclosure;

[0057] Figure 13 This is a schematic diagram of an electronic device provided in Embodiment 5 of this disclosure. Detailed Implementation

[0058] The present disclosure is further illustrated below by way of embodiments, but the present disclosure is not limited to the scope of the embodiments described herein.

[0059] The prefixes such as "first" and "second" used in this disclosure are merely for distinguishing different descriptive objects and do not limit the position, order, priority, quantity, or content of the described objects. The use of ordinal numbers and other prefixes used to distinguish descriptive objects in this disclosure does not constitute a limitation on the described objects. The description of the described objects is given in the context of the embodiments, and the use of such prefixes should not constitute unnecessary restrictions. Furthermore, in the description of this embodiment, unless otherwise stated, "multiple" means two or more.

[0060] like Figure 1The diagram shown is a schematic of the implementation environment of the image classification method provided in this application embodiment, which includes a terminal 110 and a server 120.

[0061] Terminal 110 is an electronic device with image acquisition function. This electronic device can be a smartphone, tablet, personal computer equipped with a camera, or recognition device (such as a smart camera or road recognition probe), etc. Figure 1 In this context, terminal 110 refers to a mobile phone equipped with a camera.

[0062] Optionally, the terminal 110 may have a pre-installed application with image classification functionality, or it may follow a social media account that provides image classification services (such as a public account), or it may run a mini-program that depends on other applications (such as a mini-program in an instant messaging application). When image classification is required, the terminal 110 uploads the image to be classified through the pre-installed application, social media account, or mini-program.

[0063] Terminal 110 and server 120 are connected via wired or wireless network.

[0064] Server 120 is a single server, a server cluster consisting of several servers, or a cloud computing center. In one possible implementation, server 120 is the backend server for a pre-installed application in terminal 110, the backend server for a social media account, or the backend server for a mini-program.

[0065] In this embodiment, server 120 is used for image classification. Optionally, server 120 stores an image classification model for image classification.

[0066] In one possible application scenario, such as Figure 1 As shown, terminal 110 captures an image to be classified through a camera and sends the image to server 120. Server 120 inputs the image to be classified into a visual language model, obtains the image classification result output by the visual language model, and feeds back the image classification result to terminal 110.

[0067] In another possible application scenario, when the terminal 110 stores an image classification model, the terminal 110 can use the local image classification model to obtain the classification result of the image to be classified and upload the classification result to the server 120.

[0068] Optionally, the terminal 110 or the server 120 implements image classification functionality through a neural network chip.

[0069] Optionally, the aforementioned wireless or wired networks use standard communication technologies and / or protocols. The network is typically the Internet, but can also be any network, including but not limited to Local Area Networks (LANs), Metropolitan Area Networks (MANs), Wide Area Networks (WANs), mobile, wired or wireless networks, private networks, or any combination of virtual private networks. In some embodiments, technologies and / or formats, including Hypertext Markup Language (HTML) and Extensible Markup Language (XML), are used to represent data exchanged over the network. Furthermore, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Networks (VPNs), and Internet Protocol Security (IPsec) can be used to encrypt all or some links. In other embodiments, custom and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0070] It should be understood that Figure 1 The number of terminal devices and servers shown is merely illustrative. Depending on implementation needs, there can be any number of terminal devices and servers. For example, server 120 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0071] The following is an introduction and explanation of some terms used in this disclosure:

[0072] Multimodal learning

[0073] Multimodal learning is a subfield of deep learning that focuses on training models to process and integrate various types (modalities) of data, such as text, images, audio, and video. Single-modal data may carry only partial information and lack contextual understanding. Multimodal models interpret and combine information from different data sources to achieve a comprehensive and robust understanding that more closely resembles how humans interact with the world.

[0074] Visual Language Model (VLM)

[0075] Visual Models (VLMs) are multimodal models capable of jointly processing and understanding visual and textual data, combining computer vision and natural language processing. VLMs can take both images and text as input and transform them into a shared embedding space through cross-modal alignment. VLMs have a wide range of applications, such as image captioning, visual question answering, image search and retrieval, object detection and segmentation, etc.

[0076] Comparative learning

[0077] Contrastive learning is a machine learning technique that maps data into an embedding space where similar data are close together and dissimilar data are far apart. Contrastive learning defines similar samples as positive pairs and dissimilar samples as negative pairs, and learns this embedding space by minimizing the distance between positive pairs and maximizing the distance between negative pairs. Contrastive learning is widely used in computer vision tasks such as image recognition, object detection, image segmentation, and image retrieval.

[0078] Zero-sample testing

[0079] Zero-shot detection is a task that detects new objects that have not been seen during training, relying on the semantic relationship between seen and unseen classes.

[0080] Transformer

[0081] A deformer is a deep neural network model that incorporates a multi-head attention mechanism and a feedforward neural network.

[0082] ViT

[0083] Vision Transformer is a variant of Transformer used for image processing in computer vision.

[0084] Token

[0085] The basic unit of information representation within a Transformer is typically a high-dimensional vector used to represent a rectangular region (patch) on an image, which is also referred to as a vector label in this embodiment.

[0086] CLIP

[0087] CLIP, a visual language model, integrates text and images. Its core idea is to learn visual concepts from natural language supervision. CLIP employs a dual-encoder architecture, where a Vision Transformer-based image encoder processes images, and a Transformer-based text encoder processes text cues. The CLIP model is trained on a large dataset containing 400 million image-text pairs to align image and text embeddings into the same embedding space. CLIP learns its multimodal embedding space by maximizing the cosine similarity of matching image-text pairs and minimizing the cosine similarity of mismatched pairs. Unlike computer vision models trained on specific datasets for specific tasks, CLIP is highly effective for zero-shot image classification without requiring dataset-specific or task-specific training.

[0088] SigLIP

[0089] A visual language model using a similar architecture to CLIP, but replacing CLIP's softmax-based contrastive loss with a sigmoid loss. The sigmoid loss does not require global normalization across the entire batch, thus significantly optimizing memory efficiency and computational cost, allowing for larger batch sizes. SigLIP achieves higher efficiency and accuracy than its predecessor and serves as a better alternative for a similarly wide range of tasks as CLIP.

[0090] MobileCLIP

[0091] A visual language model is proposed to bring the power of large image-text foundational models, such as MobileCLIP, to resource-constrained mobile devices. MobileCLIP employs a hybrid convolutional Transformer architecture enhanced through structural reparameterization to achieve low latency and small size. MobileCLIP also utilizes a dual-encoder architecture, using FastViT-based MCI as the hybrid visual Transformer and Text-RepMixer as the hybrid text encoder. A significant optimization in learning efficiency comes from a multimodal reinforcement training strategy that transfers knowledge from a high-performance CLIP model ensemble and an image caption generator.

[0092] Visual language models (VLMs) such as CLIP, SigLIP, and MobileCLIP perform well on zero-shot image classification tasks, demonstrating a strong ability to classify images into untrained categories. Although these VLMs have been pre-trained on large datasets, their generalization knowledge may still be insufficient in specific scenarios, and many special cases can lead to confusion between the object to be classified and other objects. For example, an orange robotic arm might sometimes be identified as a human hand by some CLIP variants. Eliminating this confusion requires a large number of cues for improvement. Users may need to evaluate many different descriptions to find the most relevant one as a classification cue.

[0093] Therefore, this disclosure uses instance classification as a supplement to semantic classification, taking sample images corresponding to multiple categories as instance images, using a pre-trained visual language model to obtain the corresponding instance vector labels of the instance images, and using the instance vector labels to calculate similarity with the image vector labels of the image to be identified in order to predict the category of the image to be identified.

[0094] Example 1

[0095] Figure 2 This is a flowchart illustrating an image classification method provided in Embodiment 1 of this disclosure. As shown in Figure 2, the method includes:

[0096] S10, obtain the similarity between the prompt data and the image to be classified based on the visual language model, wherein the prompt data includes at least one instance image and the prompt data corresponds to at least one category;

[0097] S20, determine the category of the image to be classified based on the category of the prompt data corresponding to the maximum value in the similarity.

[0098] The visual language model is a pre-trained visual language model with an image encoder and a text encoder. This pre-trained model can obtain aligned text-image vector pairs; that is, text encoded by the text encoder yields text vectors, and images encoded by the image encoder yield image vectors. The text and image vectors are aligned within the same embedding space to facilitate subsequent similarity calculations. The pre-trained visual language model can employ, for example, SigLIP or MobileCLIP.

[0099] The prompt data includes at least one instance image and may also include text prompts, that is, it may include both instance images and text prompts. The prompt data corresponds to at least one category; for example, the instance images include images of animals from multiple categories such as cats and dogs.

[0100] Since a pre-trained visual language model can convert both the image to be classified and the prompt data into aligned vector labels, the similarity between these vector labels can be used as the similarity between the image to be classified and the prompt data. The similarity between vector labels can be calculated using cosine similarity, Euclidean distance, Manhattan distance, or other methods. For example, the CLIP model uses cosine similarity.

[0101] The category of the image to be classified is determined based on the category of the prompt data corresponding to the maximum similarity between the prompt data and the image to be classified. For example, if the image to be classified has the highest similarity with an instance image of the category "cat" among multiple instance images, then the category of the image to be classified is determined to be "cat".

[0102] like Figure 3 , 4 As shown, obtaining the similarity between the prompt data and the image to be classified based on the visual language model specifically includes:

[0103] S11, The encoder based on the visual language model obtains the first vector label and image vector label corresponding to the prompt data and the image to be classified, respectively;

[0104] S12, obtain the similarity between the first vector label and the image vector label as the similarity between the corresponding prompt data and the image to be classified.

[0105] Correspondingly, the prompt data is input into the encoder of a pre-trained visual language model to obtain a first vector label. When the prompt data includes text prompts and instance images, the first vector label includes an instance vector label corresponding to the instance image and a text vector label corresponding to the text prompt. Specifically, as shown... Figure 3 As shown, when the prompt data only includes instance images, the first vector label only includes instance vector labels. The instance vector labels can be obtained by inputting the instance image into the image encoder of a pre-trained visual language model, as shown below. Figure 4 As shown, when the prompt data includes an instance image and a text prompt, the first vector label includes an instance vector label and a text vector label. The text prompt is input into the text encoder of a pre-trained visual language model to obtain the text vector label, and the instance image is input into the image encoder of the pre-trained visual language model to obtain the instance vector label.

[0106] The similarity is calculated using cosine similarity. The cue data can be known category data, such as existing instance images and text cues with category labels, or newly added instance images, such as images that previously failed to be classified. The first vector label can be obtained by inputting the cue data into the visual language model when detecting the image to be classified, or it can be pre-calculated and stored by the encoder of a pre-trained visual language model. If the first vector label is pre-calculated and stored, it can be directly loaded when transforming the image to be classified without recalculation, resulting in a faster response speed for the entire image classification method.

[0107] like Figure 3 As shown, taking the example that the prompt data only contains instance images, the cosine similarity between each instance vector label and the image vector label is calculated. The similarity between the instance vector label and the image vector label is used as the similarity between the corresponding instance image and the image to be classified. Then, based on the category of the instance image corresponding to the maximum value of the similarity, the category of the image to be classified is determined.

[0108] As another optional implementation, determining the category of the image to be classified based on the category of the prompt data corresponding to the maximum value in the similarity score specifically includes:

[0109] S21, merge the similarities of the corresponding categories in the similarity to obtain a similarity sequence;

[0110] S22, the category of the image to be classified is determined based on the category of the prompt data corresponding to the maximum value in the similarity sequence.

[0111] Taking instance images as the only cue data as an example, instance images can contain multiple categories, and each category has at least one corresponding instance image. Therefore, after calculating the cosine similarity between each instance vector label and the image vector label, the similarities between instance vector labels and image vector labels of the same category are grouped together, and the similarities of each group are averaged. For example, when instance images include animal pictures such as cats, dogs, and horses, the similarities between multiple cat instance images and the image to be classified are averaged to obtain the similarity between the cue data of category cat and the image to be classified. Similarly, the similarities between the cue data of category dog ​​and the image to be classified, and the similarities between the cue data of category horse and the image to be classified are obtained, thus obtaining a similarity sequence. The category of the cue data corresponding to the maximum value in the similarity sequence is determined as the category of the image to be classified. Averaging the similarities of corresponding categories is equivalent to first averaging the instance vector labels of corresponding categories and then calculating the similarity, because this process is a linear transformation. The two approaches yield the same numerical results, but the physical significance of the latter lies in the fact that the averaged instance vector labels are general and can represent a category, while a single image may have specific characteristics.

[0112] Image classification using visual language models typically employs semantic classification based on textual descriptions for general objects. However, for objects like human hands, skin color and posture can vary, and textual descriptions struggle to capture all possibilities. Therefore, instance classification using instance images can significantly improve the accuracy of zero-shot classification, particularly in cases where the model consistently fails. Instance classification can quickly correct classification failures for specific or unknown objects by adding or removing samples from the instance image set, without requiring model retraining or fine-tuning. Telling the model what the target object looks like using instance images is far more accurate than describing it with abstract statements; therefore, the image-based method in this embodiment effectively eliminates the ambiguity of words and phrases, resulting in more accurate classifications.

[0113] Example 2

[0114] This embodiment 2 is based on the above embodiment 1. When the prompt data includes multiple types of data, that is, when it simultaneously includes text prompts and instance images, and the first vector marker includes instance vector markers corresponding to instance images and text vector markers corresponding to text prompts, there can be multiple processing methods for the step of merging the similarity of the same category in the similarity in the above embodiment 1 to obtain a similarity sequence.

[0115] As one implementation method, such as Figure 5 , 6 As shown, step S21 specifically includes:

[0116] S211, merge the similarity scores of the corresponding categories between the instance image and the image to be classified to obtain a first similarity sequence;

[0117] Specifically, such as Figure 5 As shown, the similarity scores of the instance vector labels and image vector labels within the same category are combined and averaged to obtain a first similarity sequence corresponding to multiple categories. For example, instance images include... The corresponding instance vector is then labeled as Let m+1 be the number of instance images, and I be the image vector label. Then, the cosine similarity between the instance vector label and the image vector label is obtained as follows: If they are all different categories, there is no need to merge them. If they are in the same category, they need to be merged and averaged to obtain the first similarity sequence.

[0118] S212, merge the similarity scores of the text prompt and the image to be classified into the corresponding category to obtain a second similarity sequence;

[0119] Specifically, the similarity scores of corresponding categories between the text vector tags and image vector tags are combined and averaged to obtain a second similarity sequence corresponding to multiple categories, for example:

[0120] The text prompt corresponds to n+1 represents the number of text prompts; the corresponding text vector is labeled as follows: The image vector is labeled I, and the cosine similarity between the text vector label and the image vector label is obtained as follows: If they are all different categories, there is no need to merge them. If they are in the same category, the cosine similarity of the same category needs to be averaged to obtain the second similarity sequence.

[0121] S213, normalize the first similarity sequence and the second similarity sequence respectively to obtain a normalized first similarity sequence and a normalized second similarity sequence;

[0122] S214, the similarity scores of corresponding categories in the normalized first similarity sequence and the normalized second similarity sequence are merged to obtain the similarity sequence;

[0123] Specifically, for example, if both the first similarity sequence and the second similarity sequence include the categories "hand" and "robotic hand" respectively, the first similarity sequence is first normalized to obtain the probability of the first similarity between the two categories. , ;in, Furthermore, the second similarity sequence is also normalized to obtain the probability of the second similarity between the two categories. , ,in To classify into categories and Merging to obtain , and Merging to obtain To obtain the above similarity sequence { , Finally, the category of the image to be classified is determined based on the larger similarity in the similarity sequence: whether it is a hand or a robotic hand.

[0124] As another implementation method

[0125] S211, merge the similarity scores of the corresponding categories between the instance image and the image to be classified to obtain a first similarity sequence;

[0126] S212, merge the similarity scores of the text prompt and the image to be classified into the corresponding category to obtain a second similarity sequence;

[0127] The two steps described above are the same as those in the above implementation method, and will not be repeated here.

[0128] S213', Normalize the first similarity sequence and the second similarity sequence together to obtain a normalized similarity sequence;

[0129] S214', merge the similarities of corresponding categories in the normalized similarity sequence to obtain the similarity sequence.

[0130] Specifically, for example, if both the first and second similarity sequences include the categories "hand" and "robotic hand" respectively, the similarity scores between the two categories in the first similarity sequence and the similarity scores between the two categories in the second similarity sequence are normalized together to obtain the corresponding probabilities. , , , ;and Then sort them into categories of the same type. and Merging to obtain , and Merging to obtain Based on the similarity sequence { The larger the similarity between the two values, the more likely the image will be classified as a hand or a robotic hand.

[0131] In practical applications, both of the above similarity processing methods can be used. However, the matching degree between text-image vector labels differs from that between image-image vector labels. Even though the text and image vector labels are aligned, common normalization weakens the contribution of either side. Therefore, when text and instance vector labels correspond to the exact same category, separate normalization is generally used. Conversely, when the categories corresponding to text and instance vector labels are not entirely the same—for example, when the text prompt contains multiple categories while the instance image contains only a few—common normalization is generally used.

[0132] This embodiment discloses a case where text prompts and instance images are used together as the first vector label, adding an instance classification mechanism in addition to the standard usage of pre-trained VLMs, namely semantic classification. Instance images can significantly improve classification accuracy, as they convey information about the specific context of the application, avoiding ambiguity and being more effective than text prompts. Instance images are encoded by the image encoder of the original VLM, which does not change the structure or parameters of the original model, and therefore does not affect its generalization ability. Low cost and flexibility are achieved because no training or fine-tuning is required. Instance images can be dynamically updated at runtime and are independent of specific models or environments.

[0133] Example 3

[0134] In the third embodiment of this application, the image to be classified is a stitched image, and the stitched image includes at least two stitched regions.

[0135] like Figure 7 As shown, from Figure 7 The system can identify four regions of interest: a robotic arm, a robotic hand, a human hand, and a soft square bowl. Since these four regions of interest are of different sizes, they are scaled down to obtain a stitched area, which is then stitched together to form the image shown. Figure 8 The stitched image shown.

[0136] Figure 8 The image includes four stitching regions, preferably of the same size. To increase the processing speed of image classification, when detecting multiple images, multiple key regions from multiple images, multiple regions from a single image, or the entire image can be extracted as a Region of Interest (ROI) to obtain stitching regions. These stitching regions are then stitched together to obtain a stitched image. Classifying the stitched image using the methods described in Examples 1 and 2 allows for the simultaneous identification of the category corresponding to each stitched region within the multiple stitched regions.

[0137] Specifically, there are many mature techniques available for extracting regions of interest (ROIs) from the original image to be classified, which will not be elaborated upon here. The extracted ROIs can be scaled to form stitched regions of the same size, and there are many methods available for this, such as interpolation.

[0138] like Figure 9 As shown, when determining whether an image is a stitched image, an additional judgment can be made when inputting the image to be classified. Preferably, if it is a stitched image, it can be set to have a bounding box when forming the stitched image. The bounding box is used to determine the stitching area. Figure 9 In the process, it is possible to determine whether the image to be classified is a stitched image based on whether it has a bounding box.

[0139] Therefore, the similarity between the cue data and the image to be classified obtained based on the visual language model is, in other words, the similarity between the cue data and the at least two spliced ​​regions obtained based on the visual language model. More specifically, as... Figure 10 As shown, it includes the following steps:

[0140] S201, the encoder based on the pre-trained visual language model converts the stitched image into an image embedding; the stitched image is input into the image encoder of the pre-trained visual language model to obtain the overall image embedding, the size of which is the same as the size of the first vector label and the image vector label in embodiments 1 and 2 above.

[0141] S202, extract at least two region embeddings from the image embedding, each region embedding corresponding to a stitched region; that is, after inputting the stitched image as the image to be classified into the image encoder, the image embedding is then segmented into region embeddings, each region embedding corresponding to a stitched region. At this point, the stitched regions are of the same size, which facilitates segmentation. Specifically, region embedding segmentation can be performed based on bounding boxes.

[0142] S203, the at least two region embeddings are converted to the size of image embeddings to obtain at least two region vector labels; the segmented region embeddings are converted to the size of image embeddings, i.e., the size of the image vector labels in embodiments 1 and 2 above, to obtain at least two region vector labels corresponding to at least two splicing regions. The size of each region vector label is the same as the size of the first vector label, i.e., the same as the size of the image vector label. Therefore, the similarity between the region vector labels and the first vector label can be calculated as the similarity between the splicing region and the prompt data.

[0143] S204, obtain the similarity between the first vector marker and the at least two region vector markers as the similarity between the prompt data and the at least two stitched regions. In the above steps, at least two region vector markers corresponding to at least two stitched regions have been obtained through one input image encoder process. Subsequently, the same similarity calculation and category determination process can be performed to obtain the category of the stitched region corresponding to each of the at least two region vector markers.

[0144] Furthermore, determining the category of the image to be classified based on the category of the prompt data corresponding to the maximum value in the similarity specifically includes determining the category of the at least two spliced ​​regions based on the category of the prompt data corresponding to the maximum value in the similarity.

[0145] like Figure 11 As shown, to achieve the above steps, the encoder of the pre-trained visual language model needs to have independent, modular, and separable pooling layers. Specifically, a Region of Interest Align (ROI Align) layer is added between the feature extraction layer and the pooling layer of the image encoder of the pre-trained visual language model to modify its output. The ROI Align layer is a neural network layer used for object detection and instance segmentation tasks. Its main function is to accurately extract the features corresponding to each region of interest from the feature map. The specific structure of ROI Align includes: a segmentation function, used to segment the region of interest into sub-windows of the same size according to the output size; a sampling point calculation function, used to select four sampling points in each sub-window and calculate the features of each sampling point by performing bilinear interpolation using neighboring pixels; and a pooling function, used to take the maximum (max pooling) or average (average pooling) value of the four sampling points of each window as the final value of that window.

[0146] Leveraging the separability of image encoders with modular pooling layers, the stitched image is converted into an image embedding using the image encoder described above (with the pooling layer removed). Then, a ROI Align layer is used to extract region embeddings from the corresponding ROI locations in the converted image embedding. The region embeddings are also upsampled to the size of the standard image embedding using ROI Align. Finally, a pooling layer is used to pool the upsampled region embeddings into region vector labels. The final region vector labels still reside in the embedding space aligned with the text / instance / image vector labels.

[0147] Region embeddings are extracted from the original intermediate result, i.e., the image embedding, and then upsampled to obtain region features. Specifically, in the ROI Align layer, the image embedding is divided into fixed-size units according to the output size, i.e., the standard dimension of the image embedding. Then, bilinear interpolation is used to calculate the value of the corresponding sampling point, and finally, pooling is performed on the sampling points to obtain region features.

[0148] Furthermore, in this embodiment, if the text prompt and instance image are existing categories, that is, the text vector labels and instance vector labels have been calculated before, these labels can be directly loaded for calculating cosine similarity. If it is a newly added category, the text prompt and instance image are labeled and calculated based on the text encoder and image encoder.

[0149] In this embodiment, multiple regions of interest are extracted from the original image to be classified and stitched together to form a stitched image. The stitched image is then encoded to obtain an image embedding. Region embeddings are extracted from the image embeddings. Through processing, the region embeddings are transformed into region vector labels of the same size as the image vector labels. By calculating the cosine similarity, the classification of multiple regions of interest can be processed simultaneously, thus speeding up the processing.

[0150] To evaluate the performance of the above scheme, the MobileCLIP-S2 variant was used as the pre-trained language model and tested on the Caltech101 dataset (excluding "BACKGROUND_Google" images), the CIFAR-10 test dataset, and the EuroSAT dataset, which are classic datasets commonly used in the field of image classification.

[0151] In each dataset, images were randomly split into 30% instance images and 70% test images (images to be classified). The labels of the test images were used as text prompts for zero-shot testing. The text prompts were a set of prompts in the format "a photo of a{category_name}", where "category_name" refers to the image's category name. Performance was also tested using regions of interest (ROIs). The results are shown in the table below.

[0152] Table 1: Performance of the MobileCLIP-S2-based classification mechanism on the Caltech101 dataset for image classification and region classification.

[0153]

[0154] Table 2: Performance of the MobileCLIP-S2-based classification mechanism on the CIFAR-10 dataset for image and region classification.

[0155]

[0156] Table 3: Performance of the MobileCLIP-S2-based classification mechanism on the EuroSAT dataset for image classification and region classification.

[0157]

[0158] The instance images in the above tests were not deliberately selected but randomly chosen, yet instance classification still demonstrated high accuracy. Moreover, the accuracy of instance classification was significantly higher than that of semantic classification, especially on the EuroSAT dataset in Table 3. As can be seen from the three tables above, adding instance classification to semantic classification significantly improves classification accuracy compared to semantic classification alone.

[0159] As can be seen from the three tables above, using region vector labeling, i.e., classifying using stitched images, can significantly reduce image encoding latency, thus significantly improving classification speed. Although the accuracy of region vector labeling is lower than that of image vector labeling on some datasets, it can greatly reduce runtime, achieving a good latency-accuracy trade-off. Moreover, in semantic classification, the accuracy of region vector labeling is close to that of image vector labeling, indicating that classification using region vector labeling preserves the regional information of the image well and maintains the image-text cross-modal characteristics of the base model.

[0160] Example 4

[0161] Corresponding to the aforementioned image classification method embodiments, this disclosure also provides image classification device embodiments. Figure 12 This is a schematic diagram of a module of an image classification apparatus disclosed in an embodiment of the present disclosure. The apparatus includes:

[0162] Similarity acquisition module 1 is used to obtain the similarity between prompt data and image to be classified based on a visual language model, wherein the prompt data includes at least one instance image and the prompt data corresponds to at least one category;

[0163] Category determination module 2 is used to determine the category of the image to be classified based on the category of the prompt data corresponding to the maximum value in the similarity.

[0164] The visual language model is a pre-trained visual language model with an image encoder and a text encoder. This pre-trained model can obtain aligned text-image vector pairs; that is, text encoded by the text encoder yields text vectors, and images encoded by the image encoder yield image vectors. The text and image vectors are aligned within the same embedding space to facilitate subsequent similarity calculations. The pre-trained visual language model can employ, for example, SigLIP or MobileCLIP.

[0165] The prompt data includes at least one instance image and may also include text prompts, that is, it may include both instance images and text prompts. The prompt data corresponds to at least one category; for example, the instance images include images of animals from multiple categories such as cats and dogs.

[0166] Since a pre-trained visual language model can convert both the image to be classified and the prompt data into aligned vector labels, the similarity between these vector labels can be used as the similarity between the image to be classified and the prompt data. The similarity between vector labels can be calculated using cosine similarity, Euclidean distance, Manhattan distance, or other methods. For example, the CLIP model uses cosine similarity.

[0167] The category of the image to be classified is determined based on the category of the prompt data corresponding to the maximum similarity between the prompt data and the image to be classified. For example, if the image to be classified has the highest similarity with an instance image of the category "cat" among multiple instance images, then the category of the image to be classified is determined to be "cat".

[0168] Specifically, the similarity determination module 1 includes:

[0169] The conversion module 11 is used to obtain the first vector label and image vector label corresponding to the prompt data and the image to be classified, respectively, based on the encoder of the visual language model;

[0170] The similarity calculation module 12 is used to obtain the similarity between the first vector label and the image vector label as the similarity between the corresponding prompt data and the image to be classified.

[0171] Correspondingly, the prompt data is input into the encoder of a pre-trained visual language model to obtain a first vector label. When the prompt data includes text prompts and instance images, the first vector label for the instance image includes an instance vector label corresponding to the instance image and a text vector label corresponding to the text prompt. Specifically, as shown... Figure 3As shown, when the prompt data only includes instance images, the first vector label only includes instance vector labels. The instance vector labels can be obtained by inputting the instance image into the image encoder of a pre-trained visual language model, as shown below. Figure 4 As shown, when the prompt data includes an instance image and a text prompt, the first vector label includes an instance vector label and a text vector label. The text prompt is input into the text encoder of a pre-trained visual language model to obtain the text vector label, and the instance image is input into the image encoder of the pre-trained visual language model to obtain the instance vector label.

[0172] The similarity is calculated using cosine similarity. The cue data can be known category data, such as existing instance images and text cues with category labels, or newly added instance images, such as images that previously failed to be classified. The first vector label can be obtained by inputting the cue data into the visual language model when detecting the image to be classified, or it can be pre-calculated and stored by the encoder of a pre-trained visual language model. If the first vector label is pre-calculated and stored, it can be directly loaded when transforming the image to be classified without recalculation, resulting in a faster response speed for the entire image classification method.

[0173] like Figure 3 As shown, taking the example that the prompt data only contains instance images, the cosine similarity between each instance vector label and the image vector label is calculated. The similarity between the instance vector label and the image vector label is used as the similarity between the corresponding instance image and the image to be classified. Then, based on the category of the instance image corresponding to the maximum value of the similarity, the category of the image to be classified is determined.

[0174] The category determination module 2 includes:

[0175] The similarity sequence determination module 21 is used to merge the similarities of corresponding categories in the similarity to obtain a similarity sequence;

[0176] The determination module 22 is used to determine the category of the image to be classified based on the category of the prompt data corresponding to the maximum value in the similarity sequence.

[0177] Taking instance images as the only cue data as an example, instance images can contain multiple categories, and each category has at least one corresponding instance image. Therefore, after calculating the cosine similarity between each instance vector label and the image vector label, the similarities between instance vector labels and image vector labels of corresponding categories are grouped together, and the similarities of each group are averaged. For example, when instance images include animal pictures such as cats, dogs, and horses, the similarities between multiple cat instance images and the image to be classified are averaged to obtain the similarity between cue data of category cat and the image to be classified. Similarly, the similarities between cue data of category dog ​​and the image to be classified, and the similarities between cue data of category horse and the image to be classified are obtained, thus obtaining a similarity sequence. The category of the cue data corresponding to the maximum value in the similarity sequence is determined as the category of the image to be classified.

[0178] When the prompt data includes multiple types of data, that is, when it includes both text prompts and instance images, and the first vector label includes instance vector labels corresponding to the instance images and text vector labels corresponding to the text prompts, there can be multiple processing methods for the step of merging the similarities of the same category in the similarity in the above embodiment 1 to obtain a similarity sequence.

[0179] The similarity sequence determination module 21 further includes:

[0180] The first similarity sequence determination module 211 is used to merge the similarity of the corresponding categories between the instance image and the image to be classified to obtain the first similarity sequence;

[0181] Specifically, such as Figure 5 As shown, the similarity scores of the instance vector labels and image vector labels within the same category are combined and averaged to obtain a first similarity sequence corresponding to multiple categories. For example, instance images include... The corresponding instance vector is then labeled as Let m+1 be the number of instance images, and I be the image vector label. Then, the cosine similarity between the instance vector label and the image vector label is obtained as follows: If they are all different categories, there is no need to merge them. If they are in the same category, they need to be merged and averaged to obtain the first similarity sequence.

[0182] The second similarity sequence determination module 212 is used to merge the similarity of the text prompt and the image to be classified into a second similarity sequence.

[0183] Specifically, the similarity scores of corresponding categories between the text vector tags and image vector tags are combined and averaged to obtain a second similarity sequence corresponding to multiple categories, for example:

[0184] The text prompt corresponds to n+1 represents the number of text prompts; the corresponding text vector is labeled as follows: The image vector is labeled I, and the cosine similarity between the text vector label and the image vector label is obtained as follows: If they are all different categories, there is no need to merge them. If they are in the same category, the cosine similarity of the same category needs to be averaged to obtain the second similarity sequence.

[0185] The normalization module 213 is used to normalize the first similarity sequence and the second similarity sequence respectively to obtain a normalized first similarity sequence and a normalized second similarity sequence;

[0186] The merging module 214 is used to merge the similarities of corresponding categories in the normalized first similarity sequence and the normalized second similarity sequence to obtain the similarity sequence;

[0187] Specifically, for example, if both the first similarity sequence and the second similarity sequence include the categories "hand" and "robotic hand" respectively, the first similarity sequence is first normalized to obtain the probability of the first similarity between the two categories. , ;in, Furthermore, the second similarity sequence is also normalized to obtain the probability of the second similarity between the two categories. , ,in To classify into categories and Merging to obtain , and Merging to obtain To obtain the above similarity sequence { , Finally, the category of the image to be classified is determined based on the larger similarity in the similarity sequence: whether it is a hand or a robotic hand.

[0188] In another implementation, the similarity sequence determination module 21 includes:

[0189] The first similarity sequence determination module 211 is used to merge the similarity of the corresponding categories between the instance image and the image to be classified to obtain the first similarity sequence;

[0190] The second similarity sequence determination module 212 is used to merge the similarity of the text prompt and the image to be classified into a second similarity sequence.

[0191] The two steps described above are the same as those in the above implementation method, and will not be repeated here.

[0192] The normalization module 213 is further configured to perform normalization processing on the first similarity sequence and the second similarity sequence to obtain a normalized similarity sequence;

[0193] The merging module 214 is used to merge the similarities of corresponding categories in the normalized similarity sequence to obtain the similarity sequence.

[0194] Specifically, for example, if both the first and second similarity sequences include the categories "hand" and "robotic hand" respectively, the similarity scores between the two categories in the first similarity sequence and the similarity scores between the two categories in the second similarity sequence are normalized together to obtain the corresponding probabilities. , , , ;and Then sort them into categories of the same type. and Merging to obtain , and Merging to obtain Based on the similarity sequence { The larger the similarity between the two values, the more likely the image will be classified as a hand or a robotic hand.

[0195] In practical applications, both of the above similarity processing methods can be used. However, the matching degree between text-image vector labels differs from that between image-image vector labels. Even though the text and image vector labels are aligned, common normalization weakens the contribution of either side. Therefore, when text and instance vector labels correspond to the exact same category, separate normalization is generally used. Conversely, when the categories corresponding to text and instance vector labels are not entirely the same—for example, when the text prompt contains multiple categories while the instance image contains only a few—common normalization is generally used.

[0196] This embodiment discloses a case where text prompts and instance images are used together as the first vector label, adding an instance classification mechanism in addition to the standard usage of pre-trained VLMs, namely semantic classification. Instance images can significantly improve classification accuracy, as they convey information about the specific context of the application, avoiding ambiguity and being more effective than text prompts. Instance images are encoded by the image encoder of the original VLM, which does not change the structure or parameters of the original model, and therefore does not affect its generalization ability. Low cost and flexibility are achieved because no training or fine-tuning is required. Instance images can be dynamically updated at runtime and are independent of specific models or environments.

[0197] When the image to be classified is a spliced ​​image, the similarity determination module 1 is further used to obtain the similarity between the prompt data and the at least two spliced ​​regions based on the visual language model;

[0198] The category determination module 2 is further configured to determine the category of the at least two spliced ​​regions based on the category of the prompt data corresponding to the maximum value in the similarity.

[0199] The conversion module 11 is further configured to convert the stitched image into an image embedding based on the encoder of the pre-trained visual language model;

[0200] The category determination module 2 also includes:

[0201] Extraction module 202 extracts at least two region embeddings from the image embedding, each region embedding corresponding to a splicing region;

[0202] The second conversion module 203 converts the at least two region embeddings into the size of the image embedding to obtain at least two region vector labels;

[0203] The similarity calculation module 12 is further configured to obtain the similarity between the first vector marker and the at least two region vector markers as the similarity between the prompt data and the at least two spliced ​​regions.

[0204] To achieve the above steps, the encoder of the pre-trained visual language model needs to have independent, modular, and separable pooling layers. Specifically, a Region of Interest (ROI) Alignment (ROI Align) layer is added between the feature extraction layer and the pooling layer of the image encoder of the pre-trained visual language model to modify its output. The ROI Align layer is a neural network layer used for object detection and instance segmentation tasks, and its main function is to accurately extract the features corresponding to each region of interest from the feature map.

[0205] Leveraging the separability of image encoders with modular pooling layers, the stitched image is converted into an image embedding using the image encoder described above (with the pooling layer removed). Then, a ROI Align layer is used to extract region embeddings from the corresponding ROI locations in the converted image embedding. The region embeddings are also upsampled to the size of the standard image embedding using ROI Align. Finally, a pooling layer is used to pool the upsampled region embeddings into region vector labels. The final region vector labels still reside in the embedding space aligned with the text / instance / image vector labels.

[0206] Region embeddings are extracted from the original intermediate result, i.e., the image embedding, and then upsampled to obtain region features. In the ROIAlign layer, the image embedding is divided into fixed-size units according to the output size, i.e., the standard dimension of the image embedding. Then, bilinear interpolation is used to calculate the value of the corresponding sampling point. Finally, pooling is performed on the sampling points to obtain region features.

[0207] Furthermore, in this embodiment, if the text prompt and instance image are existing categories, that is, the text vector labels and instance vector labels have been calculated before, these labels can be directly loaded for calculating cosine similarity. If it is a newly added category, the text prompt and instance image are labeled and calculated based on the text encoder and image encoder.

[0208] In this embodiment, multiple regions of interest are extracted from the original image to be classified and stitched together to form a stitched image. The stitched image is then encoded to obtain an image embedding. Region embeddings are extracted from the image embeddings. Through processing, the region embeddings are transformed into region vector labels of the same size as the image vector labels. By calculating the cosine similarity, the classification of multiple regions of interest can be processed simultaneously, thus speeding up the processing.

[0209] Example 5

[0210] Figure 13 This is a schematic diagram of the structure of an electronic device according to an example embodiment of the present disclosure. The electronic device includes a memory, a processor, and a computer program stored in the memory and used to run on the processor. When the processor executes the computer program, it implements the image classification method described in any of the above embodiments. Figure 13 The electronic device 90 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0211] like Figure 13 As shown, the electronic device 90 can be manifested as a general-purpose computing device, such as a server device. The components of the electronic device 90 may include, but are not limited to: at least one processor 91, at least one memory 92, and a bus 93 connecting different system components (including memory 92 and processor 91).

[0212] Bus 93 includes a data bus, an address bus, and a control bus.

[0213] The memory 92 may include volatile memory, such as random access memory (RAM) 921 and / or cache memory 922, and may further include read-only memory (ROM) 923.

[0214] The memory 92 may also include a program tool 925 (or utility) having a set (at least one) program module 824, such program module 924 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0215] The processor 91 executes various functional applications and data processing by running computer programs stored in the memory 92, such as the image classification method provided in any of the above embodiments.

[0216] Electronic device 90 can also communicate with one or more external devices 94 (e.g., keyboard, pointing device, etc.). This communication can be performed through input / output (I / O) interface 95. Furthermore, electronic device 90 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public network, such as the Internet) via network adapter 96. As shown, network adapter 96 communicates with other modules of electronic device 90 via bus 93. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with electronic device 90, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.

[0217] It should be noted that although several units / modules or sub-units / modules of the electronic device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0218] Example 6

[0219] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image classification method provided in any of the above embodiments.

[0220] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0221] Example 7

[0222] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the image classification method described in any of the preceding embodiments.

[0223] The program code for executing the computer program product of this disclosure can be written in any combination of one or more programming languages, and the program code can be executed entirely on a user device, partially on a user device, as a stand-alone software package, partially on a user device and partially on a remote device, or entirely on a remote device.

[0224] While specific embodiments of this disclosure have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of this disclosure is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of this disclosure, but all such changes and modifications fall within the scope of protection of this disclosure.

Claims

1. An image classification method, characterized in that, Includes the following steps: The similarity between the prompt data and the image to be classified is obtained based on a visual language model. The prompt data includes at least one instance image and corresponds to at least one category. The category of the image to be classified is determined based on the category of the prompt data corresponding to the maximum value in the similarity score.

2. The image classification method as described in claim 1, characterized in that, The similarity between the prompt data and the image to be classified, obtained based on the visual language model, specifically includes: The encoder based on the visual language model obtains the first vector label and the image vector label corresponding to the prompt data and the image to be classified, respectively; The similarity between the first vector label and the image vector label is obtained as the similarity between the corresponding prompt data and the image to be classified.

3. The image classification method as described in claim 2, characterized in that, The process of determining the category of the image to be classified based on the category of the prompt data corresponding to the maximum value in the similarity score specifically includes: The similarity scores of the corresponding categories are merged to obtain a similarity sequence; The category of the image to be classified is determined based on the category of the prompt data corresponding to the maximum value in the similarity sequence.

4. The image classification method as described in claim 3, characterized in that, The prompt data also includes at least one text prompt. The step of merging the similarities of corresponding categories in the similarity data to obtain a similarity sequence specifically includes: The similarity scores of the instance image and the image to be classified are combined to obtain the first similarity sequence; The similarity scores of the text prompt and the image to be classified are combined to obtain a second similarity sequence; The first similarity sequence and the second similarity sequence are normalized to obtain the normalized first similarity sequence and the normalized second similarity sequence, respectively. The similarity sequence is obtained by merging the similarity scores of corresponding categories in the normalized first similarity sequence and the normalized second similarity sequence; or, The first similarity sequence and the second similarity sequence are normalized together to obtain a normalized similarity sequence; The similarity sequences are obtained by merging the similarity scores of corresponding categories in the normalized similarity sequences.

5. The image classification method as described in claim 2, characterized in that, The image to be classified is a stitched image, and the stitched image includes at least two stitched regions; The similarity between the prompt data and the image to be classified, obtained based on the visual language model, specifically includes: The similarity between the prompt data and the at least two spliced ​​regions is obtained based on the visual language model. The step of determining the category of the image to be classified based on the category of the prompt data corresponding to the maximum value in the similarity score specifically includes: The categories of the at least two spliced ​​regions are determined based on the category of the prompt data corresponding to the maximum value in the similarity.

6. The image classification method as described in claim 5, characterized in that, The similarity between the prompt data and the at least two spliced ​​regions is obtained based on the visual language model; specifically, this includes: An encoder based on a pre-trained visual language model converts the stitched image into an image embedding. At least two region embeddings are extracted from the image embedding, and each region embedding corresponds to a splicing region; The at least two region embeddings are converted to the size of the image embedding to obtain at least two region vector labels; The similarity between the first vector marker and the at least two region vector markers is obtained as the similarity between the prompt data and the at least two spliced ​​regions.

7. The image classification method as described in claim 6, characterized in that, The image encoder of the pre-trained visual language model has independent, modular, and separable pooling layers, and an interest region alignment layer is added between the feature extraction layer and the pooling layer of the image encoder of the pre-trained visual language model. The interest region alignment layer is used to convert the region embedding into the size of the image embedding to obtain the region vector label.

8. The image classification method as described in claim 5, characterized in that, The at least two splicing areas are the same size.

9. An image classification device, characterized in that, include: A similarity acquisition module is used to obtain the similarity between prompt data and the image to be classified based on a visual language model. The prompt data includes at least one instance image and corresponds to at least one category. The category determination module is used to determine the category of the image to be classified based on the category of the prompt data corresponding to the maximum value in the similarity.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and for running on the processor, characterized in that, When the processor executes the computer program, it implements the image classification method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the image classification method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the image classification method as described in any one of claims 1-8.

Images