Image classification methods, systems, media and electronic devices

Abstract

This invention provides an image classification method, system, medium, and electronic device. The method includes constructing an image classification model, which comprises a text representation extractor, a visual representation extractor, and a learnable metric module; constructing extended category labels; extracting support visual representations of support images and query visual representations of query images based on the visual representation extractor; calculating a first similarity matrix between the text representations and support visual representations based on the learnable metric module, and adjusting the parameters of the learnable metric module; calculating a second similarity matrix between the text representations and query visual representations based on the learnable metric module, and adjusting the parameters of the text representation extractor, visual representation extractor, and learnable metric module, thereby classifying the image to be classified based on the trained image classification model. The image classification method, system, medium, and electronic device of this invention improves accuracy by training the image classification model through a dual-modal comparative learning approach involving both image and text.

Description

Technical Field

[0001] This invention belongs to the technical field of deep learning, and in particular relates to an image classification method, system, medium, and electronic device. Background Technology

[0002] Traditional deep learning methods require a large amount of labeled data to train neural networks, but obtaining labeled data is time-consuming and labor-intensive in many practical applications. For example, in the field of medical image processing, experts are needed to label every single image. In contrast, humans can learn to recognize and classify unfamiliar objects with only a small number of sample images. Inspired by this, a series of few-shot learning methods have been proposed, with the goal of training a model on a base class containing sufficiently labeled images, which can then generalize well to new classes using only a small number of sample images.

[0003] The few-sample image classification problem uses the following problem setting:

[0004] 1) The problem contains two non-overlapping sets of categories: the base class set and the new class set.

[0005] 2) Each base class contains a large number of labeled images, and each new class contains a small number of labeled images and a large number of unlabeled images.

[0006] This method requires training a model on a base class that can utilize a small number of labeled images and generalize well to new classes. The problem is formally defined as N-way K-shot classification, where in each classification task, the model needs to classify images N times, and for each class, there are K sample images available for reference.

[0007] Few-shot learning methods typically employ a meta-learning paradigm, comprising two phases: meta-training and meta-testing. These phases are performed on the base class training set and the new class test set, respectively, and both consistently organize the training / testing task into the N-way K-shot format described above. The example images in the task are called support images, and the images that the task requires to be classified are called query images.

[0008] In existing technologies, pure visual classification is usually performed using information from the image itself, which cannot take into account text, semantics, and other information. This limits the application scenarios and the classification accuracy cannot meet the needs of practical applications. Summary of the Invention

[0009] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide an image classification method, system, medium, and electronic device that trains the image classification model through image and text dual-modal comparative learning, thereby effectively improving the accuracy of image classification.

[0010] In a first aspect, the present invention provides an image classification method, the method comprising the following steps: constructing an image classification model, the image classification model including a text representation extractor, a visual representation extractor, and a learnable metric module; constructing extended category labels, and extracting text representations of the extended category labels based on the text representation extractor; extracting support visual representations of support images and query visual representations of query images based on the visual representation extractor; calculating a first similarity matrix between the text representations and the support visual representations based on the learnable metric module, calculating a first contrast loss based on the first similarity matrix, and adjusting the parameters of the learnable metric module based on the first contrast loss; calculating a second similarity matrix between the text representations and the query visual representations based on the adjusted learnable metric module, calculating a second contrast loss based on the second similarity matrix, and adjusting the parameters of the text representation extractor, the visual representation extractor, and the learnable metric module based on the second contrast loss; iteratively calculating the first contrast loss and the second contrast loss and adjusting the parameters of the image classification model based on the adjusted image classification model, so as to classify the image to be classified based on the trained image classification model.

[0011] In one implementation of the first aspect, constructing extended category labels includes the following steps:

[0012] Get category labels;

[0013] Build template;

[0014] Based on the mask template, the category label is used to generate extended category labels with or without a mask.

[0015] In one implementation of the first aspect, extracting the text representation of the extended category label based on the text representation extractor includes the following steps:

[0016] The extended category tags are segmented into words;

[0017] The embedding vector of each word is extracted based on the word segmentation encoding neural network;

[0018] The text representation is generated using a neural network based on the embedding vector.

[0019] In one implementation of the first aspect, the learnable metric module employs a bilinear layer or a neural network.

[0020] In one implementation of the first aspect, the visual representation extractor employs a convolutional neural network ResNet or a Transformer-based neural network ViT, and the text representation extractor employs a BERT or RoBERTa model.

[0021] In one implementation of the first aspect, classifying the image to be classified based on a trained image classification model includes the following steps:

[0022] The learnable metric module is adjusted based on supporting image and text representations;

[0023] Based on the adjusted learnable metric module, a second similarity matrix between the image to be classified and the support image is obtained;

[0024] The category label with the highest similarity value is selected as the classification result of the image to be classified.

[0025] In a second aspect, the present invention provides an image classification system, the system comprising a first construction module, a second construction module, a representation extraction module, a first adjustment module, a second adjustment module, and an iteration module;

[0026] The first building module is used to build an image classification model, which includes a text representation extractor, a visual representation extractor, and a learnable metric module.

[0027] The second building module is used to build extended category labels and extract text representations of the extended category labels based on the text representation extractor;

[0028] The representation extraction module is used to extract the supporting visual representation of the supporting image based on the visual representation extractor, and to extract the query visual representation of the query image.

[0029] The first adjustment module is used to calculate a first similarity matrix between the text representation and the supporting visual representation based on the learnable metric module, calculate a first contrast loss based on the first similarity matrix, and adjust the parameters of the learnable metric module based on the first contrast loss.

[0030] The second adjustment module is used to calculate a second similarity matrix between the text representation and the query visual representation based on the adjusted learnable metric module, calculate a second contrast loss based on the second similarity matrix, and adjust the parameters of the text representation extractor, the visual representation extractor, and the learnable metric module based on the second contrast loss.

[0031] The iterative module is used to iteratively calculate the first contrast loss and the second contrast loss based on the adjusted image classification model, and to adjust the parameters of the image classification model, so as to classify the image to be classified based on the trained image classification model.

[0032] Thirdly, the present invention provides an electronic device comprising a processor and a memory.

[0033] The memory is used to store computer programs;

[0034] The processor is used to execute the computer program stored in the memory, so that the electronic device performs the image classification method described above.

[0035] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that the program, when executed by an electronic device, implements the above-described image classification method.

[0036] As described above, the image classification method, system, medium, and electronic device of the present invention have the following beneficial effects:

[0037] (1) The image classification model is trained by image and text dual-modal contrast learning, which effectively improves the accuracy of image classification.

[0038] (2) By adopting a two-layer training strategy, the learnable metric module becomes task-specific, providing the model's transferability to new classes;

[0039] (3) By using extended category labels to extract text features, the role of semantic information is fully utilized, enriching the application scenarios, and is especially suitable for small sample image classification tasks. Attached Figure Description

[0040] Figure 1 The flowchart shown is an embodiment of the image classification method of the present invention;

[0041] Figure 2 The diagram shows a framework schematic of the image classification method of the present invention in one embodiment;

[0042] Figure 3 The diagram shown is a structural schematic of the image classification system of the present invention in one embodiment.

[0043] Figure 4 The diagram shown is a structural schematic of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0044] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0045] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0046] The image classification method, system, medium, and electronic device of this invention train the image classification model through image-text bimodal contrastive learning. By employing a two-layer training strategy, the learnable metric module becomes task-specific, providing transferability of the model to new classes, thereby effectively improving the accuracy of image classification. The contrastive learning utilizes contrastive loss to bring positive sample pairs closer and negative sample pairs further apart in the metric space. In the image-text bimodal contrastive learning, a positive sample pair refers to the sample image and its matching labeled category, while a negative sample pair refers to the sample image and other unmatched category labels in the task.

[0047] The technical solutions of the present invention will now be described in detail with reference to the accompanying drawings.

[0048] like Figure 1 and Figure 2 As shown, in one embodiment, the image classification method of the present invention includes the following steps:

[0049] Step S1: Construct an image classification model, which includes a text representation extractor, a visual representation extractor, and a learnable metric module.

[0050] Specifically, the image classification model of the present invention comprises three parts: a text representation extractor, a visual representation extractor, and a learnable metric module. The text representation extractor is used to extract textual representations of textual information. The visual representation extractor is used to extract visual representations of the image. The learnable metric module is used to calculate the similarity matrix between two representations.

[0051] In one embodiment, the visual representation extractor employs a convolutional neural network ResNet or a Transformer-based neural network ViT, and the text representation extractor employs a BERT or RoBERTa model. The learnable metric module employs a bilinear layer or a neural network. For example, the bilinear layer is represented as M(x,y) = x T Ay, where A is a learnable matrix. The neural network is represented as M(x,y) = NeuralNet([x,y]), where [x,y] refers to the concatenation of vectors x and y. This neural network is learnable.

[0052] Step S2: Construct extended category labels and extract text representations of the extended category labels based on the text representation extractor.

[0053] Specifically, in order to extract text representations that are more relevant to the image classification task and richer in semantic information, this invention uses extended category labels to extract text representations. First, category labels are obtained and a template is constructed. Then, based on the template, extended category labels with or without masks are generated from the category labels.

[0054] In one embodiment, the template can be represented as: [CLS]The appearance of [INPUT] is [MASK].[SEP]. Here, [CLS] and [SEP] are special word segments added at the beginning and end, and [MASK] is a masked word segment. Replacing [INPUT] in the template with a category tag generates an extended category tag containing the mask.

[0055] The text representation extractor takes the extended category labels as input, segments the extended category labels into words, obtains the embedding vector for each word, and finally generates the text representation based on the embedding vectors. The text representation extractor includes a word segmentation encoding neural network, such as a pre-trained language model encoder, for extracting embedding vectors from the words. Preferably, the pre-trained language model encoder is a Transformer-structured encoder. Simultaneously, this invention employs a neural network, such as a language model head, to project the embedding vectors into a text representation with the same dimension as the visual representation.

[0056] Step S3: Extract the supporting visual representation of the supporting image based on the visual representation extractor, and extract the query visual representation of the query image.

[0057] Specifically, both the supporting image and the query image are input into the visual representation extractor to obtain the corresponding supporting visual representation and query visual representation.

[0058] Step S4: Calculate a first similarity matrix between the text representation and the supporting visual representation based on the learnable metric module, calculate a first contrast loss based on the first similarity matrix, and adjust the parameters of the learnable metric module based on the first contrast loss.

[0059] Specifically, to improve the transferability of the model trained on the base class to new classes, this invention employs a learnable metric module to calculate the similarity matrix. For different classification tasks, the parameters in the learnable metric module are adaptively adjusted based on the support images to improve the model's performance on each specific few-shot classification task.

[0060] In this invention, the learnable metric module calculates a first similarity matrix between the text representation and the supporting visual representation, calculates a first contrast loss based on the first similarity matrix, and adjusts the parameters of the learnable metric module at the inner layer based on the first contrast loss.

[0061] For a sample image set I and a category set C (of size |C|), the calculated similarity matrix S = |I| × |C|, where |I| is the size of the sample image set I, |C| is the size of the category set C, and S... ij Let be the similarity between the i-th sample image and the j-th category label text, where i∈[1,|I|], j∈[1,|C|]). The formula for calculating the contrast loss based on the similarity matrix is ​​as follows:

[0062]

[0063] Among them, t i The index of the true class label corresponding to the i-th sample image (i.e., image I). i The true category is C i ), τ is the temperature coefficient.

[0064] Step S5: Calculate the second similarity matrix between the text representation and the query visual representation based on the adjusted learnable metric module, calculate the second contrast loss based on the second similarity matrix, and adjust the parameters of the text representation extractor, the visual representation extractor, and the learnable metric module based on the second contrast loss.

[0065] Specifically, a second similarity matrix between the text representation and the query visual representation is calculated based on the adjusted learnable metric module. A second contrast loss is calculated based on the second similarity matrix. The parameters of the text representation extractor, the visual representation extractor, and the learnable metric module are adjusted on the outer layer based on the second contrast loss.

[0066] Similarly, the algorithm for calculating the contrast loss based on the similarity matrix is ​​the same as above, and will not be repeated here.

[0067] Step S6: Iteratively calculate the first contrast loss and the second contrast loss based on the adjusted image classification model, and adjust the parameters of the image classification model to classify the image to be classified based on the trained image classification model.

[0068] Specifically, the parameters of the image classification model are adjusted twice through steps S5 and S6. To further improve the accuracy of the image classification model, steps S4-S6 are iteratively executed based on the adjusted image classification model until a preset number of iterations is reached or the first contrast loss and the second contrast loss meet preset requirements, thereby completing the training of the image classification model.

[0069] After the image classification model is trained, classifying the image to be classified based on the trained image classification model includes the following steps:

[0070] a) Adjust the learnable metric module based on supporting image and text representations;

[0071] b) Based on the adjusted learnable metric module, obtain the second similarity matrix between the image to be classified and the support image;

[0072] c) Select the category label with the highest similarity value as the classification result of the image to be classified.

[0073] The image classification method of the present invention will be further illustrated below through specific embodiments.

[0074] Take the 5-way 1-shot classification task on the miniImageNet dataset as an example.

[0075] In the visual branch, a ResNet-12 network pre-trained on the miniImageNet training set serves as the visual representation extractor, extracting 640-dimensional visual representations from the support and query images, each with a size of 84*84*3. In the text branch, a pre-trained RoBERTa-base language model with fixed parameters is used as the text representation extractor, followed by a language model head to project the 768-dimensional masked word embedding vectors into a 640-dimensional text representation consistent with the visual representation dimensions. The learnable metric module is implemented using a bilinear layer.

[0076] During the meta-training phase, the SGD (Stochastic Gradient Descent) optimizer was used to train on the training set for 80 epochs, with each epoch using 300 specific 5-class classification tasks. In each specific classification task, each class had 1 support image and 16 query images. The model first performed 25 inner-layer updates on the support image set, and then performed 1 outer-layer update on the query image set. The inner-layer learning rate was set to a fixed 0.5. In each inner-layer update, the similarity matrix between all support images and the corresponding representations of the 5 class labels was calculated, and then the contrastive loss was calculated to update the model. The outer-layer initial learning rate was set to 1e-3, using a cosine annealing learning rate adjustment strategy. The outer-layer update calculated the similarity matrix between all query images and the corresponding representations of the 5 class labels, and then the contrastive loss was calculated to update the model.

[0077] In the meta-testing phase, the model's classification performance was tested using 1000 specific 5-class classification tasks. In each specific task, each class contained one support image and 16 query images. The model first performed 25 inner-layer updates on the support image set, and then tested classification accuracy on the query image set. The inner-layer operation was the same as in the meta-training phase. The outer layer calculated the similarity matrix between all query images and the corresponding representations of the 5 class labels, then selected the class label with the highest similarity for each query image as the classification result, and finally calculated the accuracy for all query images.

[0078] Tables 1 and 2 show the performance comparison of different image classification models. As can be seen from the tables, on the 5-way 1-shot classification task of the miniImageNet dataset, the classification accuracy of this invention is 69.52%. On four commonly used few-sample image classification datasets, the classification accuracy of this invention reaches the highest in both 1-shot and 5-shot settings.

[0079] Table 1. Comparison of the performance of different image classification models

[0080]

[0081] Table 2. Comparison of the performance of different image classification models

[0082]

[0083] The scope of protection of the image classification method described in this embodiment is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principle of this invention is included within the scope of protection of this invention.

[0084] This invention also provides an image classification system that can implement the image classification method described in this invention. However, the implementation device of the image classification system described in this invention includes, but is not limited to, the structure of the image classification system listed in this embodiment. All structural modifications and substitutions of the prior art made in accordance with the principles of this invention are included within the protection scope of this invention.

[0085] like Figure 3 As shown, in one embodiment, the image classification system of the present invention includes a first construction module 31, a second construction module 32, a representation extraction module 33, a first adjustment module 34, a second adjustment module 35, and an iteration module 36.

[0086] The first construction module 31 is used to construct an image classification model, which includes a text representation extractor, a visual representation extractor, and a learnable metric module.

[0087] The second construction module 32 is connected to the first construction module 31 and is used to construct extended category labels and extract text representations of the extended category labels based on the text representation extractor.

[0088] The representation extraction module 33 is connected to the first construction module 31 and is used to extract the supporting visual representation of the supporting image and the query visual representation of the query image based on the visual representation extractor.

[0089] The first adjustment module 34 is connected to the second construction module 32 and the representation extraction module 33, and is used to calculate a first similarity matrix between the text representation and the supporting visual representation based on the learnable metric module, calculate a first contrast loss based on the first similarity matrix, and adjust the parameters of the learnable metric module based on the first contrast loss.

[0090] The second adjustment module 35 is connected to the second construction module 32 and the representation extraction module 33, and is used to calculate the second similarity matrix between the text representation and the query visual representation based on the adjusted learnable metric module, calculate the second contrast loss based on the second similarity matrix, and adjust the parameters of the text representation extractor, the visual representation extractor and the learnable metric module based on the second contrast loss.

[0091] The iteration module 36 is connected to the first adjustment module 34 and the second adjustment module 35, and is used to iteratively calculate the first contrast loss and the second contrast loss based on the adjusted image classification model, and adjust the parameters of the image classification model, so as to classify the image to be classified based on the trained image classification model.

[0092] The structure and principle of the first construction module 31, the second construction module 32, the representation extraction module 33, the first adjustment module 34, the second adjustment module 35 and the iteration module 36 correspond one-to-one with the steps in the above image classification method, so they will not be described again here.

[0093] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, or methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or modules or units may be electrical, mechanical, or other forms.

[0094] The modules / units described as separate components may or may not be physically separate. The components shown as modules / units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules / units can be selected to achieve the objectives of the embodiments of the present invention, depending on actual needs. For example, the functional modules / units in the various embodiments of the present invention may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0095] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0096] This invention also provides a computer-readable storage medium. Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing a processor. The program can be stored in a computer-readable storage medium, which is a non-transitory medium, such as random access memory, read-only memory, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof. The storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. This available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state drive (SSD)).

[0097] This invention also provides an electronic device. The electronic device includes a processor and a memory.

[0098] The memory is used to store computer programs.

[0099] The memory includes various media capable of storing program code, such as ROM, RAM, magnetic disk, USB flash drive, memory card, or optical disk.

[0100] The processor is connected to the memory and is used to execute the computer program stored in the memory so that the electronic device performs the image classification method described above.

[0101] Preferably, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0102] like Figure 4As shown, the electronic device of the present invention is embodied in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: one or more processors or processing units 41, a memory 42, and a bus 43 connecting different system components (including the memory 42 and the processing unit 41).

[0103] Bus 43 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0104] Electronic devices typically include a variety of computer-readable media. These media can be any available media that can be accessed by the electronic device, including volatile and non-volatile media, and removable and non-removable media.

[0105] Memory 42 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 421 and / or cache memory 422. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 423 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 4 Not shown; usually referred to as a "hard drive"). Although Figure 4 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 43 via one or more data media interfaces. Memory 42 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0106] A program / utility 424 having a set (at least one) of program modules 4241 may be stored, for example, in memory 42. Such program modules 4241 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 4241 typically perform the functions and / or methods described in the embodiments of the present invention.

[0107] The electronic device can also communicate with one or more external devices (e.g., keyboard, pointing device, display, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed through input / output (I / O) interface 44. Furthermore, the electronic device can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) through network adapter 45. Figure 4 As shown, network adapter 45 communicates with other modules of the electronic device via bus 43. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0108] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. An image classification method, characterized in that, The method includes the following steps: Construct an image classification model, which includes a text representation extractor, a visual representation extractor, and a learnable metric module; Construct extended category labels, and extract text representations of the extended category labels based on the text representation extractor; Based on the visual representation extractor, the supporting visual representation of the supporting image is extracted, and the query visual representation of the query image is extracted. The first similarity matrix between the text representation and the supporting visual representation is calculated based on the learnable metric module, a first contrast loss is calculated based on the first similarity matrix, and the parameters of the learnable metric module are adjusted based on the first contrast loss. The second similarity matrix between the text representation and the query visual representation is calculated based on the adjusted learnable metric module. A second contrast loss is calculated based on the second similarity matrix. The parameters of the text representation extractor, the visual representation extractor, and the learnable metric module are adjusted based on the second contrast loss. Based on the adjusted image classification model, the first contrast loss and the second contrast loss are calculated iteratively, and the parameters of the image classification model are adjusted to achieve the classification of the image to be classified based on the trained image classification model; The process of classifying an image based on a trained image classification model includes the following steps: The learnable metric module is adjusted based on supporting image and text representations; Based on the adjusted learnable metric module, a second similarity matrix between the image to be classified and the support image is obtained; The category label with the highest similarity value is selected as the classification result of the image to be classified.

2. The image classification method according to claim 1, characterized in that: Building extended category tags involves the following steps: Get category labels; Build template; Based on the template, the category labels are used to generate extended category labels with or without masks.

3. The image classification method according to claim 1, characterized in that: Extracting the text representation of the extended category label based on the text representation extractor includes the following steps: The extended category tags are segmented into words; The embedding vector of each word is extracted based on the word segmentation encoding neural network; The text representation is generated using a neural network based on the embedding vector.

4. The image classification method according to claim 1, characterized in that: The learnable metric module employs a bilinear layer or a neural network.

5. The image classification method according to claim 1, characterized in that: The visual representation extractor uses a convolutional neural network ResNet or a Transformer-based neural network ViT, while the text representation extractor uses a BERT or RoBERTa model.

6. An image classification system, characterized in that, The system includes a first construction module, a second construction module, a characterization extraction module, a first adjustment module, a second adjustment module, and an iteration module; The first building module is used to build an image classification model, which includes a text representation extractor, a visual representation extractor, and a learnable metric module. The second building module is used to build extended category labels and extract text representations of the extended category labels based on the text representation extractor; The representation extraction module is used to extract the supporting visual representation of the supporting image based on the visual representation extractor, and to extract the query visual representation of the query image. The first adjustment module is used to calculate a first similarity matrix between the text representation and the supporting visual representation based on the learnable metric module, calculate a first contrast loss based on the first similarity matrix, and adjust the parameters of the learnable metric module based on the first contrast loss. The second adjustment module is used to calculate a second similarity matrix between the text representation and the query visual representation based on the adjusted learnable metric module, calculate a second contrast loss based on the second similarity matrix, and adjust the parameters of the text representation extractor, the visual representation extractor, and the learnable metric module based on the second contrast loss. The iterative module is used to iteratively calculate the first contrast loss and the second contrast loss based on the adjusted image classification model, and to adjust the parameters of the image classification model, so as to classify the image to be classified based on the trained image classification model. The process of classifying an image based on a trained image classification model includes the following steps: The learnable metric module is adjusted based on supporting image and text representations; Based on the adjusted learnable metric module, a second similarity matrix between the image to be classified and the support image is obtained; The category label with the highest similarity value is selected as the classification result of the image to be classified.

7. An electronic device, characterized in that, The electronic device includes: a processor and a memory; The memory is used to store computer programs; The processor is configured to execute the computer program stored in the memory to cause the electronic device to perform the image classification method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by an electronic device, the program implements the image classification method of any one of claims 1 to 5.

Images