An image classification method and product based on multi-modal small sample continuous learning

Abstract

This application provides an image classification method and product based on multimodal few-shot continuous learning, relating to the field of image processing technology. The method includes: acquiring an input image; acquiring input data based on the input image, the input data including at least visual features and text prototypes corresponding to the visual features; inputting the visual features and text prototypes into a semantic fusion network to fuse information between the visual features and text prototypes, obtaining an adaptive prototype, the adaptive prototype including at least an adaptive visual prototype and an adaptive query representation; and obtaining the category probability of the query image based on the adaptive visual prototype and the adaptive query representation. This application, by fusing semantic information and visual information hidden in the category text during few-shot continuous learning using a semantic fusion network, enables the model to consider the influence of semantic information during few-shot continuous learning, thereby greatly mitigating the catastrophic forgetting problem in the few-shot continuous learning process.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to an image classification method and product based on multimodal few-shot continuous learning. Background Technology

[0002] Few-shot continuous learning, as a paradigm that simultaneously addresses both continuous learning and few-shot learning problems, has become a challenging issue in machine learning. An ideal few-shot continuous learning model should be able to distinguish all known classes when a new class appears, where each class contains only a small amount of labeled data.

[0003] However, existing few-shot continuous learning methods only consider visual modalities, focusing on feature learning based on visual information of a single modality of an image, ignoring semantic relationships between categories. This often leads to overlap between the distribution of new categories and old categories, making them difficult to distinguish and resulting in a severe catastrophic forgetting problem. Therefore, how to alleviate the catastrophic forgetting problem of few-shot continuous learning has become a pressing technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] This application provides an image classification method and product based on multimodal few-shot continuous learning, aiming to address the problem of catastrophic forgetting in few-shot continuous learning.

[0005] The first aspect of this application provides an image classification method based on multimodal few-shot continuous learning, applied to the t-th continuous learning task, where t≥1, the method comprising:

[0006] Obtain an input image, which includes at least a query image and a support image;

[0007] Based on the input image, input data is obtained, and the input data includes at least visual features and the text prototypes corresponding to the visual features.

[0008] The visual features and the text prototype are input into a semantic fusion network to perform information fusion on the visual features and the text prototype to obtain an adaptive prototype, which includes at least an adaptive visual prototype and an adaptive query representation.

[0009] Based on the adaptive visual prototype and the adaptive query representation, the category probability of the query image is obtained.

[0010] In one optional implementation, the adaptive prototype further includes an adaptive text prototype, and after obtaining the adaptive prototype, the method further includes:

[0011] Based on the adaptive visual prototype, the adaptive text prototype and the first real label, a semantic consistency loss is obtained, where the first real label is used to characterize the matching relationship between the adaptive visual prototype and the text prototype corresponding to the adaptive visual prototype.

[0012] Based on the adaptive query representation, the adaptive visual prototype, and the second true label, the query loss is obtained, where the second true label is the true category label of the query representation.

[0013] Based on the semantic consistency loss and the query loss, the semantic fusion network is optimized to obtain an optimized semantic fusion network.

[0014] The optimized semantic fusion network is used as the semantic fusion network for the (t+1)th continuous learning task.

[0015] In one optional implementation, based on the input image, input data is acquired, including:

[0016] Based on the category to which the supported image belongs, semantic information is obtained;

[0017] The support image and the query image are input into the convolutional sub-network to obtain the visual prototype corresponding to the support image and the query representation corresponding to the query image, respectively.

[0018] The semantic information is input into the projection subnetwork to obtain the text prototype;

[0019] The visual feature is obtained by concatenating the query representation and the visual prototype.

[0020] In one optional implementation, the semantic fusion network includes a self-attention module and a forward propagation layer. The visual features and the text prototype are input into the semantic fusion network to fuse information from the visual features and the text prototype, obtaining an adaptive prototype, including:

[0021] The visual feature and the text prototype are input into the self-attention module, so that one of the visual feature and the text prototype can perform information fusion and information enhancement on the other based on the self-attention mechanism to obtain the first visual feature and the first text prototype.

[0022] The first visual feature, the first text prototype, and the text prototype are summed and normalized to obtain the enhanced first visual feature and the enhanced first text prototype.

[0023] The enhanced first visual feature and the enhanced first text prototype are input into the forward propagation layer to be mapped to the latent space to obtain the second visual feature and the second text prototype.

[0024] The second visual feature, the second text prototype, and the enhanced first text prototype are summed and normalized to obtain the adaptive prototype.

[0025] In one optional implementation, based on the adaptive visual prototype and the adaptive query representation, the category probability of the query image is obtained, including:

[0026] Based on the adaptive visual prototype and the adaptive query representation, obtain the cosine distance between the adaptive query representation and the adaptive visual prototype;

[0027] Based on the cosine distance, the probability that the adaptive query representation belongs to the image category corresponding to the adaptive visual prototype is obtained, and is used as the category probability.

[0028] In an optional implementation, after obtaining the cosine distance between the adaptive query representation and the adaptive visual prototype, the method further includes:

[0029] Based on the cosine distance, an adaptive visual prototype of the target is obtained, wherein the adaptive visual prototype of the target is the adaptive visual prototype with the smallest cosine distance to the adaptive query representation.

[0030] The image category corresponding to the target adaptive visual prototype is used as the category of the query image.

[0031] In one optional implementation, when t=1, the semantic fusion network is obtained as follows:

[0032] Obtain a training dataset, which includes at least the original support images, the original query images, and the original semantic information;

[0033] The training dataset is input into the initial projection subnetwork and the initial convolutional subnetwork to obtain the original input data;

[0034] The original input data is fed into the initial semantic fusion network to obtain the original adaptive prototype;

[0035] Based on the original adaptive prototype and the original real labels, obtain the basic training loss;

[0036] Based on the aforementioned basic training loss, the initial projection subnetwork, the initial convolutional subnetwork, and the initial semantic fusion network are trained to obtain a first semantic fusion network, a projection subnetwork, and a convolutional network. The projection subnetwork and the convolutional network are used to acquire the input data in all continuous learning tasks.

[0037] The first semantic fusion network is used as the semantic fusion network at t=1.

[0038] A second aspect of this application provides an image classification device based on multimodal few-shot continuous learning, applied to the t-th continuous learning task, where t≥1, the device comprising:

[0039] The acquisition module is used to acquire an input image, which includes at least a query image and a support image;

[0040] The underlying processing module is used to acquire input data based on the input image, wherein the input data includes at least visual features and the text prototypes corresponding to the visual features;

[0041] The semantic fusion module is used to input the visual features and the text prototype into the semantic fusion network to perform information fusion on the visual features and the text prototype to obtain an adaptive prototype, wherein the adaptive prototype includes at least an adaptive visual prototype and an adaptive query representation.

[0042] The classification module is used to obtain the category probability of the query image based on the adaptive visual prototype and the adaptive query representation.

[0043] In one alternative embodiment, the device further includes:

[0044] The first loss module is used to obtain semantic consistency loss based on the adaptive visual prototype, the adaptive text prototype and the first real label, wherein the first real label is used to characterize the matching relationship between the adaptive visual prototype and the text prototype corresponding to the adaptive visual prototype.

[0045] The second loss module is used to obtain the query loss based on the adaptive query representation, the adaptive visual prototype, and the second true label, wherein the second true label is the true category label of the query representation;

[0046] An optimization module is used to optimize the semantic fusion network based on the semantic consistency loss and the query loss to obtain an optimized semantic fusion network.

[0047] An update module is used to use the optimized semantic fusion network as the semantic fusion network for the (t+1)th continuous learning task.

[0048] In one optional implementation, the underlying processing module includes:

[0049] The semantic acquisition submodule is used to acquire semantic information based on the category to which the supported image belongs;

[0050] The first low-level processing submodule is used to input the support image and the query image into the convolutional subnetwork to obtain the visual prototype corresponding to the support image and the query representation corresponding to the query image, respectively.

[0051] The second low-level processing submodule is used to input the semantic information into the projection subnetwork to obtain the text prototype;

[0052] The connection submodule is used to connect the query representation and the visual prototype to obtain the visual features.

[0053] In one optional implementation, the semantic fusion module includes:

[0054] The self-attention submodule is used to input the visual feature and the text prototype into the self-attention module, so that one of the visual feature and the text prototype performs information fusion and information enhancement on the other based on the self-attention mechanism to obtain the first visual feature and the first text prototype.

[0055] The first enhancement submodule is used to sum and normalize the first visual feature, the first text prototype and the text prototype to obtain the enhanced first visual feature and the enhanced first text prototype.

[0056] The forward propagation submodule is used to input the enhanced first visual feature and the enhanced first text prototype into the forward propagation layer to map to the latent space to obtain the second visual feature and the second text prototype.

[0057] The second enhancement submodule is used to sum and normalize the second visual feature, the second text prototype, and the enhanced first text prototype to obtain the adaptive prototype.

[0058] In one optional implementation, the classification module includes:

[0059] The distance calculation submodule is used to obtain the cosine distance between the adaptive query representation and the adaptive visual prototype based on the adaptive visual prototype and the adaptive query representation;

[0060] The category probability acquisition submodule is used to obtain the probability that the adaptive query representation belongs to the image category corresponding to the adaptive visual prototype based on the cosine distance, and use it as the category probability.

[0061] In one optional implementation, the classification module further includes:

[0062] The target adaptive visual prototype acquisition submodule is used to acquire the target adaptive visual prototype based on the cosine distance, wherein the target adaptive visual prototype is the adaptive visual prototype with the smallest cosine distance to the adaptive query representation.

[0063] The classification submodule is used to use the image category corresponding to the target adaptive visual prototype as the category of the query image.

[0064] In one alternative embodiment, the device further includes:

[0065] The training set acquisition module is used to acquire the training dataset, which includes at least the original supporting images, the original query images, and the original semantic information.

[0066] The raw input data acquisition module is used to input the training dataset into the initial projection sub-network and the initial convolutional sub-network to acquire the raw input data.

[0067] The initial semantic fusion module is used to input the original input data into the initial semantic fusion network to obtain the original adaptive prototype;

[0068] The basic loss module is used to obtain the basic training loss based on the original adaptive prototype and the original true labels;

[0069] The training module is used to train the initial projection sub-network, the initial convolutional sub-network, and the initial semantic fusion network based on the basic training loss to obtain a first semantic fusion network, a projection sub-network, and a convolutional network. The projection sub-network and the convolutional network are used to acquire the input data in all continuous learning tasks.

[0070] The first update module is used to use the first semantic fusion network as the semantic fusion network at t=1.

[0071] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the image classification method based on multimodal few-shot continuous learning as described in the first aspect.

[0072] A fourth aspect of this application provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the image classification method based on multimodal few-shot continuous learning as described in the first aspect.

[0073] Beneficial effects:

[0074] This application provides an image classification method and product based on multimodal few-shot continuous learning, applied to the t-th continuous learning task, t≥1. The method includes: acquiring an input image, the input image including at least a query image and support images; acquiring input data based on the input image, the input data including at least visual features and text prototypes corresponding to the visual features; inputting the visual features and the text prototypes into a semantic fusion network to perform information fusion on the visual features and the text prototypes to obtain an adaptive prototype, the adaptive prototype including at least an adaptive visual prototype and an adaptive query representation; and acquiring the category probability of the query image based on the adaptive visual prototype and the adaptive query representation.

[0075] This application addresses the catastrophic forgetting problem in few-shot continuous learning by fusing semantic and visual information hidden in categorical texts using a semantic fusion network. This allows the model to consider the influence of semantic information during few-shot continuous learning, thereby greatly mitigating the problem of catastrophic forgetting. Attached Figure Description

[0076] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0077] Figure 1 This is a schematic diagram of a baseline model for image classification based on a single modality in the existing technology;

[0078] Figure 2 This is a flowchart of an image classification method based on multimodal few-shot continuous learning proposed in one embodiment of this application;

[0079] Figure 3 This is a schematic diagram of an image classification model based on multimodal few-shot continuous learning proposed in an embodiment of this application;

[0080] Figure 4 This is a schematic diagram of the network architecture of an image classification method based on multimodal few-shot continuous learning proposed in one embodiment of this application;

[0081] Figure 5 This is a schematic diagram of a semantic fusion network architecture proposed in one embodiment of this application;

[0082] Figure 6This is a schematic diagram of an image classification device based on multimodal few-shot continuous learning proposed in an embodiment of this application;

[0083] Figure 7 This is a schematic diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0084] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0085] In related technologies, few-shot continuous learning, as a paradigm that simultaneously solves the problems of continuous learning and few-shot learning, has become a challenging problem in machine learning. An ideal few-shot continuous learning model should be able to distinguish all known categories when a new category appears, where each category contains only a small amount of labeled data.

[0086] Figure 1 A schematic diagram of a baseline model for image classification based on a single modality in the prior art is shown, such as... Figure 1 As shown, existing few-shot continuous learning methods only consider visual modalities and classify query images based on visual similarity scores between query images and visual prototypes corresponding to support images. This process focuses on feature learning based on visual information of a single modality of the image, ignoring semantic relationships between categories. This makes it difficult to distinguish the distribution of new categories from old categories, resulting in a serious catastrophic forgetting problem.

[0087] In view of this, this application proposes an image classification method based on multimodal few-shot continuous learning. Figure 2 The flowchart of an image classification method based on multimodal few-shot continuous learning proposed in this application is shown, as follows: Figure 2 As shown, when applied to the t-th continuous learning task (t≥1), the method includes the following steps:

[0088] S101. Obtain the input image.

[0089] In this embodiment, the query image is the image to be classified input in each continuous learning task. In each continuous learning task, the query image is not an image already existing in the database. Specifically, in step S101, images of multiple categories input to the current continuous learning task are acquired. These images are sampled to obtain a query image and supporting images, which together constitute the input image. Specifically, from the images of multiple categories input to the current continuous learning task, a first preset number of images are extracted from each category as supporting images, and a second preset number of images are extracted from each category as query images. The query image and supporting images are then input together as input images into the network architecture of the continuous learning task to perform image classification on the query image. It should be noted that the first and second preset numbers can be determined according to actual conditions, and this application does not impose any limitations on them.

[0090] S102. Based on the input image, obtain input data.

[0091] In this embodiment, input data is obtained based on the input image. The input data includes at least visual features and text prototypes corresponding to the visual features. Specifically, the input data consists of multimodal basic features obtained based on the query image, support images, and semantic information. These multimodal basic features include query representations, visual prototypes, and text prototypes. Both query representations and visual prototypes are image features. The query representations and visual prototypes are combined to form the visual features. The query representation is the latent representation (vector representation in a shared latent space) obtained by processing the query image through a backbone network. The visual prototype is the average latent representation of each category of images in the support images. The text prototype is the latent representation of semantic information.

[0092] In specific implementation step S102, while acquiring the supporting images, semantic information corresponding to the category to which the supporting images belong is acquired. The semantic information is calculated based on the text labels of each category to which the supporting images belong, using a pre-trained word embedding model. This yields a dataset (including query images, supporting images, and semantic information) for acquiring input data. Next, input data is acquired based on this dataset.

[0093] Specifically, the data from the dataset is input into a pre-trained backbone network to obtain input data. The backbone network includes a convolutional subnetwork and a projection subnetwork. On one hand, the support images and query images from the dataset are input into the convolutional subnetwork. The convolutional subnetwork obtains the latent representation of the query image in a shared latent space, serving as the query representation for the query image. The convolutional subnetwork also obtains the average latent representation of the support images for each category in the shared latent space, serving as the visual prototype for each category. On the other hand, semantic information is input into the projection subnetwork. Affine layers in the projection subnetwork project the semantic information (word embeddings for each category) into the shared latent space to obtain the text prototype. After obtaining the visual prototype and query representation, since both are latent representations of images, the query representation and the visual prototype are concatenated to obtain the visual features.

[0094] In one optional implementation, the backbone network is initially trained during the basic learning task phase and used to acquire the input data in each subsequent learning task. This basic learning task phase, preceding all subsequent learning tasks, acquires the initially trained backbone network in the following manner:

[0095] First, a training dataset is obtained, which includes at least original support images, original query images, and original semantic information. The original query images and original support images are distinct from the query images and support images. In the basic learning phase, the original query images and original support images are images sampled from multiple categories in the database. Specifically, in each training cycle of the basic learning task phase, a third preset number of categories are sampled from the database, and a fourth preset number of images are extracted from each category to form original support images. Images are extracted from each category that is the same as the original support images to serve as original query images. Based on the original support images, the original semantic information corresponding to the category to which the original support images belong is obtained. The original query images, original support images, and original semantic information are then combined to form the training dataset. It should be noted that the aforementioned third and fourth preset numbers can be determined according to actual circumstances, and this application does not impose any restrictions on them.

[0096] Subsequently, in each training cycle, the training dataset is input into the initial backbone network to obtain the original input data. This initial backbone network includes an initial projection subnetwork and an initial convolutional subnetwork. Specifically, the original support images and the original query images from the training dataset are input into the initial convolutional subnetwork. Based on the initial convolutional subnetwork, the latent representation of the original query image in the shared latent space is obtained as the original query representation corresponding to the original query image. Based on the initial convolutional subnetwork, the average latent representation of the original support images for each category in the shared latent space is obtained as the original visual prototype corresponding to each category. The original semantic information is input into the initial projection subnetwork, and the original semantic information is projected onto the shared latent space through a linear affine layer in the initial projection subnetwork to obtain the original text prototype. The original text prototype, the original query representation, and the original visual prototype are combined into the original input data.

[0097] The original input data is input into the initial semantic fusion network to obtain the original adaptive prototype. Based on the original adaptive prototype and the original real labels, a basic training loss is obtained, which includes semantic consistency loss and query loss. The query loss distinguishes different categories of query images, and the semantic consistency loss encourages the alignment of visual and textual prototypes and enhances the semantic information of the original visual prototype. Based on the basic training loss, the initial projection subnetwork, the initial convolutional subnetwork, and the initial semantic fusion network are trained to obtain the first semantic fusion network, the projection subnetwork, and the convolutional subnetwork. The projection subnetwork and the convolutional subnetwork are used to obtain the input data in all continuous learning tasks. It should be noted that the process of training the backbone network and the initial semantic fusion network based on the basic training loss is the same as the training and optimization process of the semantic fusion network in the continuous learning stage. The difference is that in the basic training task stage, the initial backbone network and the initial semantic fusion network are trained and optimized simultaneously, while in the continuous learning task stage, only the semantic fusion network is trained and optimized. Therefore, for details, please refer to the process of training and optimizing the semantic fusion network in the continuous learning task stage in S103 below, which will not be repeated here.

[0098] In the basic learning task stage, the text prototype in the original adaptive prototype provides anchor points for the latent space, so that the visual prototype in the original adaptive prototype obtained by the model during the optimization process tends to align with the text prototype. The projection sub-network and convolutional sub-network obtained from the basic learning task are applied to the continuous learning task, which helps to accelerate the speed of continuous learning.

[0099] S103. Input the visual features and the text prototype into a semantic fusion network to perform information fusion on the visual features and the text prototype to obtain an adaptive prototype.

[0100] The visual features (including visual prototypes and query representations) and the text prototypes are input into a semantic fusion network to perform information fusion on the visual features and the text prototypes, obtaining an adaptive prototype. The adaptive prototype includes at least an adaptive visual prototype and an adaptive query representation; furthermore, it also includes an adaptive text prototype. The adaptive prototype is obtained according to the following formula:

[0101]

[0102] Among them, e q For the query representation, e v For the aforementioned visual prototype, e t For the text prototype, e′ q For the adaptive query representation, e′ v For the adaptive vision prototype, e′ t This is the adaptive text prototype. This refers to the semantic fusion network.

[0103] In the specific implementation step S103, Figure 5 A schematic diagram of a semantic fusion network architecture proposed in an embodiment of this application is shown, as follows: Figure 5 As shown, the semantic fusion network includes a self-attention module and a forward propagation layer. First, multimodal information containing visual features and text prototypes is fused and enhanced based on the self-attention mechanism of the self-attention module. Specifically, the visual features and the text prototypes are input into the self-attention module, so that one of the visual features and the text prototypes fuses and enhances the other based on the self-attention mechanism, resulting in a first visual feature and a first text prototype. Subsequently, the first visual feature and the first text prototype are enhanced by introducing the text prototype into the information fused by the self-attention mechanism. Specifically, the first visual feature, the first text prototype, and the text prototype are summed and normalized to obtain an enhanced first visual feature and an enhanced first text prototype.

[0104] The enhanced first visual feature and the enhanced first text prototype are input into the forward propagation layer. The forward propagation layer maps the multimodal information (enhanced first visual feature and enhanced first text prototype) to a common latent space to obtain the second visual feature and the second text prototype. Finally, the second visual feature and the second text prototype are enhanced by introducing the enhanced first text prototype into the information mapped by the forward propagation layer. Specifically, the second visual feature, the second text prototype, and the enhanced first text prototype are summed and normalized to obtain an adaptive text prototype and an adaptive visual feature. The adaptive visual feature is then decomposed into an adaptive visual prototype and an adaptive query representation to obtain an adaptive prototype, which includes at least an adaptive visual prototype, an adaptive query representation, and an adaptive text prototype.

[0105] Based on the semantic fusion module, visual features and text prototypes are fused and enhanced. The resulting adaptive prototype integrates image information and semantic information. The adaptive learning mechanism based on the semantic fusion module enhances the semantic association between visual prototypes and image categories, effectively mitigating the catastrophic forgetting problem in continuous learning.

[0106] In one alternative implementation, the semantic fusion network needs to be optimized in each continuous learning task. Specifically, in the t-th continuous learning task, the input data is fed into the semantic fusion network to obtain an adaptive visual prototype, an adaptive text prototype, and an adaptive query representation. Then, the overall loss is calculated based on the adaptive visual prototype, the adaptive text prototype, and the adaptive query representation, and the semantic fusion network in the t-th continuous learning task is trained and optimized based on the overall loss.

[0107] First, based on the adaptive visual prototype, the adaptive text prototype, and the first ground truth label, a semantic consistency loss is obtained. The first ground truth label is used to characterize the matching relationship between the adaptive visual prototype and the corresponding text prototype. The semantic consistency loss is obtained according to the following formula:

[0108] m v =argmax t (e′ v ·e′ t );

[0109]

[0110] Among them, e′ v For the adaptive vision prototype, e′ t For the adaptive text prototype, m v It is the adaptive vision prototype e′ vWith the adaptive text prototype e′ t The maximum calculated matching probability between them, g v It is the first true label, L m For the semantic consistency loss, CrossEntropy is the cross-entropy function, (e′) v ·e′ t ) is e′ v With e′ t The cosine distance between them.

[0111] After semantic fusion, the adaptive text prototype is used as the anchor point of the shared latent space, and it is expected that the adaptive visual prototype is aligned with the adaptive text prototype as the latent space anchor point, thereby reducing the estimation error in the case of very few samples. Therefore, the embodiments of this application optimize the semantic fusion network by obtaining the semantic consistency loss through semantic knowledge regularization in the above formula, so that the adaptive visual prototype obtained based on the optimized semantic fusion network is aligned with the adaptive text prototype, and the visual prototype is enhanced by semantic information.

[0112] In this study, since the distribution of adaptive text prototypes in the latent space is sparser than that of adaptive visual prototypes, the semantic fusion network is optimized using semantic consistency loss (i.e., semantic-level regularization). This ensures that categories with similar semantics are closely distributed, while opposing categories are restricted to positions far apart from each other. Consequently, the distribution of adaptive text prototypes in the latent space becomes denser and more aligned with the distribution of adaptive visual prototypes. Optimizing the semantic fusion network through semantic-level regularization based on semantic consistency loss enhances its ability to distinguish images belonging to new categories from existing old categories in the database, thereby significantly mitigating the catastrophic forgetting problem in few-shot continuous learning.

[0113] Based on the adaptive query representation, the adaptive visual prototype, and the second true label, a query loss is obtained, where the second true label is the true category label of the query representation. The query loss is obtained according to the following formula:

[0114] c q =argmax v (e′ q ·e′ v );

[0115]

[0116] Among them, e′ q For the adaptive query representation, e′ v For the adaptive vision prototype, c q y is the predicted category label of the adaptive query representation. qL is the second true label. c For the semantic consistency loss, CrossEntropy is the cross-entropy function, (e′) q ·e′ v ) is e′ q With e′ v The cosine distance between them.

[0117] In this embodiment, the category of the query image is classified based on the nearest neighbor principle using an adaptive query representation. The semantic fusion module is optimized by obtaining the query loss to improve the classification accuracy based on the adaptive query representation.

[0118] Subsequently, based on the semantic consistency loss and the query loss, the overall loss is obtained, which is calculated according to the following formula:

[0119] L = L c +λ·L m ;

[0120] Where L is the total loss, L c For the query loss, L m Let λ be the semantic consistency loss, and λ be the control factor.

[0121] Figure 3 This illustration shows a schematic diagram of an image classification model based on multimodal few-shot continuous learning proposed in an embodiment of this application, as follows: Figure 3 As shown, based on the overall loss, the semantic fusion network is optimized to obtain an optimized semantic fusion network. This optimized semantic fusion network performs semantic fusion on the query image based on the t-th continuous learning task, and optimizes the semantic fusion performance using semantic knowledge regularization based on the semantic fusion result. Simultaneously, it optimizes the semantic fusion performance based on the classification of the query image (…). Figure 3 The semantic fusion network obtained by optimizing the visual similarity score in the image is used as the semantic fusion network for the (t+1)th continuous learning task. In the (t+1)th continuous learning task, it is used to perform subsequent semantic fusion on the query image of the (t+1)th continuous learning task.

[0122] In this embodiment, the semantic fusion network is optimized in both the continuous learning task stage and the basic learning task stage. Specifically, when t=1, the semantic fusion network is the first semantic fusion network; when t=2, the semantic fusion network is obtained by optimizing the first semantic fusion network based on the overall loss; when t>2, the semantic fusion network in the t-th continuous learning task is obtained by optimizing the semantic fusion network in the (t-1)-th continuous learning task based on the overall loss. The first semantic fusion network is obtained by optimizing the initial semantic fusion network based on the training dataset in the basic learning task stage. During this process, the initial backbone network is also optimized. For details of the process of obtaining the first semantic fusion network from the initial semantic fusion network, please refer to the process of obtaining the initially trained backbone network in S102 above, which will not be repeated here.

[0123] S104. Based on the adaptive visual prototype and the adaptive query representation, obtain the category probability of the query image.

[0124] In specific implementation step S104, the adaptive query representation is classified using the nearest neighbor principle. Based on the adaptive visual prototype and the adaptive query representation, without loss of generality, the cosine distance between the adaptive query representation and the adaptive visual prototype is obtained so that the adaptive query representation can be assigned to the category of the adaptive visual prototype with the smallest distance to the adaptive query representation in the latent space. Based on the cosine distance, the probability that the adaptive query representation belongs to the image category corresponding to the adaptive visual prototype is obtained as the category probability. The smaller the cosine distance between the adaptive query representation and an adaptive visual prototype in the latent space, the greater the probability that the query image corresponding to the adaptive query representation belongs to the category of that adaptive visual prototype; conversely, the larger the cosine distance between the adaptive query representation and an adaptive visual prototype in the latent space, the smaller the probability that the query image corresponding to the adaptive query representation belongs to the category of that adaptive visual prototype. The query image can be classified using this category probability.

[0125] In one optional implementation, after obtaining the cosine distance between the adaptive query representation and the adaptive visual prototype, based on the cosine distance, the adaptive visual prototype with the smallest cosine distance to the adaptive query representation is obtained as the target adaptive visual prototype. The adaptive query representation has the highest probability of belonging to the category of the target adaptive visual prototype. The image category corresponding to the target adaptive visual prototype is used as the category of the query image.

[0126] To enable those skilled in the art to more clearly understand this application, Figure 4 This illustration shows a schematic diagram of the network architecture of an image classification method based on multimodal few-shot continuous learning proposed in an embodiment of this application. Figure 4 Taking the following examples, the image classification method based on multimodal few-shot continuous learning described in this application will be described in detail.

[0127] Retrieve a preset number of image categories from the database, and sample images from each image category to obtain the original support images. Figure 4 Supporting images for the basic learning task stage and the original query image ( Figure 4 (The query images in the basic learning task stage) are used to obtain the original semantic information based on the category of the original supporting images. Figure 4 For example, corresponding to Figure 4 The image shows the original semantic information corresponding to the categories of the three original support images: "orange," "bulldog," and "traffic sign." The original query image, original support images, and original semantic information were used as the training dataset to train the initial backbone network and the initial semantic fusion network during the basic learning task phase.

[0128] In each training cycle of the basic learning task, the original support image and the original query image from the training dataset are respectively input into the initial convolutional subnetwork ( Figure 4 The convolutional neural network in the model obtains the hidden representation of the original query image in the shared hidden space based on the initial convolutional sub-network, which serves as the original query representation corresponding to the original query image. Figure 4 In the basic learning task stage, the query representation is obtained based on the initial convolutional sub-network to obtain the average latent representation of the original support image of each category in the shared latent space, which serves as the original visual prototype for each category. Figure 4 The visual prototype in the basic learning task stage; inputting the original semantic information into the initial projection subnetwork ( Figure 4 The projection operator in the initial projection subnetwork projects the original semantic information into the shared latent space through the linear affine layer to obtain the original text prototype. The original text prototype, the original query representation, and the original visual prototype are then combined into the original input data.

[0129] The original input data is fed into the initial semantic fusion network. Figure 4 In the basic learning task stage, the semantic fusion Transformer obtains the original adaptive prototype (including the original adaptive query features, the original adaptive visual prototype, and the original adaptive text prototype, corresponding to...). Figure 4The semantic fusion Transformer outputs three times during the basic learning task phase. Based on the original adaptive query features, original adaptive visual prototypes, and original ground truth labels, a semantic consistency loss is obtained. This loss is used to optimize the network model parameters during the basic learning task phase, encouraging alignment between visual and text prototypes. Based on the original adaptive text prototypes and original ground truth labels, a query loss is obtained. This loss is used to optimize the network model parameters during the basic learning task phase to distinguish different categories of query images and enhance the visual prototypes using semantic information. Based on the training dataset, the initial projection sub-network, the initial convolutional sub-network, and the initial semantic fusion network are trained to obtain the first semantic fusion network, the projection sub-network, and the convolutional network.

[0130] The parameters of the fixed projection subnetwork and convolutional subnetwork are applied to each subsequent continuous learning task, and the semantic fusion network is further optimized in subsequent continuous learning tasks. For the t-th continuous learning task, taking t=1 as an example (i.e., the first continuous learning task), the query image and support images are first obtained. These query images and support images are different from the original query image and original support images mentioned above. They are images sampled from multiple image categories input to this continuous learning task. At the same time, semantic information is obtained based on the category to which the support image belongs. Figure 4 For example, corresponding to Figure 4 The semantic information of the category to which the supporting image belongs is "dome". The supporting image and the query image are respectively input into a convolutional sub-network. Based on the convolutional sub-network, the latent representation of the query image in the shared latent space is obtained, serving as the query representation for the query image. The average latent representation of the supporting images for each category in the shared latent space is also obtained based on the convolutional sub-network, serving as the visual prototype for each category. The semantic information is input into a projection sub-network, and the semantic information (word embeddings for each category) is projected into the shared latent space through a linear affine layer in the projection sub-network to obtain the text prototype. After obtaining the visual prototype and query representation, since both are latent representations of images, the query representation and the visual prototype are concatenated to obtain the visual feature. This visual feature and the semantic prototype constitute the input data.

[0131] The input data is fed into a first semantic fusion network to fuse and enhance visual features and text prototypes, resulting in an adaptive prototype (including an adaptive visual prototype, an adaptive text prototype, and an adaptive query representation) output from the first continuous learning task. The adaptive query representation is classified using the nearest neighbor principle. Based on the adaptive visual prototype and the adaptive query representation, without loss of generality, the cosine distance between the adaptive query representation and the adaptive visual prototype is obtained. Based on the cosine distance, the probability that the adaptive query representation belongs to the image category corresponding to the adaptive visual prototype is obtained as the category probability. This category probability can be used to classify the query image.

[0132] Based on the adaptive visual prototype, the adaptive text prototype, and the first ground truth label, a semantic consistency loss is obtained. Based on the adaptive query representation, the adaptive visual prototype, and the second ground truth label, a query loss is obtained. The semantic consistency loss and the query loss are then combined into a total loss, and the first semantic fusion network is optimized based on this total loss. The optimized semantic fusion network is used as the semantic fusion network for the (t+1)th continuous learning task. Then, the (t+1)th continuous learning task begins, using the optimized semantic fusion network from the (t)th continuous learning task as the semantic fusion network for the (t+1)th continuous learning task. The query images for the (t+1)th continuous learning task are classified according to the above steps, and the optimized semantic fusion network for the (t+1)th continuous learning task is obtained simultaneously. This achieves few-shot continuous learning, continuously optimizing the semantic fusion network while classifying continuous query images.

[0133] This application provides an image classification method based on multimodal few-shot continuous learning, applied to the t-th continuous learning task, t≥1. The method includes: acquiring an input image, the input image including at least a query image and support images; acquiring input data based on the input image, the input data including at least visual features and text prototypes corresponding to the visual features; inputting the visual features and the text prototypes into a semantic fusion network to perform information fusion on the visual features and the text prototypes to obtain an adaptive prototype, the adaptive prototype including at least an adaptive visual prototype and an adaptive query representation; and acquiring the category probability of the query image based on the adaptive visual prototype and the adaptive query representation.

[0134] This application addresses the catastrophic forgetting problem in few-shot continuous learning by fusing semantic and visual information hidden in categorical texts using a semantic fusion network. This allows the model to consider the influence of semantic information during few-shot continuous learning, thereby greatly mitigating the problem of catastrophic forgetting.

[0135] Based on the same inventive concept, this application discloses an image classification device based on multimodal few-shot continuous learning, applied to the t-th continuous learning task, where t≥1. Figure 6 A schematic diagram of an image classification device based on multimodal few-shot continuous learning is shown, such as... Figure 6 As shown, the device includes:

[0136] The acquisition module is used to acquire an input image, which includes at least a query image and a support image;

[0137] The underlying processing module is used to acquire input data based on the input image, wherein the input data includes at least visual features and the text prototypes corresponding to the visual features;

[0138] The semantic fusion module is used to input the visual features and the text prototype into the semantic fusion network to perform information fusion on the visual features and the text prototype to obtain an adaptive prototype. The adaptive prototype includes at least an adaptive visual prototype, an adaptive query representation, and an adaptive text prototype.

[0139] The classification module is used to obtain the category probability of the query image based on the adaptive visual prototype and the adaptive query representation.

[0140] In one alternative embodiment, the device further includes:

[0141] The first loss module is used to obtain semantic consistency loss based on the adaptive visual prototype, the adaptive text prototype and the first real label, wherein the first real label is used to characterize the matching relationship between the adaptive visual prototype and the text prototype corresponding to the adaptive visual prototype.

[0142] The second loss module is used to obtain the query loss based on the adaptive query representation, the adaptive visual prototype, and the second true label, wherein the second true label is the true category label of the query representation;

[0143] An optimization module is used to optimize the semantic fusion network based on the semantic consistency loss and the query loss to obtain an optimized semantic fusion network.

[0144] An update module is used to use the optimized semantic fusion network as the semantic fusion network for the (t+1)th continuous learning task.

[0145] In one optional implementation, the underlying processing module includes:

[0146] The semantic acquisition submodule is used to acquire semantic information based on the category to which the supported image belongs;

[0147] The first low-level processing submodule is used to input the support image and the query image into the convolutional subnetwork to obtain the visual prototype corresponding to the support image and the query representation corresponding to the query image, respectively.

[0148] The second low-level processing submodule is used to input the semantic information into the projection subnetwork to obtain the text prototype;

[0149] The connection submodule is used to connect the query representation and the visual prototype to obtain the visual features.

[0150] In one optional implementation, the semantic fusion module includes:

[0151] The self-attention submodule is used to input the visual feature and the text prototype into the self-attention module, so that one of the visual feature and the text prototype performs information fusion and information enhancement on the other based on the self-attention mechanism to obtain the first visual feature and the first text prototype.

[0152] The first enhancement submodule is used to sum and normalize the first visual feature, the first text prototype and the text prototype to obtain the enhanced first visual feature and the enhanced first text prototype.

[0153] The forward propagation submodule is used to input the enhanced first visual feature and the enhanced first text prototype into the forward propagation layer to map to the latent space to obtain the second visual feature and the second text prototype.

[0154] The second enhancement submodule is used to sum and normalize the second visual feature, the second text prototype, and the enhanced first text prototype to obtain the adaptive prototype.

[0155] In one optional implementation, the classification module includes:

[0156] The distance calculation submodule is used to obtain the cosine distance between the adaptive query representation and the adaptive visual prototype based on the adaptive visual prototype and the adaptive query representation;

[0157] The category probability acquisition submodule is used to obtain the probability that the adaptive query representation belongs to the image category corresponding to the adaptive visual prototype based on the cosine distance, and use it as the category probability.

[0158] In one optional implementation, the classification module further includes:

[0159] The target adaptive visual prototype acquisition submodule is used to acquire the target adaptive visual prototype based on the cosine distance, wherein the target adaptive visual prototype is the adaptive visual prototype with the smallest cosine distance to the adaptive query representation.

[0160] The classification submodule is used to use the image category corresponding to the target adaptive visual prototype as the category of the query image.

[0161] In one alternative embodiment, the device further includes:

[0162] The training set acquisition module is used to acquire the training dataset, which includes at least the original supporting images, the original query images, and the original semantic information.

[0163] The raw input data acquisition module is used to input the training dataset into the initial projection sub-network and the initial convolutional sub-network to acquire the raw input data.

[0164] The initial semantic fusion module is used to input the original input data into the initial semantic fusion network to obtain the original adaptive prototype;

[0165] The basic loss module is used to obtain the basic training loss based on the original adaptive prototype and the original true labels;

[0166] The training module is used to train the initial projection sub-network, the initial convolutional sub-network, and the initial semantic fusion network based on the basic training loss to obtain a first semantic fusion network, a projection sub-network, and a convolutional network. The projection sub-network and the convolutional network are used to acquire the input data in all continuous learning tasks.

[0167] The first update module is used to use the first semantic fusion network as the semantic fusion network at t=1.

[0168] Based on the same inventive concept, this application discloses an electronic device. Figure 7 A schematic diagram of the electronic device proposed in an embodiment of this application is shown, such as... Figure 7 As shown, the electronic device 100 includes a memory 110 and a processor 120. The memory 110 and the processor 120 are connected via a bus. The memory 110 stores at least one instruction, at least one program, code set, or instruction set. The at least one instruction, the at least one program, the code set, or the instruction set can be loaded and run on the processor 120 to implement the steps in the image classification method based on multimodal few-shot continuous learning disclosed in the embodiments of this application.

[0169] Based on the same inventive concept, embodiments of this application disclose a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set thereon. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the steps in the image classification method based on multimodal few-shot continuous learning disclosed in embodiments of this application.

[0170] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0171] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, electronic devices, and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0172] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0173] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0174] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0175] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0176] The above provides a detailed description of the image classification method and product based on multimodal few-shot continuous learning provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An image classification method based on multimodal few-shot continuous learning, characterized in that, Applied to the t-th continuous learning task, t≥1, the method includes: Obtain an input image, which includes at least a query image and a support image; Based on the input image, input data is obtained, and the input data includes at least visual features and the text prototypes corresponding to the visual features; The visual features and the text prototype are input into a semantic fusion network to perform information fusion on the visual features and the text prototype to obtain an adaptive prototype, which includes at least an adaptive visual prototype and an adaptive query representation. Based on the adaptive visual prototype and the adaptive query representation, the category probability of the query image is obtained; The adaptive prototype also includes an adaptive text prototype. After obtaining the adaptive prototype, the method further includes: Based on the adaptive visual prototype, the adaptive text prototype and the first real label, a semantic consistency loss is obtained, where the first real label is used to characterize the matching relationship between the adaptive visual prototype and the text prototype corresponding to the adaptive visual prototype. Based at least on the semantic consistency loss, the semantic fusion network is optimized to obtain an optimized semantic fusion network, thereby improving the semantic fusion network's ability to distinguish between images belonging to a new category and existing old categories in the database; The semantic consistency loss is obtained according to the following formula: ； ； in, This is the adaptive vision prototype. This is the adaptive text prototype. It is the adaptive vision prototype With the adaptive text prototype The maximum calculated matching probability between them It is the first real label. For the semantic consistency loss, The cross-entropy function, for and The cosine distance between them.

2. The image classification method based on multimodal few-shot continuous learning according to claim 1, characterized in that, The adaptive prototype also includes an adaptive text prototype. After obtaining the adaptive prototype, the method further includes: Based on the adaptive query representation, the adaptive visual prototype, and the second true label, the query loss is obtained, where the second true label is the true category label of the query representation. Based on the semantic consistency loss and the query loss, the semantic fusion network is optimized to obtain an optimized semantic fusion network. The optimized semantic fusion network is used as the semantic fusion network for the (t+1)th continuous learning task.

3. The image classification method based on multimodal few-shot continuous learning according to claim 1, characterized in that, Based on the input image, input data is obtained, including: Based on the category to which the supported image belongs, semantic information is obtained; The support image and the query image are input into the convolutional sub-network to obtain the visual prototype corresponding to the support image and the query representation corresponding to the query image, respectively. The semantic information is input into the projection subnetwork to obtain the text prototype; The visual feature is obtained by concatenating the query representation and the visual prototype.

4. The image classification method based on multimodal few-shot continuous learning according to claim 1, characterized in that, The semantic fusion network includes a self-attention module and a forward propagation layer. The visual features and the text prototype are input into the semantic fusion network to fuse information from the visual features and the text prototype, obtaining an adaptive prototype, including: The visual feature and the text prototype are input into the self-attention module, so that one of the visual feature and the text prototype can perform information fusion and information enhancement on the other based on the self-attention mechanism to obtain the first visual feature and the first text prototype. The first visual feature, the first text prototype, and the text prototype are summed and normalized to obtain the enhanced first visual feature and the enhanced first text prototype. The enhanced first visual feature and the enhanced first text prototype are input into the forward propagation layer to be mapped to the latent space to obtain the second visual feature and the second text prototype. The second visual feature, the second text prototype, and the enhanced first text prototype are summed and normalized to obtain the adaptive prototype.

5. The image classification method based on multimodal few-shot continuous learning according to claim 1, characterized in that, Based on the adaptive visual prototype and the adaptive query representation, the category probability of the query image is obtained, including: Based on the adaptive visual prototype and the adaptive query representation, obtain the cosine distance between the adaptive query representation and the adaptive visual prototype; Based on the cosine distance, the probability that the adaptive query representation belongs to the image category corresponding to the adaptive visual prototype is obtained, and is used as the category probability.

6. The image classification method based on multimodal few-shot continuous learning according to claim 5, characterized in that, After obtaining the cosine distance between the adaptive query representation and the adaptive visual prototype, the method further includes: Based on the cosine distance, an adaptive visual prototype of the target is obtained, wherein the adaptive visual prototype of the target is the adaptive visual prototype with the smallest cosine distance to the adaptive query representation. The image category corresponding to the target adaptive visual prototype is used as the category of the query image.

7. The image classification method based on multimodal few-shot continuous learning according to claim 1, characterized in that, When t=1, the semantic fusion network is obtained as follows: Obtain a training dataset, which includes at least the original support images, the original query images, and the original semantic information; The training dataset is input into the initial projection subnetwork and the initial convolutional subnetwork to obtain the original input data; The original input data is fed into the initial semantic fusion network to obtain the original adaptive prototype; Based on the original adaptive prototype and the original real labels, obtain the basic training loss; Based on the aforementioned basic training loss, the initial projection subnetwork, the initial convolutional subnetwork, and the initial semantic fusion network are trained to obtain a first semantic fusion network, a projection subnetwork, and a convolutional network. The projection subnetwork and the convolutional network are used to acquire the input data in all continuous learning tasks. The first semantic fusion network is used as the semantic fusion network at t=1.

8. An image classification device based on multimodal few-shot continuous learning, characterized in that, For the t-th continuous learning task, t≥1, the apparatus includes: The acquisition module is used to acquire an input image, which includes at least a query image and a support image; The underlying processing module is used to acquire input data based on the input image, wherein the input data includes at least visual features and the text prototypes corresponding to the visual features; The semantic fusion module is used to input the visual features and the text prototype into the semantic fusion network to perform information fusion on the visual features and the text prototype to obtain an adaptive prototype, wherein the adaptive prototype includes at least an adaptive visual prototype and an adaptive query representation. A classification module is used to obtain the category probability of the query image based on the adaptive visual prototype and the adaptive query representation; The device further includes: The first loss module is used to obtain semantic consistency loss based on the adaptive visual prototype, the adaptive text prototype and the first real label, wherein the first real label is used to characterize the matching relationship between the adaptive visual prototype and the text prototype corresponding to the adaptive visual prototype. The first processing module is used to optimize the semantic fusion network based at least on the semantic consistency loss to obtain an optimized semantic fusion network, so as to improve the ability of the semantic fusion network to distinguish between images belonging to a new category and old categories already existing in the database; The semantic consistency loss is obtained according to the following formula: ； ； in, This is the adaptive vision prototype. This is the adaptive text prototype. It is the adaptive vision prototype With the adaptive text prototype The maximum calculated matching probability between them It is the first real label. For the semantic consistency loss, The cross-entropy function, for and The cosine distance between them.

9. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the image classification method based on multimodal few-shot continuous learning as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps in the image classification method based on multimodal few-shot continuous learning as described in any one of claims 1-7.

Images