Construction method of image classification system based on knowledge perception complementation
By constructing a knowledge-based complementary image classification system, which combines general visual knowledge with task-specific information, we have achieved high accuracy and robustness of the image classification system in low-sample scenarios, and solved the problem of information fusion inconsistency in existing technologies.
Patent Information
- Application Number
- CN202610032180.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-12
- Publication Date
- 2026-05-01
AI Technical Summary
Existing technologies fail to effectively integrate general prior knowledge with task-specific cues and lack a unified coordination and optimization framework, resulting in limited accuracy and robustness in image classification.
A knowledge-aware complementary image classification system is constructed. By introducing a knowledge adaptation module to combine general visual knowledge with task-specific information, and using joint training loss to achieve synergistic optimization of general visual features and task-specific knowledge, the image classification system is configured to use a prior-guided complementary fusion mechanism to determine the target probability distribution.
It enhances the generalization and robustness of the image classification system in low-sample scenarios, and improves the accuracy of image classification.
Smart Images
Figure CN121962730A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision, specifically to multimodal image classification technology in the field of computer vision, and more specifically, to a method for constructing an image classification system based on knowledge-perception complementarity. Background Technology
[0002] In recent years, multimodal learning has made significant breakthroughs in the field of joint vision-language modeling. Large-scale pre-trained models, represented by CLIP, have achieved high-quality semantic alignment between visual and language modalities through comparative learning on massive image-text pairs. These models exhibit excellent cross-task generalization capabilities in zero-shot and few-shot scenarios, and even with limited supervised data, they can still achieve good transferability through cross-modal alignment. However, how to efficiently adapt such pre-trained models to downstream few-shot tasks remains a core challenge. Due to the scarcity of labeled samples, cross-modal semantic alignment becomes more difficult, making it challenging to accurately capture task-specific discriminative semantics while maintaining the original general knowledge of CLIP.
[0003] To address the aforementioned issues, recent research has largely focused on calibration at the logit prediction level. This involves introducing additional modules to learn task-related logit biases, thereby improving the model's adaptability to downstream tasks. For example, some studies have proposed using auxiliary features to enhance multimodal few-shot classification performance, others have proposed integrating multiple pre-trained models to improve CLIP's discriminative power, and still others have proposed training-free strategies (such as cache-based feature construction) to reduce computational costs. These methods typically rely on two types of information sources: general prior knowledge acquired from large-scale pre-training and task-specific cues extracted from limited labeled data. The former provides a cross-task transferable semantic foundation, while the latter carries the local discriminative patterns for the current task.
[0004] Nevertheless, existing methods still have significant limitations: they often treat general priors and task-specific information in isolation, lacking a systematic integration of these two types of knowledge sources. While general priors have broad applicability, they may not perfectly match the current task; and while task-specific information is targeted, it is prone to overfitting or insufficient representation due to data scarcity. Because of the lack of a unified optimization mechanism to coordinate the use of these two complementary information sources, existing adaptation strategies often lead to inconsistent knowledge utilization and optimization objectives, thus affecting the overall performance and robustness of the model under limited sample conditions.
[0005] In summary, existing technologies fail to effectively integrate general prior knowledge with task-specific cues and lack a unified coordination and optimization framework, resulting in the inability to fully utilize the two types of complementary information when performing image classification, thus limiting the accuracy and robustness of image classification.
[0006] It should be noted that the background information presented here is only for illustrating relevant information about the present invention to aid in understanding the technical solutions of the present invention, and does not imply that the relevant information is necessarily prior art. In the absence of evidence indicating that the relevant information was disclosed before the filing date of this invention, the relevant information should not be considered prior art. Summary of the Invention
[0007] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide a method for constructing an image classification system based on knowledge perception complementarity and an image classification method.
[0008] The objective of this invention is achieved through the following technical solutions.
[0009] According to a first aspect of the present invention, a method for constructing an image classification system based on knowledge-aware complementarity is provided. The method includes: step S1, obtaining a small-sample training set, wherein the training set includes multiple images of various categories, and each image corresponds to a text description indicating the category of the image; step S2, constructing an initial system, the initial system including a multimodal model, a multilayer perceptron, a knowledge adaptation module, and a fusion module, wherein: the multimodal model is used to extract image features and text features corresponding to the image, and predict a first probability distribution of the image based on the image features and text features; the multilayer perceptron is used to enhance the image features extracted by the multimodal model to obtain enhanced image features corresponding to the image; the knowledge adaptation module stores multiple learnable image features and the corresponding text features of each image feature. The label vector and knowledge adaptation module are used to extract visual features of the image to calculate the second probability distribution of the image, and to calculate the similarity between the enhanced image features corresponding to the image and the stored image features to obtain the similarity matrix of the image. Based on the image similarity matrix and the label vector corresponding to each stored image feature, the third probability distribution of the image is calculated, and the prior probability distribution of the image is calculated based on the second and third probability distributions of the image. The fusion module is used to calculate the target probability distribution of the image based on the first probability distribution and the prior probability distribution of the image. Step S3: The initial system is iteratively trained multiple times using a small sample training set until convergence to obtain the image classification system. In each iteration of training, a preset loss function is used to calculate the loss to update the parameters of the multilayer perceptron and the knowledge adaptation module.
[0010] According to some embodiments of the present invention, the knowledge adaptation module includes a cached feature subspace, a pre-trained visual model, a linear probing submodule, a similarity calculation submodule, a multiplication calculation submodule, and a joint calculation submodule, wherein: the cached feature subspace is used to store multiple learnable image features and a label vector corresponding to each image feature; the pre-trained visual model is used to extract visual features of the image; the linear probing submodule is used to calculate a second probability distribution of the image based on the visual features of the image; the similarity calculation submodule is used to calculate the similarity between the enhanced image features corresponding to the image and each stored image feature to obtain a similarity matrix of the image; the multiplication calculation submodule is used to calculate a third probability distribution of the image based on the similarity matrix of the image and the label vector corresponding to each stored image feature; and the joint calculation submodule is used to calculate the prior probability distribution of the image based on the second probability distribution and the third probability distribution of the image.
[0011] According to some embodiments of the present invention, the method further includes: configuring the similarity calculation submodule to calculate the similarity between the enhanced image features corresponding to the image and each stored image feature in the following manner:
[0012]
[0013] in, Indicates the first The similarity matrix of the images, This represents the number of learnable image features stored in the cached feature subspace. Indicates the first Enhanced image features of the image. This represents the learnable first feature stored in the cache feature subspace. Image features.
[0014] According to some embodiments of the present invention, the method further includes:
[0015] The multimodal model is configured to compute the first probability distribution of the image as follows:
[0016]
[0017] in, Indicates the first The first probability distribution of the images, The first part of the multimodal model extraction is represented by the first part. Textual features of the image The first part of the multimodal model extraction is represented by the first part. Image features of the image;
[0018] The linear probe submodule is configured to calculate the second probability distribution of the image as follows:
[0019]
[0020] in, Indicates the first The second probability distribution of the images The weights of the linear probe submodules are represented. The first image extracted by the pre-trained visual model. Visual features of the image;
[0021] The multiplication calculation submodule is configured to calculate the third probability distribution of the image as follows:
[0022]
[0023] in, Indicates the first The third probability distribution of the images, This indicates the first feature stored in the cache feature subspace. Label vectors corresponding to the features of each image.
[0024] According to some embodiments of the present invention, the method further includes: configuring the joint computation submodule to compute the prior probability distribution of the image in the following manner:
[0025]
[0026] in, Indicates the first Prior probability distribution of the images This represents the first weighting coefficient. This represents the second weighting coefficient.
[0027] According to some embodiments of the present invention, the preset loss function is:
[0028]
[0029] in,
[0030]
[0031]
[0032]
[0033] in, Indicates the total loss. Indicates hyperparameters, Indicates the first loss. This indicates the second loss. Indicates the third loss. This indicates the number of classes contained in a small sample training set. This indicates the number of images contained in each category. Natural logarithm function, Represents the normalized exponential function, Indicates the first Category labels for the images.
[0034] According to some embodiments of the present invention, the method further includes: configuring the fusion module to calculate the target probability distribution of the image in the following manner:
[0035]
[0036] in, Indicates the first Target probability distribution of the image.
[0037] According to a second aspect of the present invention, an image classification method is provided, the method comprising: step T1, obtaining an image to be classified and its corresponding text prompts, wherein the text prompts include multiple categories to indicate the possible categories to which the image to be classified may belong; step T2, processing the image to be classified using an image classification system constructed as described in the first aspect of the present invention to obtain a target probability distribution of the image, wherein the target probability distribution includes a predicted probability corresponding to each category, and the category with the highest predicted probability is taken as the classification result of the image to be classified.
[0038] Compared with the prior art, the advantages of the present invention are: (1) a knowledge adaptation module is introduced, in which general visual knowledge is combined with specific task information to achieve fine-grained control and semantic alignment at the logits level, thereby enhancing the generalization and robustness of the image classification system in few-sample scenarios; (2) a joint training loss is set to achieve synergistic optimization between general visual features and specific task knowledge, so that general features and specific task space are gradually aligned, thereby enhancing the generalization and robustness of the image classification system in few-sample scenarios; (3) the image classification system is configured to use a priori-guided complementary fusion mechanism to determine the target probability distribution in order to achieve image classification, thereby improving the accuracy of image classification. Attached Figure Description
[0039] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:
[0040] Figure 1 This is a flowchart illustrating a method for constructing an image classification system according to an embodiment of the present invention;
[0041] Figure 2This is a schematic diagram of the composition structure of the knowledge adaptation module according to an embodiment of the present invention;
[0042] Figure 3 This is a schematic diagram of the image classification method according to an embodiment of the present invention. Detailed Implementation
[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the invention.
[0044] As mentioned in the background section, in the prior art, due to the failure to effectively integrate general prior knowledge and task-specific cues, and the lack of a unified coordination and optimization framework, the two types of complementary information cannot be fully utilized when performing image classification, thus limiting the accuracy and robustness of image classification.
[0045] To address the aforementioned issues, the inventors propose a knowledge-aware complementary multimodal few-shot learning framework for constructing image classification systems. In this framework, an initial system is built upon a zero-shot multimodal model, and a knowledge adaptation module is integrated into this initial system to combine general visual knowledge with task-specific information, thereby achieving fine-grained control and semantic alignment at the probability distribution (logits) level. Furthermore, a joint training loss is introduced to achieve synergistic optimization between general visual features and task-specific knowledge. Even further, the image classification system is configured to employ a prior-guided complementary fusion mechanism to determine the target probability distribution for image classification. Based on this learning framework, the inventors propose a method for constructing an image classification system. This method includes steps S1-S3. In step S1, a few-sample training set is obtained, which includes a small number of image samples and corresponding text descriptions for each image sample. In step S2, an initial system is constructed, comprising a multimodal model, a multilayer perceptron, a knowledge adaptation module, and a fusion module. The multimodal model is used to predict the first probability distribution of the image, the multilayer perceptron is used to enhance the image features extracted by the multimodal model, the knowledge adaptation module is used to predict the prior probability distribution of the image based on the image and its corresponding enhanced image features, and the fusion module is used to calculate the target probability distribution of the image based on the first probability distribution and the prior probability distribution. In step S3, the initial system is iteratively trained multiple times using the few-sample training set until convergence to obtain the image classification system.
[0046] In summary, such as Figure 1As shown, a method for constructing an image classification system based on knowledge-aware complementarity is provided. The method includes: Step S1, obtaining a small-sample training set, wherein the training set includes multiple images of various categories, and each image corresponds to a text description indicating the category of the image; Step S2, constructing an initial system, wherein the system includes a multimodal model, a multilayer perceptron, a knowledge adaptation module, and a fusion module, wherein: the multimodal model is used to extract image features and text features corresponding to the image, and predict a first probability distribution of the image based on the image features and text features; the multilayer perceptron is used to enhance the image features extracted by the multimodal model to obtain enhanced image features corresponding to the image; the knowledge adaptation module stores multiple learnable image features and label vectors corresponding to each image feature. The knowledge adaptation module is used to extract visual features of an image to predict the second probability distribution of the image, and calculate the similarity between the enhanced image features corresponding to the image and the stored image features to obtain the similarity matrix of the image. Based on the image similarity matrix and the label vector corresponding to each stored image feature, the third probability distribution of the image is calculated, and the prior probability distribution of the image is calculated based on the second and third probability distributions of the image. The fusion module is used to calculate the target probability distribution of the image based on the first and prior probability distributions of the image. Step S3: The initial system is iteratively trained multiple times using a small sample training set until convergence is obtained to obtain the image classification system. In each iteration of training, a preset loss function is used to calculate the loss to update the parameters of the multilayer perceptron and the knowledge adaptation module.
[0047] Before describing the present invention, let's first introduce how existing multimodal few-shot learning methods achieve image classification.
[0048] To address few-shot tasks, researchers have constructed a unified multimodal few-shot learning framework. This framework analyzes the principles of various few-shot learning methods and the learning objectives of models from the perspective of logit bias. In simple terms, the CLIP-based multimodal few-shot learning framework can be represented as:
[0049]
[0050] in, This represents the predicted probability distribution corresponding to the image. This represents the text features corresponding to the image extracted by the CLIP model. , This represents the text encoder in the CLIP model. This represents the text description corresponding to the image. This represents the image features corresponding to the image extracted by the CLIP model. , This represents the visual encoder in the CLIP model. Represents an image. This represents the probability distribution (logit) corresponding to the image obtained through the CLIP model. Indicates the weighting coefficient. This represents the logit bias derived from the limited training images. Therefore, the core idea of the CLIP-based multimodal few-shot learning framework is to calibrate the probability distribution of the CLIP model's output using a learnable logit bias. This calibration mechanism allows the CLIP model to better adapt to specific downstream tasks, achieving more accurate image classification with only a few labeled samples.
[0051] Different CLIP-based multimodal few-shot learning methods learn different forms of logit bias to achieve more accurate image classification. For example, cue-based optimization methods can be described as follows:
[0052]
[0053] in, The logit bias is learned by optimizing the cue embedding through a text encoder.
[0054] While existing multimodal few-shot learning methods can improve the adaptability of CLIP models to downstream tasks, these methods often rely on general prior knowledge (e.g., by using additional pre-trained models as adaptation modules) or task-specific information (e.g., through feature caching mechanisms). These methods typically fail to establish a unified learning framework to jointly optimize prior knowledge and task-specific information. This separate approach may lead to suboptimal knowledge integration and potentially result in unreasonable logit bias, thereby affecting the final image classification performance.
[0055] To better understand the present invention, each step will be described in detail below with reference to specific embodiments.
[0056] I. Step S1
[0057] In step S1, a small sample training set is obtained, wherein the training set includes multiple images of various categories, and each image has a corresponding text description to indicate the category of the image.
[0058] In this context, images and their corresponding text descriptions can form image-text pairs. For example, if the image category is "dog" and the text description is "a photo of dog", then the corresponding image-text pair is {image, a photo of dog}. Similarly, if the image category is "bird" and the text description is "a photo of bird", then the corresponding image-text pair is {image, a photo of bird}. The representation of other image-text pairs is similar, and will not be elaborated further here.
[0059] II. Step S2
[0060] In step S2, an initial system is constructed, which includes a multimodal model, a multilayer perceptron, a knowledge adaptation module, and a fusion module. The individual modules of the initial system are described below.
[0061] 2.1 Multimodal Model
[0062] The multimodal model is used to extract image features and text features corresponding to the image, and to calculate the first probability distribution of the image based on the image features and text features. The multimodal model employs the CLIP model, which includes a visual encoder and a text encoder. The visual encoder is used to extract image features, and the text encoder is used to extract text features.
[0063] The multimodal model is configured to compute the first probability distribution (CLIPlogit) of the image as follows:
[0064]
[0065] in, Indicates the first The first probability distribution of the images, The first part of the multimodal model extraction is represented by the first part. Textual features of the image The first part of the multimodal model extraction is represented by the first part. Image features of the image.
[0066] For the CLIP model, it is configured to extract image and text features corresponding to the image in the following manner:
[0067]
[0068]
[0069] in, Represents a visual encoder. Indicates a text encoder. Indicates the first Zhang Image Indicates the first The text descriptions corresponding to the images.
[0070] 2.2 Multilayer Perceptron
[0071] The multilayer perceptron is used to enhance the image features extracted by the multimodal model to obtain the enhanced image features corresponding to the image.
[0072] 2.3 Knowledge Adaptation Module
[0073] The knowledge adaptation module stores multiple learnable image features and label vectors corresponding to each image feature. The knowledge adaptation module is used to extract visual features of the image to calculate the second probability distribution of the image, and calculate the similarity between the enhanced image features corresponding to the image and each stored image feature to obtain the similarity matrix of the image. Based on the image similarity matrix and the label vectors corresponding to each stored image feature, the module calculates the third probability distribution of the image, and calculates the prior probability distribution of the image based on the second probability distribution and the third probability distribution of the image.
[0074] To better understand the knowledge adaptation module, the following will combine... Figure 2 Let me explain.
[0075] According to one embodiment of the present invention, such as Figure 2 As shown, the Knowledge Adaptation Module (KAM) includes a cached feature subspace, a pre-trained visual model, a linear probing (LP) submodule, a similarity calculation submodule, a multiplication calculation submodule, and a joint calculation submodule. The cached feature subspace stores multiple learnable image features and the label vector corresponding to each image feature. The pre-trained visual model extracts visual features from the image. The linear probing submodule calculates the second probability distribution of the image based on its visual features. The similarity calculation submodule calculates the similarity between the enhanced image features corresponding to the image and each stored image feature to obtain the image's similarity matrix. The multiplication calculation submodule calculates the third probability distribution of the image based on the image's similarity matrix and the label vector corresponding to each stored image feature. The joint calculation submodule calculates the prior probability distribution of the image based on its second and third probability distributions.
[0076] It should be noted that the multiple image features originally stored in the cached feature subspace are obtained by processing a small sample training set using a multimodal model, and the label vector corresponding to each image feature is represented by a one-hot vector. It should also be noted that the pre-trained visual model uses a MoCov3 model with RN50 as its backbone to extract the visual features of the images.
[0077] According to one embodiment of the present invention, the method further includes: configuring the similarity calculation submodule to calculate the similarity between the enhanced image features corresponding to the image and each stored image feature in the following manner:
[0078]
[0079] in, Indicates the first The similarity matrix of the images, This represents the number of learnable image features stored in the cached feature subspace. Indicates the first Enhanced image features of the image. This represents the learnable first feature stored in the cache feature subspace. Image features.
[0080] According to one embodiment of the present invention, the method further includes: configuring the linear detection submodule to calculate a second probability distribution of the image in the following manner:
[0081]
[0082] in, Indicates the first The second probability distribution of the images The weights of the linear probe submodules are represented. The first image extracted by the pre-trained visual model. Visual features of the images. The initial weights of the linear detection submodule are determined as follows:
[0083]
[0084]
[0085] in, This represents the original weights of the linear probe submodule. This represents the mean of the visual features extracted by the pre-trained visual model for all images corresponding to the first category. This represents the mean of the visual features extracted by the pre-trained visual model for all images corresponding to the second category. The first image extracted by the pre-trained visual model. The mean of the visual features of all images corresponding to a category. This indicates the number of images contained in each category. Indicates the first The first in the class Visual features of the image.
[0086] Furthermore, the multiplication calculation submodule is configured to calculate the third probability distribution of the image in the following manner:
[0087]
[0088] in, Indicates the first The third probability distribution of the images, This indicates the first feature stored in the cache feature subspace. Label vectors corresponding to the features of each image.
[0089] According to one embodiment of the present invention, the method further includes: configuring the joint computation submodule to compute the prior probability distribution (KAM logit) of the image in the following manner:
[0090]
[0091] in, Indicates the first Prior probability distribution of the images This represents the first weighting coefficient. This represents the second weighting coefficient. and This is used to integrate general knowledge and task-specific knowledge at the logit level to obtain prior probability distributions, and and The value is determined based on the importance of general knowledge and task-specific knowledge; for example, setting... , .
[0092] Combination Figure 2 As described in the previous embodiments, the learnable cached feature subspace and the pre-trained visual model work together to support the task adaptation process. The cached feature subspace learns and extracts task-specific semantic patterns from a small number of image samples, while the pre-trained visual model provides general visual features. In the knowledge adaptation module, the image features stored in the cached feature subspace are used as semantic anchors to guide the visual features extracted by the pre-trained visual model to maintain consistency with the distribution of the specific task, thereby achieving more accurate logit-level adaptation. This enables the knowledge adaptation module to generate a more accurate prior probability distribution.
[0093] 2.4 Fusion Module
[0094] The fusion module (PGC Fusion) is used to calculate the target probability distribution (total logit) of the image based on the first probability distribution and the prior probability distribution of the image.
[0095] According to one embodiment of the present invention, the method further includes: configuring the fusion module to calculate the target probability distribution of the image in the following manner:
[0096]
[0097] in, Indicates the first Target probability distribution of the image.
[0098] III. Step S3
[0099] In step S3, the initial system is trained iteratively multiple times using a small sample training set until convergence. During each iteration, a preset loss function is used to calculate the loss and update the parameters of the multilayer perceptron and the knowledge adaptation module. It should be noted that the learnable parameters in the knowledge adaptation module are updated during each iteration. Figure 2 It can be seen that the learnable parameters in the knowledge adaptation module include multiple image features stored in the cached feature subspace, as well as the weight parameters in the linear detection submodule.
[0100] According to one embodiment of the present invention, the preset loss function is:
[0101]
[0102] in,
[0103]
[0104]
[0105]
[0106] in, Indicates the total loss. Indicates hyperparameters, Indicates the first loss. This indicates the second loss. Indicates the third loss. This indicates the number of classes contained in a small sample training set. This indicates the number of images contained in each category. Natural logarithm function, Represents the normalized exponential function, Indicates the first Category labels for the images.
[0107] As can be seen from the foregoing embodiments, this invention proposes a multimodal few-shot learning framework based on knowledge-aware complementarity to construct an image classification system. The core idea of this learning framework is to establish a unified task adaptation paradigm, introduce a knowledge adaptation module (KAM) to construct an image classification system based on the zero-shot CLIP model, combine general visual knowledge with specific task knowledge from limited samples, and improve the generalization and robustness of image classification tasks in few-shot scenarios by performing fine-grained control and semantic alignment at the logits level.
[0108] Specifically, during the initial system training process, the multimodal model processes each image and its corresponding text description in the few-sample training set to obtain the first probability distribution for each image; the multilayer perceptron enhances the image features extracted by the multimodal model for each image; the knowledge adaptation module calculates the second, third, and prior probability distributions for each image based on each image and its enhanced image features; and a joint training loss is constructed based on the first, second, and third probability distributions for each image to update the parameters of the multilayer perceptron and the learnable parameters in the knowledge adaptation module, thereby achieving collaborative optimization between general visual features and task-specific knowledge.
[0109] Based on the image classification system constructed using the method described in the foregoing embodiments, this invention also proposes an image classification method, comprising: step T1, acquiring the image to be classified and its corresponding text prompts, wherein the text prompts include multiple categories to indicate the possible categories to which the image to be classified belongs; step T2, processing the image to be classified using the image classification system constructed using the method described in the foregoing embodiments to obtain the target probability distribution of the image, wherein the target probability distribution includes the predicted probability corresponding to each category, and the category with the highest predicted probability is taken as the classification result of the image to be classified. The text prompt corresponding to the image to be classified can be represented as "a photo of {bird,dog,fish,fox,…,cat}". It should be noted that the target probability distribution also needs to be Softmax normalized, and the final classification result is determined by the predicted probability corresponding to each category in the normalized target probability distribution.
[0110] To better understand the image classification method proposed in this invention, the following will be combined with... Figure 3 Let me explain.
[0111] Depend on Figure 3As can be seen, the specific process of the image classification system performing the classification task is as follows: the CLIP model extracts the image features and text features corresponding to the image to be classified, and calculates the first probability distribution of the image to be classified based on the image features and text features; the image features of the image to be classified are passed to the multilayer perceptron for enhancement processing to obtain the enhanced image features of the image to be classified; the enhanced image features of the image to be classified are then passed to the knowledge adaptation module to calculate the prior probability distribution of the image to be classified; finally, the target probability distribution of the image to be classified is calculated through the fusion module, and the classification result of the image to be classified is determined by the predicted probability corresponding to each category in the target probability distribution after Softmax normalization.
[0112] To verify that the image classification method proposed in this invention has higher classification accuracy than existing technologies, the inventors set up a comparative experiment under the same small sample task scenario and obtained the experimental results shown in Table 1.
[0113] As shown in Table 1, compared to the zero-shot CLIP method, all other methods have better classification accuracy, and the image classification method proposed in this invention has the highest classification accuracy. This indicates that this invention effectively improves the accuracy of image classification by fusing general prior knowledge with task-specific cues. Specifically, CoOp and CLIP-Adapter primarily achieve initial improvements by adjusting text representation or introducing bias through a lightweight adapter; Tip-Adapter-F and CaFo shift towards sample feature reuse based on caching mechanisms and begin fusing features from multiple models, further driving performance improvements; AMU-Tuning introduces an uncertainty-aware fusion mechanism to dynamically integrate auxiliary information.
[0114] Table 1
[0115]
[0116] The beneficial effects of the present invention are as follows: (1) A knowledge adaptation module is introduced, in which general visual knowledge is combined with specific task information, thereby achieving fine-grained control and semantic alignment at the logits level, which enhances the generalization and robustness of the image classification system in few-sample scenarios; (2) A joint training loss is set to achieve synergistic optimization between general visual features and specific task knowledge, so that general features and specific task space are gradually aligned, which enhances the generalization and robustness of the image classification system in few-sample scenarios; (3) The image classification system is configured to use a priori-guided complementary fusion mechanism to determine the target probability distribution in order to achieve image classification, which improves the accuracy of image classification.
[0117] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently, or even in a different order, as long as the required function can be achieved.
[0118] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.
[0119] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can include, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.
[0120] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.
Claims
1. A method for constructing an image classification system based on knowledge-perceptual complementarity, characterized in that, The method includes: Step S1: Obtain a small sample training set, wherein the training set includes multiple images of various categories, and each image has a corresponding text description to indicate the category of the image; Step S2: Construct an initial system, which includes a multimodal model, a multilayer perceptron, a knowledge adaptation module, and a fusion module, wherein: The multimodal model is used to extract image features and text features corresponding to the image, and predict the first probability distribution of the image based on the image features and text features; The multilayer perceptron is used to enhance the image features extracted by the multimodal model to obtain the enhanced image features corresponding to the image. The knowledge adaptation module stores multiple learnable image features and label vectors corresponding to each image feature. The knowledge adaptation module is used to extract visual features of the image to calculate the second probability distribution of the image, calculate the similarity between the enhanced image features corresponding to the image and each stored image feature to obtain the similarity matrix of the image, calculate the third probability distribution of the image based on the image similarity matrix and the label vectors corresponding to each stored image feature, and calculate the prior probability distribution of the image based on the second probability distribution and the third probability distribution of the image. The fusion module is used to calculate the target probability distribution of the image based on the first probability distribution and the prior probability distribution of the image. Step S3: Use a small sample training set to iterate the initial system multiple times until convergence is obtained to obtain an image classification system. During each iteration of training, a preset loss function is used to calculate the loss in order to update the parameters of the multilayer perceptron and the knowledge adaptation module.
2. The method according to claim 1, characterized in that, The knowledge adaptation module includes a cached feature subspace, a pre-trained visual model, a linear detection submodule, a similarity calculation submodule, a multiplication calculation submodule, and a joint calculation submodule, wherein: The cached feature subspace is used to store multiple learnable image features and the label vector corresponding to each image feature; The pre-trained visual model is used to extract visual features from the image; The linear detection submodule is used to calculate the second probability distribution of the image based on its visual features; The similarity calculation submodule is used to calculate the similarity between the enhanced image features corresponding to the image and the stored image features to obtain the similarity matrix of the image; The multiplication calculation submodule is used to calculate the third probability distribution of the image based on the image similarity matrix and the stored label vector corresponding to each image feature; The joint computation submodule is used to calculate the prior probability distribution of the image based on the second and third probability distributions of the image.
3. The method according to claim 2, characterized in that, The method further includes configuring the similarity calculation submodule to calculate the similarity between the enhanced image features corresponding to the image and each stored image feature in the following manner: in, Indicates the first The similarity matrix of the images, This represents the number of learnable image features stored in the cached feature subspace. Indicates the first Enhanced image features of the image. This represents the learnable first feature stored in the cache feature subspace. Image features.
4. The method according to claim 3, characterized in that, The method further includes: The multimodal model is configured to compute the first probability distribution of the image as follows: in, Indicates the first The first probability distribution of the images, The first part of the multimodal model extraction is represented by the first part. Textual features of the image The first part of the multimodal model extraction is represented by the first part. Image features of the image; The linear probe submodule is configured to calculate the second probability distribution of the image as follows: in, Indicates the first The second probability distribution of the images The weights of the linear probe submodules are represented. The first image extracted by the pre-trained visual model. Visual features of the image; The multiplication calculation submodule is configured to calculate the third probability distribution of the image as follows: in, Indicates the first The third probability distribution of the images, This indicates the first feature stored in the cache feature subspace. Label vectors corresponding to the features of each image.
5. The method according to claim 4, characterized in that, The method further includes configuring the joint computation submodule to compute the prior probability distribution of the image in the following manner: in, Indicates the first Prior probability distribution of the images This represents the first weighting coefficient. This represents the second weighting coefficient.
6. The method according to claim 5, characterized in that, The preset loss function is: in, in, Indicates the total loss. Indicates hyperparameters, Indicates the first loss. This indicates the second loss. Indicates the third loss. This indicates the number of classes contained in a small sample training set. This indicates the number of images contained in each category. Natural logarithm function, Represents the normalized exponential function, Indicates the first Category labels for the images.
7. The method according to claim 6, characterized in that, The method further includes: configuring the fusion module to calculate the target probability distribution of the image in the following manner: in, Indicates the first Target probability distribution of the image.
8. An image classification method, characterized in that, The method includes: Step T1: Obtain the image to be classified and its corresponding text prompts, wherein the text prompts include multiple categories to indicate the possible category to which the image to be classified may belong; Step T2: The image to be classified is processed using the image classification system constructed by the method described in any one of claims 1-7 to obtain the target probability distribution of the image, wherein the target probability distribution includes the predicted probability corresponding to each category, and the category with the highest predicted probability is taken as the classification result of the image to be classified.
9. A computer-readable storage medium, characterized in that, It contains a computer program that can be executed by a processor to implement the steps of the method according to any one of claims 1-8.
10. An electronic device, characterized in that, include: One or more processors, and memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method according to any one of claims 1-8 by executing the executable instructions.