A method and apparatus for image classification using a semantic-aware knowledge base

Abstract

Embodiments of the present application provide a method and device for image classification using a semantic-aware knowledge base, relating to the field of image classification. The method comprises: extracting visual features and index features of a target image through a visual feature extraction module and an index feature extraction module in an image classification model; determining the similarity between each code word in the knowledge base and the index features, and determining K similar code words and their visual features according to the similarity; fusing the visual features corresponding to the K similar code words according to the similarity between the code words and the index features, to obtain the fused prior features of the target image; and enhancing the visual features of the target image, inputting the enhanced visual features into a classification head of the image classification model, and obtaining the class prediction result of the target image. In this way, the semantic-level prior knowledge is integrated through feature fusion, so that the image classification model can obtain significant performance improvement without retraining.

Description

Technical Field

[0001] This application relates to the field of image classification technology, and in particular to a method and apparatus for image classification using a semantically aware knowledge base. Background Technology

[0002] In related technologies, accurate and efficient semantic recognition is the foundation for machine perception in image classification. However, models in these technologies often experience performance degradation when faced with complex imaging distortions and imbalanced data distributions. Pre-trained models may experience performance degradation due to various imaging distortions, and the imbalanced data distribution may make it difficult for the model to learn accurate and stable classification boundaries.

[0003] Therefore, given the shortcomings of the existing technologies, an effective method for image classification is urgently needed. Summary of the Invention

[0004] To address the aforementioned issues, this application provides a method and apparatus for image classification using a semantically aware knowledge base, aiming to efficiently and accurately classify images using a semantically aware knowledge base.

[0005] A first aspect of this application provides a method for image classification using a semantically aware knowledge base, the method comprising: The visual features of the target image are extracted by the visual feature extraction module in the image classification model, and the index features of the target image are extracted by the index feature extraction module; the visual features represent the pixel-level features of the target image; the index features represent the semantic-level features of the target image. Determine the similarity between each codeword in the knowledge base and the index features of the target image; According to the order of similarity from large to small, the top K codewords are determined as K similar codewords, and the visual features corresponding to each of the K similar codewords are extracted from the knowledge base; Based on the similarity between the K similar codewords and the index features of the target image, the visual features corresponding to the K similar codewords are fused to obtain the fused prior features of the target image; Using the fusion prior features of the target image as prior knowledge of the target image, the visual features of the target image are enhanced to obtain the enhanced visual features of the target image; The enhanced visual features of the target image are input into the classification head of the image classification model to obtain the category prediction result of the target image.

[0006] Optionally, the method further includes: Given that the total number of image categories in the sample image library is C, iterate through the C image categories to determine the total number of sample images for each image category; Based on the total number of sample images from low to high, the C image categories are sequentially determined as the c-th image category; The number of codewords for the c-th image category is determined based on the distribution of index features of multiple sample images belonging to the c-th image category. Based on the number of codewords in the c-th image category, determine multiple codewords for the c-th image category.

[0007] Optionally, the number of codewords for the c-th image category is determined based on the distribution of index features of multiple sample images belonging to the c-th image category, including: Determine the number Kc of cluster center vectors for the c-th image category; Based on the number Kc of cluster center vectors of the c-th image category, cluster the index features of multiple sample images belonging to the c-th image category to obtain Kc cluster center vectors of the c-th image category; Based on the similarity between every two cluster center vectors in the Kc cluster center vectors of the c-th image category, determine the maximum similarity among the Kc cluster center vectors of the c-th image category; If the maximum similarity among the Kc cluster center vectors of the c-th image category is less than or equal to the clustering threshold, the number of codewords for the c-th image category is determined to be Kc. If the maximum similarity among the Kc cluster center vectors of the c-th image category is greater than the clustering threshold, the number Kc of cluster center vectors of the c-th image category is reduced. Based on the number of cluster center vectors after the reduction of the c-th image category, the index features of multiple sample images belonging to the c-th image category are clustered again until the maximum similarity between multiple cluster center vectors of the c-th image category is less than or equal to the clustering threshold. The number of cluster center vectors used in the last clustering of the c-th image category is determined as the number of codewords of the c-th image category.

[0008] Optionally, based on the number of codewords in the c-th image category, multiple codewords for the c-th image category are determined, including: Using the N cluster center vectors obtained from the last clustering of the c-th image category, the N codewords of the c-th image category are determined according to the following steps: For the n-th cluster center vector among the N cluster center vectors, the fine-tuned index feature that is most similar to the n-th cluster center vector among the fine-tuned index features of multiple sample images belonging to the c-th image category is determined as the n-th codeword of the c-th image category; The method further includes: The visual feature extraction module extracts the visual features of the sample image corresponding to the nth codeword of the cth image category; Store the N codewords and corresponding visual features of the c-th image category into the knowledge base.

[0009] Optionally, determining the number Kc of cluster center vectors for the c-th image category includes: Based on the total number of codewords K in the knowledge base, the total number of allocated codewords, and the total number of remaining image categories, determine the number Kc of cluster center vectors for the c-th image category; After obtaining the number of codewords for the c-th image category, the total number of allocated codewords is updated. After determining multiple codewords for the c-th image category, the total number of remaining image categories is updated. The updated total number of allocated codewords and the updated total number of remaining image categories are used to determine the number of cluster center vectors for the (c+1)-th image category. After determining multiple codewords for each of the C image categories, if the total number of allocated codewords is less than the total number of codewords K in the knowledge base, the clustering threshold is increased. Based on the increased clustering threshold, multiple codewords for each of the C image categories are re-determined until the total number of allocated codewords is greater than or equal to the total number of codewords K in the knowledge base.

[0010] Optionally, the method further includes: The distribution of index features of multiple sample images belonging to the c-th image category is fitted by a Gaussian mixture model to obtain Nc Gaussian components. Each Gaussian component has a mean vector and a clustering weight. The clustering weight of a Gaussian component represents the proportion of the number of sample images with the mean vector of the Gaussian component as the cluster center vector relative to the total number of sample images in the c-th image category. The average similarity of the c-th image category is determined based on the similarity between the mean vectors of every two Gaussian components in the Nc Gaussian components. The normalized entropy of the c-th image category is determined based on the clustering weights of the Nc Gaussian components. The larger the normalized entropy of the c-th image category and the lower the average similarity of the c-th image category, the more balanced the distribution of the clustering weights of the Nc Gaussian components and the less similar the cluster center vectors corresponding to the Nc Gaussian components are. If the normalized entropy of the c-th image category is not greater than the normalized entropy threshold, or the average similarity is not less than the similarity threshold, it is determined that multi-class center vectors do not need to be enabled for the c-th image category. The class center vectors for supervised contrastive learning of image categories that do not require multi-class center vectors are determined. These class center vectors are then used as micro-call class center vectors for fine-tuning the index feature extraction module. The index feature extraction module is trained by supervised contrastive learning on a neural network using a sample image library and C class center vectors, with each of the C class center vectors corresponding to one of the C image categories.

[0011] Optionally, the method further includes: If the normalized entropy of the c-th image category is greater than the normalized entropy threshold and the average similarity is less than the similarity threshold, it is determined that a multi-class center vector needs to be enabled for the c-th image category. When it is necessary to enable multi-class center vectors for the c-th image category, the Nc+1 class center vectors for the c-th image category are determined according to the following steps: The class center vector for supervised contrastive learning of the c-th image category is determined; the index feature extraction module is obtained by supervising the neural network with supervised contrastive learning using a sample image library and C class center vectors, and the C class center vectors correspond one-to-one with the C image categories; The weight of the category center vector for supervised contrastive learning of the c-th image category is reduced from 1 to a first weight less than 1. And, based on the clustering weights of the Nc Gaussian components, Nc corresponding second weights are assigned to the Nc Gaussian components, and all Nc second weights are less than the first weight. The category center vector of the c-th image category with the first weight and the Nc cluster center vectors with Nc second weights are determined as the Nc+1 category center vectors of the c-th image category. Input the i-th sample image belonging to the c-th image category into the index feature extraction module to obtain the index features of the i-th sample image belonging to the c-th image category; Based on the weighted similarity between the index features of the i-th sample image belonging to the c-th image category and the Nc+1 category center vectors of the c-th image category, the target category center vector of the i-th sample image belonging to the c-th image category is determined. The target category center vector is used as the micro-call category center vector corresponding to the i-th sample image of the c-th image category that requires multi-category center vectors.

[0012] Optionally, the method further includes: Based on the micro-call category center vectors corresponding to each sample image, the projection head module of the index feature extraction module is fine-tuned using the sample image library. During fine-tuning, the micro-call category center vector corresponding to each sample image is used as the positive optimization target for that sample image, making the index features of that sample image closer to its corresponding micro-call category center vector and closer to the index features of other sample images belonging to the same image category as that sample image. Simultaneously, the index features of that sample image are made further away from the category center vectors corresponding to other image categories and the index features of sample images from other image categories, resulting in a fine-tuned index feature extraction module. The sample image library is input into the fine-tuned index feature extraction module to obtain the fine-tuned index features of multiple sample images in C image categories.

[0013] A second aspect of this application provides an apparatus for image classification using a semantically aware knowledge base, the apparatus comprising: The feature extraction module is used to extract visual features of the target image through the visual feature extraction module in the image classification model, and to extract index features of the target image through the index feature extraction module; the visual features represent the pixel-level features of the target image; the index features represent the semantic-level features of the target image. A similarity determination module is used to determine the similarity between each codeword in the knowledge base and the index features of the target image; The similar codeword determination module is used to determine the top K codewords as K similar codewords in descending order of similarity, and extract the visual features corresponding to each of the K similar codewords from the knowledge base; The prior knowledge determination module is used to fuse the visual features corresponding to the K similar codewords based on the similarity between the K similar codewords and the index features of the target image, so as to obtain the fused prior features of the target image. The visual feature enhancement module is used to enhance the visual features of the target image by using the fused prior features of the target image as prior knowledge of the target image, so as to obtain the enhanced visual features of the target image. The category prediction module is used to input the enhanced visual features of the target image into the classification head of the image classification model to obtain the category prediction result of the target image.

[0014] Optionally, the apparatus for image classification using a semantically aware knowledge base further includes: The module for determining the total number of sample images is used to traverse C image categories and determine the total number of sample images for each image category when the total number of image categories in the sample image library is C. The image category sorting module is used to sequentially determine the c-th image category from C image categories according to the order of the total number of sample images from low to high. The codeword count determination module is used to determine the codeword count of the c-th image category based on the distribution of index features of multiple sample images belonging to the c-th image category. The codeword determination module is used to determine multiple codewords for the c-th image category based on the number of codewords for the c-th image category.

[0015] Optionally, the codeword quantity determination module includes: The first cluster center vector quantity determination submodule is used to determine the quantity Kc of cluster center vectors for the c-th image category; The cluster center vector determination submodule is used to cluster the index features of multiple sample images belonging to the c-th image category according to the number Kc of cluster center vectors of the c-th image category, so as to obtain Kc cluster center vectors of the c-th image category. The cluster center vector similarity determination submodule is used to determine the maximum similarity among the Kc cluster center vectors of the c-th image category based on the similarity between every two cluster center vectors in the Kc cluster center vectors of the c-th image category. The first codeword quantity determination submodule is used to determine the codeword quantity of the c-th image category as Kc when the maximum similarity among the Kc cluster center vectors of the c-th image category is less than or equal to the clustering threshold. The second cluster center vector number determination submodule is used to reduce the number Kc of cluster center vectors of the c-th image category when the maximum similarity among the Kc cluster center vectors of the c-th image category is greater than the clustering threshold. The second codeword quantity determination submodule is used to cluster the index features of multiple sample images belonging to the c-th image category again according to the number of cluster center vectors after the reduction of the c-th image category, until the maximum similarity between the multiple cluster center vectors of the c-th image category is less than or equal to the clustering threshold, and the number of cluster center vectors used in the last clustering of the c-th image category is determined as the codeword quantity of the c-th image category.

[0016] Optionally, the codeword determination module includes: The codeword determination submodule is used to determine the N codewords of the c-th image category using the N cluster center vectors obtained from the last clustering of the c-th image category, according to the following steps: For the n-th cluster center vector among the N cluster center vectors, the fine-tuned index feature that is most similar to the n-th cluster center vector among the index features of multiple fine-tuned sample images belonging to the c-th image category is determined as the n-th codeword of the c-th image category; The module also includes: A visual feature extraction submodule is used to extract the visual features of the sample image corresponding to the nth codeword of the cth image category through the visual feature extraction module; The storage submodule is used to store the N codewords and corresponding visual features of the c-th image category into the knowledge base.

[0017] Optionally, the first cluster center vector number determination submodule includes: The first cluster center vector number determination unit is used to determine the number Kc of the cluster center vectors of the c-th image category based on the total number K of codewords in the knowledge base, the total number of allocated codewords, and the total number of remaining image categories. The second cluster center vector number determination unit is used to update the total number of allocated codewords after obtaining the number of codewords for the c-th image category, and to update the total number of remaining image categories after determining multiple codewords for the c-th image category. The updated total number of allocated codewords and the updated total number of remaining image categories are used to determine the number of cluster center vectors for the (c+1)-th image category. The clustering threshold adjustment unit is used to, after determining multiple codewords for each of the C image categories, increase the clustering threshold if the total number of allocated codewords is less than the total number of codewords K in the knowledge base, and redetermine multiple codewords for each of the C image categories based on the increased clustering threshold, until the total number of allocated codewords is greater than or equal to the total number of codewords K in the knowledge base.

[0018] Optionally, the apparatus for image classification using a semantically aware knowledge base further includes: The Gaussian component determination module is used to fit the distribution of index features of multiple sample images belonging to the c-th image category using a Gaussian mixture model to obtain Nc Gaussian components. Each Gaussian component has a mean vector and a clustering weight. The clustering weight of a Gaussian component represents the proportion of the number of sample images with the mean vector of the Gaussian component as the cluster center vector relative to the total number of sample images in the c-th image category. The average similarity determination module is used to determine the average similarity of the c-th image category based on the similarity between the mean vectors of every two Gaussian components in Nc Gaussian components. The normalized entropy determination module is used to determine the normalized entropy of the c-th image category based on the clustering weights of the Nc Gaussian components. The larger the normalized entropy of the c-th image category and the lower the average similarity of the c-th image category, the more balanced the distribution of the clustering weights of the Nc Gaussian components and the less similar the cluster center vectors corresponding to the Nc Gaussian components are. The first multi-class center vector activation determination module is used to determine that multi-class center vectors do not need to be enabled for the c-th image category when the normalized entropy of the c-th image category is not greater than the normalized entropy threshold, or the average similarity is not less than the similarity threshold. The first micro-call category center vector determination module is used to determine the category center vectors for supervised contrastive learning of image categories that do not require multi-category center vectors. The category center vectors for supervised contrastive learning of image categories that do not require multi-category center vectors are used as micro-call category center vectors for fine-tuning the index feature extraction module. The index feature extraction module is obtained by supervising the neural network with a sample image library and C category center vectors, and the C category center vectors correspond one-to-one with the C image categories.

[0019] Optionally, the apparatus for image classification using a semantically aware knowledge base further includes: The second multi-class center vector activation determination module is used to determine that multi-class center vectors need to be enabled for the c-th image category when the normalized entropy of the c-th image category is greater than the normalized entropy threshold and the average similarity is less than the similarity threshold. The category center vector determination module is used to determine Nc+1 category center vectors for the c-th image category when it is necessary to enable multi-category center vectors for the c-th image category through the following sub-modules: the first category center vector determination sub-module is used to determine the category center vectors for supervised contrastive learning of the c-th image category; the index feature extraction module is obtained by supervising the neural network with a sample image library and C category center vectors, and the C category center vectors correspond one-to-one with the C image categories; The weight allocation submodule is used to reduce the weight of the category center vector of the c-th image category used for supervised contrastive learning from 1 to a first weight less than 1, and to assign Nc second weights to the Nc Gaussian components according to the clustering weights of the Nc Gaussian components, wherein all Nc second weights are less than the first weight. The second category center vector determination submodule is used to determine the category center vector of the c-th image category with the first weight and the Nc cluster center vectors with Nc second weights as the Nc+1 category center vectors of the c-th image category. The first sample image input module is used to input the i-th sample image belonging to the c-th image category into the index feature extraction module to obtain the index features of the i-th sample image belonging to the c-th image category; The target category center vector determination module is used to determine the target category center vector of the i-th sample image belonging to the c-th image category based on the weighted similarity between the index features of the i-th sample image belonging to the c-th image category and the Nc+1 category center vectors of the c-th image category. The second micro-call category center vector determination module is used to use the target category center vector as the micro-call category center vector corresponding to the i-th sample image of the c-th image category that needs to enable multi-category center vectors.

[0020] Optionally, the apparatus for image classification using a semantically aware knowledge base further includes: The index feature extraction module fine-tuning module is used to fine-tune the projection head module of the index feature extraction module based on the micro-call category center vector corresponding to each sample image and using the sample image library. During the fine-tuning process, the micro-call category center vector corresponding to each sample image is used as the positive optimization target for that sample image, making the index features of that sample image closer to its corresponding micro-call category center vector and closer to the index features of other sample images belonging to the same image category as that sample image. Simultaneously, the index features of that sample image are made further away from the category center vectors corresponding to other image categories and the index features of sample images from other image categories, resulting in a fine-tuned index feature extraction module. The fine-tuned index feature determination module is used to input the sample image library into the fine-tuned index feature extraction module to obtain the fine-tuned index features of each of the multiple sample images in C image categories.

[0021] A third aspect of this application provides an electronic device, the electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, it implements the method for image classification using a semantically aware knowledge base as described in the first aspect of this application.

[0022] A fourth aspect of this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of image classification using a semantically aware knowledge base according to the first aspect of this application.

[0023] A fifth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the method of image classification using a semantically aware knowledge base according to the first aspect of this application.

[0024] The image classification method using a semantically aware knowledge base according to embodiments of this application first extracts visual features of the target image through a visual feature extraction module in an image classification model, and extracts index features of the target image through an index feature extraction module. The visual features represent the pixel-level features of the target image, and the index features represent the semantic-level features of the target image. Then, the similarity between each codeword in the knowledge base and the index features of the target image is determined. Next, the top K codewords with the highest similarity are selected as the K most similar codewords. Visual features corresponding to the K similar codewords are extracted from the knowledge base. Then, based on the similarity between the K similar codewords and the index features of the target image, the visual features corresponding to the K similar codewords are fused to obtain the fused prior features of the target image. Then, using the fused prior features of the target image as prior knowledge of the target image, the visual features of the target image are enhanced to obtain the enhanced visual features of the target image. Finally, the enhanced visual features of the target image are input into the classification head of the image classification model to obtain the category prediction result of the target image.

[0025] This application simultaneously extracts both visual and index features from the target image, thus processing the target image at both the pixel and semantic levels. At the semantic level, prior knowledge of the target image is obtained from codewords in the knowledge base that are similar to the index features of the target image. This prior knowledge is then used to enhance the visual features of the target image, resulting in a more accurate category prediction. By integrating semantically similar prior knowledge through feature fusion, the image classification model achieves significant performance improvements without retraining. Furthermore, the semantically structured codeword organization and adaptive prior integration method in this application overcome the inherent bottlenecks of traditional prototype and retrieval methods in related technologies, providing stronger support for research on hierarchical knowledge structures and machine perception systems. Attached Figure Description

[0026] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is a flowchart of a method for image classification using a semantically aware knowledge base, as proposed in an embodiment of this application; Figure 2 This is a schematic diagram of a framework for image classification using a semantically aware knowledge base, as proposed in an embodiment of this application. Figure 3 This is a schematic diagram of a knowledge base-enhanced semantic recognition framework proposed in one embodiment of this application; Figure 4 This is a schematic diagram of an experimental result proposed in one embodiment of this application; Figure 5 This is a structural block diagram of an apparatus for image classification using a semantically aware knowledge base, provided in an embodiment of this application. Figure 6 This is a schematic diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0027] The embodiments of this application will now be described in detail. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0028] In related technologies, image classification techniques that introduce external storage typically rely on only one codeword per image category, lacking a mechanism for systematically organizing and compressing complex external information. Furthermore, the knowledge bases in these methods cannot adapt to the heterogeneity of the distribution characteristics of different semantic categories. These limitations restrict the generalization ability of the methods in related technologies in complex real-world environments, resulting in poor integration of semantic recognition and image classification technologies.

[0029] Therefore, in order to at least partially solve one or more of the above problems and other potential problems, embodiments of this application propose a method for image classification using a semantically aware knowledge base. This method can not only integrate prior knowledge at the semantic level to improve the image classification effect, but also adjust the allocation of codewords in different image categories according to the actual situation, thereby further improving the processing effect on heterogeneous semantic distribution.

[0030] Please refer to the details. Figure 1 , Figure 1 This is a flowchart illustrating a method for image classification using a semantically aware knowledge base, as proposed in an embodiment of this application. Figure 1 As shown, the method may include steps S101 to S106: Step S101: Extract visual features of the target image through the visual feature extraction module in the image classification model, and extract index features of the target image through the index feature extraction module; the visual features represent the pixel-level features of the target image; the index features represent the semantic-level features of the target image. Step S102: Determine the similarity between each codeword in the knowledge base and the index features of the target image; Step S103: According to the order of similarity from large to small, determine the top K codewords as K similar codewords, and extract the visual features corresponding to each of the K similar codewords from the knowledge base; Step S104: Based on the similarity between the K similar codewords and the index features of the target image, fuse the visual features corresponding to the K similar codewords to obtain the fused prior features of the target image; Step S105: Using the fusion prior features of the target image as the prior knowledge of the target image, enhance the visual features of the target image to obtain the enhanced visual features of the target image; Step S106: Input the enhanced visual features of the target image into the classification head of the image classification model to obtain the category prediction result of the target image.

[0031] The technical scenario addressed in this application primarily involves image classification tasks with complex and diverse semantic structures and heterogeneity across image categories. Existing methods often produce unstable results due to samples near classification boundaries, or they apply a single global strategy across the entire dataset, ignoring category heterogeneity and failing to adapt to the specific distribution characteristics of each category. Therefore, this application constructs a visual feature extraction module and an index feature extraction module within the image classification model to extract pixel-level and semantic-level features of the target image. Furthermore, based on the similarity between codewords and index features in a pre-built knowledge base, the visual features corresponding to the K codewords most similar to the target image's index features are fused to obtain semantic-level fused prior features. These fused prior features are then used as prior knowledge to enhance the visual features of the target image, determining the enhanced visual features. Finally, image classification is performed based on the enhanced visual features of the target image to obtain the category prediction result.

[0032] The image category predicted by the method in this application is a prediction made after incorporating semantic-level fusion prior features. Therefore, it can not only classify target images based on visual feature information at the pixel level, but also further assist in understanding the specific content in the target image by combining semantically similar prior knowledge. This allows for a global understanding of the information contained in the target image. The category prediction results obtained in this way are more accurate and efficient, and can more accurately grasp image features, thus realizing an image classification scheme with high recognition accuracy and strong robustness.

[0033] For the specific structure, please refer to Figure 2 , Figure 2 This is a schematic diagram of a framework for image classification using a semantically aware knowledge base proposed in an embodiment of this application. The lower half is knowledge base-assisted semantic recognition. The validation set, i.e., the target image, is input into the model and undergoes index feature extraction (Retrieval Index Extraction) and visual feature extraction (Classification Feature Extraction) respectively. Then, prior retrieval is performed to obtain codewords and their corresponding visual features as prior knowledge. The prior knowledge and visual features are then fused (Feature Fusion) and input into the classification head for classification, thereby determining the category prediction result (RecognitionResult) of the target image.

[0034] Step S101: Extract visual features of the target image through the visual feature extraction module in the image classification model, and extract index features of the target image through the index feature extraction module; the visual features represent the pixel-level features of the target image; the index features represent the semantic-level features of the target image.

[0035] In this embodiment, the trained image classification model includes at least two modules: a visual feature extraction module and an index feature extraction module. Visual features refer to the pixel-level features of the target image, a commonly used technique in related technologies. This involves analyzing the pixel information of the target image to determine its category. In this application, visual features are extracted through the visual feature extraction module. On the other hand, this application also extracts index features of the target image through the index feature extraction module. Index features represent the semantic features of the target image, performing semantic analysis to understand the overall image content and thus assisting visual features in more accurate image classification.

[0036] Step S102: Determine the similarity between each codeword in the knowledge base and the index features of the target image.

[0037] In this embodiment, after extracting the index features of the target image, further analysis is required. Specifically, the knowledge base of the pre-trained image classification model in this application contains multiple image classification categories, and each category contains one or more codewords. These codewords are interpretable representations constructed from the training data of each image category through methods such as prototype learning and representation learning. In this application, the similarity between each codeword stored in the knowledge base and the index features of the target image is used to determine the degree of similarity between the index features of the target image and each codeword, i.e., with each image category, and this similarity is then used as a reference for image classification of the target image.

[0038] Step S103: According to the order of similarity from large to small, the top K codewords are determined as K similar codewords, and the visual features corresponding to each of the K similar codewords are extracted from the knowledge base.

[0039] In this embodiment, the similarity between the index features of the target image and each codeword varies. The greater the similarity, the more similar the index features of the target image are to the features of the codeword. Therefore, in this application, the K codewords in the knowledge base that are most similar to the index features of the target image are selected. That is, the K codewords ranked in descending order of similarity are selected as the codewords that are most similar to the index features of the target image. The visual features corresponding to these K codewords are extracted from the knowledge base.

[0040] Step S104: Based on the similarity between the K similar codewords and the index features of the target image, the visual features corresponding to the K similar codewords are fused to obtain the fused prior features of the target image.

[0041] In this embodiment, since the similarity between the K similar codewords and the index features of the target image may also differ, the visual features corresponding to the K similar codewords are fused according to their similarity. In an optional embodiment, different weights can be assigned to the K similar codewords according to their similarity, thereby weighted fusing of their respective visual features to finally determine the fused prior features of the target image. This is the analysis result of the target image obtained from the semantic level, which is used to assist in the classification of the target image.

[0042] Step S105: Using the fusion prior features of the target image as the prior knowledge of the target image, enhance the visual features of the target image to obtain the enhanced visual features of the target image.

[0043] In this embodiment of the application, after determining the fusion prior features of the target image, the fusion prior features of the target image can be used as prior knowledge. Based on the prior knowledge, the visual features of the target image are enhanced. Thus, based on the pixel-level visual features, semantically similar prior knowledge is combined to obtain the enhanced visual features of the target image. The enhanced visual features of the target image are content that incorporates semantic features, thereby enabling more accurate classification of the target image.

[0044] Step S106: Input the enhanced visual features of the target image into the classification head of the image classification model to obtain the category prediction result of the target image.

[0045] In this embodiment, the enhanced visual features of the target image are input into the classification head of the image classification model, and the classification head outputs the category prediction result of the target image. Since the semantic prior knowledge is combined, the category prediction result of the target image obtained in this application is more accurate and more robust.

[0046] In conjunction with the above embodiments, in one implementation, this application also provides a method for image classification using a semantically aware knowledge base, which specifically includes the following: First, given that the total number of image categories in the sample image library is C, we iterate through the C image categories to determine the total number of sample images for each category.

[0047] In this embodiment, since codewords represent the semantic information of each image category, it is necessary to determine the codewords for each image category during the construction of the knowledge base required for the image classification model. This ensures that the index features of the target image can be accurately analyzed to obtain prior knowledge when processing the target image classification task. In determining the codewords for each image category in the database, it is first necessary to analyze the total number C of image categories in the training sample image library, traversing all C image categories to determine the total number of sample images contained in each category.

[0048] This is because the difficulty of collecting image data of different categories in the real world varies, easily leading to long-tail datasets with imbalanced distribution of samples across different categories. In related technologies, the processing of long-tail data often weakens the model's ability to distinguish the tail categories as the long-tail distribution intensifies, i.e., the data imbalance ratio increases, thus reducing the overall performance of the model. Therefore, in this application, to more accurately handle long-tail data, the total number of samples for each image category in the sample image library is first counted during the codeword determination process.

[0049] Then, according to the order of the total number of sample images from low to high, the C image categories are sequentially determined as the c-th image category.

[0050] In this embodiment, after counting the total number of samples in each image category, all C image categories in the sample image library are sorted in order of increasing total number of sample images. The codewords are determined starting from the image category with the lowest total number of sample images. This avoids ignoring long-tail data during training, which could affect the final classification result.

[0051] Next, based on the distribution of index features of multiple sample images belonging to the c-th image category, the number of codewords for the c-th image category is determined.

[0052] In this embodiment, before determining the codeword in the c-th image category, it is necessary to first determine the number of codewords contained in that image category. Since codewords represent each image in each image category at the semantic level, related technologies often assign the same number of codewords to different image categories. However, this approach ignores the complex intra-class diversity and inter-class heterogeneity in real-world data, thereby reducing the adaptability to image categories with complex potential substructures. For example, suppose there are two subcategories with significantly different features within the same image category. If only one codeword is assigned to this image category, the selection of that codeword needs to consider the features of both subcategories simultaneously, resulting in a compromise. However, this codeword cannot fully represent the features of either subcategory. If two codewords are assigned to this image category, one codeword can be assigned to each of the two significantly different subcategories, thus enabling a more accurate matching of the index features of the target image in subsequent image classification processes.

[0053] From another perspective, suppose that in a certain image category, the features of each image sample are very similar. If multiple codewords are assigned to the same image category in the same way, then the features of the multiple codewords of the image samples will all be very similar. In this case, such a redundant allocation method will occupy more database space and computing resources, which is not worth the effort.

[0054] Therefore, in this application, before determining the codewords for each image category, the number of codewords for each image category is first determined, thereby avoiding allocating too many or too few codewords to each image category. This ensures the accuracy of image classification while also achieving model simplification and lightweighting.

[0055] Finally, based on the number of codewords in the c-th image category, multiple codewords for the c-th image category are determined.

[0056] In this embodiment, after determining the number of codewords for the c-th image category, one or more codewords corresponding to the c-th image category are determined based on the determined number of codewords. It should be noted that for each image category, at least one codeword is assigned to it to represent the semantic information of that image category.

[0057] In conjunction with the above embodiments, in one implementation, this application also provides a method for image classification using a semantically aware knowledge base. Based on the distribution of index features of multiple sample images belonging to the c-th image category, the number of codewords for the c-th image category is determined. Specifically, this includes the following: First, determine the number Kc of cluster center vectors for the c-th image category.

[0058] In this embodiment, to determine the number of codewords for the c-th image category, the following method proposed in this application can be used: First, determine the number of cluster center vectors Kc in the c-th image category. A cluster center represents the center to which image samples in each category converge, that is, the representative of the image samples in each category. Related technologies use a single category center to make samples of the same category converge towards the center. However, this method cannot produce optimal results when the image data exhibits a multimodal distribution. For example, as mentioned above, in the same image category, there are two subclasses with significantly different features; setting only one cluster center is clearly insufficient to characterize the features of that image category. Therefore, in this application, before determining the number of codewords for the c-th image category, the number of naturally formed cluster center vectors Kc in the c-th image category is first determined.

[0059] Then, according to the number Kc of cluster center vectors of the c-th image category, the index features of multiple sample images belonging to the c-th image category are clustered to obtain Kc cluster center vectors of the c-th image category.

[0060] In this embodiment of the application, after determining the number Kc of cluster center vectors of the c-th image category, multiple sample images belonging to the c-th image category are clustered according to their index features, thereby completing the clustering of sample images in the c-th image category at the semantic level to obtain the clustering result, which is the cluster center vector corresponding to the Kc cluster centers of the c-th image category.

[0061] Then, based on the similarity between every two cluster center vectors in the Kc cluster center vectors of the c-th image category, the maximum similarity among the Kc cluster center vectors of the c-th image category is determined.

[0062] In this embodiment, for the multiple cluster centers naturally formed in each image category, not every cluster center is significantly different. There may be cases where multiple cluster centers are quite similar. In such cases, these similar cluster centers cannot all be identified as codewords for that image category. For example, each cluster center is like a mountain peak, and other sample images tend to gravitate towards these peaks. These peaks may be hills or mountains, so they need to be compared and distinguished.

[0063] Therefore, specifically, in this application, based on the cluster center vectors of the Kc cluster centers of the c-th image category, the similarity between each cluster center vector is calculated pairwise, and the maximum value in the calculation results is selected as the maximum similarity between the Kc cluster center vectors of the c-th image category. This maximum similarity characterizes the degree of similarity between the two most similar cluster centers in the c-th image category.

[0064] Next, if the maximum similarity among the Kc cluster center vectors of the c-th image category is less than or equal to the clustering threshold, the number of codewords for the c-th image category is determined to be Kc.

[0065] In this embodiment, a clustering threshold is set for the comparison results. The clustering threshold represents the tolerance for similarity. If the similarity between two cluster center vectors is not higher than the clustering threshold, then the two cluster feature vectors are considered to be dissimilar and are in an independent, multimodal state. That is, when the maximum similarity between the Kc cluster center vectors of the c-th image category is less than or equal to the clustering threshold, the two most similar cluster center vectors in the c-th image category are considered to be dissimilar. Therefore, the other cluster center vectors in the c-th image category are also dissimilar, thus determining the number of codewords in the c-th image category as Kc, which is the number of these cluster center vectors.

[0066] Subsequently, if the maximum similarity among the Kc cluster center vectors of the c-th image category is greater than the clustering threshold, the number Kc of cluster center vectors of the c-th image category is reduced.

[0067] In this embodiment, when the similarity between two cluster center vectors is higher than the clustering threshold, the two cluster feature vectors are considered to be similar. That is, when the maximum similarity between the Kc cluster center vectors of the c-th image category is greater than the clustering threshold, it indicates that there are at least two relatively similar cluster centers among the Kc cluster centers. Therefore, the number of cluster centers needs to be reduced to avoid wasting storage resources.

[0068] Finally, based on the number of cluster center vectors after the reduction of the c-th image category, the index features of multiple sample images belonging to the c-th image category are clustered again until the maximum similarity between multiple cluster center vectors of the c-th image category is less than or equal to the clustering threshold. The number of cluster center vectors used in the last clustering of the c-th image category is determined as the number of codewords of the c-th image category.

[0069] In this embodiment, after reducing the number of cluster center vectors for the c-th image category, the above steps are repeated. The index features of multiple sample images in the c-th image category are clustered based on the new number of cluster centers. The similarity between each pair of new cluster center vectors is calculated, and this similarity is compared with a clustering threshold until only one cluster center remains or the maximum similarity between multiple cluster center vectors is no higher than the clustering threshold. At this point, the number of cluster center vectors used in the last clustering process is determined as the number of codewords for the c-th image category. This ensures that multimodal features in highly heterogeneous image categories are maintained while also overcoming storage resource limitations to achieve a lightweight database.

[0070] In conjunction with the above embodiments, in one implementation, this application also provides a method for image classification using a semantically aware knowledge base, which determines multiple codewords for the c-th image category based on the number of codewords in the c-th image category, specifically including the following: First, using the N cluster center vectors obtained from the last clustering of the c-th image category, the N codewords of the c-th image category are determined according to the following steps: For the n-th cluster center vector among the N cluster center vectors, the fine-tuned index feature that is most similar to the n-th cluster center vector among the fine-tuned index features of multiple sample images belonging to the c-th image category is determined as the n-th codeword of the c-th image category; In this embodiment, after determining the number of codewords in the c-th image category in the above steps, it is also necessary to determine the specific codewords in the c-th image category. This is done by using the N cluster center vectors used in the last clustering process of the c-th image category as a benchmark. These N cluster centers represent the N heterogeneous subcategories contained in the c-th image category. Then, based on the fine-tuned index features of each sample image in the c-th image category, it is compared with the fine-tuned index features of each cluster center vector. The fine-tuned index features of the sample image that is most similar to the fine-tuned index features of the n-th cluster center vector are determined as the n-th codeword of the c-th image category. In this way, all N codewords of the c-th image category are determined.

[0071] Then, the method further includes: The visual feature extraction module extracts the visual features of the sample image corresponding to the nth codeword of the cth image category; the N codewords of the cth image category and their corresponding visual features are stored in the knowledge base.

[0072] In this embodiment, since the determination of prior knowledge at the semantic level through index features is for the purpose of enhancing and fusing with visual features at the pixel level to determine the enhanced visual features, when storing codewords for each image category in the database, the data feature extraction module also needs to extract the visual features of the sample images corresponding to each codeword, so that the codewords and visual features are in one-to-one correspondence. The N codewords for each image category and their corresponding visual features are stored in the knowledge base, so that when performing the target image classification task, codewords similar to the index features of the target image can be determined, and then the prior knowledge of the target image can be determined based on its corresponding visual features.

[0073] The knowledge base construction process described in this application can be found in [reference needed]. Figure 2 The upper part shows the semantic-aware knowledge base construction process. Specifically, the images in the SupportSet are first input into the Semantic-sensitive Index Learning module and the Classification Feature Learning module to obtain the index features and classification features corresponding to each sample image. The Semantic-sensitive Index Learning module is used to learn index features with class discrimination capabilities, while the Classification Feature Learning module is used to extract visual classification features for image recognition.

[0074] Building upon this, the Category-balanced KB Construction module adaptively allocates a corresponding number of codewords to each image category based on the sample distribution characteristics of different image categories. For example, Figure 2 Image categories 1 correspond to 3 codewords and are therefore represented by 3 red squares; image categories 2 correspond to 2 codewords and are therefore represented by 2 blue squares; and so on for other categories. For each image category, the category-balanced knowledge base construction module can perform clustering modeling based on the feature distribution of samples in that category, and determine the codewords and their corresponding classification features for that category based on the clustering results.

[0075] Furthermore, during the codeword allocation process, the codeword allocation result can be adjusted by incorporating the knowledge base state (KB State) to ensure that the limited knowledge base capacity can more evenly cover different categories and their internal differences. Finally, the determined codewords and their corresponding classification features are jointly stored in the external storage (KB) knowledge base, thereby providing support for subsequent prior knowledge retrieval and semantic recognition of the target image.

[0076] In conjunction with the above embodiments, in one implementation, this application also provides a method for image classification using a semantically aware knowledge base, determining the number Kc of cluster center vectors for the c-th image category, specifically including the following: First, based on the total number of codewords K in the knowledge base, the total number of allocated codewords, and the total number of remaining image categories, the number of cluster center vectors Kc for the c-th image category is determined.

[0077] In this embodiment, when determining the number Kc of cluster center vectors for the c-th image category, the total number of codewords stored in the knowledge base needs to be considered. The number of codewords contained in a knowledge base is not infinite; it is limited by the knowledge base's capacity. Furthermore, lightweight knowledge bases are a requirement for image classification model construction during technological development. Therefore, in this application, the number of codewords that can still be allocated needs to be determined based on the total number K of codewords in the knowledge base and the total number of allocated codewords. Furthermore, the number of cluster center vectors Kc that can be allocated to the c-th image category needs to be determined based on the total number of remaining image categories for which codewords have not yet been determined. In an optional embodiment, an equal distribution method can be used, dividing the number of allocatable codewords by the total number of remaining image categories, and using the result as the initial number Kc of cluster center vectors allocated to the c-th image category.

[0078] Then, after obtaining the number of codewords for the c-th image category, the total number of allocated codewords is updated, and after determining multiple codewords for the c-th image category, the total number of remaining image categories is updated. The updated total number of allocated codewords and the updated total number of remaining image categories are used to determine the number of cluster center vectors for the (c+1)-th image category.

[0079] In this embodiment, after determining the data volume Kc of the cluster center vector of the c-th image category, the number of codewords and the specific codewords in the c-th image category can be determined according to the above steps, thereby completing the codeword determination for the c-th image category and proceeding to the codeword determination process for the (c+1)-th image category. Before this process begins, the total number of allocated codewords needs to be updated, that is, the number of codewords for the c-th image category is added to the total number of allocated codewords. Furthermore, the total number of remaining image categories is also updated, that is, 1 is subtracted from the total number of remaining image categories. Then, based on the updated total number of allocated codewords and the updated total number of remaining image categories, the number of cluster center vectors for the (c+1)-th image category is determined.

[0080] Finally, after determining the multiple codewords for each of the C image categories, if the total number of allocated codewords is less than the total number of codewords K in the knowledge base, the clustering threshold is increased. Based on the increased clustering threshold, the multiple codewords for each of the C image categories are re-determined until the total number of allocated codewords is greater than or equal to the total number of codewords K in the knowledge base.

[0081] In this embodiment of the application, after determining the codewords for each of the C image categories in the knowledge base, since the number of codewords allocated to each image category may be less than the number of cluster centers initially allocated during the codeword determination process, there may still be a surplus of codewords available for allocation after the codewords for all C image categories are determined. That is, the total number of allocated codewords is less than the total number of codewords K in the knowledge base. In this case, there may be a waste of codeword resources, so it is necessary to reallocate the codewords to make full use of the resources.

[0082] Specifically, by increasing the clustering threshold, the similarity criterion between each pair of cluster center vectors in each image category is broadened, as mentioned above. Two cluster center vectors that might have been considered similar before may now be considered dissimilar after the clustering threshold is increased. This further refines the types of cluster centers in the image category, resulting in a richer selection of codewords. In this way, the codewords in all C image categories in the knowledge base are redefined. This process of redefining codewords is repeated until the total number of allocated codewords is greater than or equal to the total number of codewords K in the knowledge base. At this point, it is considered that the codeword resources in the knowledge base have been fully utilized, and a more refined and accurate codeword allocation has been determined.

[0083] In conjunction with the above embodiments, in one implementation, this application also provides a method for image classification using a semantically aware knowledge base, which specifically includes the following: First, the distribution of index features of multiple sample images belonging to the c-th image category is fitted using a Gaussian mixture model to obtain Nc Gaussian components. Each Gaussian component has a mean vector and a clustering weight. The clustering weight of a Gaussian component represents the proportion of the number of sample images with the mean vector of that Gaussian component as the cluster center vector relative to the total number of sample images in the c-th image category.

[0084] In this embodiment, the technical scenario addressed is that the processed image category contains multiple evenly distributed and significantly heterogeneous subcategories. In such cases, if the semantic information of an image category is represented by only one codeword, semantic information may be lost, thereby reducing the accuracy of the image classification task. Conversely, if the image category does not contain evenly distributed or significantly heterogeneous subcategories, the prior knowledge of the target image can be determined using conventional semantic recognition methods.

[0085] Therefore, this application further determines the image categories in the knowledge base at the semantic level. First, this application uses a Gaussian mixture model to fit the distribution of index features of multiple sample images belonging to the c-th image category, thus obtaining Nc Gaussian components. Each Gaussian component has a mean vector and a clustering weight. The mean vector is the feature vector exhibited by the Gaussian component, while the clustering weight is the proportion of the number of sample images in the c-th image category whose mean vector is used as the cluster center vector, relative to the total number of sample images in the c-th image category. In short, it uses the Gaussian mixture model to classify the possible clustering situations of the c-th image category at the semantic level.

[0086] In an alternative embodiment, modeling can be performed using a Gaussian mixture model (GMM): ,in, The sample feature distribution for each category, This represents the number of Gaussian components obtained after modeling image category c using a Gaussian Mixture Model (GMM). Cluster weights Let represent the mean vector and covariance matrix of the i-th component.

[0087] Then, the average similarity of the c-th image category is determined based on the similarity between the mean vectors of every two Gaussian components in the Nc Gaussian components.

[0088] In this embodiment of the application, the average similarity of the c-th image category is calculated based on the similarity between the mean vectors of every two Gaussian components in each of the Nc Gaussian components. This average similarity is used to measure the degree of similarity between the clusters formed by the Nc Gaussian components, characterize the cohesion of the clusters, and determine whether the Nc Gaussian components are relatively similar or have differences.

[0089] In one alternative embodiment, the average similarity The following formula can be used for calculation: ,in, .

[0090] Next, the normalized entropy of the c-th image category is determined based on the clustering weights of the Nc Gaussian components. The larger the normalized entropy of the c-th image category and the lower the average similarity of the c-th image category, the more balanced the distribution of the clustering weights of the Nc Gaussian components and the less similar the clustering center vectors corresponding to the Nc Gaussian components are.

[0091] In this embodiment, the normalized entropy of the c-th image category is also determined based on the clustering weights of the Nc Gaussian components. The normalized entropy characterizes the degree of imbalance among the cluster distributions in the c-th image category. A larger normalized entropy value indicates a more balanced distribution among different clusters in the image category, meaning the sample images are more evenly distributed across different clusters. For example, suppose an image category contains 100 sample images and 2 Gaussian components. If the clusters represented by these two Gaussian components each contain 50 sample images, then the sample images are evenly distributed, and the normalized entropy value is larger. However, if the clusters represented by these two Gaussian components contain 10 and 90 sample images respectively, then the sample images are unevenly distributed, and the normalized entropy value is smaller.

[0092] In one alternative embodiment, the entropy value is normalized. The following formula can be used for calculation: , where the value ranges from 0 to 1.

[0093] Subsequently, if the normalized entropy of the c-th image category is not greater than the normalized entropy threshold, or the average similarity is not less than the similarity threshold, it is determined that multi-class center vectors do not need to be enabled for the c-th image category.

[0094] In this embodiment, normalized entropy reflects the imbalance of the distribution of each cluster, and average similarity reflects the cohesion within each cluster. Therefore, based on the set normalized entropy threshold and similarity threshold, the c-th image category can be divided into four cases: First, if the normalized entropy is close to 0, it indicates that the image category consists of only one cluster, or the weight of one of the clusters is close to 100%. In this case, enabling multi-class center vectors would be somewhat redundant. If the normalized entropy is small and the average similarity is low, it indicates that although the different clusters are far apart and have significant differences, the weight distribution is unbalanced, and the clusters with larger weights can represent the characteristics of this image category. If the normalized entropy is large and the average similarity is high, although the weight distribution of each cluster is relatively balanced, the clusters are close to each other and have small differences. Therefore, in the latter two cases, multi-class center vectors are not needed.

[0095] Finally, the category center vectors for supervised contrastive learning of image categories that do not require multi-category center vectors are determined. These category center vectors are then used as micro-call category center vectors for fine-tuning the index feature extraction module. The index feature extraction module is trained by supervised contrastive learning on a neural network using a sample image library and C category center vectors, with each of the C category center vectors corresponding to one of the C image categories.

[0096] In this embodiment of the application, the index feature extraction module is trained through supervised contrastive learning. Each of the C image categories has a category center vector. The goal is to make each image in the sample image library close to the category center vector of its own category, while moving away from the category center vectors of other categories. The index feature extraction module is trained through supervised contrastive learning.

[0097] After enabling multi-class centroid vectors in this application, the index feature extraction module needs to be fine-tuned. This fine-tuning process requires training not only on categories with enabled multi-class centroid vectors but also on categories without enabled centroid vectors. Therefore, in this application, after determining the centroid vectors for supervised contrastive learning for image categories that do not require multi-class centroid vectors, these centroid vectors are used as the fine-tuning centroid vectors for fine-tuning the index feature extraction module.

[0098] At the same time, the image categories that enable multi-class center vectors are also used to fine-tune the index feature extraction module. The purpose of fine-tuning the index features is to make each image in the sample image library, when it belongs to an image category that does not require multi-class center vectors, closer to the micro-call category center vector of its own category, while moving away from the category center vectors of other categories; when it belongs to an image category that requires multi-class center vectors, it closer to its corresponding micro-call cluster center vector, while moving away from the category center vectors of other categories.

[0099] In conjunction with the above embodiments, in one implementation, this application also provides a method for image classification using a semantically aware knowledge base, which specifically includes the following: First, if the normalized entropy of the c-th image category is greater than the normalized entropy threshold and the average similarity is less than the similarity threshold, it is determined that a multi-class center vector needs to be enabled for the c-th image category.

[0100] In this embodiment of the application, continuing from the above, when the normalized entropy of the c-th image category is large, greater than the normalized entropy threshold, and the average similarity is small, less than the similarity threshold, it indicates that the weight distribution among the clusters in the c-th image category is uniform, and the distance between them is large and the differences are obvious. If the semantic features of the image category are described by only one codeword, the meaning may be unclear. Therefore, at this time, it is necessary to enable the multi-category center vector proposed in this application for the c-th image category.

[0101] Then, if it is necessary to enable multi-class center vectors for the c-th image category, the Nc+1 class center vectors for the c-th image category are determined according to the following steps: In this embodiment of the application, when determining to enable multiple category center vectors, it is first necessary to determine the multiple category center vectors existing in the c-th image category. The multiple category center vectors not only include the original category center vectors of the c-th image category, but also need to include new category center vectors determined according to each Gaussian component.

[0102] First, the category center vector for supervised contrastive learning of the c-th image category is determined; the index feature extraction module is obtained by supervising the neural network with supervised contrastive learning using the sample image library and the C category center vectors, and the C category center vectors correspond one-to-one with the C image categories.

[0103] In this embodiment, when determining the multiple category center vectors for supervised contrastive learning of the c-th image category, it is first necessary to determine the original category center vectors for supervised contrastive learning of the c-th image category. These original category center vectors correspond to a total of C categories in the knowledge base, each category having an original category center vector for supervised contrastive learning. Furthermore, the initial index feature extraction module of the image classification model in this application is obtained by using the sample image library and the C category center vectors to train the neural network through supervised contrastive learning. Therefore, it can be said that these C category center vectors represent the initial semantic features of the C image categories.

[0104] Then, the weight of the category center vector for supervised contrastive learning of the c-th image category is reduced from 1 to a first weight less than 1, and Nc second weights are assigned to the Nc Gaussian components according to the clustering weights of the Nc Gaussian components, where all Nc second weights are less than the first weights.

[0105] In this embodiment, since a multi-class center vector is enabled for the c-th image category, the semantic features of the entire c-th image category can no longer be represented by the original single class center vector. Therefore, it is necessary to combine other newly added class center vectors for supervised contrastive learning. Specifically, the weights of the original class center vectors used for supervised contrastive learning are reduced to a first weight less than 1, which can be 0.5 in an optional embodiment. At the same time, the clustering weights of the Nc Gaussian components of the c-th image category are the Nc second weights corresponding to these Nc Gaussian components, so that the sum of the weights of the original class center vectors used for supervised contrastive learning and the Nc Gaussian components is finally 1.

[0106] Next, the category center vector of the c-th image category with the first weight and the Nc cluster center vectors with Nc second weights are determined as the Nc+1 category center vectors of the c-th image category.

[0107] In this embodiment, after weight allocation, the category center vector of the c-th image category can be determined. Specifically, the category center vector of the c-th image category with the first weight, and the Nc cluster center vectors with Nc second weights, are determined as a total of Nc+1 category center vectors for the c-th image category. Furthermore, based on these Nc+1 category center vectors that explicitly characterize the semantic features of each cluster in the c-th image category, the index feature extraction module can be fine-tuned to obtain an index feature extraction module that more clearly parses the semantic features of the c-th image category.

[0108] Then, the i-th sample image belonging to the c-th image category is input into the index feature extraction module to obtain the index features of the i-th sample image belonging to the c-th image category.

[0109] In this embodiment of the application, during the fine-tuning of the index feature extraction module, the i-th sample image belonging to the c-th image category is first input into the index feature extraction module for index feature extraction, thereby obtaining the index features of the i-th sample image belonging to the c-th image category. The index features are used for subsequent comparison and calculation with the category center vector to determine the target category center vector of the i-th sample image.

[0110] Finally, based on the weighted similarity between the index features of the i-th sample image belonging to the c-th image category and the Nc+1 category center vectors of the c-th image category, the target category center vector of the i-th sample image belonging to the c-th image category is determined; the target category center vector is used as the micro-call category center vector corresponding to the i-th sample image of the c-th image category that requires the use of multi-category center vectors.

[0111] In this embodiment, after determining the index features of the i-th sample image, the target category center vector of the i-th sample image belonging to the c-th image category is determined by calculating the weighted similarity between the i-th sample image and the Nc+1 category center vectors of the c-th image category. For the c-th image category, since it contains Nc+1 category center vectors and each category center vector exhibits obvious heterogeneity, the weighted similarity is calculated to determine which category center vector should be selected as the optimization direction in the training process for the i-th sample image, and this vector is used as the micro-call category center vector in the subsequent fine-tuning process. That is, the target category center vector of the i-th sample image belonging to the c-th image category is determined as the micro-call category center vector corresponding to the i-th sample image of the c-th image category.

[0112] In conjunction with the above embodiments, in one implementation, this application also provides a method for image classification using a semantically aware knowledge base, which specifically includes the following: First, based on the micro-call category center vector corresponding to each sample image, the projection head module of the index feature extraction module is fine-tuned using the sample image library. During the fine-tuning process, the micro-call category center vector corresponding to each sample image is used as the positive optimization target for that sample image, making the index features of that sample image closer to its corresponding micro-call category center vector and closer to the index features of other sample images belonging to the same image category as that sample image. Simultaneously, the index features of that sample image are made further away from the category center vectors corresponding to other image categories and the index features of sample images from other image categories, resulting in the fine-tuned index feature extraction module.

[0113] In this embodiment, after determining the micro-call category center vector corresponding to each sample image, the projection head module of the index feature extraction module can be fine-tuned. During the fine-tuning process using the sample image library, the index features of the sample images extracted by the index feature extraction module can be closer to the category center vector of their own category and farther away from the category center vectors of other image categories when they belong to an image category that does not require the use of multi-category center vectors; when they belong to an image category that requires the use of multi-category center vectors, they can be closer to their own micro-call category center vector and the index features of other sample images of the same image category, and farther away from the category center vectors of other image categories and the index features of each sample image belonging to other image categories.

[0114] By employing the above method, samples belonging to the same image category can be constrained to form a more compact distribution in the index feature space, guiding them towards the corresponding category center or intra-category cluster center; simultaneously, samples belonging to different image categories maintain a larger feature interval in the index feature space. Thus, the index features exhibit clearer intra-class clustering and inter-class separability, resulting in a trained index feature extraction module. Finally, the sample image library is input into the finely tuned index feature extraction module to obtain the finely tuned index features for multiple sample images of C image categories.

[0115] In this embodiment, each sample image in the sample image library is input into the fine-tuned index feature extraction module to obtain the fine-tuned index features of each of the multiple sample images in all C image categories.

[0116] In an optional embodiment, the Nc+1 category center vectors in this application are denoted as... ,in, This represents the original class center vector, in At this point, the weight of the original class center vector is assigned a value of 1 and then normalized to the weights of the other Nc Gaussian vectors to obtain the new weights. This corresponds to Nc+1 category centers, specifically, among which These are the weights of the original class center vectors. Weights for other Gaussian vectors At this point, the target of the sample is selected using the following formula: in, The contrast feature vector of the sample is used to determine the category center vector that is most similar to the i-th sample image by weighted cosine similarity between the sample and multiple category centers. This vector is then used as the target category center vector of the i-th sample image and serves as the optimization direction for the sample.

[0117] In an optional embodiment, the image recognition model proposed in this application can be found in [reference needed]. Figure 3 , Figure 3 This is a schematic diagram of a knowledge base-enhanced semantic recognition framework proposed in the embodiments of this application. The upper part is the training phase, and the lower part is the inference phase. During the training phase, the semantic recognition model is trained using a sample image library (TrainingSet). Then, in the semantic knowledge construction phase, sample images from the sample image library (TrainingSet) are input into the knowledge base (KBConstruction) to determine their index features, category labels, keyframes, and semantic features. Then, in the inference stage, the target image is input into the index feature extraction module to obtain the index feature. The codeword with the highest priority weight is determined based on the feature similarity, and its corresponding semantic features are fused to obtain prior knowledge. Finally, the semantic features fused with prior knowledge are input into the classification head, and the recognition result is finally output.

[0118] In an optional embodiment, the proposed image classification method using a semantically aware knowledge base was experimentally validated against other benchmark methods to evaluate image classification and recognition performance, as detailed in Tables 1 to 4. The last row of the tables shows the results of the proposed RseNet model combined with an external knowledge base, with Table 1 representing the experimental results of recognition accuracy (%) using the Cifar-100 dataset at different imbalance ratios (IR).

[0119] Table 1

[0120] Table 2 shows the experimental results of the recognition accuracy (%) using the Cifar-100 dataset at different heterogeneity levels (α).

[0121] Table 2

[0122] Table 3 shows the experimental results of recognition accuracy (%) using the ImageNet-100 dataset at different imbalance ratios (IR).

[0123] Table 3

[0124] Table 4 shows the experimental results of recognition accuracy (%) using the ImageNet-100 dataset at different heterogeneity levels (α).

[0125] Table 4

[0126] The results in Tables 1 to 4 show that, under different imbalance ratios and heterogeneity levels, the image classification method using a semantically aware knowledge base proposed in this application achieves an average accuracy improvement of 2.85% on the Cifar-100 dataset and 3.48% on the ImageNet-100 dataset compared to the baseline method (ResNet) without a database. These experimental results demonstrate that the image classification method using a semantically aware knowledge base proposed in this application can effectively correct image features, thereby significantly improving the robustness of the underlying model.

[0127] In an optional embodiment, the proposed image classification method using a semantically aware knowledge base was experimentally validated under different knowledge base sizes and imbalance ratios, as detailed in Tables 5 and 6. In these tables, the first row represents the baseline model, and the second row represents the proposed method. As shown in Table 5, the proposed method achieves an average retrieval accuracy of 60.226%, exceeding the baseline model's 59.954%. This indicates that the fine-tuning of the multi-class centers proposed in this application results in a more reasonable distribution of the method in the feature space and enables the generation of more representative codewords. Furthermore, at a smaller knowledge base size, such as the 200-level model shown in Table 5, the accuracy is improved by approximately 0.66%, demonstrating the significant advantage of the proposed method: effectively covering the feature space and capturing the core data distribution with fewer codewords. Furthermore, although both methods show a downward trend as the knowledge base size increases, the method proposed in this application still outperforms the extreme model, for example, improving performance by 0.53% at a scale of 1000 and by 0.13% at a scale of 2000. This demonstrates that by introducing multiple category centers, this application can better capture intra-class diversity, thereby ensuring more accurate matching of the nearest codeword when processing the target image.

[0128] Table 5

[0129] Table 6 shows that in scenarios with imbalanced data distribution, although the performance of both approaches decreases as the imbalance ratio increases and the long-tail distribution becomes more pronounced, the method proposed in this application consistently outperforms the baseline model. This demonstrates the robustness of this application in scenarios with long-tailed data distribution. In extreme cases, such as when IR=100, it effectively mitigates performance degradation and exhibits superior generalization ability.

[0130] Table 6

[0131] In an optional embodiment, this application compares the proposed image classification method using a semantically aware knowledge base with a direct retrieval method through experimental verification. See details in the attached document. Figure 4 , Figure 4 This is a schematic diagram of an experimental result proposed in an embodiment of this application, wherein, Figure 4 The leftmost image (a) shows the experimental results on the Cifar-100 dataset. Figure 4The right side (b) of the ImageNet-100 dataset shows the experimental results. The results of the red solid-line square nodes represent the experimental results of the proposed method, while the results of the blue dashed-line circular nodes represent the experimental results of the direct retrieval method. As can be seen from the figure, when the imbalance ratio increases, i.e., the long-tail distribution intensifies, the accuracy of both methods decreases. However, the accuracy of the proposed method consistently remains above that of the direct retrieval method. Furthermore, compared to… Figure 4 Experimental results on different datasets (left and right sides) show that the proposed method exhibits a more significant performance improvement on the Cifar-100 dataset. This is because the feature distribution of the Cifar-100 dataset is more complex, making direct retrieval methods more susceptible to interference or bias. However, in this case, the proposed method effectively combines visual features with index features and leverages prior knowledge to enhance visual features. Therefore, when the discriminative power of the retrieved features is less than ideal (as in the Cifar-100 dataset), the complementary advantages of the proposed method for image classification using a semantically aware knowledge base are maximized, resulting in a more significant performance improvement.

[0132] Based on the same design concept, one embodiment of this application provides an apparatus for image classification using a semantically aware knowledge base. (Reference) Figure 5 , Figure 5 This is a structural block diagram of an apparatus for image classification using a semantically aware knowledge base, provided in an embodiment of this application. Figure 5 As shown, the device includes: The feature extraction module is used to extract visual features of the target image through the visual feature extraction module in the image classification model, and to extract index features of the target image through the index feature extraction module; the visual features represent the pixel-level features of the target image; the index features represent the semantic-level features of the target image. A similarity determination module is used to determine the similarity between each codeword in the knowledge base and the index features of the target image; The similar codeword determination module is used to determine the top K codewords as K similar codewords in descending order of similarity, and extract the visual features corresponding to each of the K similar codewords from the knowledge base; The prior knowledge determination module is used to fuse the visual features corresponding to the K similar codewords based on the similarity between the K similar codewords and the index features of the target image, so as to obtain the fused prior features of the target image. The visual feature enhancement module is used to enhance the visual features of the target image by using the fused prior features of the target image as prior knowledge of the target image, so as to obtain the enhanced visual features of the target image. The category prediction module is used to input the enhanced visual features of the target image into the classification head of the image classification model to obtain the category prediction result of the target image.

[0133] Optionally, the apparatus for image classification using a semantically aware knowledge base further includes: The module for determining the total number of sample images is used to traverse C image categories and determine the total number of sample images for each image category when the total number of image categories in the sample image library is C. The image category sorting module is used to sequentially determine the c-th image category from C image categories according to the order of the total number of sample images from low to high. The codeword count determination module is used to determine the codeword count of the c-th image category based on the distribution of index features of multiple sample images belonging to the c-th image category. The codeword determination module is used to determine multiple codewords for the c-th image category based on the number of codewords for the c-th image category.

[0134] Optionally, the codeword quantity determination module includes: The first cluster center vector quantity determination submodule is used to determine the quantity Kc of cluster center vectors for the c-th image category; The cluster center vector determination submodule is used to cluster the index features of multiple sample images belonging to the c-th image category according to the number Kc of cluster center vectors of the c-th image category, so as to obtain Kc cluster center vectors of the c-th image category. The cluster center vector similarity determination submodule is used to determine the maximum similarity among the Kc cluster center vectors of the c-th image category based on the similarity between every two cluster center vectors in the Kc cluster center vectors of the c-th image category. The first codeword quantity determination submodule is used to determine the codeword quantity of the c-th image category as Kc when the maximum similarity among the Kc cluster center vectors of the c-th image category is less than or equal to the clustering threshold. The second cluster center vector number determination submodule is used to reduce the number Kc of cluster center vectors of the c-th image category when the maximum similarity among the Kc cluster center vectors of the c-th image category is greater than the clustering threshold. The second codeword quantity determination submodule is used to cluster the index features of multiple sample images belonging to the c-th image category again according to the number of cluster center vectors after the reduction of the c-th image category, until the maximum similarity between the multiple cluster center vectors of the c-th image category is less than or equal to the clustering threshold, and the number of cluster center vectors used in the last clustering of the c-th image category is determined as the codeword quantity of the c-th image category.

[0135] Optionally, the codeword determination module includes: The codeword determination submodule is used to determine the N codewords of the c-th image category using the N cluster center vectors obtained from the last clustering of the c-th image category, according to the following steps: For the n-th cluster center vector among the N cluster center vectors, the fine-tuned index feature that is most similar to the n-th cluster center vector among the fine-tuned index features of multiple sample images belonging to the c-th image category is determined as the n-th codeword of the c-th image category; The module also includes: A visual feature extraction submodule is used to extract the visual features of the sample image corresponding to the nth codeword of the cth image category through the visual feature extraction module; The storage submodule is used to store the N codewords and corresponding visual features of the c-th image category into the knowledge base.

[0136] Optionally, the first cluster center vector number determination submodule includes: The first cluster center vector number determination unit is used to determine the number Kc of the cluster center vectors of the c-th image category based on the total number K of codewords in the knowledge base, the total number of allocated codewords, and the total number of remaining image categories. The second cluster center vector number determination unit is used to update the total number of allocated codewords after obtaining the number of codewords for the c-th image category, and to update the total number of remaining image categories after determining multiple codewords for the c-th image category. The updated total number of allocated codewords and the updated total number of remaining image categories are used to determine the number of cluster center vectors for the (c+1)-th image category. The clustering threshold adjustment unit is used to, after determining multiple codewords for each of the C image categories, increase the clustering threshold if the total number of allocated codewords is less than the total number of codewords K in the knowledge base, and redetermine multiple codewords for each of the C image categories based on the increased clustering threshold, until the total number of allocated codewords is greater than or equal to the total number of codewords K in the knowledge base.

[0137] Optionally, the apparatus for image classification using a semantically aware knowledge base further includes: The Gaussian component determination module is used to fit the distribution of index features of multiple sample images belonging to the c-th image category using a Gaussian mixture model to obtain Nc Gaussian components. Each Gaussian component has a mean vector and a clustering weight. The clustering weight of a Gaussian component represents the proportion of the number of sample images with the mean vector of the Gaussian component as the cluster center vector relative to the total number of sample images in the c-th image category. The average similarity determination module is used to determine the average similarity of the c-th image category based on the similarity between the mean vectors of every two Gaussian components in Nc Gaussian components. The normalized entropy determination module is used to determine the normalized entropy of the c-th image category based on the clustering weights of the Nc Gaussian components. The larger the normalized entropy of the c-th image category and the lower the average similarity of the c-th image category, the more balanced the distribution of the clustering weights of the Nc Gaussian components and the less similar the cluster center vectors corresponding to the Nc Gaussian components are. The first multi-class center vector activation determination module is used to determine that multi-class center vectors do not need to be enabled for the c-th image category when the normalized entropy of the c-th image category is not greater than the normalized entropy threshold, or the average similarity is not less than the similarity threshold. The first micro-call category center vector determination module is used to determine the category center vectors for supervised contrastive learning of image categories that do not require multi-category center vectors. The category center vectors for supervised contrastive learning of image categories that do not require multi-category center vectors are used as micro-call category center vectors for fine-tuning the index feature extraction module. The index feature extraction module is obtained by supervising the neural network with a sample image library and C category center vectors, and the C category center vectors correspond one-to-one with the C image categories.

[0138] Optionally, the apparatus for image classification using a semantically aware knowledge base further includes: The second multi-class center vector activation determination module is used to determine that multi-class center vectors need to be enabled for the c-th image category when the normalized entropy of the c-th image category is greater than the normalized entropy threshold and the average similarity is less than the similarity threshold. The category center vector determination module is used to determine Nc+1 category center vectors for the c-th image category when it is necessary to enable multi-category center vectors for the c-th image category, through the following sub-modules: The first category center vector determination submodule is used to determine the category center vector for supervised contrastive learning of the c-th image category; the index feature extraction module is obtained by supervising the neural network with supervised contrastive learning using a sample image library and C category center vectors, and the C category center vectors correspond one-to-one with the C image categories; The weight allocation submodule is used to reduce the weight of the category center vector of the c-th image category used for supervised contrastive learning from 1 to a first weight less than 1, and to assign Nc second weights to the Nc Gaussian components according to the clustering weights of the Nc Gaussian components, wherein all Nc second weights are less than the first weight. The second category center vector determination submodule is used to determine the category center vector of the c-th image category with the first weight and the Nc cluster center vectors with Nc second weights as the Nc+1 category center vectors of the c-th image category. The first sample image input module is used to input the i-th sample image belonging to the c-th image category into the index feature extraction module to obtain the index features of the i-th sample image belonging to the c-th image category; The target category center vector determination module is used to determine the target category center vector of the i-th sample image belonging to the c-th image category based on the weighted similarity between the index features of the i-th sample image belonging to the c-th image category and the Nc+1 category center vectors of the c-th image category. The second micro-call category center vector determination module is used to use the target category center vector as the micro-call category center vector corresponding to the i-th sample image of the c-th image category that needs to enable multi-category center vectors.

[0139] Optionally, the apparatus for image classification using a semantically aware knowledge base further includes: The index feature extraction module fine-tuning module is used to fine-tune the projection head module of the index feature extraction module based on the micro-call category center vector corresponding to each sample image and using the sample image library. During the fine-tuning process, the micro-call category center vector corresponding to each sample image is used as the positive optimization target for that sample image, making the index features of that sample image closer to its corresponding micro-call category center vector and closer to the index features of other sample images belonging to the same image category as that sample image. Simultaneously, the index features of that sample image are made further away from the category center vectors corresponding to other image categories and the index features of sample images from other image categories, resulting in a fine-tuned index feature extraction module. The fine-tuned index feature determination module is used to input the sample image library into the fine-tuned index feature extraction module to obtain the fine-tuned index features of each of the multiple sample images in C image categories.

[0140] Based on the same design concept, another embodiment of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method for image classification using a semantically aware knowledge base as described in any of the above embodiments of this application.

[0141] Based on the same design concept, another embodiment of this application provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps in the method for image classification using a semantically aware knowledge base as described in any of the above embodiments of this application.

[0142] Based on the same design concept, another embodiment of this application provides an electronic device, such as... Figure 6 As shown. Figure 6 This is a schematic diagram of an electronic device according to an embodiment of this application. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When executed by the processor, the program implements the steps of the image classification method using a semantically aware knowledge base described in any of the above embodiments of this application.

[0143] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0144] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0145] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0146] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0147] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0148] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0149] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0150] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0151] The above provides a detailed description of the method and apparatus for image classification using a semantically aware knowledge base provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for image classification using a semantically aware knowledge base, characterized in that, The method includes: The visual features of the target image are extracted by the visual feature extraction module in the image classification model, and the index features of the target image are extracted by the index feature extraction module; the visual features represent the pixel-level features of the target image; the index features represent the semantic-level features of the target image. Determine the similarity between each codeword in the knowledge base and the index features of the target image; According to the order of similarity from large to small, the top K codewords are determined as K similar codewords, and the visual features corresponding to each of the K similar codewords are extracted from the knowledge base; Based on the similarity between the K similar codewords and the index features of the target image, the visual features corresponding to the K similar codewords are fused to obtain the fused prior features of the target image; Using the fusion prior features of the target image as prior knowledge of the target image, the visual features of the target image are enhanced to obtain the enhanced visual features of the target image; The enhanced visual features of the target image are input into the classification head of the image classification model to obtain the category prediction result of the target image.

2. The method for image classification using a semantically aware knowledge base according to claim 1, characterized in that, The method further includes: Given that the total number of image categories in the sample image library is C, iterate through the C image categories to determine the total number of sample images for each image category; Based on the total number of sample images from low to high, the C image categories are sequentially determined as the c-th image category; The number of codewords for the c-th image category is determined based on the distribution of index features of multiple sample images belonging to the c-th image category. Based on the number of codewords in the c-th image category, determine multiple codewords for the c-th image category.

3. The method for image classification using a semantically aware knowledge base according to claim 2, characterized in that, The number of codewords for the c-th image category is determined based on the distribution of index features of multiple sample images belonging to the c-th image category, including: Determine the number Kc of cluster center vectors for the c-th image category; Based on the number Kc of cluster center vectors of the c-th image category, cluster the index features of multiple sample images belonging to the c-th image category to obtain Kc cluster center vectors of the c-th image category; Based on the similarity between every two cluster center vectors in the Kc cluster center vectors of the c-th image category, determine the maximum similarity among the Kc cluster center vectors of the c-th image category; If the maximum similarity among the Kc cluster center vectors of the c-th image category is less than or equal to the clustering threshold, the number of codewords for the c-th image category is determined to be Kc. If the maximum similarity among the Kc cluster center vectors of the c-th image category is greater than the clustering threshold, the number Kc of cluster center vectors of the c-th image category is reduced. Based on the number of cluster center vectors after the reduction of the c-th image category, the index features of multiple sample images belonging to the c-th image category are clustered again until the maximum similarity between multiple cluster center vectors of the c-th image category is less than or equal to the clustering threshold. The number of cluster center vectors used in the last clustering of the c-th image category is determined as the number of codewords of the c-th image category.

4. The method for image classification using a semantically aware knowledge base according to claim 2, characterized in that, Based on the number of codewords in the c-th image category, determine multiple codewords for the c-th image category, including: Using the N cluster center vectors obtained from the last clustering of the c-th image category, the N codewords of the c-th image category are determined according to the following steps: For the n-th cluster center vector among the N cluster center vectors, the fine-tuned index feature that is most similar to the n-th cluster center vector among the fine-tuned index features of multiple sample images belonging to the c-th image category is determined as the n-th codeword of the c-th image category; The method further includes: The visual feature extraction module extracts the visual features of the sample image corresponding to the nth codeword of the cth image category; Store the N codewords and corresponding visual features of the c-th image category into the knowledge base.

5. The method for image classification using a semantically aware knowledge base according to claim 3, characterized in that, Determining the number Kc of cluster center vectors for the c-th image category includes: Based on the total number of codewords K in the knowledge base, the total number of allocated codewords, and the total number of remaining image categories, determine the number Kc of cluster center vectors for the c-th image category; After obtaining the number of codewords for the c-th image category, the total number of allocated codewords is updated. After determining multiple codewords for the c-th image category, the total number of remaining image categories is updated. The updated total number of allocated codewords and the updated total number of remaining image categories are used to determine the number of cluster center vectors for the (c+1)-th image category. After determining multiple codewords for each of the C image categories, if the total number of allocated codewords is less than the total number of codewords K in the knowledge base, the clustering threshold is increased. Based on the increased clustering threshold, multiple codewords for each of the C image categories are re-determined until the total number of allocated codewords is greater than or equal to the total number of codewords K in the knowledge base.

6. The method for image classification using a semantically aware knowledge base according to claim 4, characterized in that, The method further includes: The distribution of index features of multiple sample images belonging to the c-th image category is fitted by a Gaussian mixture model to obtain Nc Gaussian components. Each Gaussian component has a mean vector and a clustering weight. The clustering weight of a Gaussian component represents the proportion of the number of sample images with the mean vector of the Gaussian component as the cluster center vector relative to the total number of sample images in the c-th image category. The average similarity of the c-th image category is determined based on the similarity between the mean vectors of every two Gaussian components in the Nc Gaussian components. The normalized entropy of the c-th image category is determined based on the clustering weights of the Nc Gaussian components. The larger the normalized entropy of the c-th image category and the lower the average similarity of the c-th image category, the more balanced the distribution of the clustering weights of the Nc Gaussian components and the less similar the cluster center vectors corresponding to the Nc Gaussian components are. If the normalized entropy of the c-th image category is not greater than the normalized entropy threshold, or the average similarity is not less than the similarity threshold, it is determined that multi-class center vectors do not need to be enabled for the c-th image category. The class center vectors for supervised contrastive learning of image categories that do not require multi-class center vectors are determined. These class center vectors are then used as micro-call class center vectors for fine-tuning the index feature extraction module. The index feature extraction module is trained by supervised contrastive learning on a neural network using a sample image library and C class center vectors, with each of the C class center vectors corresponding to one of the C image categories.

7. The method for image classification using a semantically aware knowledge base according to claim 6, characterized in that, The method further includes: If the normalized entropy of the c-th image category is greater than the normalized entropy threshold and the average similarity is less than the similarity threshold, it is determined that a multi-class center vector needs to be enabled for the c-th image category. When it is necessary to enable multi-class center vectors for the c-th image category, the Nc+1 class center vectors for the c-th image category are determined according to the following steps: The class center vector for supervised contrastive learning of the c-th image category is determined; the index feature extraction module is obtained by supervising the neural network with supervised contrastive learning using a sample image library and C class center vectors, and the C class center vectors correspond one-to-one with the C image categories; The weight of the category center vector for supervised contrastive learning of the c-th image category is reduced from 1 to a first weight less than 1. And, based on the clustering weights of the Nc Gaussian components, Nc corresponding second weights are assigned to the Nc Gaussian components, and all Nc second weights are less than the first weight. The category center vector of the c-th image category with the first weight and the Nc cluster center vectors with Nc second weights are determined as the Nc+1 category center vectors of the c-th image category. Input the i-th sample image belonging to the c-th image category into the index feature extraction module to obtain the index features of the i-th sample image belonging to the c-th image category; Based on the weighted similarity between the index features of the i-th sample image belonging to the c-th image category and the Nc+1 category center vectors of the c-th image category, the target category center vector of the i-th sample image belonging to the c-th image category is determined. The target category center vector is used as the micro-call category center vector corresponding to the i-th sample image of the c-th image category that requires multi-category center vectors.

8. The method for image classification using a semantically aware knowledge base according to claim 6, characterized in that, The method further includes: Based on the micro-call category center vectors corresponding to each sample image, the projection head module of the index feature extraction module is fine-tuned using the sample image library. During fine-tuning, the micro-call category center vector corresponding to each sample image is used as the positive optimization target for that sample image, making the index features of that sample image closer to its corresponding micro-call category center vector and closer to the index features of other sample images belonging to the same image category as that sample image. Simultaneously, the index features of that sample image are made further away from the category center vectors corresponding to other image categories and the index features of sample images from other image categories, resulting in a fine-tuned index feature extraction module. The sample image library is input into the fine-tuned index feature extraction module to obtain the fine-tuned index features of multiple sample images in C image categories.

9. An apparatus for image classification using a semantically aware knowledge base, characterized in that, The device includes: The feature extraction module is used to extract visual features of the target image through the visual feature extraction module in the image classification model, and to extract index features of the target image through the index feature extraction module; the visual features represent the pixel-level features of the target image; the index features represent the semantic-level features of the target image. A similarity determination module is used to determine the similarity between each codeword in the knowledge base and the index features of the target image; The similar codeword determination module is used to determine the top K codewords as K similar codewords in descending order of similarity, and extract the visual features corresponding to each of the K similar codewords from the knowledge base; The prior knowledge determination module is used to fuse the visual features corresponding to the K similar codewords based on the similarity between the K similar codewords and the index features of the target image, so as to obtain the fused prior features of the target image. The visual feature enhancement module is used to enhance the visual features of the target image by using the fused prior features of the target image as prior knowledge of the target image, so as to obtain the enhanced visual features of the target image. The category prediction module is used to input the enhanced visual features of the target image into the classification head of the image classification model to obtain the category prediction result of the target image.

Images