Image classification method and device, equipment and medium

Abstract

The invention discloses an image classification method and device, equipment and a medium, and relates to the technical field of computer vision. The method comprises the following steps: acquiring a to-be-classified image; inputting the to-be-classified image into a first-level sub-model of the large model, and performing coarse classification on the to-be-classified image according to the first-level category set through the first-level sub-model to obtain a coarse classification result of the to-be-classified image; inputting a coarse classification result of the to-be-classified image into a second-level sub-model of the large model, and performing fine classification on the to-be-classified image according to a second-level category set corresponding to the coarse classification result through the second-level sub-model to obtain a fine classification result and a confidence score of the to-be-classified image; determining an uncertain quantization function value according to the confidence score; and if the uncertain quantization function value is greater than a preset threshold value, inputting a fine classification result of the to-be-classified image into a three-level sub-model of the large model, and performing multi-dimensional verification through the three-level sub-model to obtain a final classification result of the to-be-classified image. Therefore, the method can improve the accuracy of image classification.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to an image classification method, apparatus, device and medium. Background Technology

[0002] Image classification is one of the core tasks in the field of computer vision. Currently, image classification is typically performed using the following method: First, a large-scale sample image set with classification labels is constructed. Then, a machine learning model is trained using the sample image set to obtain an image classification model. In practical applications, the image to be classified is simply input into the image classification model to complete the classification.

[0003] However, in practical applications, due to the complexity of the image acquisition background, the randomness of the angle, and the variability of the lighting environment, there is often noise interference in the images to be classified, resulting in a low accuracy of image classification. Summary of the Invention

[0004] To address the aforementioned issues, this application provides an image classification method, apparatus, device, and medium that can improve the accuracy of image classification.

[0005] The embodiments of this application disclose the following technical solutions:

[0006] In a first aspect, this application discloses an image classification method, the method comprising:

[0007] Obtain the image to be classified;

[0008] The image to be classified is input into a first-level sub-model of the large model. The first-level sub-model performs a coarse classification of the image to be classified based on the first-level category set to obtain the coarse classification result of the image to be classified.

[0009] The coarse classification result of the image to be classified is input into the secondary sub-model of the large model. The secondary sub-model performs fine classification on the image to be classified according to the secondary category set corresponding to the coarse classification result, and obtains the fine classification result and confidence score of the image to be classified.

[0010] The value of the uncertainty quantization function is determined based on the confidence score.

[0011] If the value of the uncertain quantization function is greater than a preset threshold, the fine classification result of the image to be classified is input into the third-level sub-model of the large model, and multi-dimensional verification is performed through the third-level sub-model to obtain the final classification result of the image to be classified.

[0012] Optionally, the step of performing coarse classification on the image to be classified based on the primary category set includes:

[0013] Determine the scene category corresponding to the image to be classified;

[0014] Based on the scenario category, determine the primary target prompt words from the primary prompt word library;

[0015] Using the primary target cue words, the image to be classified is coarsely classified according to the primary category set.

[0016] Optionally, the step of performing fine classification on the image to be classified based on the set of secondary categories corresponding to the coarse classification result includes:

[0017] Based on the scenario category, determine the secondary target prompt words in the secondary prompt word library;

[0018] Using the secondary target cue words, the image to be classified is further classified according to the secondary category set corresponding to the coarse classification result.

[0019] Optionally, the step of performing fine classification on the image to be classified based on the set of secondary categories corresponding to the coarse classification result includes:

[0020] Determine the standard image set corresponding to the secondary category set corresponding to the coarse classification result; the standard image set includes multiple standard images, and each standard image corresponds to a fine classification result;

[0021] The image to be classified is further subdivided by determining the similarity between the multiple standard images and the image to be classified.

[0022] Optionally, the formula for determining the value of the uncertain quantization function is as follows:

[0023] U(I) = α × H(p) + β × D KL (p||p) coarse ) + γ × Sim feat (I, I) proto );

[0024] Where U(I) is the value of the uncertainty quantization function, α is the first coefficient, H(p) is the probability entropy, which is related to the confidence score, β is the second coefficient, and D... KL Let p be the KL divergence function, and p be the probability distribution of the fine-grained classification. coarse The probability distribution for coarse classification, γ is the third coefficient, Sim feat Let I be the similarity function, and let I be the image feature of the image to be classified. proto The image features of the standard image corresponding to the fine classification results.

[0025] Optionally, the multi-dimensional verification through the three-level sub-model includes:

[0026] The contradictory information between the image to be classified and the standard image corresponding to the fine classification result is determined through the three-level sub-model;

[0027] Based on the contradictory information, the fine classification results of the image to be classified are verified in multiple dimensions, including biological features, environmental context, functional purpose and material texture.

[0028] Optionally, the method further includes:

[0029] If the value of the uncertain quantization function is less than or equal to a preset threshold, then the fine classification result of the image to be classified is determined as the final classification result.

[0030] Secondly, this application discloses an image classification device, the device comprising: an image acquisition module, a primary classification module, a secondary classification module, a numerical determination module, and a tertiary classification module;

[0031] The image acquisition module is used to acquire the image to be classified;

[0032] The first-level classification module is used to input the image to be classified into the first-level sub-model of the large model, and through the first-level sub-model, perform coarse classification on the image to be classified according to the first-level category set to obtain the coarse classification result of the image to be classified.

[0033] The secondary classification module is used to input the coarse classification result of the image to be classified into the secondary sub-model of the large model. Through the secondary sub-model, the image to be classified is further classified according to the secondary category set corresponding to the coarse classification result, so as to obtain the fine classification result and confidence score of the image to be classified.

[0034] The numerical determination module is used to determine the value of the uncertain quantization function based on the confidence score;

[0035] The three-level classification module is used to input the fine classification result of the image to be classified into the three-level sub-model of the large model if the value of the uncertain quantization function is greater than a preset threshold, and to perform multi-dimensional verification through the three-level sub-model to obtain the final classification result of the image to be classified.

[0036] Optionally, the primary classification module is specifically used for: determining the scene category corresponding to the image to be classified; determining primary target prompt words in the primary prompt word library according to the scene category; and using the primary target prompt words to perform coarse classification of the image to be classified according to the primary category set.

[0037] Optionally, the secondary classification module is specifically used to: determine secondary target prompt words in the secondary prompt word library according to the scene category; and use the secondary target prompt words to perform fine classification on the image to be classified according to the secondary category set corresponding to the coarse classification result.

[0038] Optionally, the secondary classification module is specifically used to: determine the standard image set corresponding to the secondary category set corresponding to the coarse classification result; the standard image set includes multiple standard images, and each standard image corresponds to a fine classification result; and perform fine classification on the image to be classified by determining the similarity between the multiple standard images and the image to be classified.

[0039] Optionally, the formula for determining the value of the uncertain quantization function is as follows:

[0040] U(I) = α × H(p) + β × D KL (p||p) coarse ) + γ × Sim feat (I, I) proto );

[0041] Where U(I) is the value of the uncertainty quantization function, α is the first coefficient, H(p) is the probability entropy, which is related to the confidence score, β is the second coefficient, and D... KL Let p be the KL divergence function, and p be the probability distribution of the fine-grained classification. coarse The probability distribution for coarse classification, γ is the third coefficient, Sim feat Let I be the similarity function, and let I be the image feature of the image to be classified. proto The image features of the standard image corresponding to the fine classification results.

[0042] Optionally, the three-level classification module is specifically used to: determine the contradictory information between the image to be classified and the standard image corresponding to the fine classification result through the three-level sub-model; and perform multi-dimensional verification of the fine classification result of the image to be classified based on the contradictory information, wherein the dimensions include biological features, environmental context, functional use and material texture.

[0043] Optionally, the device further includes: a fourth classification module;

[0044] The fourth classification module is used to determine the fine classification result of the image to be classified as the final classification result if the value of the uncertain quantization function is less than or equal to a preset threshold.

[0045] Thirdly, this application discloses an image classification device, the device comprising: a memory and a processor;

[0046] The memory is used to store programs;

[0047] The processor is configured to execute the program to implement the various steps of the image classification method as described in the first aspect.

[0048] Fourthly, this application discloses a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the various steps of the image classification method as described in the first aspect.

[0049] Compared with the prior art, this application has the following advantages:

[0050] This application provides an image classification method, apparatus, device, and medium. The method includes: acquiring an image to be classified; inputting the image to be classified into a first-level sub-model of a large model, and performing coarse classification on the image to be classified according to the first-level category set through the first-level sub-model to obtain a coarse classification result of the image to be classified; inputting the coarse classification result of the image to be classified into a second-level sub-model of the large model, and performing fine classification on the image to be classified according to the second-level category set corresponding to the coarse classification result through the second-level sub-model to obtain a fine classification result and a confidence score of the image to be classified; determining an uncertain quantization function value based on the confidence score; if the uncertain quantization function value is greater than a preset threshold, then inputting the fine classification result of the image to be classified into a third-level sub-model of the large model, and performing multi-dimensional verification through the third-level sub-model to obtain the final classification result of the image to be classified. Therefore, this application first uses a first-level sub-model to perform coarse classification of the image to be classified, categorizing it into a broad category range (i.e., coarse classification result, such as "animals"). Then, a second-level sub-model performs fine classification within this broad category range (i.e., fine classification result, such as distinguishing between "cats" and "dogs" within "animals"). This reduces the difficulty of each layer of image classification and improves the overall accuracy of image classification. Furthermore, the second-level sub-model not only outputs the fine classification result but also a confidence score. If the uncertainty quantization function value determined by the confidence score exceeds a preset threshold, a third-level sub-model is used for multi-dimensional verification, further improving the overall accuracy of image classification. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 A flowchart illustrating an image classification method provided in this application embodiment; Figure 2A flowchart illustrating another image classification method provided in this application embodiment; Figure 3 A schematic diagram of an image classification device provided in an embodiment of this application; Figure 4 This is a schematic diagram of a computer-readable medium provided in an embodiment of this application. Detailed Implementation

[0053] As described earlier, image classification is currently typically performed using the following method: First, a large-scale sample image set with classification labels is constructed. Then, a machine learning model is trained using this sample image set to obtain an image classification model. In practical applications, the image to be classified is simply input into the image classification model to complete the classification. However, in real-world applications, the complexity of the image acquisition background, the randomness of the angle, and the variability of the lighting environment often lead to noise interference in the images to be classified, resulting in low accuracy in image classification.

[0054] Through research, the inventors have proposed an image classification method, apparatus, device, and medium. This application first uses a first-level sub-model to perform coarse classification of the image to be classified, categorizing it into a broad category range (i.e., coarse classification result, such as "animals"). Then, a second-level sub-model performs fine classification within this broad category range (i.e., fine classification result, such as distinguishing between "cats" and "dogs" within "animals"). This reduces the difficulty of each layer of image classification and improves the overall accuracy of image classification. Furthermore, the second-level sub-model not only outputs the fine classification result but also a confidence score. If the uncertainty quantization function value determined by the confidence score exceeds a preset threshold, a third-level sub-model is used for multi-dimensional verification, further improving the overall accuracy of image classification.

[0055] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0056] See Figure 1 The figure is a flowchart of an image classification method provided in an embodiment of this application. The method includes:

[0057] S101: Obtain the image to be classified.

[0058] The images to be classified can be real-time captured images, locally stored images, frame images from video streams, etc.

[0059] S102: Input the image to be classified into the first-level sub-model of the large model. Through the first-level sub-model, the image to be classified is coarsely classified according to the first-level category set to obtain the coarse classification result of the image to be classified.

[0060] Specifically, the image to be classified and the first-level prompt words (structured natural language instructions used to guide the large model to perform coarse classification) are input into the first-level sub-model of the large model (such as GPT-4V, LLaVA, Qwen-VL, etc.). The first-level sub-model performs coarse classification on the image to be classified according to the first-level category set C1 to obtain the coarse classification result c1 of the image to be classified.

[0061] For example, the first-level category set C1 may include: natural scenery, urban buildings, evidence materials, animals, plants, people, vehicles, etc.

[0062] For example, the first-level prompt word can be as follows: "[Visual semantic anchoring] Please determine which category in the {first-level category set C1} the {image to be classified} belongs to, and determine the coarse classification result of the image to be classified: {c1|c1∈C1}".

[0063] S103: Input the coarse classification result of the image to be classified into the second-level sub-model of the large model. Through the second-level sub-model, the image to be classified is further classified according to the second-level category set corresponding to the coarse classification result, so as to obtain the fine classification result and confidence score of the image to be classified.

[0064] It should be noted that while completing the coarse classification task, the first-level sub-model has already extracted rich and robust visual attributes. Therefore, it is no longer necessary to input the image to be classified into the second-level sub-model of the large model. Instead, the second-level sub-model can directly call the visual attributes.

[0065] Specifically, the coarse classification result c1 of the image to be classified and the secondary cue words (structured natural language instructions used to guide the large model in fine classification) are input into the secondary sub-model of the large model. The secondary sub-model then performs fine classification on the image to be classified based on the secondary category set C2 corresponding to the coarse classification result c1, obtaining the fine classification result c2 and a confidence score. The confidence score is a value between 0 and 1, representing the degree of confidence the secondary sub-model has in the fine classification result c2. For example, 0.95 indicates that the secondary sub-model is very confident that the fine classification result c2 of the image to be classified is "car," while 0.6 indicates that the secondary sub-model is less certain that the fine classification result c2 of the image to be classified is "car."

[0066] It is understandable that the design of the secondary category set C2 is strongly related to the coarse classification result c1, and is a subordinate term of the coarse classification result c1. For example, the secondary category set C2 corresponding to the coarse classification result c1 "vehicles" can include: cars, off-road vehicles, trucks, buses, etc.; the secondary category set C2 corresponding to the coarse classification result c1 "natural scenery" can include: mountains, deserts, rivers, grasslands, etc.

[0067] For example, the secondary prompt words can be as follows: "[Visual semantic anchoring] Please determine which category in the {secondary category set C2} corresponding to the {coarse classification result c1} belongs to the {image to be classified}, and determine the fine classification result: {c2|c2∈C2} and confidence score."

[0068] It should be noted that the above embodiments are examples for model application. However, during model training, i.e., in the process of training the secondary sub-model using the sample image set (including multiple sample images), it is necessary to minimize the loss function of the following formula (1):

[0069] (1)

[0070] Among them, L align To compare the alignment loss function values, f v p is the feature vector of the sample image. y p is the feature vector of the secondary prompt word corresponding to the true fine classification result of the sample image. c τ is the set of feature vectors of all secondary prompt words in the secondary category set corresponding to the true coarse classification result of the sample image, and τ is the temperature coefficient.

[0071] S104: Determine the value of the uncertain quantization function based on the confidence score.

[0072] In one specific implementation, the formula for determining the value of the uncertain quantization function is shown in formula (2) below:

[0073] U(I) = α × H(p) + β × D KL (p||p) coarse ) + γ × Sim feat (I, I) proto (2)

[0074] Where U(I) is the value of the uncertainty quantization function, α is the first coefficient, H(p) is the probability entropy (i.e., the entropy of the probability distribution p of the sub-classification, which is related to the confidence score), β is the second coefficient, and D... KL Let p be the KL divergence function, and p be the probability distribution of the fine-grained classification. coarse The probability distribution for coarse classification, γ is the third coefficient, Sim feat Let I be the difference function, and let I be the image feature of the image to be classified.proto Image features of the standard image corresponding to the fine classification results.

[0075] It is understandable that the higher H(p) is, the lower D is. KL The higher, Sim feat The higher the value, the higher the uncertainty. Therefore, the uncertainty quantification function value combines the confidence level within the second-level sub-model, the consistency of the classification results, and the degree of conformity with the standard image, forming a very comprehensive and robust uncertainty assessment framework.

[0076] S105: If the value of the uncertain quantization function is greater than the preset threshold, the fine classification result of the image to be classified is input into the third-level sub-model of the large model. The third-level sub-model is used for multi-dimensional verification to obtain the final classification result of the image to be classified.

[0077] Specifically, if the value of the uncertain quantization function is greater than the preset threshold, it indicates that the reliability of the fine classification result given by the second-level sub-model is questionable. In this case, the fine classification result c2 of the image to be classified and the third-level prompt words (structured natural language instructions used to guide the large model to perform multi-dimensional verification) are input into the third-level sub-model of the large model. Through the third-level sub-model, multi-dimensional verification is performed to obtain the final classification result of the image to be classified.

[0078] For example, a level 3 prompt could look like this: "Initial judgment: {Detailed classification result c2}. [Multi-dimensional verification] Please demonstrate from four dimensions: 1) Biological characteristics 2) Environmental context 3) Functional purpose 4) Material texture. [Reflection and correction] Based on the above, what is the final classification result? If still uncertain, please output "Rejection". It is understandable that outputting "Rejection" is crucial in high-risk applications, avoiding the potentially serious consequences of forced classification.

[0079] It is understood that biological characteristics refer to morphology, color, part structure, growth pattern, etc.; environmental context refers to the location, time, surrounding objects, lighting conditions, etc.; functional use refers to the object's function, how it is used, and usage scenarios, etc.; and material texture refers to whether the surface is smooth, rough, metallic, woody, fur, etc. This application does not limit the specific dimensions.

[0080] For example, if c1 = "animal", c2 = "wolf", and U(I) is greater than the preset threshold of 0.7, then the fine classification result "wolf" of the image to be classified is input into the third-level sub-model of the large model.

[0081] First, the three-level sub-model undergoes the following multi-dimensional validation:

[0082] "1) Biological characteristics: The animal has canine characteristics, and its tail appears to be too bushy and drooping, which is different from the typical wolf tail and is more like that of a large dog.

[0083] 2) Environmental Context: The image background is a dense forest. However, this area is close to human settlements, and the probability of wolves appearing in this location is relatively low.

[0084] 3) Functional Use (This dimension is not applicable to animals and may be skipped in the model): No items with obvious functional uses.

[0085] 4) Material texture: The texture of the hair is difficult to judge precisely, but it generally conforms to the characteristics of medium-length hair.

[0086] Subsequently, the three-level sub-model underwent the following reflection and revision:

[0087] "Based on the above analysis, there are doubts about the biological characteristics (tail), and the environmental context also reduces the possibility that it is a wolf."

[0088] Ultimately, the output of the three-level sub-model is: Husky.

[0089] In summary, this application provides an image classification method. First, a first-level sub-model performs coarse classification of the image to be classified, categorizing it into a broad category (i.e., coarse classification result, such as "animals"). Then, a second-level sub-model performs fine classification within this broad category (i.e., fine classification result, such as distinguishing between "cats" and "dogs" within "animals"). This reduces the difficulty of each layer of image classification and improves the overall accuracy of image classification. Furthermore, the second-level sub-model not only outputs the fine classification result but also a confidence score. If the uncertainty quantization function value determined by the confidence score exceeds a preset threshold, a third-level sub-model performs multi-dimensional verification, further improving the overall accuracy of image classification.

[0090] See Figure 2 This figure is a flowchart of another image classification method provided in an embodiment of this application. The method includes:

[0091] S201: Obtain the image to be classified.

[0092] It is understandable that step S201 is similar to step S101, so it will not be described again here.

[0093] S202: Input the image to be classified into the first-level sub-model of the large model. Through the first-level sub-model, perform the following: First, determine the scene category corresponding to the image to be classified; then, determine the first-level target prompt words in the first-level prompt word library according to the scene category; finally, use the first-level target prompt words to perform coarse classification of the image to be classified according to the first-level category set to obtain the coarse classification result of the image to be classified.

[0094] The image to be classified is input into a first-level sub-model of the larger model. The first-level sub-model then performs the following:

[0095] First, determine the scene category corresponding to the image to be classified. The scene category refers to the shooting conditions or quality status of the image. For example, scene categories include too bright lighting, too dark lighting, uneven lighting, extremely high shooting angle, extremely low shooting angle, motion blur, blurred perspective, target too small, incomplete target, cluttered background, and occlusion.

[0096] Subsequently, based on the scene category, the highest-performing primary target prompt word (e.g., the one with the highest classification accuracy reported by users) is determined from the primary prompt word library using the Upper Confidence Bound (UCB) method. For example, if the scene category is "Occlusion Exists," the primary target prompt word determined from the primary prompt word library could be as follows: "[Visual Semantic Anchoring] The image to be classified is {object}. Please determine which category {object} belongs to in {Primary Category Set C1}, and determine the coarse classification result of the image to be classified: {c1|c1∈C1}. Note that the image to be classified has an occlusion issue."

[0097] Finally, using the primary target cue words, the image to be classified is coarsely classified according to the primary category set to obtain the coarse classification result of the image to be classified.

[0098] It's important to note that training a first-level sub-model capable of handling various scene categories requires a sample image set covering all scene categories. However, while some scene categories exist in reality, deliberately collecting and labeling a large number of sample images for these categories requires significant manpower and resources. Therefore, when the sample image set lacks sufficient sample images for a certain scene category (e.g., the number of sample images for a certain scene category is less than a threshold), a style perturbation generator G(⋅) can be used to automatically generate such sample images, thus obtaining a sample image set. This eliminates the need for expending enormous manpower and resources to collect and label a large number of sample images for this scene category, thereby reducing model training costs and improving image classification accuracy.

[0099] S203: Input the coarse classification result of the image to be classified into the secondary sub-model of the large model. Through the secondary sub-model, perform the following: First, determine the secondary target prompt words in the secondary prompt word library according to the scene category; then, use the secondary target prompt words to perform fine classification of the image to be classified according to the secondary category set corresponding to the coarse classification result, and obtain the fine classification result and confidence score of the image to be classified.

[0100] The coarse classification result of the image to be classified is input into the secondary sub-model of the large model. Through the secondary sub-model, the following is executed:

[0101] First, based on the scene category determined in step S102, the UCB method is used to determine the secondary target prompt word with the best historical performance (e.g., the highest classification accuracy reported by users) from the secondary prompt word library. For example, if the scene category is: occlusion exists, the secondary target prompt word determined from the secondary prompt word library can be as follows: "[Visual semantic anchoring] Please determine which category {object} belongs to in the {secondary category set C2} corresponding to {coarse classification result c1}, and determine the fine classification result of the image to be classified: {c2|c2∈C2} and confidence score. Note that the image to be classified has an occlusion problem."

[0102] Subsequently, using secondary target cue words, the image to be classified is further classified according to the secondary category set corresponding to the coarse classification result, so as to obtain the fine classification result and confidence score of the image to be classified.

[0103] Furthermore, the fine classification of the image to be classified based on the set of secondary categories corresponding to the coarse classification result can be performed as follows: First, determine the set of standard images corresponding to the set of secondary categories corresponding to the coarse classification result; the set of standard images includes multiple standard images, and each standard image corresponds to a fine classification result. For example, if the coarse classification result c1 is "vehicle", then the set of standard images corresponding to the set of secondary categories corresponding to the coarse classification result c1 can include: a typical sedan standard image (labeled "sedan"), a typical SUV standard image (labeled "SUV"), a typical truck standard image (labeled "truck"), etc. Subsequently, by determining the similarity (e.g., cosine similarity, Euclidean distance, etc.) between the feature vectors of multiple standard images and the feature vector of the image to be classified, the secondary category corresponding to the standard image with the highest similarity to the image to be classified is selected as the fine classification result c2.

[0104] S204: Determine the value of the uncertain quantization function based on the confidence score.

[0105] It is understandable that step S204 is similar to step S104, so it will not be described again here.

[0106] S205: Determine whether the value of the uncertain quantization function is greater than a preset threshold. If yes, proceed to step S206; otherwise, proceed to step S207.

[0107] S206: Input the fine classification result of the image to be classified into the three-level sub-model of the large model, and perform the following through the three-level sub-model: First, determine the contradictory information between the image to be classified and the standard image corresponding to the fine classification result; then, based on the contradictory information, perform multi-dimensional verification of the fine classification result of the image to be classified, including biological features, environmental context, functional use and material texture, to obtain the final classification result of the image to be classified.

[0108] If the quantization function value is uncertain and exceeds a preset threshold, the fine classification result of the image to be classified is input into the third-level sub-model of the larger model (e.g., the Thinking model with a self-reflection mechanism). Through the third-level sub-model, the following is executed:

[0109] First, identify the contradiction information (contradiction_list) between the image to be classified and the standard image corresponding to the fine classification result. Contradictory information refers to evidence identified in the image to be classified that contradicts the fine classification result c2. For example: the object in the image to be classified appears to be metallic while the fine classification result c2 identifies it as plastic; the background environment in the image to be classified does not match the common habitat of the fine classification result c2, etc.

[0110] Subsequently, based on the contradictory information, the fine classification results of the image to be classified were validated in multiple dimensions to obtain the final classification result of the image. The dimensions include biological characteristics, environmental context, functional purpose, and material texture.

[0111] S207: Determine the fine classification result of the image to be classified as the final classification result.

[0112] In summary, this application provides an image classification method. First, a first-level sub-model performs coarse classification of the image to be classified, categorizing it into a broad category (i.e., coarse classification result, such as "animals"). Then, a second-level sub-model performs fine classification within this broad category (i.e., fine classification result, such as distinguishing between "cats" and "dogs" within "animals"). This reduces the difficulty of each layer of image classification and improves the overall accuracy of image classification. Furthermore, the second-level sub-model not only outputs the fine classification result but also a confidence score. If the uncertainty quantization function value determined by the confidence score exceeds a preset threshold, a third-level sub-model performs multi-dimensional verification, further improving the overall accuracy of image classification.

[0113] See Figure 3 The figure is a schematic diagram of an image classification device provided in an embodiment of this application. The image classification device 300 includes: an image acquisition module 301, a primary classification module 302, a secondary classification module 303, a numerical determination module 304, and a tertiary classification module 305.

[0114] Image acquisition module 301 is used to acquire images to be classified;

[0115] The first-level classification module 302 is used to input the image to be classified into the first-level sub-model of the large model. Through the first-level sub-model, the image to be classified is coarsely classified according to the first-level category set to obtain the coarse classification result of the image to be classified.

[0116] The secondary classification module 303 is used to input the coarse classification result of the image to be classified into the secondary sub-model of the large model. Through the secondary sub-model, the image to be classified is further classified according to the secondary category set corresponding to the coarse classification result, so as to obtain the fine classification result and confidence score of the image to be classified.

[0117] The numerical determination module 304 is used to determine the value of the uncertain quantization function based on the confidence score;

[0118] The three-level classification module 305 is used to input the fine classification result of the image to be classified into the three-level sub-model of the large model if the uncertain quantization function value is greater than the preset threshold. The three-level sub-model is then used for multi-dimensional verification to obtain the final classification result of the image to be classified.

[0119] In one specific implementation, the primary classification module 302 is specifically used to: determine the scene category corresponding to the image to be classified; determine the primary target prompt word in the primary prompt word library according to the scene category; and use the primary target prompt word to perform coarse classification of the image to be classified according to the primary category set.

[0120] In one specific implementation, the secondary classification module 303 is specifically used to: determine secondary target prompt words in the secondary prompt word library according to the scene category; and use the secondary target prompt words to perform fine classification on the image to be classified according to the secondary category set corresponding to the coarse classification result.

[0121] In one specific implementation, the secondary classification module 303 is specifically used to: determine the standard image set corresponding to the secondary category set corresponding to the coarse classification result; the standard image set includes multiple standard images, and each standard image corresponds to a fine classification result; and perform fine classification on the image to be classified by determining the similarity between the multiple standard images and the image to be classified.

[0122] In one specific implementation, the formula for determining the value of the uncertain quantization function is as follows:

[0123] U(I) = α × H(p) + β × D KL (p||p) coarse ) + γ × Sim feat (I, I) proto );

[0124] Where U(I) is the value of the uncertainty quantization function, α is the first coefficient, H(p) is the probability entropy, which is related to the confidence score, β is the second coefficient, and D... KL Let p be the KL divergence function, and p be the probability distribution of the fine-grained classification. coarse The probability distribution for coarse classification, γ is the third coefficient, Sim feat Let I be the similarity function, and let I be the image feature of the image to be classified. protoImage features of the standard image corresponding to the fine classification results.

[0125] In one specific implementation, the three-level classification module 305 is specifically used to: determine the contradictory information between the image to be classified and the standard image corresponding to the fine classification result through the three-level sub-model; and perform multi-dimensional verification of the fine classification result of the image to be classified based on the contradictory information, including biological characteristics, environmental context, functional purpose and material texture.

[0126] In one specific implementation, the image classification device 300 further includes: a fourth classification module;

[0127] The fourth classification module is used to determine the fine classification result of the image to be classified as the final classification result if the uncertain quantization function value is less than or equal to a preset threshold.

[0128] In summary, this application provides an image classification device. First, a first-level sub-model performs coarse classification of the image to be classified, categorizing it into a broad category range (i.e., coarse classification result, such as "animals"). Then, a second-level sub-model performs fine classification within this broad category range (i.e., fine classification result, such as distinguishing between "cats" and "dogs" within "animals"). This reduces the difficulty of each layer of image classification and improves the overall accuracy of image classification. Furthermore, the second-level sub-model not only outputs the fine classification result but also a confidence score. If the uncertainty quantization function value determined by the confidence score exceeds a preset threshold, a third-level sub-model performs multi-dimensional verification, further improving the overall accuracy of image classification.

[0129] This application also provides corresponding image classification devices and computer-readable media for implementing the image classification method provided in this application.

[0130] The image classification device includes a memory and a processor. The memory is used to store instructions or code, and the processor is used to execute the instructions or code to enable the device to perform an image classification method according to any embodiment of this application.

[0131] See Figure 3 This figure is a schematic diagram of a computer-readable medium provided in an embodiment of this application. The computer-readable medium 300 stores a computer program 311, which, when executed by a processor, implements the above-described... Figure 1 The steps of an image classification method.

[0132] It should be noted that, in the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0133] It should be noted that the machine-readable medium described above in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0134] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0135] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

[0136] While several specific implementation details are included in the foregoing discussion, these should not be construed as limiting the scope of this application. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0137] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. An image classification method characterized by, The method comprises: acquiring an image to be classified; inputting the image to be classified into a first sub-model of a large model, performing coarse classification on the image to be classified according to a first category set through the first sub-model, and obtaining a coarse classification result of the image to be classified; inputting the coarse classification result of the image to be classified into a second sub-model of the large model, performing fine classification on the image to be classified according to a second category set corresponding to the coarse classification result through the second sub-model, and obtaining a fine classification result and a confidence score of the image to be classified; determining an uncertainty quantification function value according to the confidence score; if the uncertainty quantification function value is greater than a preset threshold, inputting the fine classification result of the image to be classified into a third sub-model of the large model, performing multi-dimensional verification through the third sub-model, and obtaining a final classification result of the image to be classified.

2. The method of claim 1, wherein, The coarse classification of the image to be classified according to the first category set comprises: determining a scene category corresponding to the image to be classified; determining a first target prompt word in a first prompt word library according to the scene category; performing coarse classification on the image to be classified according to the first category set by using the first target prompt word.

3. The method of claim 2, wherein, The fine classification of the image to be classified according to the second category set corresponding to the coarse classification result comprises: determining a second target prompt word in a second prompt word library according to the scene category; performing fine classification on the image to be classified according to the second category set corresponding to the coarse classification result by using the second target prompt word.

4. The method of claim 3, wherein, The fine classification of the image to be classified according to the second category set corresponding to the coarse classification result comprises: determining a standard image set corresponding to the second category set corresponding to the coarse classification result; the standard image set comprises a plurality of standard images, and each standard image corresponds to a fine classification result; performing fine classification on the image to be classified by respectively determining the similarity of the plurality of standard images and the image to be classified.

5. The method of claim 4, wherein, The determination formula of the uncertainty quantification function value is as follows: U(I) = a x H(p) + b x D + g x Sim(I, I) KL (p || p coarse ) + g x Sim(I, I) feat (I, I proto ); wherein U(I) is the uncertain quantization function value, a is a first coefficient, H(p) is a probability entropy, the probability entropy is related to the confidence score, β is a second coefficient, D KL is a KL divergence function, p is a probability distribution of fine classification, p coarse is a probability distribution of coarse classification, γ is a third coefficient, Sim feat is a similarity function, I is an image feature of the image to be classified, I proto is an image feature of a standard image corresponding to the fine classification result.

6. The method of claim 4, wherein, The multi-dimensional verification through the third sub-model comprises: determining, through the third sub-model, contradictory information between the image to be classified and a standard image corresponding to the fine classification result; performing multi-dimensional verification on the fine classification result of the image to be classified according to the contradictory information, wherein the dimensions include biological characteristics, environmental context, functional use and material texture.

7. The method according to any one of claims 1 to 6, characterized in that, The method further comprises: if the uncertainty quantification function value is less than or equal to the preset threshold, determining the fine classification result of the image to be classified as the final classification result.

8. An image classification apparatus characterized by comprising: The device comprises an image acquisition module, a first classification module, a second classification module, a numerical value determination module and a third classification module. The image acquisition module is configured to acquire an image to be classified. The first classification module is configured to input the image to be classified into a first sub-model of a large model, perform coarse classification on the image to be classified according to a first category set through the first sub-model, and obtain a coarse classification result of the image to be classified. The secondary classification module is configured to input the coarse classification result of the image to be classified into a secondary sub-model of the large model, perform fine classification on the image to be classified according to a secondary category set corresponding to the coarse classification result through the secondary sub-model, and obtain a fine classification result and a confidence score of the image to be classified. The numerical determination module is configured to determine an uncertainty quantification function value according to the confidence score. The tertiary classification module is configured to input the fine classification result of the image to be classified into a tertiary sub-model of the large model if the uncertainty quantification function value is greater than a preset threshold, perform multi-dimensional verification through the tertiary sub-model, and obtain a final classification result of the image to be classified.

9. An image classification device, characterized by, The device comprises a memory and a processor. The memory is configured to store a program. The processor is configured to execute the program to implement each step of the image classification method according to any one of claims 1 to 7.

10. A computer readable medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, implements each step of the image classification method according to any one of claims 1 to 7.

Images