A method and device for improving image classification by using multimodal models

Abstract

The present application provides a method and device for improving image classification by using a multimodal model. The method includes: step S1, obtaining multiple frames of images in an image to be classified or a video to be classified; step S2, outputting multiple possible classification results and the probability of each classification result through a pre-trained unimodal classification model; step S3, inputting the image, classification result and the probability of each classification result into a given multimodal model, and driving the multimodal model to conduct multiple rounds of self-questioning and self-answering on the image content to obtain multiple question-and-answer results; step S4, driving the multimodal model to select the best matching classification from the classification results based on the question-and-answer results. The present application improves the accuracy and interpretability of image classification.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and in particular, relates to a method and device for improving image classification by using a multimodal model. Background Art

[0002] Image classification and video classification are two basic tasks in computer vision. Their purpose is to assign images or videos to predefined categories based on their content. Image classification and video classification are widely used, such as smart security, video content analysis, etc. The performance of image classification and video classification is usually measured by Top1 and Top5 indicators, which respectively indicate whether the first or top five categories predicted by the model contain the true category. The Top1 indicator reflects the accuracy of the model, while the Top5 indicator reflects the robustness of the model. With the development of deep learning, the models of image classification and video classification have been continuously evolving, from traditional convolutional neural networks (CNN) to visual transformers (ViT) based on attention mechanisms. The Top5 indicator has reached a very high level, close to 100%. However, the Top1 indicator still has a lot of room for improvement, especially in some fine-grained classification tasks, such as distinguishing different breeds of dogs or different kinds of flowers, or distinguishing videos of different scenes or different types of actions. This is because the differences between some categories are very small, and relying solely on the visual features of the image or video may not be enough to make accurate judgments.

[0003] In traditional image classification and video classification tasks, the focus is mainly on the category with the highest score (Top1), while ignoring other potentially correct categories in the Top5. This practice may cause the model to ignore some categories with high potential, resulting in false detection. On the other hand, if the objects in an image or video are very blurry or occluded, or if objects of one category are very similar to objects of other categories, the model may choose a wrong category without considering other potentially correct categories. These situations will affect the accuracy and reliability of the model, leading to user dissatisfaction and distrust.

[0004] In the prior art, multimodality can also be used for image classification or video classification. However, although the multimodal large model can process image and text information, its pre-training data may not contain all categories, or the data of certain categories is very small, resulting in its weak recognition ability for these categories. In addition, the internal structure and working principle of the multimodal large model are very complex. If it is used directly to output the real category, the hallucination phenomenon of the multimodal large model is relatively serious. Summary of the invention

[0005] In order to solve the above technical problems, the present application provides a method and device for improving image classification by using a multimodal model to assist, aiming to improve the Top1 accuracy by combining the advantages of traditional classification models and multimodal large models, thereby improving the accuracy and reliability of image classification and video classification and reducing the occurrence of false detections.

[0006] The first aspect of the present application is a method for improving image classification by using a multimodal model, which mainly includes:

[0007] Step S1, obtaining an image to be classified or multiple frames of images in a video to be classified;

[0008] Step S2: Output multiple possible classification results and the probability of each classification result through a pre-trained unimodal classification model;

[0009] Step S3: input the image, classification results and the probability of each classification result into a given multimodal model, and drive the multimodal model to perform multiple rounds of self-questioning and self-answering on the image content to obtain multiple question-answering results;

[0010] Step S4: driving the multimodal model to select the best matching classification from the classification results based on the question and answer results.

[0011] Preferably, in step S3, driving the multimodal model to perform multiple rounds of self-questioning and self-answering on the image content includes:

[0012] The multimodal model is driven to form questions and answers related to the classification results regarding the local structures, attributes, and relationships between the local structures in the image.

[0013] Preferably, in step S3, the rounds of self-questioning and self-answering include at least 5 times.

[0014] Preferably, step S4 further includes:

[0015] Step S5: weight the best matching classification given by the multimodal model to the classification result probability given by the unimodal classification model through the following formula to form the index value of each classification, and take the classification result with the highest index value as the final classification of the image:

[0016] P i =(1+T i ) i ;

[0017]

[0018] Among them, P i is the index value of the i-th category, p i is the probability of the i-th classification given by the unimodal classification model, and a is between 0.3 and 0.8.

[0019] The second aspect of the present application provides a device for improving image classification by using a multimodal model, which mainly includes:

[0020] An image acquisition module, used to acquire an image to be classified or multiple frames of images in a video to be classified;

[0021] A unimodal classification module is used to output multiple possible classification results and the probability of each classification result through a pre-trained unimodal classification model;

[0022] A multimodal question-answering module, used to input the image, classification results and the probability of each classification result into a given multimodal model, and drive the multimodal model to perform multiple rounds of self-questioning and self-answering on the image content to obtain multiple question-answering results;

[0023] The multimodal classification module is used to drive the multimodal model to select the best matching classification from the classification results based on the question and answer results.

[0024] Preferably, in the multimodal question-answering module, driving the multimodal model to perform multiple rounds of self-questioning and self-answering on the image content includes:

[0025] The multimodal model is driven to form questions and answers related to the classification results regarding the local structures, attributes, and relationships between the local structures in the image.

[0026] Preferably, the self-questioning and self-answering rounds include at least 5 times.

[0027] Preferably, the device further comprises:

[0028] The unimodal and multimodal classification weighting module is used to weight the best matching classification given by the multimodal model to the classification result probability given by the unimodal classification model through the following formula to form the index value of each classification, and take the classification result with the highest index value as the final classification of the image:

[0029] P i =(1+T i ) i ;

[0030]

[0031] Among them, P i is the index value of the i-th category, p i is the probability of the i-th classification given by the unimodal classification model, and a is between 0.3 and 0.8.

[0032] In a third aspect of the present application, a computer device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for improving image classification by using a multimodal model as described in any one of the above items.

[0033] In a fourth aspect of the present application, a readable storage medium stores a computer program, which, when executed by a processor, is used to implement the method of using a multimodal model to assist in improving image classification as described above.

[0034] This application improves the accuracy and interpretability of image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flowchart of a preferred embodiment of the method of the present application for using a multimodal model to assist in improving image classification.

[0036] Figure 2 This is an example of an input image according to a preferred embodiment of the present application.

[0037] Figure 3 It is a schematic diagram of the structure of a computer device of a terminal or server suitable for implementing the implementation mode of the present application. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical scheme and advantages of the implementation of this application clearer, the technical scheme in the implementation of this application will be described in more detail in combination with the drawings in the implementation of this application. In the drawings, the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The described implementation is a part of the implementation of this application, not all of the implementations. The implementation described below with reference to the drawings is exemplary and is intended to be used to explain this application, and cannot be understood as a limitation on this application. Based on the implementation in this application, all other implementations obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. The implementation of this application is described in detail below in combination with the drawings.

[0039] According to the first aspect of the present application, a method for improving image classification by using a multimodal model mainly includes:

[0040] Step S1, obtaining an image to be classified or multiple frames of images in a video to be classified;

[0041] Step S2: Output multiple possible classification results and the probability of each classification result through a pre-trained unimodal classification model;

[0042] Step S3: input the image, classification results and the probability of each classification result into a given multimodal model, and drive the multimodal model to perform multiple rounds of self-questioning and self-answering on the image content to obtain multiple question-answering results;

[0043] Step S4: driving the multimodal model to select the best matching classification from the classification results based on the question and answer results.

[0044] The present application is applicable to image classification, video classification or audio classification. For image classification, in step S1, an input image can be obtained from a camera or offline. For video classification, in step S1, several frames can be discretized and extracted to be classified into image classification. For audio classification, before step S1, the audio signal can be converted into a visual spectrogram or other form of image representation.

[0045] In some optional implementations, after step S1, the image or video is further preprocessed to adapt to different input formats and size requirements.

[0046] In step S2, the present application uses a unimodal image classification or video classification model trained under a given task to classify images or videos, such as ResNet or ViT, to predict the input image or video and obtain the top categories and corresponding probabilities. Among them, the unimodal image classification or video classification model can select different models and parameters, and perform training and testing according to different tasks and data sets to achieve the best classification effect. Some commonly used evaluation indicators, such as accuracy, recall rate, F1 value, etc., can be used for the classification model to measure the performance and quality of the model.

[0047] It is understandable that the classification algorithm in deep learning refers to the automatic identification and classification of data into different categories through a neural network model. The basic principle of this algorithm is to learn and extract features from the input data through a multi-layer neural network, and then map the input data to predefined categories through the classifier of the last layer. During the training process, the algorithm continuously adjusts the neural network parameters to minimize the gap between the prediction and the actual label, thereby improving the classification accuracy. The classification algorithm of deep learning has achieved great success in the fields of image recognition, natural language processing, etc., and has demonstrated powerful performance in many practical applications. This application uses classification models such as ResNet or ViT, which are conventional unimodal classification models, and can also be called traditional models or traditional classification models.

[0048] In this step, the existing unimodal classification model usually outputs the top five classifications and classification probabilities, such as Figure 2 For the image shown, the top 5 categories and probabilities w predicted by the unimodal image classification model can be:

[0049] Lawn mower 0.42;

[0050] Tractor 0.26;

[0051] harvester 0.11;

[0052] Plow 0.09;

[0053] Tractor 0.06.

[0054] After configuring the parameters of the unimodal model, the number of output categories can be modified.

[0055] Then in step S3, the application inputs the image and the classification results of step S2 into a large multimodal model. A large multimodal model refers to a type of model based on deep neural networks and natural language processing technology. Their purpose is to use a large amount of multimodal data (such as images, text, audio, video, etc.) to learn a general multimodal representation that can be effectively reasoned and generated on different multimodal tasks. Large multimodal models usually do not require manual design and selection of features, but instead automatically learn and extract features, and use large language models as powerful inference engines to achieve zero-sample or few-sample multimodal capabilities. Examples of large multimodal models include GPT-4V, CogVLM, Qwen-VL, etc.

[0056] The multimodal model is an existing model. This application only calls its interface, inputs data and parameters to obtain its output results. As described in the background technology, if the image is only input into the multimodal large model, it is possible that the large model will have hallucination problems and cannot perform the desired classification task well. For this reason, in step S3, this application not only sends the image and the output results of the unimodal classification model to the multimodal model, but also sets the input parameters, that is, multiple rounds of self-questioning and self-answering on the image content. In actual operation, you can directly input instructions such as "You are an artificial intelligence visual assistant that can recognize a single image. You will receive a picture and five possible categories of the picture. Your task is to generate five sets of questions and answers that can identify the given image category. The design of the questions should be related to the image and the five categories. Through five rounds of question-and-answer sessions, the categories are judged in detail to obtain the true category." This is the meaning of "drive" in step S3. For the above questions, the multimodal model will give question-and-answer results.

[0057] In some optional embodiments, in step S3, driving the multimodal model to conduct multiple rounds of self-questioning and self-answering on the image content includes: driving the multimodal model to form questions and answers related to the classification results on the local structures, attributes, and relationships between local structures in the image.

[0058] In this embodiment, the top 5 categories and images or videos are input into the multimodal large model together, and the multimodal large model is asked to ask and answer 5 questions based on the top categories to determine what the real category is. These questions can be about details, attributes, relationships, etc. in the image or video, such as "What is the fur color of this dog?", "What are the people in this video doing?", etc. When setting the input instructions, it is necessary to fully consider the characteristics of different categories and images or videos, and design different question templates and strategies to generate more targeted and high-quality questions. The purpose of these questions is to let the multimodal large model focus on the features related to the category in the image or video. The multimodal large model will answer these questions and provide key information to support the final classification decision.

[0059] In some optional implementations, in step S3, the rounds of self-questioning and self-answering include at least 5. In alternative implementations, more rounds of questions and answers can also be given, which can be set according to specific circumstances.

[0060] for Figure 2 Given an image example, multimodal question answering generation for the top 5 categories:

[0061] 'Q1':'Does this image show a large vehicle designed primarily for agricultural tasks'.

[0062] 'A1':'Yes, it shows a large vehicle that looks designed for farm use'.

[0063] 'Q2': 'Is the attachment on the front of the vehicle designed for tilling or turning the soil'.

[0064] 'A2': 'No, this attachment does not appear to be designed for tilling or turning soil'.

[0065] 'Q3': 'Does the vehicle have typical features for mowing or maintaining lawns'.

[0066] 'A3': 'No, it does not have the features of a lawn mower as it looks too sturdy and lacks a cutting deck or blades for maintaining the lawn'.

[0067] 'Q4': 'Is the vehicle in this image designed to tow other vehicles or objects'.

[0068] 'A4': 'Not a primary use, although it may have the potential to tow some small tools or trailers, this is not its primary function'.

[0069] Finally, in step S4, based on the above question-and-answer results, the multimodal model will automatically think about and give the best matching classification. The "driving" here refers to giving the multimodal model the best matching classification from the classification results given by the unimodal classification model, rather than arbitrarily specifying it, such as "You are an artificial intelligence visual assistant that can recognize a single image. You will receive a picture and five rounds of conversation content related to the picture. Your task is to give your final predicted category from five options after five rounds of question-and-answer sessions and explain your thinking process." Driving instructions.

[0070] In this step, the multimodal large model can exclude and judge the potential correct categories in the top 5 according to the question-answer results, and give the thinking process, which can be the evaluation of each question-answer pair by the multimodal large model, such as "this answer supports / opposes this category", "this answer is very convincing / not convincing enough", etc. Alternatively, it can also combine the prediction results of the traditional classification model as a reference for classification, so as to comprehensively judge the final category, and also give the confidence and reason for the classification to improve the reliability and explainability of the classification.

[0071] against Figure 2 The image example shown, in step S4, drives the multimodal model to think, for example, "Based on the answers provided, the vehicle in the picture is not a plow, mower, harvester or trailer. Since it is a large vehicle designed primarily for agricultural tasks but does not have any specialized harvesting equipment, tilling or mowing attributes, and is not primarily used for towing, the most suitable category is a tractor, especially considering the frame-like attachments, it may be a tool carrier or a linkage for connecting various agricultural devices." Based on this thinking result, "tractor" is selected from the unimodal classification model as the optimal classification.

[0072] In some optional implementations, step S4 further includes:

[0073] Step S5: weight the best matching classification given by the multimodal model to the classification result probability given by the unimodal classification model through the following formula to form the index value of each classification, and take the classification result with the highest index value as the final classification of the image:

[0074] P i =(1+T i ) i ;

[0075]

[0076] Among them, P i is the index value of the i-th category, p iis the probability of the i-th classification given by the unimodal classification model, and a is between 0.3 and 0.8.

[0077] like Figure 2 In the example of the image shown, the result given by multimodality is that the image belongs to the tractor classification, which means that the second classification needs to increase its weight. The original probability is 0.26. Assuming that a is 0.5, the corrected P2 = (1 + 0.5) * 0.26 = 0.39. The probability of the lawn mower, which originally ranked first, is still 0.42. In the end, the image is identified as a lawn mower. Of course, if a is 0.8, the corrected P2 is about 0.47, which is higher than the probability value of 0.42 for the lawn mower. The final classification will be a tractor. The specific value of a needs to be set according to the confidence of the unimodal classification model and the multimodal model. For example, in a specific implementation, the value of a is determined by the following formula:

[0078] a=s1+s2*e m-n ;

[0079] Among them, m is the confidence of the multimodal model, n is the confidence of the unimodal classification model, s1 and s2 are adjustment parameters, which are intended to ensure that the final calculated result is within a preset range, such as the above-mentioned 0.3-0.8. If the calculated result exceeds the limited range, the endpoint value can be used instead. For example, when the confidence of the multimodal model is 90%, the confidence of the unimodal classification model is 80%, s1 is 0.1, and s2 is 0.5, then a is approximately equal to 0.65 after calculation. Conversely, when the confidence of the multimodal model is 80%, the confidence of the unimodal classification model is 90%, s1 is 0.1, and s2 is 0.5, then a is approximately equal to 0.55 after calculation.

[0080] In this embodiment, the confidence of the unimodal classification model is obtained by conventional calculation after the model training is completed. The confidence of the multimodal model can be given by expert evaluation or by the multimodal model itself. For example, when driving the multimodal model to think, the driving instruction is set to the following example: You are an artificial intelligence visual assistant that can recognize a single image. You will receive a picture and five rounds of conversation content related to the picture. Your task is to give your final prediction category from five options after five rounds of question and answer sessions, and give the credibility / accuracy of your prediction, whose value is limited to between 0-1. This credibility / accuracy can be used to characterize the confidence of the multimodal model.

[0081] This application uses a method that uses a multimodal model to assist in improving the accuracy of image classification and video classification, and improves the accuracy and explainability of classification through multimodal question answering and thinking process.

[0082] The second aspect of the present application provides a device for improving image classification by using a multimodal model, which mainly includes:

[0083] An image acquisition module, used to acquire an image to be classified or multiple frames of images in a video to be classified;

[0084] A unimodal classification module is used to output multiple possible classification results and the probability of each classification result through a pre-trained unimodal classification model;

[0085] A multimodal question-answering module, used to input the image, classification results and the probability of each classification result into a given multimodal model, and drive the multimodal model to perform multiple rounds of self-questioning and self-answering on the image content to obtain multiple question-answering results;

[0086] The multimodal classification module is used to drive the multimodal model to select the best matching classification from the classification results based on the question and answer results.

[0087] In some optional implementations, in the multimodal question-answering module, driving the multimodal model to perform multiple rounds of self-questioning and self-answering on image content includes:

[0088] The multimodal model is driven to form questions and answers related to the classification results regarding the local structures, attributes, and relationships between the local structures in the image.

[0089] In some optional implementations, the self-questioning and self-answering rounds include at least 5 times.

[0090] In some optional embodiments, the device further comprises:

[0091] The unimodal and multimodal classification weighting module is used to weight the best matching classification given by the multimodal model to the classification result probability given by the unimodal classification model through the following formula to form the index value of each classification, and take the classification result with the highest index value as the final classification of the image:

[0092] P i =(1+T i ) i ;

[0093]

[0094] Among them, P i is the index value of the i-th category, p i is the probability of the i-th classification given by the unimodal classification model, and a is between 0.3 and 0.8.

[0095] In a third aspect of the present application, a computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a method for improving image classification by assisting with a multimodal model.

[0096] In a fourth aspect of the present application, a readable storage medium stores a computer program, which, when executed by a processor, is used to implement the method of using a multimodal model to assist in improving image classification as described above. The computer-readable storage medium may be included in the device described in the above embodiment; or it may exist alone without being assembled into the device. The computer-readable storage medium carries one or more programs, and when the one or more programs are executed by the device, the data is processed according to the above method.

[0097] Reference below Figure 3 , which shows a schematic structural diagram of a computer device 400 suitable for implementing the embodiments of the present application. Figure 3 The computer device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0098] like Figure 3 As shown, the computer device 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage part 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the device 400 are also stored. The CPU 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0099] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed, so that a computer program read therefrom is installed into the storage section 408 as needed.

[0100] In particular, according to the embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 409, and / or installed from the removable medium 411. When the computer program is executed by the central processing unit (CPU) 401, the above functions defined in the method of the present application are executed. It should be noted that the computer storage medium of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, - but not limited to - a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, an apparatus or a device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, an apparatus or a device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0101] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0102] The modules or units involved in the embodiments described in this application may be implemented by software or hardware. The modules or units described may also be set in a processor, and the names of these modules or units do not constitute limitations on the modules or units themselves in some cases.

[0103] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for improving image classification using a multimodal model, characterized in that: include: Step S1, obtaining an image to be classified or multiple frames of images in a video to be classified; Step S2: Output multiple possible classification results and the probability of each classification result through a pre-trained unimodal classification model; Step S3: input the image, classification results and the probability of each classification result into a given multimodal model, and drive the multimodal model to perform multiple rounds of self-questioning and self-answering on the image content to obtain multiple question-answering results; Step S4: driving the multimodal model to select the best matching classification from the classification results based on the question and answer results; Among them, in step S3, driving the multimodal model to perform multiple rounds of self-questioning and self-answering on the image content includes: The multimodal model is driven to form questions and answers related to the classification results regarding the local structures, attributes, and relationships between the local structures in the image.

2. The method for improving image classification using a multimodal model as claimed in claim 1, characterized in that: In step S3, the self-questioning and self-answering rounds include at least 5 times.

3. The method for improving image classification using a multimodal model as claimed in claim 1, characterized in that: Step S4 further includes: Step S5: weight the best matching classification given by the multimodal model to the classification result probability given by the unimodal classification model by the following formula to form an index value for each classification, and take the classification result with the highest index value as the final classification of the image: P i =(1+T i )p i ; Among them, P i is the index value of the i-th category, p i is the probability of the i-th classification given by the unimodal classification model, and a is between 0.3 and 0.

8.

4. A device for improving image classification using a multimodal model, characterized in that: include: An image acquisition module, used to acquire an image to be classified or multiple frames of images in a video to be classified; A unimodal classification module is used to output multiple possible classification results and the probability of each classification result through a pre-trained unimodal classification model; A multimodal question-answering module, used to input the image, classification results and the probability of each classification result into a given multimodal model, and drive the multimodal model to perform multiple rounds of self-questioning and self-answering on the image content to obtain multiple question-answering results; A multimodal classification module, used to drive the multimodal model to select the best matching classification from the classification results based on the question and answer results; Wherein, in the multimodal question-answering module, driving the multimodal model to conduct multiple rounds of self-questioning and self-answering on the image content includes: The multimodal model is driven to form questions and answers related to the classification results regarding the local structures, attributes, and relationships between the local structures in the image.

5. The device for improving image classification using a multimodal model as claimed in claim 4, characterized in that: The self-questioning and self-answering rounds include at least 5 times.

6. The device for improving image classification using a multimodal model as claimed in claim 4, characterized in that: The device also includes: The unimodal and multimodal classification weighting module is used to weight the best matching classification given by the multimodal model to the classification result probability given by the unimodal classification model through the following formula to form the index value of each classification, and take the classification result with the highest index value as the final classification of the image: P i =(1+T i )p i ; Among them, P i is the index value of the i-th category, p i is the probability of the i-th classification given by the unimodal classification model, and a is between 0.3 and 0.8.

Images