A multimodal model visual perception ability detection method and terminal

Abstract

The present invention relates to the field of machine learning technology, and discloses a method and terminal for detecting the visual perception ability of a multimodal model, the method comprising: obtaining multiple detection images, detecting objects contained in each of the detection images; for each of the detection images, constructing positive prompt answer pairs and negative prompt answer pairs according to the objects contained therein to form a detection data set; obtaining a model to be detected, detecting the performance of the model to be detected according to the detection data set, and outputting a detection result. The present invention not only simplifies the construction of the problem by constructing positive prompt answer pairs and negative prompt answer pairs for each image, but also requires that the model cannot rely solely on random guessing when understanding the image content and its context, forcing the model to correctly understand the pattern and the problem in order to obtain a better detection result, and effectively solves the problem that the ability of some models cannot be accurately measured when performing detection.

Description

Technical Field

[0001] The present invention relates to the field of machine learning technology, and in particular to a multimodal model visual perception ability detection method, terminal and readable storage medium. Background Art

[0002] Detecting the visual perception capabilities of multimodal models is not only crucial to improving the transparency and reliability of the models, but also points the way for model optimization, ensuring that they can play a better role in a wide range of application scenarios. By revealing the strengths and weaknesses of different models, it helps define the potential and limitations of existing models in practical applications. Only when we have a clear understanding of the applicability and performance of each model in different work scenarios can we improve the reliability of model use.

[0003] In the prior art, some technologies have been proposed to detect the visual perception capabilities of multimodal models. For example, some technologies focus on the importance of each modality, focus on the intrinsic detection of pre-trained multimodal models, and study how existing visual language models use information from two modalities. By masking a specific level of information in a single modality and then predicting the hidden content of the other modality, the model's ability to process two modalities is detected.

[0004] There are also some technologies that are based on model sample selection detection, using image captions as detection datasets to study the impact of various multimodal context configurations on multimodal language models, and by designing different methods to select images and different caption assignment strategies to construct different multimodal context configurations.

[0005] There are also some technologies dedicated to detecting the capabilities of certain dimensions of multimodal language models, detecting the robustness of the model by designing adversarial examples, and deeply exploring the stability and reliability of the model when facing carefully constructed challenging inputs.

[0006] At present, unified detection work on the visual perception capabilities of multimodal language models is still relatively scarce. The detection strategies of existing technologies usually rely on standard public datasets to evaluate the performance of a single model on multiple datasets. There is a lack of unified detection of multiple models. In addition, when performing detection, the detection methods of existing technologies have the problem that the models over-rely on their own biases and make blind guesses about the detection problems, resulting in some models that are too powerful on the language side and lack perception on the visual side being overly exaggerated, making it impossible to accurately measure the capabilities of the models. Summary of the invention

[0007] The purpose of the present invention is to provide a method, terminal and readable storage medium for detecting the visual perception ability of a multimodal model, aiming to solve the problem that the prior art cannot accurately measure the ability of some models when detecting the visual perception ability of a multimodal model.

[0008] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0009] The present invention provides a method for detecting visual perception ability of a multimodal model, and the method comprises:

[0010] Acquire multiple detection images, and detect objects contained in each of the detection images;

[0011] For each of the detection images, constructing positive prompt answer pairs and negative prompt answer pairs according to the objects contained therein to form a detection data set;

[0012] The model to be detected is obtained, the performance of the model to be detected is detected according to the detection data set, and the detection result is output.

[0013] Furthermore, the detecting the object contained in each of the detection images specifically includes:

[0014] Obtain an object detection model, and input each of the detection images into the object detection model in turn;

[0015] The object detection model outputs multiple objects for each of the detection images.

[0016] Furthermore, the object detection model outputs multiple objects in each of the detection images, and then further includes:

[0017] The object detection model outputs the confidence of each of the objects in each of the detection images;

[0018] For each of the detection images, the objects are screened out according to the corresponding confidence levels.

[0019] Furthermore, for each of the detection images, a positive prompt answer pair and a negative prompt answer pair are constructed according to the objects contained therein, specifically including:

[0020] For each of the detection images, constructing a positive prompt answer pair and a negative prompt answer pair of the object detection visual perception subtask according to the objects contained therein;

[0021] For each of the detection images, constructing a positive prompt answer pair and a negative prompt answer pair of the visual perception subtask according to the object attributes included in the image;

[0022] For each of the detection images, positive prompt-answer pairs and negative prompt-answer pairs of the object counting visual perception subtask are constructed according to the objects contained therein.

[0023] Furthermore, for each of the detection images, a positive prompt answer pair and a negative prompt answer pair of the object detection visual perception subtask are constructed according to the objects contained therein, specifically including:

[0024] For each of the detection images, the following positive prompt answer pairs are constructed based on the objects contained:

[0025] Question: Judgment Is it in the picture? Answer: Yes;

[0026] in, Indicates the objects contained in the detected image;

[0027] For each of the detection images, the following negative prompt answer pairs are constructed based on the objects contained:

[0028] Question: Judgment Is it in the picture? Answer: No;

[0029] in, Indicates detecting objects that are not contained in the image.

[0030] Furthermore, for each of the detection images, a positive prompt answer pair and a negative prompt answer pair of the object attribute recognition visual perception subtask are constructed according to the objects contained therein, specifically including:

[0031] For each of the detection images, the following positive prompt answer pairs are constructed based on the objects contained:

[0032] Question: Judgment Is the object in the picture? Answer: Yes.

[0033] Among them, object represents an object contained in a detection image. Indicates the actual color of the object object;

[0034] For each of the detection images, the following negative prompt answer pairs are constructed based on the objects contained:

[0035] Question: Judgment Is the object in the picture? Answer: No.

[0036] in, is a color randomly selected from a set of common colors, removing color.

[0037] Furthermore, for each of the detection images, a positive prompt answer pair and a negative prompt answer pair of the object counting visual perception subtask are constructed according to the objects contained therein, specifically including:

[0038] For each of the detection images, the following positive prompt answer pairs are constructed based on the objects contained:

[0039] Question: Judgment Is the object in the picture? Answer: Yes.

[0040] Among them, object represents an object contained in a detection image. Indicates the number of objects;

[0041] For each of the detection images, the following negative prompt answer pairs are constructed based on the objects contained:

[0042] Question: Judgment Is the object in the picture? Answer: No.

[0043] in, It is randomly selected from common numbers. Number other than .

[0044] Furthermore, detecting the performance of the model to be detected according to the detection data set and outputting the detection result specifically includes:

[0045] Inputting the detection data set into the model to be detected to obtain the answer output of the model to be detected;

[0046] A confusion matrix is ​​constructed according to the answer output, and the confusion matrix is ​​output as the detection result. In addition, to achieve the above purpose, the present invention also provides a terminal, which includes: a memory, a processor, and a multimodal model visual perception ability detection program stored in the memory and executable on the processor, wherein when the multimodal model visual perception ability detection program is executed by the processor, the terminal is controlled to implement the steps of the multimodal model visual perception ability detection method described above.

[0047] In addition, to achieve the above-mentioned purpose, the present invention also provides a readable storage medium, which stores a multimodal model visual perception ability detection program, and when the multimodal model visual perception ability detection program is executed by a processor, it implements the steps of a multimodal model visual perception ability detection method as described above.

[0048] The present invention adopts the above technical solution to achieve the following effects:

[0049] The present invention constructs positive prompt answer pairs and negative prompt answer pairs for each image, which not only simplifies the construction of the problem, but also requires that the model cannot rely solely on random guessing when understanding the image content and its context, forcing the model to correctly understand the pattern and the problem in order to obtain better detection results, and effectively solves the problem that the ability of some models cannot be accurately measured during detection. The constructed detection data set has the advantages of uniform label distribution, simple indicator calculation, and sufficient data volume. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a flowchart of the steps of a method for detecting visual perception ability of a multimodal model in a preferred embodiment of the present invention;

[0051] Figure 2 is a schematic diagram of question-answer pairs for different subtasks in a preferred embodiment of the present invention;

[0052] Figure 3 It is a schematic diagram of input modes of different models during detection in a preferred embodiment of the present invention;

[0053] Figure 4 A schematic diagram of the answer results of the test in the preferred embodiment of the present invention;

[0054] Figure 5 A schematic diagram of the accuracy results of the test in a preferred embodiment of the present invention;

[0055] Figure 6 A schematic diagram of an operating environment of a preferred embodiment of a terminal of the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0057] Embodiment 1

[0058] See also Figure 1 and Figure 2 Embodiment 1 of the present application is a method for detecting visual perception ability of a multimodal model, which comprises the steps of:

[0059] S1. Acquire multiple detection images and detect objects contained in each detection image.

[0060] In this embodiment, the detection task includes multiple subtasks, wherein the subtasks include an object detection (ObjectDetection) visual perception subtask, an object attribute recognition (ObjectColor) visual perception subtask and an object counting (ObjectCount) visual perception subtask.

[0061] Detecting the objects contained in each image is completed in the object detection visual perception subtask. Among them, the object detection visual perception subtask is the most basic visual perception ability detection task, which mainly involves correctly identifying objects in visual information, but usually does not involve in-depth understanding of objects; the object attribute recognition visual perception subtask is a more difficult detection task than object recognition, requiring the model to recognize the color attributes of the target object; the object counting visual perception subtask is the most challenging detection task. The model not only needs to accurately identify each object, but also needs to calculate their number on this basis.

[0062] In this embodiment, for the object detection visual perception subtask, the object detection image is input into the object detection model. The object detection model is a pre-trained expert model with high object detection capability. The object detection capability of the expert model must be equal to or even stronger than the general multimodal model being tested. It can not only serve as a benchmark for detecting the multimodal language model's object detection visual perception capability, but also has the function of automatically extracting objects from images.

[0063] After the object detection model detects multiple objects from the object detection image, in order to further improve the accuracy and reliability of the recognition result and to better form a detection data set, this embodiment sorts the detected objects from high to low according to the confidence scores of the multiple objects in the same detection image according to the object detection model, thereby obtaining an object list:

[0064] ;

[0065] in, Represents a list of objects, Represents the first object, represents the confidence of the first object, Indicates objects, Indicates The confidence level of an object, Indicates objects, Indicates The confidence level of an object.

[0066] After obtaining the object list, the object list is filtered by the confidence threshold, and only objects with confidence greater than the confidence threshold are retained. In this embodiment, the confidence threshold is specifically 0.7. This screening step helps to filter out low-confidence detection results, thereby enhancing the accuracy of object recognition. In addition, in order to ensure that too many labels are not generated in a single image, the present invention limits the number of labels for each image and removes objects outside the clipping threshold. Specifically, in this embodiment, the clipping threshold is 2.

[0067] S2. For each of the detection images, construct positive prompt answer pairs and negative prompt answer pairs according to the objects contained therein to form a detection data set.

[0068] Please refer to Figure 2 In this embodiment, according to the intercepted object list Positive and negative prompt-answer pairs are constructed for the detection images. The prompt-answer pairs ensure that the model needs to be able to correctly answer questions about the content of the image and identify wrong answers that are opposite to these questions. In this way, the model needs to rely on the content of the image when answering questions, rather than just relying on guesswork. For example, for each image, the following positive and negative prompt-answer pairs are constructed:

[0069] Question: Judgment Is it in the picture? Answer: Yes.

[0070] Question: Judgment Is it in the picture? Answer: No.

[0071] in, represents the detection of objects present in the image, and Indicates the detection of objects that do not exist in the image. They can be obtained through language model association or by querying the similar intentions of existing objects. For example, if the existing object is a cat, the non-existent object can be a dog. By constructing positive and negative prompt answer pairs, the model can effectively avoid blind guessing of the problem. In the context of a single image, the model must accurately answer the positive and negative prompt answer pairs to ensure its understanding and reasoning ability of the image content.

[0072] For the object attribute recognition visual perception subtask, an advanced multimodal model is used to analyze the objects detected by the object detection model and generate corresponding color descriptions , that is, the main color attributes of objects in the image are identified through the learning ability of the model, thereby providing accurate color labels for constructing the data set. It is worth noting that the multimodal model used to generate color descriptions here is usually a domain SOTA model. The SOTA model refers to the most advanced model with the strongest detection capability in the neighborhood, which is stronger than the model being tested. Based on the detected color description, a prompt answer pair about the attributes of the object in the image can be constructed. For each image, positive and negative prompt answer pairs are also formulated to avoid blind guessing by the model. Specifically, in this embodiment, the positive and negative prompt answer pairs for the visual perception subtask of object attribute recognition are:

[0073] Question: Judgment Is the object in the picture? The answer is: yes.

[0074] Question: Judgment Is the object in the picture? The answer is: No.

[0075] Among them, object represents a target object detected by an object detection model. Indicates the actual color of the target object. It is a color randomly selected from a set of common colors. When selecting, the actual color of the target object is removed as a negative prompt. This method ensures the diversity and complexity of the negative prompt answer pairs in the detection data. It aims to test whether the model can accurately identify and distinguish the actual color attributes of objects in the image, and also tests the robustness of the model when dealing with misleading information.

[0076] For the object counting visual perception subtask, when constructing the prompt-answer pairs for the object counting task, first, the object list obtained by the object detection model is , count the number of each object and get the object counting information list:

[0077] ;

[0078] in, Represents a list of object count information. Represents the object count information of the first object, Indicates Object count information for objects, Indicates objects, Indicates The object counting information of each object. It is worth noting that this can be used to count both a complete list of objects and an incomplete list of objects. For example, in order to ensure both the confidence of the object and the diversity of the count, the object list can be first screened for confidence, and then counted to obtain a list of object count information. After that, the object count information is screened according to the value of each object to force the model to correctly understand the count instead of guessing based on the counted number. For example, since most objects usually have only one, in order to avoid the model blindly guessing correctly for questions with a number of 1, the object with a number of 1 can be screened out. However, in order to avoid the model blindly guessing incorrectly for questions with a number of 1, a number can be randomly selected and screened based on the selected number.

[0079] Then, for each entity object object, a positive and negative prompt answer pair about the image is also constructed to avoid blind guessing by the model:

[0080] Question: Judgment Is the object in the picture? The answer is: yes.

[0081] Question: Judgment Is the object in the picture? The answer is: No.

[0082] Among them, object represents a target object detected by an object detection model. represents the number of target objects in the image, It is randomly selected from common numbers and does not correspond to the actual number of objects in the image, ensuring the diversity of negative prompt answer pairs in the detection data, thereby testing whether the model can accurately identify the number of specific objects in the image and at the same time testing the model's ability to handle misleading information.

[0083] S3. Acquire a model to be detected, detect the performance of the model to be detected according to the detection data set, and obtain and output a detection result.

[0084] The detection dataset created by the above method has three advantages. The first is that it is automatically constructed and the data volume is sufficient. Compared with the manually constructed prompt answer pair dataset, the automated construction method greatly improves efficiency and significantly reduces the consumption of time and human resources. In addition, the automated process reduces the possibility of human error and bias, thereby ensuring the consistency and diversity of the dataset. The detection dataset of this embodiment constructs 10,000 question and answer pairs for each of the object detection, object color attribute recognition, and object counting visual perception subtasks, thereby providing a more comprehensive and reliable detection basis.

[0085] The second is the multi-dimensional visual perception ability detection. The data set covers three visual perception dimensions from easy to difficult, including object recognition, attribute recognition, and counting, so that the visual perception performance of the model can be finely detected. In terms of visual perception difficulty, the difficulty of object detection, object attribute recognition, and object counting is gradually increasing. Object detection is a basic visual perception task, which mainly involves correctly identifying different objects in the image. It requires the model to have certain image processing capabilities, but usually does not involve in-depth understanding of the object. Object attribute recognition adds the requirement of understanding the specific attributes of the object on the basis of object detection, and requires the model to have a deeper understanding of the details of the image, which increases the difficulty of the task. Object counting is the most complex task among the three, requiring the model not only to recognize the objects in the image, but also to accurately calculate their number. It puts higher requirements on the model's visual processing ability and spatial reasoning ability. It requires the model to be able to distinguish and recognize multiple identical or similar objects in the image and accurately count them. This is a relatively more complex and challenging task in visual perception.

[0086] The third is concise question design and balanced answer distribution, an intuitive and concise performance measurement method. This dataset uses binary (yes / no) output to quantify the performance of the detection model. Each test image corresponds to a pair of positive and negative prompt answers, ensuring a balanced distribution of "yes" and "no" answers in the dataset. This not only simplifies the construction of the questions, but also requires the model to not rely solely on random guessing when understanding the image content and its context. This yes / no answer format facilitates intuitive understanding and quantitative analysis of the model's response, simplifies the scoring process, and makes detection more accurate and objective. Furthermore, by constructing a confusion matrix, the performance of the model in different scenarios can be analyzed in detail, reflecting the model's understanding of the image content more accurately.

[0087] In this embodiment, a total of 7 currently advanced multimodal language models are detected on the proposed dataset. These models cover different scales from large to small parameter amounts. Specifically, the models to be detected include four models with large parameter amounts, namely BLIP2-T5 (4B), InstructBLIP (instruction fine-tuning bootstrapped language image pre-training model) (7B), LLava (Large Language and Vision Assistant, language and vision large model) (7B) and Bliva (Betterhandling of text-rich visual questions with LLM, a model for processing text-rich visual questions based on LLM (Large Language Model)) (7B), as well as three models with small parameter amounts, namely VL-T5, VL-Bart and FewVLM (few-shots Vision-Language Model, a small number of example visual language model).

[0088] Among them, the VL-T5 model is a VL (visual language) model that uses T5 (Text-to-Text Transfer Transformer) as the language model, the VL-Bart is a VL model that uses Bart (Bidirectional and Auto-Regressive Transformers) as the language model, and the BLIP2-T5 model is a BLIP2 (Bootstrapping Language-Image Pre-training) model that uses T5.

[0089] When performing detection, in order to facilitate quantitative performance statistics, for the large multimodal language model of detection, the idea of ​​the prompt design of the present invention is to let the model answer "yes" or "no", so please refer to Figure 3 In order to quickly apply the data set for model detection, the present invention adopts a prompt learning method to guide the multimodal language model to adapt to the data set by introducing discrete prompts. Specifically, for smaller multimodal language models such as VL-Bart, VL-T5 and FewVLM models, this method follows the original prompt template setting of the model. For example, for the FewVLM model, the template construction of the text prompt is:

[0090] .

[0091] As for the more advanced and parameterized multimodal language models, such as InstructBLIP, LLava, Bliva, BLIP2T5, etc., these models have excellent text understanding capabilities and can flexibly handle text prompts. Therefore, the final prompt consists of two parts, including a concise question [Q] and a prompt description "Please answer yes or no". Two prompts are designed for each test image, and the difference between the two lies in the question. The label answer of one question is "yes" and the label answer of the other question is "no". When the model can answer these two questions correctly, it actually shows that the model's understanding of the image and the relevant knowledge stored are more reliable, and it can effectively avoid the model giving answers by random guesses.

[0092] Please refer to Figure 4 and Figure 5 , Figure 4 is the answer result obtained by the present invention on each model to be detected, Figure 5 The components and accuracy of each model to be detected. Among them, the BLIP2-T5 model, VL-T5 and FewVLM models use T5 as the language model, VL-Bart uses Bart as the language model, and the InstructBLIP model and Bliva model replace the language model with FLANT5 (Instruction finetuning-T5, T5 based on instruction fine-tuning) model. FLANT5XXL is used as the subscript of the two models, and XXL indicates the size of FLANT5. The LLava model uses LLMA (Large Language Model MetaAI) as the language model.

[0093] In addition, the BLIP2-T5 model, InstructBLIP model and Bliva model have added a Q-Former component, and the Bliva model and LLava model have added a Projection component.

[0094] The results show that some models lack visual perception and rely too much on model internal bias. In particular, multimodal language models with small parameters perform poorly in basic visual perception and reasoning, and have significant prediction bias. For example, FewVLM and VLBart perform only about 50% on various visual perception tasks, which indicates that these models may rely solely on random guessing.

[0095] Analyzing the prediction distribution of the models, we can see that the output ratio of "yes" and "no" of these models is extremely unbalanced. For example, FewVLM outputs "yes" for both positive and negative prompt answer pairs of the same test image, that is, it makes completely contradictory judgments on the same image. This shows that the model fails to effectively use visual information for reasoning when predicting, but mainly relies on learned internal biases. In contrast, the distribution of "yes" and "no" for the models with good performance is close to 1:1, indicating that they can fully consider visual information and effectively resist bias.

[0096] In addition, experimental results show that these models lack reasoning capabilities. The object counting task requires the model to not only detect objects in the image, but also to accurately count each instance. This requires the model to understand the overall scene and consider contextual information, especially when there is overlap or occlusion between objects. However, experimental results show that all models perform averagely in the object counting task.

[0097] Embodiment 2

[0098] See also Figure 6 Based on the above method, the present invention further provides a terminal, which includes a processor 10, a memory 20 and a display 30. However, it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead.

[0099] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Furthermore, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 can also be used to temporarily store data that has been output or is to be output. In one embodiment, a multimodal model visual perception ability detection program 40 is stored on the memory 20, and the multimodal model visual perception ability detection program 40 can be executed by the processor 10, thereby realizing the terminal in the present application.

[0100] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing related programs of the multimodal model visual perception ability detection method.

[0101] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface.

[0102] In one embodiment, when the processor 10 executes a multimodal model visual perception ability detection program 40 in the memory 20, the steps of a multimodal model visual perception ability detection method as described above are implemented.

[0103] Embodiment 3

[0104] This embodiment provides a storage medium, wherein the readable storage medium stores a multimodal model visual perception ability detection program, and when the multimodal model visual perception ability detection program is executed by a processor, the steps of the multimodal model visual perception ability detection method as described above are implemented.

[0105] In summary, the present invention not only simplifies the construction of the problem by constructing positive prompt answer pairs and negative prompt answer pairs for each image, but also requires the model to not rely solely on random guessing when understanding the image content and its context, forcing the model to correctly understand the pattern and the problem in order to obtain better detection results, effectively solving the problem of being unable to accurately measure the model's capabilities for some models during detection.

[0106] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or terminal including the element.

[0107] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium, and the program can include the processes of the above-mentioned method embodiments when executed. The storage medium can be a memory, a disk, an optical disk, etc.

[0108] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for detecting visual perception ability of a multimodal model, characterized in that: The multimodal model visual perception ability detection method comprises: Acquire multiple detection images, and detect objects contained in each of the detection images; For each of the detection images, constructing positive prompt answer pairs and negative prompt answer pairs according to the objects contained therein to form a detection data set; Acquire a model to be detected, detect the performance of the model to be detected according to the detection data set, and output a detection result; For each of the detection images, a positive prompt answer pair and a negative prompt answer pair are constructed according to the objects contained therein, specifically including: For each of the detection images, constructing a positive prompt answer pair and a negative prompt answer pair of the object detection visual perception subtask according to the objects contained therein; For each of the detection images, constructing a positive prompt answer pair and a negative prompt answer pair of the visual perception subtask according to the object attributes included in the image; For each of the detection images, constructing positive prompt-answer pairs and negative prompt-answer pairs of the object counting visual perception subtask according to the included objects; For each of the detection images, a positive prompt answer pair and a negative prompt answer pair of the object detection visual perception subtask are constructed according to the objects contained therein, specifically including: For each of the detection images, the following positive prompt answer pairs are constructed based on the objects contained: Question: Judgment Is it in the picture? Answer: Yes; in, Indicates the objects contained in the detected image; For each of the detection images, the following negative prompt answer pairs are constructed based on the objects contained: Question: Judgment Is it in the picture? Answer: No; in, Indicates that the detected image does not contain objects. It is obtained through language model association or by querying the similar intentions of existing objects; For each of the detection images, constructing a positive prompt answer pair and a negative prompt answer pair of the object attribute recognition visual perception subtask according to the included objects, specifically including: For each of the detection images, the following positive prompt answer pairs are constructed based on the objects contained: Question: Judgment Is the object in the picture? Answer: Yes. Among them, object represents an object contained in a detection image. Indicates the actual color of the object object; For each of the detection images, the following negative prompt answer pairs are constructed based on the objects contained: Question: Judgment Is the object in the picture? Answer: No. in, is a color randomly selected from a set of common colors, removing Color; For each of the detection images, a positive prompt answer pair and a negative prompt answer pair of the object counting visual perception subtask are constructed according to the objects contained therein, specifically including: For each of the detection images, objects are screened out according to the number of each object contained therein; For each of the detection images, the following positive prompt answer pairs are constructed based on the objects contained: Question: Judgment Is the object in the picture? Answer: Yes. Among them, object represents an object contained in a detection image. Indicates the number of objects; For each of the detection images, the following negative prompt answer pairs are constructed based on the objects contained: Question: Judgment Is the object in the picture? Answer: No. in, It is randomly selected from common numbers. Number other than The detecting of the object contained in each of the detection images specifically includes: Obtain an object detection model, and input each of the detection images into the object detection model in turn; The object detection model outputs multiple objects in each of the detection images; The object detection model outputs a plurality of objects in each of the detection images, and then further comprises: The object detection model outputs the confidence of each of the objects in each of the detection images; For each of the detection images, the objects are screened out according to the corresponding confidence levels.

2. A method for detecting visual perception ability of a multimodal model according to claim 1, characterized in that: Detecting the performance of the model to be detected according to the detection data set and outputting the detection result specifically includes: Inputting the detection data set into the model to be detected to obtain the answer output of the model to be detected; A confusion matrix is ​​constructed according to the answer output, and the confusion matrix is ​​output as the detection result.

3. A terminal, characterized in that: The terminal includes: a memory, a processor, and a multimodal model visual perception ability detection program stored in the memory and executable on the processor. When the multimodal model visual perception ability detection program is executed by the processor, the terminal is controlled to implement the steps of a multimodal model visual perception ability detection method as described in any one of claims 1-2.

4. A readable storage medium, characterized in that: The readable storage medium stores a multimodal model visual perception ability detection program, and when the multimodal model visual perception ability detection program is executed by a processor, it implements the steps of a multimodal model visual perception ability detection method as described in any one of claims 1-2.

Images