A method and device for evaluating a multimodal large language model

Abstract

The present invention discloses an evaluation method and device for a multimodal large language model, and the method implementation includes: collecting image data to be processed, and constructing an evaluation set based on the image data to be processed, wherein the evaluation set includes multiple evaluation data; constructing evaluation criteria based on the evaluation data and the target large language model; inputting the evaluation data into the large language model to be evaluated to obtain answer information; inputting the evaluation criteria and the answer information into the evaluation model for evaluation processing to obtain the evaluation result of the large language model to be evaluated. In an embodiment of the present application, a multimodal large language model evaluation set is constructed, and the evaluation model is used as an evaluator to evaluate the ability of the multimodal large language model. It can be applicable to constructing an open task evaluation set that simulates complex scenes and complex instructions in real life, as well as the answers of the evaluation model under such tasks. The problem of the singleness of the method for evaluating a multimodal large language model can be effectively solved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal large language models, and in particular to an evaluation method, device, computer equipment and storage medium for a multimodal large language model. Background Art

[0002] Currently, the dominant evaluation framework for multimodal language models (MLLMs) mainly focuses on obtaining objective metrics from tasks with clear correct answers, but this evaluation framework cannot fully cover the full capabilities of multimodal large language models. For example, domains in multimodal contexts such as creativity, association, and ethical judgment are often difficult to simply reduce to right or wrong answers.

[0003] However, there are no standard answers to open-ended questions. When evaluating the responses of multimodal large language models to open-ended questions, it is usually necessary to use two methods: manual evaluation or evaluation by evaluation models (such as GPT-4V). However, manual evaluation is very labor-intensive, material-intensive, and time-consuming. The evaluation method using evaluation models (such as GPT-4V) cannot ensure that the evaluation criteria of the model are consistent with manual evaluation. In addition, due to the incomplete knowledge of the model, in tasks that require specific expertise (such as medical care, art, etc.), GPT-4V lacks specific knowledge and is unable to perform the role of an evaluator. Summary of the invention

[0004] Based on this, it is necessary to provide a method, device, computer equipment and storage medium for evaluating a multimodal large language model in order to solve at least one problem existing in the above-mentioned prior art.

[0005] In a first aspect, the embodiment of the present application is implemented as follows: a method for evaluating a multimodal large language model is provided, comprising the following steps:

[0006] Collecting image data to be processed, and constructing an evaluation set based on the image data to be processed, wherein the evaluation set includes a plurality of evaluation data;

[0007] Constructing an evaluation criterion based on the evaluation data and the target large language model;

[0008] Inputting the evaluation data into the large language model to be evaluated to obtain answer information;

[0009] The evaluation criteria and the answer information are input into the evaluation model for evaluation processing to obtain the evaluation result of the large language model to be evaluated.

[0010] In one embodiment, the to-be-evaluated model includes a plurality of models, and the inputting of the answer information into the evaluation model for evaluation processing to obtain the evaluation result of the to-be-evaluated large language model includes:

[0011] Determining target question information based on the answer information of each of the models to be evaluated;

[0012] Inputting the target question information, the answer information of each model to be evaluated, and the evaluation criteria into the evaluation model for voting, and obtaining the voting results of each model to be evaluated;

[0013] Based on the voting results, the evaluation results of each of the models to be evaluated are determined.

[0014] In one embodiment, the inputting the answer information into the evaluation model for evaluation processing to obtain the evaluation result of the large language model to be evaluated includes:

[0015] Determining target question information based on the answer information of each of the models to be evaluated;

[0016] Input the target question information, the answer information of the model to be evaluated, and the evaluation criteria into the evaluation model, and perform scoring according to the preset scoring rules to obtain the score of the model to be evaluated;

[0017] Based on the score of the model to be evaluated, an evaluation result of the model to be evaluated is determined.

[0018] In one embodiment, the collecting of image data to be processed includes:

[0019] Dividing the capability indicators of the large language model to be evaluated;

[0020] Based on the capability index, the latest image data in multiple fields are collected from multiple data sources as the image data to be processed.

[0021] In one embodiment, constructing an evaluation set based on the image data to be processed includes:

[0022] Based on the image data to be processed, according to a preset instruction generation strategy, a model is constructed through a preset evaluation set to generate instruction information;

[0023] Performing standardization processing on the instruction information and the image data to be processed to obtain the evaluation data;

[0024] Based on the evaluation data, the evaluation set is generated.

[0025] In one embodiment, the inputting of the image data to be processed and the prompt words for guiding the generation of instructions into the evaluation set construction model for processing, and after the instruction information is generated, the process includes:

[0026] The instruction information is subjected to consistency detection to determine whether there is an association relationship between the instruction information and the image data to be processed.

[0027] In one embodiment, after collecting the latest image data in multiple fields from multiple data sources based on the capability indicator, the method includes:

[0028] Performing quality screening on each of the latest image data according to preset screening rules;

[0029] The screened latest image data is stored for subsequent use in constructing the evaluation set.

[0030] In one embodiment, constructing an evaluation criterion based on the evaluation data, the true label, and the target large language model includes:

[0031] constructing the evaluation criteria generation guidance based on the evaluation data;

[0032] The evaluation criterion is generated by the target large language model based on the evaluation criterion generation guidance and the evaluation data.

[0033] In one embodiment, the step of generating the evaluation criteria through the target large language model based on the evaluation criteria generation guidance and the evaluation data includes:

[0034] Construct evaluation criteria to generate prompt words;

[0035] The evaluation criteria are generated by the target large language model based on the evaluation criteria generation guidance, the evaluation data and the evaluation criteria generation prompt words.

[0036] In a second aspect, a device for evaluating a multimodal large language model is provided, comprising:

[0037] An evaluation set construction module, used for collecting image data to be processed, and constructing an evaluation set based on the image data to be processed, wherein the evaluation set includes a plurality of evaluation data;

[0038] An evaluation criterion construction module, used to construct an evaluation criterion based on the evaluation data and the target large language model;

[0039] An answer information generating module, used for inputting the evaluation data into the large language model to be evaluated to obtain answer information;

[0040] The model evaluation module is used to input the evaluation criteria and the answer information into the evaluation model for evaluation processing to obtain the evaluation result of the large language model to be evaluated.

[0041] In a third aspect, a computer device is provided, comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor implements the above-mentioned multimodal large language model evaluation method when executing the computer-readable instructions.

[0042] In a fourth aspect, one or more readable storage media storing computer-readable instructions are provided. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the evaluation method of the multimodal large language model as described above.

[0043] The evaluation method, device, computer equipment and storage medium of the above-mentioned multimodal large language model, and its method implementation includes: collecting image data to be processed, and constructing an evaluation set based on the image data to be processed, wherein the evaluation set includes multiple evaluation data; constructing evaluation criteria based on the evaluation data and the target large language model; inputting the evaluation data into the large language model to be evaluated to obtain answer information; inputting the evaluation criteria and the answer information into the evaluation model for evaluation processing to obtain the evaluation result of the large language model to be evaluated. In an embodiment of the present application, a multimodal large language model evaluation set is constructed, and the evaluation model is used as an evaluator to evaluate the ability of the multimodal large language model. It can be applicable to constructing an open task evaluation set that simulates complex scenes and complex instructions in real life, as well as the answers of the evaluation model under such tasks. It can effectively solve the problem of the singleness of the method for evaluating multimodal large language models. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0045] Figure 1 It is a flow chart of a method for evaluating a multimodal large language model in one embodiment of the present invention;

[0046] Figure 2 is a structural schematic diagram of an evaluation device for a multimodal large language model in one embodiment of the present invention;

[0047] Figure 3 is a schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] In one embodiment, if Figure 1 As shown, a method for evaluating a multimodal large language model is provided, comprising the following steps:

[0050] In step S110, image data to be processed is collected, and an evaluation set is constructed based on the image data to be processed, wherein the evaluation set includes a plurality of evaluation data;

[0051] In an embodiment of the present application, the latest high-definition image data in multiple fields, such as medicine, art, etc., can be collected in real time from multiple data sources, such as social networks, columns, etc., as the image data to be processed to ensure timeliness and prevent data leakage.

[0052] It can be understood that the image data to be processed may include multiple ones.

[0053] In an embodiment of the present application, based on the latest image data to be processed collected from multiple fields, a large language model, such as GPT-4V, can be used to construct an evaluation set that meets the requirements. The evaluation data in the evaluation set is specifically in the form of picture-instruction pairs, which can be understood as each picture corresponding to an instruction.

[0054] In step S120, constructing an evaluation criterion based on the evaluation data and the target large language model;

[0055] In an embodiment of the present application, after constructing an evaluation set, the evaluation data in the evaluation set can be input into a pre-trained target large language model, such as GPT-4V, and can be manually annotated in advance in the image of the evaluation data, and a corresponding prompt word template can be constructed. At this time, the GPT-4V can output evaluation criteria that match the picture-instruction pair through the prompt word template prompt, annotations, and evaluation data of the picture-instruction pair.

[0056] It can be understood that for questions with clear answers, specific reference answers can be obtained through evaluation data, annotations, and evaluation criteria output by prompt word template prompts. For instructions that require subjective descriptions or produce ambiguous answers, the above-mentioned evaluation data, annotations, and evaluation criteria output by prompt word template prompts can provide key information for formulating answers.

[0057] In step S130, the evaluation data is input into the large language model to be evaluated to obtain answer information;

[0058] In an embodiment of the present application, the large language model to be evaluated may be a multimodal large language model, which may include one or more, and exemplarily, may be a multimodal large language model such as LLaVA-v1.5-13B, Qwen-VL-Chat, etc.

[0059] In an embodiment of the present application, the evaluation data in the constructed evaluation set is input into the large language model to be evaluated, and the pre-constructed prompt word template prompt is input to obtain the answer information output by the large language model to be evaluated based on the prompt word template prompt.

[0060] In step S140, the evaluation criteria and the answer information are input into the evaluation model for evaluation processing to obtain the evaluation result of the large language model to be evaluated.

[0061] In an embodiment of the present application, the evaluation model may be a large language model, such as GPT-4V. The evaluation criteria and the answer information are input into the GPT-4V. The GPT-4V may evaluate the capability of the model to be evaluated based on the degree of fit between the answer information output by the model to be evaluated and the evaluation criteria.

[0062] In an embodiment of the present application, a method for evaluating a large multimodal language model is provided, comprising: collecting image data to be processed, and constructing an evaluation set based on the image data to be processed, wherein the evaluation set includes multiple evaluation data; constructing evaluation criteria based on the evaluation data and the target large language model; inputting the evaluation data into the large language model to be evaluated to obtain answer information; inputting the evaluation criteria and the answer information into the evaluation model for evaluation processing to obtain the evaluation result of the large language model to be evaluated. In an embodiment of the present application, a large multimodal language model evaluation set is constructed, and the evaluation model is used as an evaluator to evaluate the ability of the large multimodal language model. It can be applied to constructing an open task evaluation set that simulates complex scenes and complex instructions in real life and the answers of the evaluation model under such tasks. Through artificial and GPT-4V, evaluation criteria are provided for automatically evaluating the answers of the multimodal language model to open questions, reducing the time, manpower, and resource costs of evaluating the answers of the multimodal language model to open questions. It can effectively solve the problem of the singleness of the method for evaluating the large multimodal language model.

[0063] In one embodiment of the present application, the to-be-evaluated model includes a plurality of models, and the inputting of the answer information into the evaluation model for evaluation processing to obtain the evaluation result of the to-be-evaluated large language model includes:

[0064] Determining target question information based on the answer information of each of the models to be evaluated;

[0065] Inputting the target question information, the answer information of each model to be evaluated, and the evaluation criteria into the evaluation model for voting, and obtaining the voting results of each model to be evaluated;

[0066] Based on the voting results, the evaluation results of each of the models to be evaluated are determined.

[0067] Specifically, take two models to be evaluated, namely, model A to be evaluated and model B to be evaluated as an example for explanation. First, a question is constructed, and the answer information of the model A to be evaluated and the model B to be evaluated to the question is obtained. Then, the question information, the answer information of each model to be evaluated and the evaluation criteria are input into the evaluation model. The evaluation model can vote according to the degree of fit between the answer information and the evaluation criteria. For example, if it is determined that the answer of model A to be evaluated is more accurate, then vote for model A to be evaluated as a win, or if the answer of model B to be evaluated is more accurate, then vote for model B to be evaluated as a win, or if the answer of model A to be evaluated is close to the answer of model B to be evaluated, then model A to be evaluated and model B to be evaluated are tied, which is a draw. Each evaluation data in the evaluation set can be subjected to the above process, and then the number of wins, losses and draws of model A to be evaluated and model B to be evaluated are counted respectively. Then, by comparing the number of wins of the two, the performance of the model to be evaluated with a large number of wins is better than that of the model to be evaluated with a small number of wins.

[0068] In one implementation scenario, assuming that the large language model includes 2, a prompt example for pairwise voting using GPT-4V is as follows:

[0069] ###You are an excellent evaluator.

[0070] ###Your task is to provide an evaluation of the given response.

[0071] ###Each evaluation will consist of *a picture*, *a question*, and *two corresponding answers*. Your task is to judge which answer is better based on **the quality** of the answer and how well it matches the picture. If you find two answers to be equally good or equally bad, feel free to choose a tie. There is no **obligation** to favor any one answer; if a decision cannot be made, a **tie will be the best choice**.

[0072] ###During your evaluation, please focus on the **semantic quality** of your answer and how well it **aligns** with the picture. Non-semantic factors, such as tone, answer format, or answer order, should not influence your evaluation. The primary focus should be on the **quality and accuracy** of the answer.

[0073] ###Please refer to relevant reference materials when making judgments.

[0074] ###You should output your voting results only on the last line: "AnswerA", "AnswerB", or "Tie".

[0075] Among them, AnswerA means that the model A to be evaluated wins, AnswerB means that model B wins, and Tie means a draw.

[0076] It can be understood that usually, there may be two models to be evaluated for voting, that is, voting in pairs. In some scenarios, there may be multiple models to be evaluated, such as 3 or 4, that is, voting on multiple models to be evaluated at the same time.

[0077] In one embodiment of the present application, the inputting the answer information into the evaluation model for evaluation processing to obtain the evaluation result of the large language model to be evaluated includes:

[0078] Determining target question information based on the answer information of each of the models to be evaluated;

[0079] Input the target question information, the answer information of the model to be evaluated, and the evaluation criteria into the evaluation model, and perform scoring according to the preset scoring rules to obtain the score of the model to be evaluated;

[0080] Based on the score of the model to be evaluated, an evaluation result of the model to be evaluated is determined.

[0081] Specifically, take two models to be evaluated, namely, model C to be evaluated and model D to be evaluated as an example for explanation, first construct a question, and obtain the answer information of the model C to be evaluated and the model D to be evaluated to the question, then input the question information, the answer information of each model to be evaluated and the evaluation criteria into the evaluation model, and the evaluation model can score according to the degree of fit between the answer information and the evaluation criteria. For example, the score range can be set to 1-10, that is, the evaluation model can be required to score the model C to be evaluated and the model D to be evaluated within the range of 1-10, and output two scores. It can be understood that each evaluation data in the evaluation set can be subjected to the above process, and then the total scores of the model C to be evaluated and the model D to be evaluated are respectively counted, and then based on the total score, if the total score of the model C to be evaluated is greater than the total score of the model D to be evaluated, then the performance of the model C to be evaluated is due to the total score of the model D to be evaluated, and if the total score of the model C to be evaluated is less than the total score of the model D to be evaluated, then the performance of the model D to be evaluated is due to the performance of the model C to be evaluated.

[0082] In one implementation scenario, assuming that the large language model includes 2, a prompt example that can be scored in pairs using GPT-4V is as follows:

[0083] ###You are a helpful feedback provider.

[0084] ###Your task is to provide feedback on both responses to the questions shown below and based on the given pictures.

[0085] ###Please rate these answers based on their helpfulness, relevance, accuracy, and thoroughness.

[0086] ###Each answer receives an overall score based on a scale of 1 to 10, where higher scores indicate better overall performance.

[0087] ###Please refer to the references given when making your judgment.

[0088] %References:{reference}

[0089] ###You only need to output the ratings for two responses.

[0090] It can be understood that usually there are two models to be evaluated that are scored, that is, paired voting. In some scenarios, there may be one or more models to be evaluated, that is, one model to be evaluated may be scored individually, or multiple models to be evaluated may be scored at the same time.

[0091] As an implementation method, when there is one model to be evaluated, a preset score threshold can be set, that is, when the score of the model to be evaluated is greater than the preset score threshold, it is considered that the performance of the model to be evaluated is good, otherwise, its performance is poor.

[0092] In one embodiment of the present application, the collecting of image data to be processed includes:

[0093] Dividing the capability indicators of the large language model to be evaluated;

[0094] Based on the capability index, the latest image data in multiple fields are collected from multiple data sources as the image data to be processed.

[0095] Specifically, based on the evaluation requirements of different large language models to be evaluated, the capability indicators of the large language models to be evaluated can be divided, for example, action recognition indicators, OCR (text recognition) indicators, scene understanding indicators, etc. It can be understood that the capability indicator divisions of different large language models to be evaluated are the same, and there are only differences in performance. For example, when the capability indicators of the large language models LLaVA-v1.5-13B and Qwen-VL-Chat to be evaluated are divided at the same time, the types of their capability indicators can be the same.

[0096] Furthermore, based on the divided capability indicators, the latest high-definition images in multiple fields can be obtained from various data sources, such as social networks, columns, etc., to ensure timeliness and prevent data leakage. For example, for OCR capability, the latest images with text content can be found from Twitter; for medical image understanding capability, the latest images in the medical field can be found from Zhihu as the image data to be processed.

[0097] It is understandable that the evaluation of the model to be evaluated can be achieved by directly collecting the image data to be processed without dividing the model's capability indicators. However, dividing the capability indicators of the model to be evaluated can simulate the levels of human cognition and divide the model's capabilities into 6 levels from shallow to deep, which helps to better align the multimodal large language model with real applications and user experience.

[0098] In an embodiment of the present application, after collecting the latest image data in multiple fields from multiple data sources based on the capability indicator, the method includes:

[0099] Performing quality screening on each of the latest image data according to preset screening rules;

[0100] The screened latest image data is stored for subsequent use in constructing the evaluation set.

[0101] Specifically, after collecting the latest image data in multiple fields from multiple data sources, Python can be used to screen the image quality. For example, for each picture, after reading and opening the picture, its resolution can be detected. If the resolution of any side of the picture is lower than the preset value, such as 512, the picture is removed to ensure that the single-side resolution of each picture is not lower than 512. The screened pictures are stored in a server, such as a local server or a specific directory on a cloud server, and can be directly called when needed.

[0102] In one embodiment of the present application, constructing an evaluation set based on the image data to be processed includes:

[0103] Based on the image data to be processed, according to a preset instruction generation strategy, a model is constructed through a preset evaluation set to generate instruction information;

[0104] Performing standardization processing on the instruction information and the image data to be processed to obtain the evaluation data;

[0105] Based on the evaluation data, the evaluation set is generated.

[0106] Specifically, the preset evaluation set construction model can be a pre-trained large language model, such as GPT-4V, according to the indicator generation strategy, that is, based on the pre-built prompt word template prompt, according to the image data to be processed and the model capability indicator, generate corresponding instruction information.

[0107] Among them, the prompt word template prompt is used to generate instruction information. For example, the prompt may be "Currently benchmarking a large multimodal language model, the specific task is (...). Based on the provided pictures and task requirements, please think of a few complex instructions."

[0108] Then, the data that passes the consistency test can be standardized, that is, the data can be aligned, for example, the data number, image storage address, and instruction can be organized into a json format to ensure that each image has a corresponding instruction, thereby forming an image-instruction pair as the evaluation data, and then forming the evaluation set.

[0109] In one embodiment of the present application, the image data to be processed and the prompt words for guiding the generation of instructions are input into the evaluation set construction model for processing, and after the instruction information is generated, the following steps are included:

[0110] The instruction information is subjected to consistency detection to determine whether there is an association relationship between the instruction information and the image data to be processed.

[0111] Specifically, the generated instruction information may be checked for consistency. For example, the instruction information may be manually checked to see whether it is related to the image content, so as to ensure semantic consistency between the image and the instruction. It can be understood that the instruction correctly describes the image content.

[0112] In one embodiment of the present application, constructing an evaluation criterion based on the evaluation data and the target large language model includes:

[0113] constructing the evaluation criteria generation guidance based on the evaluation data;

[0114] The evaluation criterion is generated by the target large language model based on the evaluation criterion generation guidance and the evaluation data.

[0115] Specifically, the evaluation data can be annotated by manual annotation to help the target large language model generate annotations for the evaluation criteria, that is, the evaluation criteria generation guidance, and then the target large language model, such as GPT-4V, can be used to generate evaluation criteria specific to the evaluation data based on the evaluation data, that is, the image-instruction pair, and the evaluation criteria generation guidance. Exemplarily, the API of GPT-4V can be called to input the evaluation data, that is, the image-instruction pair, and the criterion generation guidance to GPT-4V, and extract the answer returned by GPT-4V as the evaluation criteria specific to the evaluation data.

[0116] Furthermore, the generated evaluation criteria can be checked for consistency. For example, it is possible to manually check whether the generated criteria are relevant to the image content and whether they provide key information for answering instructions, so as to ensure that the semantics between the evaluation criteria and the data are consistent. It can be understood that the evaluation criteria can be used to evaluate the model's answer to this piece of data.

[0117] Furthermore, the data that pass the consistency detection may be standardized, that is, the data may be aligned to ensure that the data corresponds to the evaluation criteria.

[0118] In one embodiment of the present application, the step of generating the evaluation criteria based on the evaluation criteria generation guide and the evaluation data, and using the target large language model, includes:

[0119] Construct evaluation criteria to generate prompt words;

[0120] The evaluation criteria are generated by the target large language model based on the evaluation criteria generation guidance, the evaluation data and the evaluation criteria generation prompt words.

[0121] Specifically, the evaluation criteria generation guidance, the evaluation data, and the evaluation criteria generation prompt words are input into the target large language model, such as GPT-4V, and the target large language model generates specific evaluation criteria that match the picture and the instruction. It can be understood that for questions with clear answers, the evaluation criteria output by the evaluation data, the real label, and the prompt word template can obtain a specific reference answer, and for instructions that require subjective description or produce vague answers, the evaluation criteria output by the above-mentioned evaluation data, the real label, and the prompt word template can provide key information for formulating the answer.

[0122] In one implementation scenario, a picture to be processed includes person a, person b and person c, wherein the instruction is: estimate the approximate height of person c based on visual clues; the criterion generation guidance can be determined based on manual annotations, for example, if you want to know the height of person c in the picture, you can determine it by determining the heights of the other two characters in the picture, namely person a and person b, and the relative heights of the characters in the picture; and the criterion generation prompt word template prompt can be "You are an excellent evaluation reference generator. Your task is to generate a reference for evaluating the output of a multimodal large language model on instruction-image pairs. You will get a set of picture-instruction pairs and a manually annotated evaluation guidance. Please generate a reference for evaluating the answer to the picture-instruction pair." After processing based on the target large language model, the final evaluation criterion can be obtained: the height of person a is 229 cm, the height of person b is 216 cm, and the height of person c is between 165-175, which can be used as a reference answer or key information for subsequent evaluation.

[0123] In an embodiment of the present application, a multimodal large language model evaluation set is constructed, and the evaluation model is used as an evaluator to evaluate the ability of the multimodal large language model. It can be applicable to constructing an open task evaluation set that simulates complex scenes and complex instructions in real life, as well as the answers of the evaluation model under such tasks. Through manual and GPT-4V, evaluation criteria are provided for automatically evaluating the answers of the multimodal language model to open questions, reducing the time, manpower, and resource costs of evaluating the answers of the multimodal large language model to open questions. The problem of the singleness of the method for evaluating the multimodal large language model can be effectively solved.

[0124] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0125] In one embodiment, a device for evaluating a multimodal large language model is provided, and the device for evaluating a multimodal large language model corresponds one-to-one to the method for evaluating a multimodal large language model in the above embodiment. Figure 2 As shown, the evaluation device of the multimodal large language model includes an evaluation set construction module 10, an evaluation criterion construction module 20, an answer information generation module 30 and a model evaluation module 40. The detailed description of each functional module is as follows:

[0126] An evaluation set construction module 10 is used to collect image data to be processed, and to construct an evaluation set based on the image data to be processed, wherein the evaluation set includes a plurality of evaluation data;

[0127] An evaluation criteria building module 20, used to build an evaluation criteria based on the evaluation data and the target large language model;

[0128] The answer information generating module 30 is used to input the evaluation data into the large language model to be evaluated to obtain answer information;

[0129] The model evaluation module 40 is used to input the evaluation criteria and the answer information into the evaluation model for evaluation processing to obtain the evaluation result of the large language model to be evaluated.

[0130] In one embodiment of the present application, the model to be evaluated includes multiple models, and the model evaluation module 40 is also used to:

[0131] Determining target question information based on the answer information of each of the models to be evaluated;

[0132] Inputting the target question information, the answer information of each model to be evaluated, and the evaluation criteria into the evaluation model for voting, and obtaining the voting results of each model to be evaluated;

[0133] Based on the voting results, the evaluation results of each of the models to be evaluated are determined.

[0134] In one embodiment of the present application, the model evaluation module 40 is further used to:

[0135] Determining target question information based on the answer information of each of the models to be evaluated;

[0136] Input the target question information, the answer information of the model to be evaluated, and the evaluation criteria into the evaluation model, and perform scoring according to the preset scoring rules to obtain the score of the model to be evaluated;

[0137] Based on the score of the model to be evaluated, an evaluation result of the model to be evaluated is determined.

[0138] In one embodiment of the present application, the evaluation set construction module 10 is further used to:

[0139] Dividing the capability indicators of the large language model to be evaluated;

[0140] Based on the capability index, the latest image data in multiple fields are collected from multiple data sources as the image data to be processed.

[0141] In one embodiment of the present application, the evaluation set construction module 10 is further used to:

[0142] Based on the image data to be processed, according to a preset instruction generation strategy, a model is constructed through a preset evaluation set to generate instruction information;

[0143] Performing standardization processing on the instruction information and the image data to be processed to obtain the evaluation data;

[0144] Based on the evaluation data, the evaluation set is generated.

[0145] In one embodiment of the present application, the evaluation set construction module 10 is further used to:

[0146] The instruction information is subjected to consistency detection to determine whether there is an association relationship between the instruction information and the image data to be processed.

[0147] In one embodiment of the present application, the evaluation set construction module 10 is further used to:

[0148] Performing quality screening on each of the latest image data according to preset screening rules;

[0149] The screened latest image data is stored for subsequent use in constructing the evaluation set.

[0150] In one embodiment of the present application, the evaluation criteria building module 20 is further used to:

[0151] constructing the evaluation criteria generation guidance based on the evaluation data;

[0152] The evaluation criterion is generated by the target large language model based on the evaluation criterion generation guidance and the evaluation data.

[0153] In one embodiment of the present application, the evaluation criteria building module 20 is further used to:

[0154] Construct evaluation criteria to generate prompt words;

[0155] The evaluation criteria are generated by the target large language model based on the evaluation criteria generation guidance, the evaluation data and the evaluation criteria generation prompt words.

[0156] In an embodiment of the present application, a multimodal large language model evaluation set is constructed, and the evaluation model is used as an evaluator to evaluate the ability of the multimodal large language model. It can be applicable to constructing an open task evaluation set that simulates complex scenes and complex instructions in real life, as well as the answers of the evaluation model under such tasks. Through manual and GPT-4V, evaluation criteria are provided for automatically evaluating the answers of the multimodal language model to open questions, reducing the time, manpower, and resource costs of evaluating the answers of the multimodal large language model to open questions. The problem of the singleness of the method for evaluating the multimodal large language model can be effectively solved.

[0157] For the specific limitations of the evaluation device of the multimodal large language model, please refer to the limitations of the evaluation method of the multimodal large language model above, which will not be repeated here. Each module in the above-mentioned evaluation device of the multimodal large language model can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0158] In one embodiment, a computer device is provided. The computer device may be a terminal device, and its internal structure diagram may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, and a network interface connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a readable storage medium. The readable storage medium stores computer-readable instructions. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer-readable instructions are executed by the processor, a method for evaluating a multimodal large language model is implemented. The readable storage medium provided in this embodiment includes a non-volatile readable storage medium and a volatile readable storage medium.

[0159] In an embodiment of the present application, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor implements the steps of the method for evaluating a multimodal large language model as described above when executing the computer-readable instructions.

[0160] In an embodiment of the application, a readable storage medium is provided, wherein the readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the method for evaluating a multimodal large language model as described above are implemented.

[0161] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through computer-readable instructions, and the computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When the computer-readable instructions are executed, they may include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0162] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0163] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for evaluating a multimodal large language model, characterized in that: The method comprises: Collecting image data to be processed, and constructing an evaluation set based on the image data to be processed, wherein the evaluation set includes multiple evaluation data; wherein the collecting of the image data to be processed includes: dividing the capability index of the multimodal large language model to be evaluated to simulate the level of human cognition, and dividing the capability of the multimodal large language model to be evaluated into 6 levels from shallow to deep; based on the capability index, collecting the latest image data in multiple fields from multiple data sources as the image data to be processed; constructing the evaluation set based on the image data to be processed includes: based on the image data to be processed, according to the preset instruction generation strategy, generating instruction information through the preset evaluation set construction model; performing standardization processing on the instruction information and the image data to be processed to obtain the evaluation data; generating the evaluation set based on the evaluation data; wherein the evaluation data exists in the form of picture-instruction pairs, and one picture corresponds to one instruction; Based on the evaluation data and the target multimodal large language model, an evaluation criterion is constructed, including: Annotating the evaluation data by manual annotation to construct evaluation criteria generation guidance; Construct evaluation criteria to generate prompt words; Based on the evaluation criterion generation guidance, the evaluation data, and the evaluation criterion generation prompt words, the evaluation criterion is generated by the target multimodal large language model; For questions with clear answers, a specific reference answer is obtained through the evaluation data, the evaluation criteria generation guidance, and the evaluation criteria for generating prompt words output by the evaluation criteria, and for instructions that require subjective description or generate vague answers, key information for formulating answers is provided through the evaluation data, the evaluation criteria generation guidance, and the evaluation criteria for generating prompt words output by the evaluation criteria; Inputting the evaluation data into a multimodal large language model to be evaluated to obtain answer information; The evaluation criteria and the answer information are input into the evaluation model for evaluation processing to obtain an evaluation result of the multimodal large language model to be evaluated.

2. The method for evaluating a multimodal large language model according to claim 1, wherein: The multimodal large language model to be evaluated includes a plurality of models, and the evaluation criteria and the answer information are input into the evaluation model for evaluation processing to obtain the evaluation result of the multimodal large language model to be evaluated, including: Determining target question information based on the answer information of each of the multimodal large language models to be evaluated; Inputting the target question information, the answer information of each of the multimodal large language models to be evaluated, and the evaluation criteria into the evaluation model for voting, and obtaining the voting results of each of the multimodal large language models to be evaluated; Based on the voting results, the evaluation results of each of the multimodal large language models to be evaluated are determined.

3. The method for evaluating a multimodal large language model according to claim 1, wherein: The step of inputting the evaluation criteria and the answer information into the evaluation model for evaluation processing to obtain the evaluation result of the multimodal large language model to be evaluated includes: Determining target question information based on the answer information of each of the multimodal large language models to be evaluated; Inputting the target question information, the answer information of the multimodal large language model to be evaluated, and the evaluation criteria into the evaluation model, and scoring according to a preset scoring rule to obtain a score of the multimodal large language model to be evaluated; Based on the score of the multimodal large language model to be evaluated, an evaluation result of the multimodal large language model to be evaluated is determined.

4. The method for evaluating a multimodal large language model according to claim 1, wherein: The image data to be processed and the prompt words for guiding the generation of instructions are input into the evaluation set construction model for processing. After the instruction information is generated, it includes: The instruction information is subjected to consistency detection to determine whether there is an association relationship between the instruction information and the image data to be processed.

5. The method for evaluating a multimodal large language model according to claim 1, wherein: After collecting the latest image data in multiple fields from multiple data sources based on the capability indicators, the method includes: Performing quality screening on each of the latest image data according to preset screening rules; The screened latest image data is stored for subsequent use in constructing the evaluation set.

6. A multimodal large language model evaluation device, characterized in that: The device comprises: An evaluation set construction module is used to collect image data to be processed, and to construct an evaluation set based on the image data to be processed, wherein the evaluation set includes multiple evaluation data; wherein the collecting of the image data to be processed includes: dividing the capability index of the multimodal large language model to be evaluated to simulate the level of human cognition, and dividing the capability of the multimodal large language model to be evaluated into 6 levels from shallow to deep; based on the capability index, collecting the latest image data in multiple fields from multiple data sources as the image data to be processed; wherein the construction of the evaluation set based on the image data to be processed includes: based on the image data to be processed, according to the preset instruction generation strategy, generating instruction information through the preset evaluation set construction model; performing standardization processing on the instruction information and the image data to be processed to obtain the evaluation data; generating the evaluation set based on the evaluation data; wherein the evaluation data exists in the form of picture-instruction pairs, and one picture corresponds to one instruction; An evaluation criterion construction module is used to construct an evaluation criterion based on the evaluation data and the target multimodal large language model, including: Annotating the evaluation data by manual annotation to construct evaluation criteria generation guidance; Construct evaluation criteria to generate prompt words; Based on the evaluation criterion generation guidance, the evaluation data, and the evaluation criterion generation prompt words, the evaluation criterion is generated by the target multimodal large language model; For questions with clear answers, a specific reference answer is obtained through the evaluation data, the evaluation criteria generation guidance, and the evaluation criteria for generating prompt words output by the evaluation criteria, and for instructions that require subjective description or generate vague answers, key information for formulating answers is provided through the evaluation data, the evaluation criteria generation guidance, and the evaluation criteria for generating prompt words output by the evaluation criteria; An answer information generating module, used for inputting the evaluation data into the multimodal large language model to be evaluated to obtain answer information; The model evaluation module is used to input the evaluation criteria and the answer information into the evaluation model for evaluation processing to obtain the evaluation result of the multimodal large language model to be evaluated.

Images