Evaluation method and device and electronic equipment

Abstract

The invention provides an evaluation method and device and electronic equipment, and relates to the technical field of computers, in particular to the application fields of artificial intelligence, large language models, multi-modal large language models, generative search, search engines, information retrieval, question and answer systems and the like. According to the specific implementation scheme, the method comprises the steps of obtaining target data and evaluation reference information related to the target data; wherein the target data comprises a target image and a target problem; inputting the target image and the target question into a multi-modal large language model to obtain an inference answer for the target question based on the target image by using the multi-modal large language model; and based on the target question, the evaluation reference information and the reasoning answer, obtaining an evaluation result for the multi-modal large language model.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to application fields such as artificial intelligence, large language models, multimodal large language models, generative search, search engines, information retrieval, and question-answering systems, and specifically to an evaluation method, device, and electronic device. Background Art

[0002] In the field of artificial intelligence, multimodal large language models have attracted much attention due to their ability to process multiple types of data (e.g., text data, image data, etc.). However, due to the diversity of their input types and the complexity of the model structure, evaluating multimodal large language models faces huge challenges. Summary of the invention

[0003] The present disclosure provides an evaluation method, device and electronic equipment.

[0004] According to a first aspect of the present disclosure, there is provided an evaluation method, comprising:

[0005] Obtaining target data and evaluation reference information related to the target data; wherein the target data includes a target image and a target question;

[0006] Inputting the target image and the target question into the multimodal large language model, so as to obtain an inference answer to the target question based on the target image using the multimodal large language model;

[0007] Based on the target question, evaluation reference information and reasoning answers, the evaluation results for the multimodal large language model are obtained.

[0008] According to a second aspect of the present disclosure, there is provided an evaluation device, comprising:

[0009] A data acquisition unit, used to acquire target data and evaluation reference information related to the target data; wherein the target data includes a target image and a target question;

[0010] An inference unit, used for inputting a target image and a target question into a multimodal large language model, so as to obtain an inference answer to the target question based on the target image using the multimodal large language model;

[0011] The evaluation unit is used to obtain evaluation results for the multimodal large language model based on the target question, evaluation reference information and reasoning answers.

[0012] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0013] at least one processor;

[0014] a memory communicatively coupled to the at least one processor;

[0015] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method provided in the first aspect of the present disclosure.

[0016] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method provided according to the first aspect of the present disclosure.

[0017] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the method provided according to the first aspect of the present disclosure when executed by a processor.

[0018] The use of the present disclosure can improve the accuracy of the evaluation results for MLLM.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0021] Figure 1 A flowchart of an evaluation method provided by an embodiment of the present disclosure;

[0022] Figure 2 A diagram illustrating the integrity process of an evaluation method provided in an embodiment of the present disclosure;

[0023] Figure 3 A target image used in a specific example provided in an embodiment of the present disclosure;

[0024] Figure 4 A schematic diagram of an application scenario of an evaluation method provided in an embodiment of the present disclosure;

[0025] Figure 5 A schematic structural block diagram of an evaluation device provided in an embodiment of the present disclosure;

[0026] Figure 6 A schematic structural block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0028] As described in the background technology, in the field of artificial intelligence, multimodal large language models (MLLM) have attracted much attention due to their ability to process multiple types of data. However, due to the diversity of its input types and the complexity of the model structure, evaluating MLLM faces huge challenges.

[0029] At present, the evaluation of MLLM mainly relies on evaluation data in the form of multiple-choice questions and / or true-or-false questions. During the evaluation process, for product data in the form of multiple-choice questions, MLLM only needs to select a target answer from multiple given candidate answers; for evaluation data in the form of true-or-false questions, MLLM only needs to select a target result from the given correct options and wrong options. However, the inventors have found that this evaluation method mainly evaluates MLLM's recognition ability, perception ability, and reasoning ability, and lacks direct evaluation of MLLM's conversational ability and visual ability. Therefore, it is impossible to obtain accurate evaluation results for MLLM.

[0030] In view of the above problems, the embodiments of the present disclosure provide an evaluation method, which can be applied to electronic devices. The electronic device can be a server, a workbench, a mainframe computer, a conventional computer (desktop computer, laptop computer, etc.) or other similar computing devices. Figure 1 The flowchart diagram shown in the figure illustrates an evaluation method provided by an embodiment of the present disclosure. It should be noted that although the logical order is shown in the flowchart diagram, in some cases, the steps shown or described in the flowchart can also be performed in other orders.

[0031] Step S101, obtaining target data and evaluation reference information related to the target data.

[0032] The target data may include a target image and a target question.

[0033] Here, the target image can be an image used to evaluate any one of the basic visual perception ability, fine-grained image understanding ability, complex visual reasoning ability, and video understanding reasoning ability of MLLM. Correspondingly, when the target image is an image used to evaluate the basic visual perception ability of MLLM, the target question can be a question related to the target image and raised for the basic visual perception ability of MLLM; when the target image is an image used to evaluate the fine-grained image understanding ability of MLLM, the target question can be a question related to the target image and raised for the fine-grained image understanding ability of MLLM; when the target image is an image used to evaluate the complex visual reasoning ability of MLLM, the target question can be a question related to the target image and raised for the complex visual reasoning ability of MLLM; when the target image is an image used to evaluate the video understanding reasoning ability of MLLM, the target question can be a question related to the target image and raised for the video understanding reasoning ability of MLLM.

[0034] In addition, in the embodiments of the present disclosure, the evaluation reference information may include descriptive information for the target image and / or reference answers to the target questions, that is, the evaluation reference information may be descriptive information for the target image, or may be reference answers to the target questions, or may include both descriptive information for the target image and reference answers to the target questions.

[0035] Step S102 , inputting the target image and the target question into the MLLM, so as to obtain an inference answer to the target question based on the target image using the MLLM.

[0036] It can be understood that in the embodiment of the present disclosure, after the target image and the target question are input into the MLLM, the MLLM can automatically obtain an inference answer to the target question based on the target image.

[0037] Step S103, obtaining the evaluation result for MLLM based on the target question, evaluation reference information and reasoning answer.

[0038] In one example, the reasoning answer can be evaluated based on the target question and the evaluation reference information to obtain the evaluation result for the MLLM, wherein the evaluation result can be a capability score to characterize the performance of the MLLM.

[0039] The evaluation method provided by the embodiment of the present disclosure can obtain target data including target images and target questions, and give the target images and target questions as input data to MLLM, so as to use MLLM to obtain reasoning answers to the target questions based on the target images, and then obtain evaluation results for MLLM based on the target questions, evaluation reference information related to the target data, and reasoning answers. This evaluation method can not only evaluate the recognition ability, perception ability, and reasoning ability of MLLM, but also because the input data of MLLM includes target images and target questions, and the output result is the reasoning answer to the target question, in the process of MLLM obtaining the output result based on the input data, the visual ability of its conversation ability will definitely be used. Therefore, in this way, the conversation ability and visual ability of MLLM can be directly evaluated to improve the accuracy of the evaluation results for MLLM.

[0040] In addition, it should be noted that in the embodiment of the present disclosure, after obtaining multiple evaluation results for MLLM, a final evaluation result for MLLM can be obtained based on the multiple evaluation results.

[0041] It should also be noted that in the embodiment of the present disclosure, when executing step S101, the target data can be obtained first, and then the evaluation reference information related to the target data can be obtained. Moreover, while obtaining the evaluation reference information related to the target data, the target image and the target question can be input into the MLLM, so as to utilize the MLLM to obtain the inference answer to the target question based on the target image, thereby improving the evaluation efficiency of the MLLM.

[0042] In some optional implementations, step S101, i.e., “obtaining target data” may include:

[0043] Identify multiple application categories of MLLM;

[0044] Constructing a plurality of candidate data groups corresponding to a plurality of application categories one by one; wherein each of the plurality of candidate data groups includes a plurality of candidate data, and each of the plurality of candidate data includes a candidate image and a candidate question;

[0045] Each candidate data in the plurality of candidate data groups is regarded as target data.

[0046] The multiple application categories may be multiple first-level categories, and each of the multiple first-level categories has multiple second-level categories, so the multiple application categories may also be multiple second-level categories. Here, each of the multiple second-level categories has multiple third-level categories, so the multiple application categories may also be multiple third-level categories.

[0047] In an example, the above-mentioned multiple primary categories, multiple secondary categories under each of the multiple primary categories, and multiple tertiary categories under each of the multiple secondary categories can be shown in the following Table 1:

[0048] Table 1

[0049]

[0050]

[0051] Further, in one example, “constructing multiple candidate data groups corresponding to multiple application categories one by one” may include:

[0052] Obtain the number of instances matching the target category in the actual application process of MLLM; wherein the target category is each application category in the multiple application categories;

[0053] Based on the number of instances, determine the number of targets corresponding to the target category;

[0054] A candidate data group corresponding to the target category is constructed, and the candidate data group includes a target number of candidate data.

[0055] In a specific example, after obtaining the number of instances matching the target category in the actual application process of MLLM, a numerical value that is positively correlated with the number of instances and is located in a preset numerical interval can be obtained as the target number corresponding to the target category, that is, in the actual application process of MLLM, the more instances matching the target category, the larger the target number corresponding to the target category; in the actual application process of MLLM, the smaller the number of instances matching the target category, the smaller the target number corresponding to the target category. Among them, the preset numerical interval can be set according to application requirements, for example, it can be set to [30, 600], and the embodiment of the present disclosure is not limited to this. In a more specific example, after obtaining the number of instances matching the target category in the actual application process of MLLM, the product of the number of instances and the preset ratio is calculated, and when the product is less than the minimum value in the preset numerical interval, the minimum value in the preset numerical interval is used as the target number corresponding to the target category; when the product is in the preset numerical interval, the product is used as the target number corresponding to the target category; when the product is greater than the maximum value in the preset numerical interval, the maximum value in the preset numerical interval is used as the target number corresponding to the target category. The preset ratio may be specifically set according to application requirements, for example, it may be set to a ratio value less than or equal to 1, for example, 0.01, and the embodiments of the present disclosure do not limit this.

[0056] After determining the number of targets corresponding to the target category based on the number of instances, a candidate data group corresponding to the target category can be constructed, and the candidate data group includes the target number of candidate data. The candidate data may include candidate images and candidate questions, and the candidate images and candidate questions can be obtained from the application examples of MLLM, which will not be described in detail here.

[0057] Exemplarily, the multiple application categories are multiple secondary categories, and the preset ratio is 0.01.

[0058] Assume that, in the actual application of MLLM, the number of instances matching the secondary category of "natural image" is 8600. Then, when the secondary category of "natural image" is used as the target category, the number of targets corresponding to the target category can be determined to be: 8600×0.01=86, and a candidate data group corresponding to the target category is constructed, and the candidate data group includes 86 candidate data; Assume again that, in the actual application of MLLM, the number of instances matching the secondary category of "common sense recognition" is 34100. Then, when the secondary category of "common sense recognition" is used as the target category, It can be determined that the number of targets corresponding to the target category is: 34100×0.01=341, and a candidate data group corresponding to the target category is constructed, and 341 candidate data are included in the candidate data group; assuming that, in the actual application of MLLM, the number of instances matching the secondary category of "picture creation" is 8400, then when the secondary category of "common sense recognition" is used as the target category, it can be determined that the number of targets corresponding to the target category is: 8400×0.01=84, and a candidate data group corresponding to the target category is constructed, and 84 candidate data are included in the candidate data group.

[0059] Finally, multiple candidate data groups as shown in Table 2 below can be obtained.

[0060] Table 2

[0061]

[0062]

[0063] That is to say, in the above example, 13 candidate data groups can be obtained, and these 13 candidate data groups include a total of 1606 candidate data. Therefore, each candidate data in the 13 candidate data groups can be used as target data, and steps S102 and S103 can be executed. That is, each candidate data in the 1606 candidate data can be used as target data (including target images and target questions), and the target images and target questions can be input into MLLM, so as to use MLLM to obtain an inference answer to the target question based on the target image, and then obtain an evaluation result for MLLM based on the target question, the evaluation reference information related to the target data, and the inference answer.

[0064] Finally, 1606 test results can be obtained. After that, the final evaluation result for MLLM can be obtained based on the 1606 evaluation results. For example, the 1606 evaluation results can be fused to obtain a fusion result (specifically, when the evaluation result is an ability score, the fusion result can be the average of the 1606 evaluation results), and then the fusion result is used as the final evaluation result for MLLM.

[0065] Through the above method, in the embodiment of the present disclosure, multiple application categories of MLLM can be determined, and multiple candidate data groups corresponding to the multiple application categories can be constructed, and each candidate data in the multiple candidate data groups is used as the target data (including the target image and the target question) to evaluate the MLLM. In this way, it can be ensured that the evaluation of MLLM is extensive and comprehensively covers the multiple application categories of MLLM, and therefore, the accuracy of the evaluation results for MLLM can be further improved.

[0066] In addition, when constructing multiple candidate data groups corresponding to multiple application categories, it can be specifically to use each of the multiple application categories as a target category, and obtain the number of instances matching the target category in the actual application process of MLLM, and then determine the target number corresponding to the target category based on the number of instances, so as to construct a candidate data group corresponding to the target category, and make the candidate data group include the target number of candidate data. In other words, when evaluating MLLM, the number of candidate data included in each of the multiple candidate data groups used is determined based on the number of instances matched by the application category corresponding to the candidate array in the actual application process of MLLM. In this way, it can be ensured that the distribution of all candidate data is consistent with the instance distribution of MLLM in the actual application scenario, thereby making the evaluation results for MLLM more representative and practical.

[0067] In some optional implementations, step S103, i.e., “obtaining an evaluation result for the MLLM based on the target question, the evaluation reference information and the reasoning answer” may include:

[0068] Obtain multiple evaluation dimensions;

[0069] Generate assessment prompts including multiple assessment dimensions;

[0070] Using the evaluation model, following the evaluation prompts, based on the target questions, evaluation reference information and reasoning answers, the evaluation results for MLLM are obtained.

[0071] Among them, multiple evaluation dimensions may include at least one of accuracy, relevance (specifically, it may be the relevance of the inference answer to the target question) and emotional expression; the evaluation prompts are used to indicate how the evaluation model obtains the evaluation results for the MLLM based on the target question, evaluation reference information and inference answers; the evaluation model may be other MLLMs with better performance, or it may be a large language model (LLM) with better performance.

[0072] In one example, “using the assessment model, following the assessment prompts, based on the target questions, assessment reference information and reasoning answers, to obtain the assessment results for the MLLM” may include:

[0073] Using the assessment model, following the assessment prompts, based on the target questions, assessment reference information and reasoning answers, multiple assessment sub-results corresponding to multiple assessment dimensions and targeting MLLM are obtained;

[0074] Based on multiple evaluation sub-results, the evaluation results for MLLM are obtained.

[0075] Among them, for each of the multiple evaluation sub-results, the evaluation sub-result can be a single-dimensional score to characterize the performance strength of the MLLM under the evaluation dimension corresponding to the evaluation sub-result. Based on this, in a specific example, after obtaining multiple evaluation sub-results, the sum of the multiple evaluation sub-results can be used as the evaluation result for the MLLM.

[0076] In the above example, after obtaining multiple evaluation dimensions and generating evaluation prompts including multiple evaluation dimensions, the evaluation model can be used to obtain multiple evaluation sub-results corresponding to multiple evaluation dimensions and for MLLM based on the target questions, evaluation reference information and reasoning answers according to the evaluation prompts, and the evaluation results for MLLM are obtained based on the multiple evaluation sub-results. Therefore, the evaluation results for MLLM can comprehensively show the performance of MLLM in multiple evaluation dimensions and have strong reliability.

[0077] In addition, as mentioned above, in the embodiments of the present disclosure, the evaluation reference information may include description information for the target image and / or reference answers to the target questions.

[0078] The evaluation reference information may be obtained by processing the target image and / or the target problem in an automatic and / or manual way.

[0079] In one example, the target image and / or target problem may be processed by automatic annotation to obtain initial reference information, and then the initial reference information may be verified by manual annotation to obtain evaluation reference information. The automatic annotation may be to process the target image and / or target problem using other MLLMs with better performance to obtain initial reference information.

[0080] In addition, it should be noted that in the embodiments of the present disclosure, the evaluation reference information can include the reference answer to the target question only if there is an objective answer corresponding to the target question, and the reference answer can include the objective answer corresponding to the target question.

[0081] Exemplarily, when the evaluation reference information includes description information for the target image and / or a reference answer to the target question, the evaluation prompt may include:

[0082]

[0083]

[0084] Further, in one example, the evaluation reference information includes description information for the target image. Based on this, preferably, "using the evaluation model, following the evaluation prompt, based on the target question, the evaluation reference information and the reasoning answer, to obtain the evaluation result for the MLLM" may include:

[0085] Get LLM;

[0086] LLM is used as an evaluation model to utilize the evaluation model, follow the evaluation prompts, and obtain the evaluation results for MLLM based on the target questions, descriptive information, and reasoning answers.

[0087] In a specific example, the evaluation model can be used to obtain multiple evaluation sub-results corresponding to multiple evaluation dimensions and for MLLM based on the target questions, evaluation reference information and reasoning answers according to the evaluation prompts, and obtain the evaluation results for MLLM based on the multiple evaluation sub-results.

[0088] For example, the evaluation reference information is the descriptive information for the target image. Then, the evaluation model can be used to obtain a credible answer to the target question based on the descriptive information in accordance with the evaluation prompts, and the inference answer can be compared with the credible answer from multiple evaluation dimensions to obtain multiple evaluation sub-results corresponding to the multiple evaluation dimensions and for MLLM, and then the evaluation result for MLLM can be obtained based on the multiple evaluation sub-results; for another example, the evaluation reference information includes both the descriptive information for the target image and the reference answer to the target question. Then, the evaluation model can be used to obtain a credible answer to the target question based on the descriptive information in accordance with the evaluation prompts, and the inference answer can be compared with the credible answer and the reference answer from multiple evaluation dimensions to obtain multiple evaluation sub-results corresponding to the multiple evaluation dimensions and for MLLM, and then the evaluation result for MLLM can be obtained based on the multiple evaluation sub-results.

[0089] In the above examples, the evaluation reference information may include descriptive information for the target image and / or reference answers to the target questions. Moreover, when the evaluation reference information includes descriptive information for the target image, LLM may be obtained and used as an evaluation model to obtain evaluation results for MLLM based on the target question, descriptive information and reasoning answers using the evaluation model in accordance with the evaluation prompts. Since LLM usually only has the processing capability for text data but not for image data, when the evaluation reference information includes descriptive information for the target image, using LLM as an evaluation model can not only ensure the smooth progress of the evaluation process, but also save the processing process for image data, thereby improving the evaluation efficiency for MLLM.

[0090] In summary, through the above method, in the embodiment of the present disclosure, multiple evaluation dimensions can be obtained, and evaluation prompts including multiple evaluation dimensions can be generated, and then the evaluation model can be used to obtain the evaluation results for MLLM according to the evaluation prompts based on the target questions, evaluation reference information and reasoning answers. On the one hand, the evaluation results for MLLM can be made to comprehensively characterize the performance strengths and weaknesses of MLLM in multiple evaluation dimensions, that is, the evaluation results for MLLM have strong reliability; on the other hand, after obtaining multiple evaluation dimensions and generating evaluation prompts including multiple evaluation dimensions, the evaluation model can be used to obtain the evaluation results for MLLM according to the evaluation prompts based on the target questions, evaluation reference information and reasoning answers, thereby improving the evaluation efficiency for MLLM.

[0091] The following will be combined Figure 2 , the complete process of an evaluation method provided by an embodiment of the present disclosure is described.

[0092] First, multiple application categories of MLLM are determined, and multiple candidate data groups corresponding to the multiple application categories are constructed, and then each candidate data in the multiple candidate data groups is used as target data. Each candidate data group in the multiple candidate data groups includes multiple candidate data, and each candidate data in the multiple candidate data includes a candidate image and a candidate question; the target data may include a target image and a target question.

[0093] In one example, “constructing multiple candidate data groups corresponding to multiple application categories one by one” may include:

[0094] Obtain the number of instances matching the target category in the actual application process of MLLM; wherein the target category is each application category in the multiple application categories;

[0095] Based on the number of instances, determine the number of targets corresponding to the target category;

[0096] A candidate data group corresponding to the target category is constructed, and the candidate data group includes a target number of candidate data.

[0097] Thereafter, evaluation reference information related to the target data may be obtained, wherein the evaluation reference information may include description information for the target image and / or reference answers for the target question.

[0098] The evaluation reference information may be obtained by processing the target image and / or the target problem in an automatic and / or manual way.

[0099] In one example, the target image and / or target problem may be processed by automatic annotation to obtain initial reference information, and then the initial reference information may be verified by manual annotation to obtain evaluation reference information. The automatic annotation may be to process the target image and / or target problem using other MLLMs with better performance to obtain initial reference information.

[0100] In addition, it should be noted that in the embodiments of the present disclosure, the evaluation reference information can include the reference answer to the target question only if there is an objective answer corresponding to the target question, and the reference answer can include the objective answer corresponding to the target question.

[0101] Next, the target image and the target question are input into the MLLM to obtain the reasoning answer to the target question based on the target image using the MLLM.

[0102] After that, multiple evaluation dimensions can be obtained, and evaluation prompts including multiple evaluation dimensions can be generated. Then, LLM can be used as an evaluation model to utilize the evaluation model, according to the evaluation prompts, based on the target questions, evaluation reference information and reasoning answers, to obtain multiple evaluation sub-results corresponding to multiple evaluation dimensions one by one and for MLLM, and based on the multiple evaluation sub-results, obtain the evaluation results for MLLM.

[0103] For the specific functions and examples of each step in the embodiments of the present disclosure, please refer to the relevant descriptions of the corresponding steps in the aforementioned evaluation method embodiments, which will not be repeated here.

[0104] The following describes the evaluation method provided by the embodiment of the present disclosure in combination with specific examples.

[0105] In this example, the target data includes a target image and a target question. Figure 3 The target question is: What is funny about this picture? Please describe it picture by picture.

[0106] In this example, the evaluation reference information related to the target data includes descriptive information for the target image, specifically: This image shows a humorous misuse. The image is divided into three parts: on the left is a mobile phone with a blue VGA computer monitor cable plugged into the charging port of the mobile phone, which is obviously a mismatch because mobile phones usually use smaller charging cables. The upper right corner is a packaging box with "CABLE MANIA" and "For Lightning Cable" written on it, which means that this packaging box was originally designed for Lightning charging cables, not VGA cables. The lower right corner is a hand holding a VGA connector, one end of which has been transformed into a shape that can be inserted into the Lightning charging port. The whole scene seems a bit absurd and funny because this transformation is neither practical nor scientific.

[0107] The target image and the target question are fed into the MLLM to exploit the MLLM. Based on the target image, the inference answer to the target question is: The funny thing about this image is that it shows a mobile phone connected to a device that looks like a computer mouse, which in turn is connected to a device that looks like a computer keyboard. This unusual connection creates a funny scene because mobile phones are not usually connected to computer mice or keyboards. In addition, the image also shows a person holding the phone, further emphasizing this unusual connection.

[0108] After obtaining multiple evaluation dimensions, an evaluation prompt including multiple evaluation dimensions is generated, specifically:

[0109]

[0110]

[0111] Finally, LLM is used as an evaluation model. By using the evaluation model, according to the evaluation prompts, based on the target questions, evaluation reference information and reasoning answers, multiple evaluation sub-results corresponding to multiple evaluation dimensions and for MLLM are obtained. Based on the multiple evaluation sub-results, the evaluation results for MLLM are obtained, which are as follows:

[0112]

[0113] See also Figure 4 , which is a schematic diagram of an application scenario of an evaluation method provided in an embodiment of the present disclosure.

[0114] As mentioned above, the evaluation method provided by the embodiment of the present disclosure is applied to an electronic device, wherein the electronic device may be a server, a workbench, a mainframe computer, a conventional computer or other similar computing devices.

[0115] Electronic devices are used for:

[0116] Obtaining target data and evaluation reference information related to the target data; wherein the target data includes a target image and a target question;

[0117] Input the target image and the target question into the MLLM to obtain an inference answer to the target question based on the target image using the MLLM;

[0118] Based on the target question, evaluation reference information and reasoning answers, the evaluation results for MLLM are obtained.

[0119] It should be noted that in the embodiments of the present disclosure, Figure 4 The scene diagram shown is only illustrative and not restrictive. Those skilled in the art can Figure 4 Various obvious changes and / or substitutions are made to the examples, and the obtained technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0120] In order to better implement the evaluation method, the embodiment of the present disclosure also provides an evaluation device, which can be integrated in an electronic device. The electronic device can be a server, a workbench, a large computer, a conventional computer or other similar computing devices. Figure 5 The schematic structural block diagram shown is used to illustrate an evaluation device 500 provided in the disclosed embodiment.

[0121] The evaluation device 500 includes:

[0122] The data acquisition unit 501 is used to acquire target data and evaluation reference information related to the target data; wherein the target data includes a target image and a target question;

[0123] The reasoning unit 502 is used to input the target image and the target question into the MLLM, so as to obtain a reasoning answer to the target question based on the target image using the MLLM;

[0124] The evaluation unit 503 is used to obtain the evaluation result for the MLLM based on the target question, the evaluation reference information and the reasoning answer.

[0125] In some optional implementations, the evaluation unit 503 is used to:

[0126] Obtain multiple evaluation dimensions;

[0127] Generate assessment prompts including multiple assessment dimensions;

[0128] Using the evaluation model, following the evaluation prompts, based on the target questions, evaluation reference information and reasoning answers, the evaluation results for MLLM are obtained.

[0129] In some optional implementations, the evaluation unit 503 is used to:

[0130] Using the assessment model, following the assessment prompts, based on the target questions, assessment reference information and reasoning answers, multiple assessment sub-results corresponding to multiple assessment dimensions and targeting MLLM are obtained;

[0131] Based on multiple evaluation sub-results, the evaluation results for MLLM are obtained.

[0132] In some optional implementations, the evaluation reference information includes descriptive information for the target image and / or reference answers to the target questions.

[0133] In some optional implementations, the evaluation reference information includes description information for the target image;

[0134] The evaluation unit 503 is used for:

[0135] Get a large language model;

[0136] The large language model is used as the evaluation model to utilize the evaluation model, follow the evaluation prompts, and obtain the evaluation results for MLLM based on the target questions, descriptive information, and reasoning answers.

[0137] In some optional implementations, the data acquisition unit 501 is used to:

[0138] Identify multiple application categories of MLLM;

[0139] Constructing a plurality of candidate data groups corresponding to a plurality of application categories one by one; wherein each of the plurality of candidate data groups includes a plurality of candidate data, and each of the plurality of candidate data includes a candidate image and a candidate question;

[0140] Each candidate data in the plurality of candidate data groups is regarded as target data.

[0141] In some optional implementations, the data acquisition unit 501 is used to:

[0142] Obtain the number of instances matching the target category in the actual application process of MLLM; wherein the target category is each application category in the multiple application categories;

[0143] Based on the number of instances, determine the number of targets corresponding to the target category;

[0144] A candidate data group corresponding to the target category is constructed, and the candidate data group includes a target number of candidate data.

[0145] In the embodiment of the present disclosure, the specific functions and examples of each unit in the evaluation device 500 can refer to the relevant descriptions of the corresponding steps in the aforementioned evaluation method embodiment, and will not be repeated here.

[0146] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0147] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0148] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as vehicle-mounted computing devices, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0149] like Figure 6As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 to a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0150] A number of components in the electronic device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of renderers, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the electronic device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0151] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a GPU, various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as the evaluation method. For example, in some embodiments, the evaluation method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps in the evaluation method described above may be executed. Alternatively, in other embodiments, the computing unit 601 may be configured as an evaluation method in any other appropriate manner (eg, by means of firmware).

[0152] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0153] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0154] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM) or a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0155] To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having: a rendering device (e.g., a cathode ray tube (CRT) renderer or a liquid crystal display (LCD)) for rendering information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices are also used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0156] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: Local Area Network (LAN), Wide Area Network (WAN), and the Internet.

[0157] The computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0158] The embodiment of the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the evaluation method.

[0159] The embodiment of the present disclosure also provides a computer program product, including a computer program, which implements the evaluation method when executed by a processor.

[0160] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in the present disclosure can be executed in parallel, sequentially or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, and this document is not limited here. In addition, in the present disclosure, relational terms such as "first", "second", "third", etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. In addition, "multiple" in the present disclosure can be understood as at least two.

[0161] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An evaluation method, comprising: Acquire target data and evaluation reference information related to the target data; wherein the target data includes a target image and a target question; Inputting the target image and the target question into a multimodal large language model, so as to obtain an inference answer to the target question based on the target image by using the multimodal large language model; Based on the target question, the evaluation reference information and the inference answer, an evaluation result for the multimodal large language model is obtained.

2. The method according to claim 1, wherein: The obtaining of an evaluation result for the multimodal large language model based on the target question, the evaluation reference information and the inference answer includes: Obtain multiple evaluation dimensions; generating an assessment prompt including the multiple assessment dimensions; By using the evaluation model, according to the evaluation prompt, based on the target question, the evaluation reference information and the inference answer, an evaluation result for the multimodal large language model is obtained.

3. The method according to claim 2, wherein: The utilizing the evaluation model, following the evaluation prompt, based on the target question, the evaluation reference information and the inference answer, obtaining an evaluation result for the multimodal large language model, includes: Using the evaluation model, according to the evaluation prompt, based on the target question, the evaluation reference information and the inference answer, a plurality of evaluation sub-results corresponding to the plurality of evaluation dimensions and for the multimodal large language model are obtained; Based on the multiple evaluation sub-results, an evaluation result for the multimodal large language model is obtained.

4. The method according to any one of claims 2, wherein: The evaluation reference information includes description information for the target image and / or a reference answer for the target question.

5. The method according to claim 4, wherein: The evaluation reference information includes description information for the target image; The utilizing the evaluation model, following the evaluation prompt, based on the target question, the evaluation reference information and the inference answer, obtaining an evaluation result for the multimodal large language model, includes: Get a large language model; The large language model is used as an evaluation model, and the evaluation model is utilized to obtain an evaluation result for the multimodal large language model according to the evaluation prompt, based on the target question, the descriptive information and the inference answer.

6. The method according to any one of claims 1 to 5, wherein: The obtaining of target data comprises: Determining multiple application categories of the multimodal large language model; Constructing a plurality of candidate data groups corresponding one-to-one to the plurality of application categories; wherein each of the plurality of candidate data groups includes a plurality of candidate data, and each of the plurality of candidate data includes a candidate image and a candidate question; Each candidate data in the plurality of candidate data groups is used as the target data.

7. The method according to claim 6, wherein: The step of constructing a plurality of candidate data groups corresponding one-to-one to the plurality of application categories comprises: Acquire the number of instances matching a target category during the actual application of the multimodal large language model; wherein the target category is each application category of the multiple application categories; Based on the number of instances, determining the number of targets corresponding to the target category; A candidate data group corresponding to the target category is constructed, and the candidate data group includes a target number of candidate data.

8. An evaluation device, comprising: A data acquisition unit, used to acquire target data and evaluation reference information related to the target data; wherein the target data includes a target image and a target question; An inference unit, configured to input the target image and the target question into a multimodal large language model, so as to obtain an inference answer to the target question based on the target image using the multimodal large language model; An evaluation unit is used to obtain an evaluation result for the multimodal large language model based on the target question, the evaluation reference information and the inference answer.

9. The device according to claim 8, wherein: The evaluation unit is used for: Obtain multiple evaluation dimensions; generating an assessment prompt including the multiple assessment dimensions; By using the evaluation model, according to the evaluation prompt, based on the target question, the evaluation reference information and the inference answer, an evaluation result for the multimodal large language model is obtained.

10. The device according to claim 9, wherein: The evaluation unit is used for: Using the evaluation model, according to the evaluation prompt, based on the target question, the evaluation reference information and the inference answer, a plurality of evaluation sub-results corresponding to the plurality of evaluation dimensions and for the multimodal large language model are obtained; Based on the multiple evaluation sub-results, an evaluation result for the multimodal large language model is obtained.

11. The device according to any one of claims 9, wherein: The evaluation reference information includes description information for the target image and / or a reference answer for the target question.

12. The device according to claim 11, wherein The evaluation reference information includes description information for the target image; The evaluation unit is used for: Get a large language model; The large language model is used as an evaluation model, and the evaluation model is utilized to obtain an evaluation result for the multimodal large language model according to the evaluation prompt, based on the target question, the descriptive information and the inference answer.

13. The device according to any one of claims 8 to 12, wherein: The data acquisition unit is used for: Determining multiple application categories of the multimodal large language model; Constructing a plurality of candidate data groups corresponding one-to-one to the plurality of application categories; wherein each of the plurality of candidate data groups includes a plurality of candidate data, and each of the plurality of candidate data includes a candidate image and a candidate question; Each candidate data in the plurality of candidate data groups is used as the target data.

14. The device according to claim 13, wherein: The data acquisition unit is used for: Acquire the number of instances matching a target category during the actual application of the multimodal large language model; wherein the target category is each application category of the multiple application categories; Based on the number of instances, determining the number of targets corresponding to the target category; A candidate data group corresponding to the target category is constructed, and the candidate data group includes a target number of candidate data.

15. An electronic device, comprising: at least one processor; a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the method according to any one of claims 1 to 7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

17. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 7.

Images