Large-model performance assessment method and related device

Abstract

Provided in the embodiments of the present application are a large-model performance assessment method and a related device. The method comprises: acquiring structured text description of a test material, wherein the structured text description comprises information of the test material in a plurality of dimensions; inputting the structured text description and a test question into a large language model, so as to acquire an evaluation true value; inputting the test material and the test question into a large model under test, so as to acquire an answer under test of the test question; and inputting the evaluation true value, the test question and the answer under test into the large language model, so as to acquire an assessment result for the answer under test.

Description

Method for evaluating performance of large model and related device

[0001] The present application claims priority to the Chinese patent application No. 202410385861.X, filed on March 30, 2024, and entitled "Method for evaluating performance of large model and related device", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence, in particular to a method for evaluating performance of a large model and related device. BACKGROUND

[0003] Large models (such as Chat GPT) have become a highlight in the field of artificial intelligence due to their excellent performance and wide range of applications. These large models not only perform well in natural language processing tasks, but also show great application potential in image processing, video understanding, autonomous driving, and other fields. With the widespread application of large model technology, it is particularly important to build a high-quality, high-precision, and credible large model capability evaluation platform to help model developers and users better understand the performance and scope of application of the model.

[0004] In the prior art, the information of a test image is converted into a textual description, and the inference result of a large model to be tested on the test image is obtained. The inference result of the large model to be tested is judged by a large language model based on the above-mentioned textual description, and a quantifiable automatic evaluation is realized.

[0005] However, the above-mentioned textual description only contains rough information of the test image and lacks hierarchy, which is not conducive to the understanding of the large language model, and thus the performance of the large model to be tested cannot be accurately evaluated. SUMMARY

[0006] The present application provides a method for evaluating performance of a large model and related device, which is used to accurately evaluate the performance of a large model to be tested.

[0007] The first aspect of the present application provides a method for evaluating performance of a large model:

[0008] The test material is obtained, and a test question is obtained, the test question being a question based on the test material. A structured text description of the test material is obtained, the structured text description including information of multiple dimensions of the test material. The structured text description and the test question are input into a large language model to obtain an evaluation ground truth output by the large language model, the evaluation ground truth including information related to the test question in the test material or being a standard answer to the test question; the test material and the test question are input into a to-be-tested large model to obtain a to-be-tested answer to the test question output by the to-be-tested large model; and the evaluation ground truth, the test question, and the to-be-tested answer are input into the large language model to obtain an evaluation result of the to-be-tested large model on the to-be-tested answer based on the evaluation ground truth and the test question.

[0009] In the present application, since the structured text description includes information of multiple dimensions of the test material, it is more conducive to mine and reflect more key information and details in the test material, and the description is more hierarchical, which is more conducive to understanding by the large language model, and thus an accurate evaluation ground truth is obtained, and the accuracy of performance evaluation is improved.

[0010] In a possible implementation, the structured text description is verified by the large language model to determine a description error existing in the structured text description, the description error including a contradiction between information of multiple dimensions. The description error in the structured text description is corrected, and the corrected structured text description and the test question are input into the large language model to obtain the evaluation ground truth output by the large language model.

[0011] In the present application, the structured text description is also corrected based on the large language model, and the accuracy of performance evaluation is further improved.

[0012] In a possible implementation, the test material is an image of a traffic scene, the information of multiple dimensions includes scene-level information, object-level information, and region-level information, the scene-level information including a general description of the traffic scene, the object-level information including a description of each object in each object category in the traffic scene, and the region-level information including a description of a target region in the traffic scene.

[0013] In a possible implementation, obtaining the structured text description of the test material specifically includes:

[0014] The test material is input into the multi-modal large model to obtain scene-level information output by the multi-modal large model. A plurality of object annotation images are obtained, the object annotation images being obtained by annotating each object in the same object category in the test material, and the objects annotated in different object annotation images belonging to different object categories. The plurality of object annotation images are sequentially input into the multi-modal large model to obtain object-level information output by the multi-modal large model. A region annotation image is obtained, the region annotation image being obtained by annotating a target region in the test material. The region annotation image and position coordinates of the target region are input into the multi-modal large model to obtain region-level information output by the multi-modal large model.

[0015] In a possible implementation, the object categories include vehicles, road users, and roadblocks.

[0016] In a possible implementation, the test material is input into the visual pre-training model to obtain annotation information output by the visual pre-training model, the annotation information including annotations of objects in the test material, and the object annotation images and the region annotation image being obtained based on the annotation information.

[0017] In this application, the visual pre-training model is introduced to identify each object in the test material in the process of constructing the structured text description, which improves the automation level and accuracy of constructing the structured text description.

[0018] In a possible implementation, the test material is an image of a geometry problem, the information of the plurality of dimensions includes geometry element information and geometry relationship information, the geometry element information includes geometry elements appearing in the geometry problem, and the geometry relationship information includes geometry relationships between the geometry elements.

[0019] The second aspect of the present application provides a performance evaluation method of a large model:

[0020] The test material and a test question are obtained, the test question being a question raised based on the test material. A structured text description of the test material is obtained, the structured text description including information of a plurality of dimensions of the test material. The test material and the test question are input into a to-be-tested large model to obtain a to-be-tested answer to the test question output by the to-be-tested large model; and the structured text description, the test question, and the to-be-tested answer are input into a large language model to obtain an evaluation result of the large language model on the to-be-tested answer based on the structured text description and the test question.

[0021] In the present application, since the structured text description includes information of a plurality of dimensions of the test material, it is more conducive to mine and reflect more key information and details in the test material, and the description is more hierarchical, which is more conducive to understanding by the large language model, thereby improving the accuracy of performance evaluation.

[0022] In a possible implementation, the test material is an image of a traffic scene, the information of multiple dimensions includes scene-level information, object-level information, and region-level information, the scene-level information includes a global description of the traffic scene, the object-level information includes a description of each object in each object category in the traffic scene, and the region-level information includes a description of a target region in the traffic scene.

[0023] In a possible implementation, the structured text description of the test material is specifically as follows:

[0024] The test material is input into the multi-modal large model to obtain the scene-level information output by the multi-modal large model. A plurality of object annotation images are obtained, the object annotation images are obtained by annotating each object in the same object category in the test material, and the object categories to which the objects in different object annotation images belong are different. The plurality of object annotation images are sequentially input into the multi-modal large model to obtain the object-level information output by the multi-modal large model. A region annotation image is obtained, the region annotation image is obtained by annotating the target region in the test material. The region annotation image and the position coordinates of the target region are input into the multi-modal large model to obtain the region-level information output by the multi-modal large model.

[0025] In a possible implementation, the object categories include vehicles, road users, and road barriers.

[0026] In a possible implementation, the test material is input into the visual pre-training model to obtain annotation information output by the visual pre-training model, the annotation information includes annotations of objects in the test material, and the object annotation images and the region annotation image are obtained based on the annotation information.

[0027] In a possible implementation, the test material is an image of a geometry problem, the information of multiple dimensions includes geometry element information and geometry relationship information, the geometry element information includes geometry elements appearing in the geometry problem, and the geometry relationship information includes geometry relationships between the geometry elements.

[0028] The third aspect of the present application provides a computing device, comprising:

[0029] The obtaining unit is configured to obtain test material and a test question, the test question being a question raised based on the test material.

[0030] The obtaining unit is further configured to obtain a structured text description of the test material, the structured text description including information of multiple dimensions of the test material.

[0031] The processing unit is configured to input the structured text description and the test question into the large language model to obtain an evaluation true value output by the large language model, the evaluation true value including information related to the test question in the test material or a standard answer to the test question.

[0032] The processing unit is further configured to input the test material and the test question into the to-be-tested large model to obtain a to-be-tested answer to the test question output by the to-be-tested large model.

[0033] The processing unit is further configured to input the evaluation true value, the test question, and the to-be-tested answer into the large language model to obtain an evaluation result of the to-be-tested answer made by the large language model based on the evaluation true value and the test question.

[0034] In a possible implementation manner,

[0035] The processing unit is further configured to perform verification on the structured text description based on the large language model to determine a description error existing in the structured text description, the description error including a contradiction between information in multiple dimensions.

[0036] The processing unit is further configured to correct the description error in the structured text description.

[0037] The processing unit is specifically configured to input the corrected structured text description and the test question into the large language model to obtain the evaluation true value output by the large language model.

[0038] In a possible implementation manner, the test material is an image of a traffic scene, and the information in multiple dimensions includes scene-level information, object-level information, and region-level information, the scene-level information including an overall description of the traffic scene, the object-level information including a description of each object in each object category in the traffic scene, and the region-level information including a description of a target region in the traffic scene.

[0039] In a possible implementation manner,

[0040] The obtaining unit is specifically configured to input the test material into the multi-modal large model to obtain the scene-level information output by the multi-modal large model.

[0041] The obtaining unit is specifically configured to obtain a plurality of object annotation images, the object annotation images being obtained by annotating each object in a same object category in the test material, and the object categories to which the objects in different object annotation images belong being different.

[0042] The obtaining unit is specifically configured to input the plurality of object annotation images into the multi-modal large model in sequence to obtain the object-level information output by the multi-modal large model.

[0043] The acquisition unit is specifically configured to acquire a region-labeled image, where the region-labeled image is obtained by labeling a target region in the test material.

[0044] The acquisition unit is specifically configured to input the region-labeled image and the position coordinates of the target region into the multi-modal large model to acquire the region-level information output by the multi-modal large model.

[0045] In a possible implementation, the object categories include vehicles, road users, and roadblocks.

[0046] In a possible implementation,

[0047] The processing unit is further configured to input the test material into the visual pre-training model to acquire labeled information output by the visual pre-training model, where the labeled information includes labeling of objects in the test material.

[0048] The object-labeled image and the region-labeled image are acquired based on the labeled information.

[0049] In a possible implementation, the test material is an image of a geometry question, the information of the multiple dimensions includes geometry element information and geometry relationship information, the geometry element information includes geometry elements appearing in the geometry question, and the geometry relationship information includes geometry relationships between the geometry elements.

[0050] The fourth aspect of the present application provides a computing device including a processor and a memory, the processor being configured to execute instructions stored in the memory to enable the computing device cluster to perform the method in the first aspect.

[0051] The fifth aspect of the present application provides a computer program product including instructions, when the instructions are executed by a computing device, enabling the computing device to perform the method in the first aspect.

[0052] The sixth aspect of the present application provides a computer-readable storage medium including computer program instructions, when the computer program instructions are executed by a computing device, enabling the computing device to perform the method in the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0053] FIG. 1 is a schematic diagram of a system architecture in the present application;

[0054] FIG. 2 is another schematic diagram of a system architecture in the present application;

[0055] FIG. 3 is a schematic diagram of a performance evaluation method of a large model in the present application;

[0056] FIG. 4 is a schematic diagram of a test material in the present application;

[0057] FIG. 5 is a schematic diagram of acquiring scene-level information in the present application;

[0058] FIG. 6 is a schematic diagram of an object-labeled image in the present application;

[0059] FIGS. 7-9 are schematic diagrams of obtaining object-level information in the present application;

[0060] FIG. 10 is a schematic diagram of a region-labeled image in the present application;

[0061] FIG. 11 is a schematic diagram of obtaining region-level information in the present application;

[0062] FIG. 12 is a schematic diagram of evaluating answers based on a large language model in the present application;

[0063] FIG. 13 is another schematic diagram of a performance evaluation method of a large model in the present application;

[0064] FIG. 14 is a schematic diagram of evaluating answers based on a large language model in the present application;

[0065] FIG. 15 is another schematic diagram of test materials in the present application;

[0066] FIG. 16a is a schematic diagram of obtaining evaluation truth values based on a large language model in the present application;

[0067] FIG. 16b is a schematic diagram of obtaining answers of a large model to be tested to test questions in the present application;

[0068] FIG. 17 is a schematic diagram of the effect of a performance evaluation method of a large model in the present application;

[0069] FIG. 18 is a schematic diagram of a structure of a computing device in the present application;

[0070] FIG. 19 is another schematic diagram of a structure of a computing device in the present application. DETAILED DESCRIPTION

[0071] The embodiments of the present application are described below with reference to the accompanying drawings. It is obvious that the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Those skilled in the art can know that the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems as technology develops and new scenarios appear.

[0072] The terms "first", "second", and the like, as used in the specification and in the claims of the application, and in the above Description of Embodiments, unless otherwise specified, are used for distinguishing between similar objects talking about the same object are used to distinguish between similar objects talking about the same object and do not necessarily have to follow an intentionally set sequence. It is to be understood that the use of the terms so or so, data so or so, etc. in the description is not to be construed to limit the respective terms to data derived from a corresponding object, but rather have their common dictionary meaning to, for example, "identifying" or "designating" the object. Similarly, it is to be understood that the use of the descriptive term or terms "first", "second", etc. in the description, if any, are used for distinguishing between similar objects and are not necessarily to be taken in their normally understood sense, such that the

[0073] Referring to FIG. 1, FIG. 1 is a deployment diagram of an evaluation system according to the present disclosure, and the performance evaluation method of the large model in the present disclosure is executed by the evaluation system. The evaluation system can be deployed in a cloud environment, which is an entity that provides cloud services to users by utilizing underlying resources in a cloud computing mode. The cloud environment includes a cloud data center and a cloud service platform. The cloud data center includes a large number of underlying resources (including computing resources, storage resources, and network resources) owned by a cloud service provider. The computing resources included in the cloud data center can be a large number of computing devices (such as servers). The evaluation system can be independently deployed on a server or a virtual machine in the cloud data center. The evaluation system can also be distributedly deployed on multiple servers in the cloud data center, or distributedly deployed on multiple virtual machines in the cloud data center, or distributedly deployed on servers and virtual machines in the cloud data center. As shown in FIG. 1, the evaluation system is abstracted by the cloud service provider into an evaluation cloud service on the cloud service platform to provide users. After the user purchases the cloud service (which can be pre-recharged and settled according to the use of the final resources), the cloud environment provides the evaluation cloud service to the user by utilizing the evaluation system deployed in the cloud data center. The evaluation system can also be deployed in an edge environment, which refers to a collection of data centers or edge computing devices close to the user. The edge environment includes one or more edge computing devices. The evaluation system can be independently deployed on an edge computing device. The evaluation system can also be distributedly deployed on multiple edge servers, or distributedly deployed on multiple edge small stations with computing power, or distributedly deployed on edge servers and edge small stations with computing power. In addition, the evaluation system can also be deployed in other environments, such as a cluster of terminal computing devices. The evaluation system can be a software system running on a server or other computing device. Referring to FIG. 2, FIG. 2 is another deployment diagram of an evaluation system according to the present disclosure. As shown in FIG. 2, the evaluation system provided by the present disclosure can also be distributedly deployed in different environments. The evaluation system provided by the present disclosure can be logically divided into multiple parts, each part having a different function. Each part of the evaluation system can be deployed in any two or three of the terminal computing device, the edge environment, and the cloud environment. The terminal computing device includes: a terminal server, a smartphone, a notebook computer, a tablet computer, a personal desktop computer, a smart camera, and the like. The edge environment is an environment including a collection of edge computing devices close to the terminal computing device. The edge computing device includes: an edge server, an edge small station with computing power, and the like. The parts of the evaluation system deployed in different environments or devices cooperate to achieve the evaluation function. It should be understood that the present disclosure does not limit the division of the parts of the evaluation system to be deployed in specific environments. In actual applications, the evaluation system can be adaptively deployed according to the computing power of the terminal computing device, the resource occupation of the edge environment and the cloud environment, or specific application requirements.Of course, the evaluation system can also be deployed in a non-cloud environment as a toolkit for users to perform performance evaluation on large models.

[0074] Please refer to FIG. 3 for the following introduction of the flow of the performance evaluation method of the large model in the present application:

[0075] 301, obtain test materials and test questions, the test questions being questions raised based on the test materials;

[0076] The evaluation system first obtains test materials and test questions for testing the large model to be tested, wherein the test materials can be multi-modal information such as images, audio, etc., and the test questions are questions raised based on the test materials. The present application can accurately evaluate the performance of the large model to be tested in the face of an autonomous driving task. Please refer to FIG. 4, the test materials are, for example, an image of a traffic scene, and the test question is: "What is the driving suggestion in the traffic scene?".

[0077] 302, obtain a structured text description of the test materials, the structured text description including information of multiple dimensions of the test materials;

[0078] The evaluation system will obtain information of multiple dimensions of the test materials respectively according to a multi-modal large language model (MLLM), which includes scene-level information, object-level information, and region-level information, wherein the scene-level information includes a general description of the traffic scene, the object-level information includes a description of each object in each object category in the traffic scene, and the region-level information includes a description of a target region in the traffic scene.

[0079] The following describes the way in which the evaluation system obtains the above information:

[0080] 1. Obtain scene-level information

[0081] Please refer to FIG. 5, the evaluation system directly inputs the above test materials into the multi-modal large model, and instructs the multi-modal large model to describe the information in the image as much as possible. The output of the multi-modal large model is the scene-level information. For example: "On a city road in the daytime, the ego vehicle is driving forward. In front of it, a green bus occupies the lane. A silver SUV slightly ahead of the ego vehicle on the left side, the ego vehicle needs to observe whether it will change lanes. At the same time, there are some cyclists on the right side in front of the ego vehicle. There is a red and white fence on the right side of the road. The traffic light in front is currently green.".

[0082] 2. Obtain object-level information

[0083] The evaluation system inputs the test material into the visual pre-training model, and the visual pre-training model identifies and labels each object (such as a vehicle, a roadblock, etc.) in the test material. Then, the evaluation system obtains a plurality of object label images based on the labels made by the visual pre-training model, and the object label images are obtained by labeling each object in the same object category in the test material, and the object categories of the objects labeled in different object label images are different. For example, referring to FIG. 6, the evaluation system obtains object label image 1, object label image 2, and object label image 3, wherein the object label image 1 is obtained by labeling each vehicle in the test material, the object label image 2 is obtained by labeling each road user in the test material, and the object label image 3 is obtained by labeling each roadblock in the test material. Referring to FIG. 7, the evaluation system inputs the object label image 1 into the multi-modal large model, and instructs the multi-modal large model to describe the information in the label box in the image, and the output of the multi-modal large model is the description of each object in the object category of vehicle. For example: “Vehicle 1 description: a green bus directly in front of the ego vehicle, occupying the lane. Vehicle 1 explanation: the bus is a large, slow-moving vehicle that may block the ego vehicle's road and limit the forward view. Vehicle 2 description: a silver SUV is located on the left side of the ego vehicle, slightly ahead of the adjacent lane, Vehicle 2 explanation: the SUV may change lanes or maintain the current trajectory, and the ego vehicle needs to monitor to safely change lanes or turn.”

[0084] Referring to FIG. 8, the evaluation system inputs the object label image 2 into the multi-modal large model, and instructs the multi-modal large model to describe the information in the label box in the image, and the output of the multi-modal large model is the description of each object in the object category of road user, for example: “Road user 1 description: a cyclist appears on the right side of the ego vehicle, parallel to the car. Road user 1 explanation: the cyclist may behave unpredictably, so the ego vehicle must be ready to respond to their change of direction or entry into the lane.”

[0085] Referring to FIG. 9, the evaluation system inputs the object label image 3 into the multi-modal large model, and instructs the multi-modal large model to describe the information in the label box in the image, and the output of the multi-modal large model is the description of each object in the object category of roadblock, for example: “Fence 1 description: there is a red and white fence on the right side of the road, which demarcates a part of the road. Fence 1 explanation: these fences indicate that there may be a construction zone or lane closure, and care needs to be taken and the normal driving mode may need to be changed.”

[0086] The evaluation system combines the above descriptions of vehicles, road users, and roadblocks to form object-level information.

[0087] 3. Obtain region-level information

[0088] The evaluation system obtains region-annotated images based on annotations performed by a visual pre-trained model. These region-annotated images are obtained by annotating target regions in the test material. The target region can be the bounding box of an object in the image. For example, as shown in Figure 10, the target region could be the bounding box of a bus. Referring to Figure 11, the evaluation system inputs the region-annotated images into a multimodal large model, providing the location coordinates of the target region and instructing the multimodal large model to describe the information within the annotated boxes in the image. The output of the multimodal large model is region-level information, such as: "A large bus designed to carry multiple passengers. The vehicle is directly in front, indicating that the autonomous vehicle needs to maintain a safe following distance and be prepared to react appropriately when the bus stops or slows down, especially when passengers may be boarding or alighting."

[0089] The evaluation system combines information from the above multiple dimensions to obtain a structured text description.

[0090] The above describes how the evaluation system can obtain structured text descriptions on its own. In another possible implementation, users can also upload test materials, test questions, and corresponding structured text descriptions to the evaluation system themselves.

[0091] In one possible implementation, the evaluation system can also input the structured text description into a large language model, which analyzes the description to determine if any errors exist, such as contradictions between different dimensions of information. The evaluation system then corrects these errors, either manually or automatically.

[0092] 303. Input the structured text description and test questions into the large language model to obtain the evaluation truth value output by the large language model;

[0093] The evaluation system inputs structured text descriptions and test questions into a large language model, which then outputs ground truth values, representing the standard answers to the test questions. In this embodiment, the test question is: "What are the driving recommendations in this traffic scenario?", and the corresponding ground truth values ​​are: "The vehicle should maintain a safe distance from the bus, be prepared for the bus to stop, monitor for any lane-changing intentions from the SUV on the left, and be aware of the cyclist on the right. Since the traffic light is currently green, the vehicle can continue, but should remain cautious and slow down, and be prepared for possible changes in the traffic light, changes in cyclist behavior, and lane narrowing due to barriers."

[0094] 304. Input the test materials and test questions into the large model under test to obtain the test answers to the test questions output by the large model under test;

[0095] The evaluation system inputs test materials and test questions into the large-scale model under test. The large-scale model under test will output answers to the test questions, such as: "Maintain a safe distance from the bus, be aware of SUVs changing lanes on the left and cyclists on the right. When the traffic light is green, continue driving but slow down and be prepared to deal with traffic light changes, cyclist behavior, and possible lane narrowing."

[0096] 305. Input the evaluation truth, test question, and answer to be tested into the large language model to obtain the evaluation results of the large language model based on the evaluation truth and test question.

[0097] Referring to Figure 12, the evaluation system inputs the test question, the large model's response, and the evaluation truth values ​​into the large language model (e.g., a plain text version of GPT-4), and instructs the large language model to analyze the correctness of the large model's response based on the test question and the evaluation truth values, and to give a corresponding score. In this embodiment, the analysis result of the large language model is, for example: "The response of the model under test covers most of the key safety elements, such as maintaining a safe distance from the bus, monitoring the lane-changing intention of the SUV on the left, paying attention to the cyclist on the right, the status of traffic lights, and slowing down and preparing to deal with changes in traffic conditions, demonstrating an understanding of traffic safety rules. However, the response of the model under test is slightly lacking in richness and detail. For example, it does not explicitly mention 'preparing for the possible stopping of the bus,' which is an important safety consideration. Overall, the model accurately identifies and responds to most key safety factors, but lacks some detail and depth. Score (0-10 points): 7 points." In this application, the analysis result of the large language model can be in JSON format, which can be directly parsed.

[0098] It should be understood that the above performance evaluation process can be repeated. Multiple sets of test materials and test questions can be used to evaluate the large model under test, and the performance of the large model under test can be reflected by the final total score of the large model under test.

[0099] In this application, because the structured text description includes information from multiple dimensions of the test material, it is more conducive to mining and reflecting more key information and details in the test material, and the description is more hierarchical, which is more conducive to the understanding of large language models, thereby obtaining accurate evaluation truth values ​​and improving the accuracy of performance evaluation.

[0100] Please refer to Figure 13. The following describes another process for the performance evaluation method of the large model in this application:

[0101] 1301、obtain a test material and a test question, the test question being a question raised based on the test material;

[0102] The evaluation system first obtains a test material and a test question for testing the large model to be tested. For example, referring to FIG. 4, the test material is an image of a traffic scene, and the test question is “What is the driving suggestion in the traffic scene?”.

[0103] 1302、obtain a structured text description of the test material, the structured text description including information of multiple dimensions of the test material;

[0104] The evaluation system obtains information of multiple dimensions of the test material according to the multi-modal large model, which includes scene-level information, object-level information, and region-level information, which are similar to the embodiments described in FIG. 3 and will not be described here.

[0105] 1303、input the test material and the test question into the large model to be tested to obtain a test answer to the test question output by the large model to be tested;

[0106] The evaluation system inputs the test material and the test question into the large model to be tested, and the large model to be tested outputs an answer to the test question, for example: “Keep a safe distance from the bus, pay attention to the left SUV changing lanes and the right cyclist. When the traffic light is green, continue driving but slow down and prepare to respond to traffic light changes, cyclist behavior, and possible lane narrowing.”

[0107] 1304、input the structured text description, the test question, and the test answer into the large language model to obtain an evaluation result of the large language model on the test answer based on the structured text description and the test question.

[0108] Referring to FIG. 14, unlike the embodiment shown in FIG. 3, in this embodiment, the test question, the answer to the test question by the large model to be tested, and the structured text description are directly input into the large language model (such as the pure text version of GPT-4), and the large language model is instructed to analyze the correctness of the answer of the large model to be tested according to the structured text description and the test question, and give a corresponding score. It should be understood that the performance evaluation process described above can be repeated, and a plurality of test materials and test questions can be used to evaluate the large model to be tested, and the final total score of the large model to be tested can reflect the performance of the large model to be tested.

[0109] In the present application, since the structured text description includes information of multiple dimensions of the test material, it is more beneficial to mine and reflect more key information and details in the test material, and the description is more hierarchical, which is more beneficial to the understanding of the large language model, thereby improving the accuracy of performance evaluation.

[0110] The above describes the flow of the application in the automatic driving scene. Please refer to steps A01 to A05. The following describes the flow of the application in the geometry problem analysis scene:

[0111] A01, obtain test material and test question, the test question is a question based on the test material;

[0112] Please refer to FIG. 15. In the present embodiment, the test material is an image of a geometry problem, and the test question is: “Circle O is the circumcircle of triangle ABC, and angle ABO = 30 degrees. What is the size of angle ACB?”.

[0113] A02, obtain the structured text description of the test material, which includes information of multiple dimensions of the test material;

[0114] The evaluation system can also obtain the structured description text of the geometry problem. The structured description text may, for example, be obtained by professional annotators annotating the geometry problem, including but not limited to the attributes of the figure (such as straight line, circle, polygon, etc.), the positional relationship (such as parallel, perpendicular, tangent, etc.), and the application of geometric theorems or formulas. By drawing figures, writing logical explanation texts, and citing relevant geometric theorems or formulas, a set of structured description texts of geometry problems are constructed. The structured description text includes, for example, geometry element information and geometry relationship information. The geometry element information includes the geometry elements appearing in the geometry problem, for example:

[0115] Circle (O)

[0116] Triangle (A, B, C)

[0117] Triangle (A, O, B)

[0118] The geometry relationship information includes the geometry relationship between the geometry elements, for example:

[0119] Point Lies On circle (A, circle (O))

[0120] Indicates that point A is on circle O

[0121] Point Lies On circle (B, circle (0))

[0122] Indicates that point B is on circle O

[0123] Point Lies On circle(c,circle(O))

[0124] This indicates that point C lies on circle O.

[0125] A03. Input the structured text description and test questions into the large language model to obtain the true evaluation values ​​output by the large language model.

[0126] Please refer to Figure 16a. The evaluation system inputs the structured text description and test question into the large language model, and instructs the large language model to describe information relevant to the test question as much as possible based on the structured text description. The output of the large language model is the evaluation truth value, for example: "This image consists of a triangle ABC, which is inscribed in a circle, labeled circle 0. Points A, B, and C are the vertices of the triangle, all located on the circumference of the circle. The center of the circle is labeled point 0. Another triangle AOB is formed by the center point O and points A and B."

[0127] A04. Input the test materials and test questions into the large model to be tested, and obtain the test answers of the large model to be tested for the test questions.

[0128] Please refer to Figure 16b. The evaluation system inputs the test materials and test questions into the large model under test. The large model under test will output the answers to the test questions, such as: "In triangle AOB, OA = OB, angle ABO = 30°: therefore, angle AOB = 180 - 2 × angle ABO = 120°: therefore, angle ACB = 1 / 2 × angle AOB = 60°."

[0129] A05. Input the evaluation truth, test question, and answer to be tested into the large language model to obtain the evaluation results of the large language model based on the evaluation truth and test question.

[0130] The evaluation system inputs the test question, the answer of the to-be-tested large model to the test question, and the evaluation true value into a large language model (e.g., a pure text version of GPT-4), and instructs the large language model to analyze the correctness of the answer of the to-be-tested large model according to the test question and the evaluation true value, and give a corresponding score. In this embodiment, the analysis result of the large language model is, for example: "First, the to-be-tested large model points out that in triangle AOB, OA = OB. This is correct because OA and OB are both radii of circle O. The to-be-tested large model mentions that angle ABO is 30°, which is consistent with the test question. Next, the to-be-tested large model calculates the size of angle A0B as 180°-2x angle ABO. This calculation is based on the fact that in an isosceles triangle AOB, the base angles (angle ABO and angle BAO) are equal, and the sum of the interior angles of the triangle is 180°. Since angle ABO is 30°, angle A0B is indeed 180°-2x30°=120°. Finally, the to-be-tested large model claims that angle ACB is half of angle AOB, i.e., 60°. This conclusion is based on the theorem that a central angle is half of its corresponding arc angle. In this case, angle AOB is the central angle of angle ACB, so angle ACB is indeed half of angle AOB, i.e., 60°. Correctness score: 1 point."

[0131] In this application, since the structured text description includes information of multiple dimensions of the test material, it is more conducive to mine and reflect more key information and details in the test material, and the description is more hierarchical, which is more conducive to the understanding of the large language model, and thus the accurate evaluation true value is obtained, and the accuracy of performance evaluation is improved.

[0132] Please refer to steps B01 to B04, and the following will introduce another flow of the application in the geometric question analysis scene:

[0133] B01, obtain the test material and the test question, the test question is a question based on the test material;

[0134] This step is similar to the aforementioned step A01, which will not be described here.

[0135] B02, obtain the structured text description of the test material, the structured text description includes information of multiple dimensions of the test material;

[0136] This step is similar to the aforementioned step A02, which will not be described here.

[0137] B03, input the test material and the test question into the to-be-tested large model to obtain the to-be-tested answer of the to-be-tested large model to the test question;

[0138] The evaluation system inputs the test material and the test question into the large model to be tested, and the large model to be tested outputs an answer to the test question, for example: in triangle AOB, OA = OB, angle ABO = 30°: therefore, angle AOB = 180-2x angle ABO = 120°: therefore, angle ACB = 1 / 2x angle AOB = 60°.

[0139] B04, input the structured text description, the test question and the answer to be tested into the large language model to obtain the evaluation result of the large language model based on the evaluation true value and the test question to the answer to be tested.

[0140] Unlike the foregoing embodiments, in this embodiment, the evaluation true value is not needed to be obtained, but the test question, the answer of the large model to be tested to the test question and the evaluation true value are directly input into the large language model (such as the pure text version of GPT-4), and the large language model is instructed to analyze the correctness of the answer of the large model to be tested according to the test question and the structured text description, and give the corresponding score.

[0141] In this application, since the structured text description includes the information of multiple dimensions of the test material, it is more conducive to mine and reflect more key information and details in the test material, and the description is more hierarchical, which is more conducive to the understanding of the large language model, and thus improves the accuracy of performance evaluation.

[0142] In this application, when the evaluation system evaluates the performance of the model to be tested in different task scenarios (such as autonomous driving and geometric problems), the prompt engineering suitable for the current task scenario can be automatically selected, such as a customized prompt template, thereby optimizing the accuracy of evaluation.

[0143] Referring to FIG. 17, based on the method provided in this application, the performance of each large model in each aspect can be quickly and accurately evaluated, including but not limited to general perception, regional perception, driving suggestions, all, vehicles, vuinerable road users, traffic cones, traffic lights, traffic signs, barriers and miscellaneous.

[0144] The method in this application is introduced above, and the computing device in this application is introduced below:

[0145] Referring to FIG. 18, the computing device 1800 in the application includes an acquisition unit 1801 and a processing unit 1802.

[0146] The acquisition unit 1801 is configured to acquire test material and a test question, the test question being a question raised based on the test material.

[0147] The acquisition unit 1801 is further configured to acquire a structured text description of the test material, the structured text description including information of multiple dimensions of the test material.

[0148] The processing unit 1802 is configured to input the structured text description and the test question into a large language model to acquire an evaluation true value output by the large language model, the evaluation true value including information related to the test question in the test material or being a standard answer to the test question.

[0149] The processing unit 1802 is further configured to input the test material and the test question into a to-be-tested large model to acquire a to-be-tested answer to the test question output by the to-be-tested large model.

[0150] The processing unit 1802 is further configured to input the evaluation true value, the test question and the to-be-tested answer into the large language model to acquire an evaluation result of the to-be-tested large model on the to-be-tested answer based on the evaluation true value and the test question.

[0151] In a possible implementation manner,

[0152] The processing unit 1802 is further configured to verify the structured text description according to the large language model to determine a description error existing in the structured text description, the description error including a contradiction between information of multiple dimensions.

[0153] The processing unit 1802 is further configured to correct the description error in the structured text description.

[0154] The processing unit 1802 is specifically configured to input the corrected structured text description and the test question into the large language model to acquire the evaluation true value output by the large language model.

[0155] In a possible implementation manner, the test material is an image of a traffic scene, the information of multiple dimensions includes scene-level information, object-level information and region-level information, the scene-level information including an overall description of the traffic scene, the object-level information including a description of each object in each object category in the traffic scene, and the region-level information including a description of a target region in the traffic scene.

[0156] In a possible implementation manner,

[0157] The acquisition unit 1801 is specifically configured to input the test material into the multi-modal large model to acquire scene-level information output by the multi-modal large model.

[0158] The acquisition unit 1801 is specifically configured to acquire a plurality of object annotation images, the object annotation images being obtained by annotating each object in a same object category in the test material, and the object categories to which the objects in different object annotation images belong being different.

[0159] The acquisition unit 1801 is specifically configured to input the plurality of object annotation images into the multi-modal large model in sequence to acquire object-level information output by the multi-modal large model.

[0160] The acquisition unit 1801 is specifically configured to acquire a region annotation image, the region annotation image being obtained by annotating a target region in the test material.

[0161] The acquisition unit 1801 is specifically configured to input the region annotation image and the position coordinates of the target region into the multi-modal large model to acquire region-level information output by the multi-modal large model.

[0162] In a possible implementation, the object categories include vehicles, road users, and roadblocks.

[0163] In a possible implementation,

[0164] The processing unit 1802 is further configured to input the test material into the visual pre-training model to acquire annotation information output by the visual pre-training model, the annotation information including annotation of objects in the test material.

[0165] The object annotation images and the region annotation image are acquired based on the annotation information.

[0166] In a possible implementation, the test material is an image of a geometry question, the information of the plurality of dimensions includes geometry element information and geometry relationship information, the geometry element information includes geometry elements appearing in the geometry question, and the geometry relationship information includes geometry relationships between the geometry elements.

[0167] FIG. 19 is a structural schematic diagram of a computing device provided in the present application, which is used to implement the method in the foregoing various embodiments. The computing device 1900 can include one or more central processing units (CPUs) 1901 and a memory 1905 in which one or more application programs or data are stored.

[0168] The memory 1905 can be volatile memory or non-volatile memory. The program stored in the memory 1905 can include one or more modules, each of which can include a series of instruction operations in the server. Further, the central processing unit 1901 can be configured to communicate with the memory 1905 to execute the series of instruction operations in the memory 1905 on the computing device 1900. The computing device 1900 can also include one or more power supplies 1902, one or more wired or wireless network interfaces 1903, one or more input / output interfaces 1904, and / or one or more operating systems.

[0169] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0170] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0171] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0172] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0173] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, read-only memory), a random access memory (RAM, random access memory), a magnetic disk or an optical disk, and various media that can store program codes.

Claims

1. A large model performance evaluation method, characterized in that, The method comprises: obtaining test material and a test question, the test question being a question raised based on the test material; obtaining a structured text description of the test material, the structured text description comprising information of multiple dimensions of the test material; inputting the structured text description and the test question into a large language model to obtain an evaluation truth value output by the large language model, the evaluation truth value comprising information related to the test question in the test material or a standard answer to the test question; inputting the test material and the test question into a large model under test to obtain a test answer to the test question output by the large model under test; inputting the evaluation truth value, the test question and the test answer into a large language model to obtain an evaluation result of the large language model on the test answer based on the evaluation truth value and the test question.

2. The method of claim 1, wherein, The method further comprises: verifying the structured text description according to the large language model to determine a description error existing in the structured text description, the description error comprising a contradiction between the information of the multiple dimensions; correcting the description error in the structured text description; The inputting the structured text description and the test question into a large language model to obtain an evaluation truth value output by the large language model comprises: inputting the corrected structured text description and the test question into the large language model to obtain the evaluation truth value output by the large language model.

3. The method according to claim 1 or 2, characterized in that, The test material is an image of a traffic scene, and the information of the multiple dimensions comprises scene-level information, object-level information and region-level information, the scene-level information comprising a holistic description of the traffic scene, the object-level information comprising a description of each object in each object category in the traffic scene, and the region-level information comprising a description of a target region in the traffic scene.

4. The method of claim 3, wherein, The obtaining a structured text description of the test material comprises: inputting the test material into a multi-modal large model to obtain scene-level information output by the multi-modal large model; obtaining a plurality of object annotation images, the object annotation images being obtained by annotating each object in a same object category in the test material, and the object categories of the objects annotated in different object annotation images being different; inputting the plurality of object annotation images into the multi-modal large model in sequence to obtain object-level information output by the multi-modal large model; obtaining a region annotation image, the region annotation image being obtained by annotating a target region in the test material; inputting the region annotation image and position coordinates of the target region into the multi-modal large model to obtain region-level information output by the multi-modal large model.

5. The method of claim 4, wherein, The object categories comprise vehicles, road users and road barriers.

6. The method of claim 5, wherein, The method further comprises: input the test material into the visual pre-training model to obtain annotation information output by the visual pre-training model, the annotation information including annotation of an object in the test material; the object annotation image and the region annotation image are obtained based on the annotation information.

7. The method of claim 2, wherein, the test material is an image of a geometry question, and the information of multiple dimensions includes geometry element information and geometry relationship information, the geometry element information including a geometry element appearing in the geometry question, and the geometry relationship information including a geometry relationship between the geometry elements.

8. A computing device, comprising: comprise: an obtaining unit, configured to obtain test material and a test question, the test question being a question raised based on the test material; the obtaining unit is further configured to obtain a structured text description of the test material, the structured text description including information of multiple dimensions of the test material; a processing unit, configured to input the structured text description and the test question into a large language model to obtain evaluation truth value output by the large language model, the evaluation truth value including information in the test material related to the test question or being a standard answer to the test question; the processing unit is further configured to input the test material and the test question into a to-be-tested large model to obtain a to-be-tested answer to the test question output by the to-be-tested large model; the processing unit is further configured to input the evaluation truth value, the test question, and the to-be-tested answer into the large language model to obtain an evaluation result of the to-be-tested large model on the to-be-tested answer based on the evaluation truth value and the test question.

9. The device of claim 8, wherein the processing unit is further configured to verify the structured text description according to the large language model to determine a description error existing in the structured text description, the description error including a contradiction between the information of multiple dimensions; the processing unit is further configured to correct the description error in the structured text description; the processing unit is specifically configured to input the corrected structured text description and the test question into the large language model to obtain the evaluation truth value output by the large language model.

10. The apparatus of claim 8 or 9, characterized in that, the test material is an image of a traffic scene, and the information of multiple dimensions includes scene-level information, object-level information, and region-level information, the scene-level information including an overall description of the traffic scene, the object-level information including a description of each object in each object category in the traffic scene, and the region-level information including a description of a target region in the traffic scene.

11. The device of claim 10, wherein the obtaining unit is specifically configured to input the test material into a multi-modal large model to obtain the scene-level information output by the multi-modal large model; The acquisition unit is specifically configured to acquire a plurality of object-labeled images, the object-labeled images being obtained by labeling each object in a same object category in the test material, and the object-labeled images being different in object category of the labeled objects; The acquisition unit is specifically configured to sequentially input the plurality of object-labeled images into the multi-modal large model to acquire the object-level information output by the multi-modal large model; The acquisition unit is specifically configured to acquire a region-labeled image, the region-labeled image being obtained by labeling a target region in the test material; The acquisition unit is specifically configured to input the region-labeled image and position coordinates of the target region into the multi-modal large model to acquire the region-level information output by the multi-modal large model.

12. The apparatus of claim 11, wherein, The object categories include vehicles, road users, and roadblocks.

13. The device of claim 12, wherein, The processing unit is further configured to input the test material into a visual pre-training model to acquire labeled information output by the visual pre-training model, the labeled information including labeling of objects in the test material; The object-labeled images and the region-labeled image are acquired based on the labeled information.

14. The apparatus of claim 9, wherein, The test material is an image of a geometry problem, and the information of the plurality of dimensions includes geometry element information and geometry relationship information, the geometry element information including geometry elements appearing in the geometry problem, and the geometry relationship information including geometry relationships between the geometry elements.

15. A computing device, comprising: comprises a processor and a memory; The processor is configured to execute instructions stored in the memory to cause the computing device to perform the method of any one of claims 1-7.

16. A computer program product comprising instructions, characterized in that, The instructions, when executed by a computing device, cause the computing device to perform the method of any one of claims 1-7.

17. A computer-readable storage medium, characterized in that, comprise computer program instructions, which, when executed by a computing device cluster, cause the computing device cluster to perform the method of any one of claims 1-7.

Images