Method and device for evaluating large model in vehicle field, and electronic equipment

Abstract

The invention belongs to the field of vehicles, and discloses an evaluation method and device for a vehicle field large model and electronic equipment, and the method comprises the steps: calling the vehicle field large model to answer each test question in an evaluation test question set; performing multi-dimensional scoring on the vehicle field large model based on each solution result to obtain a multi-dimensional scoring result; and determining a final score of the vehicle field large model according to a multi-dimensional scoring result. According to the method, detailed multi-dimensional scoring is carried out according to the solution result of the large model in the vehicle field, four key dimensions of quality, performance, responsibility and specialty are covered, the application capability of the large model in the vehicle field can be comprehensively evaluated, and the evaluation efficiency of the large model in the vehicle field can be improved by comprehensively considering the scoring result of each dimension. According to the method, a more accurate and reliable final score is determined, the problems that a traditional evaluation method is high in subjectivity and lacks unified standards and index relevance are solved, and important technical support is provided for research, development and application of a large model in the field of vehicles.

Description

Technical Field

[0001] The present invention relates to the field of vehicles, and in particular to an evaluation method, device, electronic device and computer-readable storage medium for a large model in the field of vehicles. Background Art

[0002] With the rapid development of artificial intelligence technology, large language models (LLMs) have been widely used in various fields, especially in the automotive industry. As a technology-intensive industry, the automotive industry has an increasing demand for technologies such as intelligence, intelligent driving, and vehicle networking, and the demand for Chinese large models has also increased accordingly. However, despite the huge application potential of large models in the automotive field, the existing large model evaluation benchmarks do not cover the needs of the automotive industry. They are mainly aimed at the evaluation of general capabilities, which has led to a lack of public evaluation standards for the evaluation of large model capabilities in the industry.

[0003] The existing large-scale model evaluation methods in the vehicle field lack recognized accurate indicators and standardized processes, and the scoring system relies too much on subjective evaluation and lacks objectivity, making the scores easily affected by personal preferences and extreme values, thus lacking stability. These problems lead to poor accuracy and reliability of the evaluation results.

[0004] Therefore, how to improve the accuracy and reliability of the evaluation results of large models in the vehicle field is a technical problem that those skilled in the art need to solve. Summary of the invention

[0005] In order to solve the problem of low accuracy and reliability of evaluation results of large vehicle models in the prior art, the present invention provides an evaluation method, device, electronic device and computer-readable storage medium for large vehicle models.

[0006] A method for evaluating a large model in a vehicle domain, comprising:

[0007] Obtaining an evaluation question set; the evaluation question set includes a plurality of questions in the vehicle field;

[0008] Calling the vehicle domain big model to answer each question in the evaluation question set and obtain corresponding answer results;

[0009] Based on each of the answer results, the vehicle domain big model is scored in multiple dimensions to obtain a multi-dimensional scoring result; the multi-dimensional scoring result includes at least one of a quality dimension scoring result, a performance dimension scoring result, a responsibility dimension scoring result, and a professional dimension scoring result;

[0010] The final score of the vehicle domain large model is determined according to the multi-dimensional scoring results.

[0011] Optionally, the vehicle domain big model is scored in multiple dimensions based on each of the solution results to obtain a multi-dimensional scoring result, including:

[0012] For each dimension among all dimensions, determine the indicator scores of all indicators in the current dimension according to each of the solution results;

[0013] Calculate a first benchmark value of the current dimension according to the indicator scores of all the indicators;

[0014] Calculating a first difference between each of the indicator scores and the first benchmark value;

[0015] Determine a first weight value of each of the indicators in the current dimension according to the first difference;

[0016] The dimension score of the current dimension is calculated according to each of the indicator scores and the corresponding first weight value.

[0017] Optionally, calculating the first benchmark value of the current dimension according to the indicator scores of all the indicators includes:

[0018] The first reference value is determined according to the formula:

[0019]

[0020] Among them, d i is the first reference value of the i-th dimension, n i is the number of indicators in the i-th dimension, x im Score the indicator for the mth indicator in the i-th dimension;

[0021] The calculating of the first difference between each of the indicator scores and the first benchmark value correspondingly includes:

[0022] The first difference is determined according to the formula:

[0023] d im =x im -d i

[0024] Among them, d im is the first difference between the indicator score of the mth indicator in the i-th dimension and the first benchmark value;

[0025] The determining, according to the first difference, a first weight value of each indicator in the current dimension correspondingly includes:

[0026] The first weight value is determined according to the formula:

[0027] w im ′=1 / d im

[0028]

[0029] Among them, w im ′ is the original weight value of the mth indicator in the ith dimension, w im is the first weight value of the mth indicator in the i-th dimension;

[0030] The step of calculating the dimension score of the current dimension according to each indicator score and the corresponding first weight value includes:

[0031] The dimension score for the current dimension is determined according to the formula:

[0032]

[0033] Among them, C i Give the dimension score for the i-th dimension.

[0034] Optionally, determining the final score of the vehicle domain large model according to the multi-dimensional scoring results includes:

[0035] Calculating a second benchmark value based on the dimension score of each dimension;

[0036] Calculating a second difference between the dimension score of each dimension and the second benchmark value;

[0037] Determine a second weight value for each dimension according to the second difference;

[0038] The final score of the vehicle domain large model is determined according to the dimension score of each dimension and the corresponding second weight value.

[0039] Optionally, calculating the second benchmark value according to the dimension score of each dimension includes:

[0040] The second reference value is determined according to the formula:

[0041]

[0042] Among them, e is the second reference value, j is the number of dimensions, C i Score the dimension of the i-th dimension;

[0043] The calculating of the second difference between the dimension score of each dimension and the second reference value correspondingly includes:

[0044] The second difference is determined according to the formula:

[0045] e i =C i -e

[0046] Among them, e i is the first difference between the dimension score of the i-th dimension and the first benchmark value;

[0047] Determining the second weight value of each dimension according to the second difference value correspondingly includes:

[0048] The second weight value is determined according to the formula:

[0049] f i ′=1 / e i

[0050]

[0051] Among them, f i ′ is the original weight value of the i-th dimension, f i is the second weight value of the i-th dimension;

[0052] The final score of the vehicle domain large model is determined according to the dimension score of each dimension and the corresponding second weight value, and correspondingly includes:

[0053] The final score of the vehicle domain model is determined according to the formula:

[0054]

[0055] Among them, ||X|| is the L2 norm of the dimension score, and G is the final score of the large model in the vehicle field.

[0056] Optionally, the dimension includes at least one of a quality dimension, a performance dimension, a responsibility dimension, and a professional dimension, wherein:

[0057] The indicators in the quality dimension include at least one of context comprehension ability, prompt comprehension ability, amount of Chinese training data, and maximum context size;

[0058] Indicators within the performance dimension include average response speed and / or first sentence response time;

[0059] The indicators in the responsibility dimension include at least one of the size of the illusion window, the illusion ratio, the risk content, and compliance;

[0060] The indicators within the professional dimension include at least one of the following: vehicle model and series proficiency, vehicle speech proficiency, vehicle field basic knowledge proficiency, and vehicle industry dialogue comprehension ability.

[0061] Optionally, obtaining the assessment question set includes:

[0062] In response to the input search request, searching for documents and test questions in the vehicle field; the documents include at least one of journals, papers, and expert guidelines;

[0063] Calling a preset language model to generate a first evaluation question according to the document;

[0064] Calling the preset language model to screen and modify the test questions to obtain second evaluation test questions;

[0065] The assessment question set is generated according to the first assessment question and the second assessment question.

[0066] An evaluation device for a large vehicle model, comprising:

[0067] An acquisition module, used to acquire an evaluation question set; the evaluation question set includes a plurality of questions in the vehicle field;

[0068] An answering module, used for calling the vehicle domain large model to answer each question in the evaluation question set and obtain corresponding answer results;

[0069] A first scoring module is used to perform multi-dimensional scoring on the vehicle domain big model based on each of the answer results to obtain a multi-dimensional scoring result; the multi-dimensional scoring result includes at least one of a quality dimension scoring result, a performance dimension scoring result, a responsibility dimension scoring result, and a professional dimension scoring result;

[0070] The second scoring module is used to determine the final score of the vehicle domain large model according to the multi-dimensional scoring results.

[0071] An electronic device, comprising:

[0072] A processor and a memory, wherein the memory is used to store at least one instruction, and when the instruction is loaded and executed by the processor, the method for evaluating a large vehicle model as described in any one of the above is implemented.

[0073] A computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the evaluation method of a large vehicle domain model as described in any one of the above items is implemented.

[0074] The evaluation method of the vehicle field big model provided by the embodiment of the present invention is to call the vehicle field big model to answer each question in the evaluation test set, and perform multi-dimensional scoring on the vehicle field big model based on each answer result to obtain a multi-dimensional scoring result; finally, the final score of the vehicle field big model is determined according to the multi-dimensional scoring result. The present invention significantly improves the accuracy and objectivity of the evaluation through a systematic process and a multi-dimensional scoring mechanism. The method first constructs an evaluation test set containing multiple vehicle field questions, then uses the big model to answer these questions, and performs detailed multi-dimensional scoring based on the answer results, covering four key dimensions of quality, performance, responsibility and professionalism. The present invention can not only comprehensively evaluate the application ability of the big model in the vehicle field, but also determine a more accurate and reliable final score by comprehensively considering the scoring results of each dimension, solving the problems of strong subjectivity, lack of unified standards and indicator correlation in traditional evaluation methods, and providing important technical support for the research and development and application of big models in the vehicle field. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0076] Figure 1 A flow chart of a method for evaluating a large vehicle model provided by an embodiment of the present invention;

[0077] Figure 2 for Figure 1 A flowchart of a practical expression of S03 in a method for evaluating a large vehicle model is provided;

[0078] Figure 3 for Figure 1 A flowchart of a practical expression of S04 in a method for evaluating a large model in the vehicle field is provided;

[0079] Figure 4 for Figure 1 A flowchart of a practical representation of S01 in a method for evaluating a large model in the vehicle field is provided;

[0080] Figure 5 A schematic diagram of the structure of a large vehicle model evaluation device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0081] In order to better understand the technical solution of the present invention, the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0082] It should be clear that the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0083] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0084] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0085] With the rapid development of artificial intelligence technology, large language models have been widely used in various fields, especially in the automotive industry. As a technology-intensive industry, the automotive industry has an increasing demand for technologies such as intelligence, smart driving, and vehicle networking, and the demand for large Chinese models has also increased accordingly. However, despite the huge application potential of large models in the automotive field, the existing large model evaluation benchmarks do not cover the needs of the automotive industry. They are mainly aimed at the evaluation of general capabilities, which has led to a lack of public evaluation standards for the evaluation of large model capabilities in the industry.

[0086] The existing large-scale model evaluation methods in the vehicle field lack recognized accurate indicators and standardized processes, and the scoring system relies too much on subjective evaluation and lacks objectivity, making the scores easily affected by personal preferences and extreme values, thus lacking stability. These problems lead to poor accuracy and reliability of the evaluation results. Therefore, the present invention provides an evaluation method for large-scale models in the vehicle field to solve the above problems.

[0087] Please refer to Figure 1 , which is a flow chart of a method for evaluating a large vehicle model provided by an embodiment of the present invention, comprising the following steps:

[0088] Step S01, obtaining an assessment question set.

[0089] In this embodiment, before evaluating the large model in the vehicle field, it is necessary to first collect and construct a test question set specifically for the vehicle field. This set contains multiple professional test questions related to vehicles, which provides a standardized test basis for the evaluation and ensures that the evaluation can cover professional knowledge and skill points in the vehicle field. In this way, the ability of the large model to understand and apply vehicle field knowledge can be systematically tested and evaluated, thereby ensuring the accuracy and reliability of the evaluation results.

[0090] In some embodiments, the process of obtaining the assessment question set may specifically include:

[0091] Extract key knowledge points from resources such as professional literature, technical manuals, industry standards, journal articles, expert guides, etc. in the vehicle field, and convert these knowledge points into specific test questions. This includes determining the scope and difficulty of the test questions to ensure that they can comprehensively cover all aspects of the vehicle field, such as mechanical engineering, electronic systems, safety standards, market analysis, etc. Next, organize these test questions into a structured set, each of which is designed to test the understanding and reasoning ability of the big model in a specific field. In addition, the construction of the evaluation test set can also cooperate with domain experts to ensure the accuracy and practicality of the test questions, as well as design test questions through actual cases or simulation scenarios to more realistically simulate the problems that the big model may encounter in actual applications. Such a test set provides a standardized and systematic basis for evaluating the performance of big models in the vehicle field.

[0092] Step S02, call the vehicle domain large model to answer each question in the evaluation test set and obtain the corresponding answer result.

[0093] In this embodiment, the vehicle domain big model refers to a large artificial intelligence model that is specially trained for automotive industry data and knowledge. This model is usually based on deep learning technology and can process and understand a large amount of vehicle-related data. These data may include technical manuals, user manuals, maintenance guides, market research reports, customer feedback, vehicle performance data, etc. The purpose of the vehicle domain big model is to provide in-depth understanding and intelligent response to the vehicle domain, so that it can play a role in a variety of application scenarios, such as vehicle diagnosis, customer service, market analysis, product design and development, etc.

[0094] This embodiment uses the vehicle domain big model to answer each question in the evaluation test set, with the purpose of testing and evaluating the big model's knowledge mastery and problem-solving ability in the vehicle domain. Since the vehicle domain involves complex technology and professional knowledge, by letting the big model answer professional questions, its knowledge mastery and problem-solving ability in the vehicle domain can be intuitively measured. The output of the direct evaluation model can objectively reflect its understanding and application of vehicle domain knowledge, thereby ensuring the accuracy and reliability of the evaluation results and providing valuable feedback for the development and optimization of the big model in the vehicle domain.

[0095] Step S03, based on each answer result, the vehicle domain large model is scored in multiple dimensions to obtain a multi-dimensional scoring result.

[0096] Among them, the multi-dimensional scoring results include at least one of the quality dimension scoring results, performance dimension scoring results, responsibility dimension scoring results and professional dimension scoring results.

[0097] In this embodiment, the comprehensive performance of the model in terms of accuracy, efficiency, compliance, and professional knowledge is evaluated by analyzing the answers of the large model in the vehicle field to a series of evaluation questions. Specifically, the quality dimension score focuses on the accuracy and reliability of the answer; the performance dimension score focuses on the efficiency of the model in handling problems; the responsibility dimension score focuses on the compliance and ethics of the answer; and the professional dimension score focuses on the model's professional knowledge and skills in the vehicle field. This multi-dimensional scoring method can provide a more comprehensive and in-depth perspective to evaluate the actual application capabilities of the large model in the vehicle field.

[0098] In some embodiments, the multi-dimensional scoring of the vehicle domain large model can be achieved by executing the following steps:

[0099] 1) Set evaluation criteria: First, set specific evaluation criteria for the four dimensions of quality, performance, responsibility, and expertise. For example, the quality dimension may focus on the accuracy and reliability of the answer; the performance dimension may focus on the response speed and processing capability of the model; the responsibility dimension may focus on the compliance and ethics of the answer; and the expertise dimension may focus on the model's mastery of vehicle domain knowledge.

[0100] 2) Scoring each question: For each question in the evaluation test set, the vehicle domain big model is used to generate an answer, and then each answer is scored based on the evaluation criteria of the above dimensions combined with the scoring rules (such as a 1-5 point system). This process can be completed through manual expert review or automatic scoring using the super model as a referee.

[0101] 3) Calculate dimension scores: Calculate the scores of all questions under each dimension and get the average score of the dimension, which represents the performance of the model in this dimension.

[0102] Step S04, determining the final score of the vehicle domain large model according to the multi-dimensional scoring results.

[0103] In this embodiment, after scoring the answers of the large model in the vehicle field in terms of quality, performance, responsibility, and expertise, an overall score is calculated based on these scoring results. This final score is a quantitative assessment of the comprehensive capabilities of the model in the vehicle field, which reflects the performance and reliability of the model in actual applications. In this way, the performance of the large model can be evaluated more comprehensively to ensure that it can meet industry needs and standards in actual deployment and use.

[0104] This embodiment not only considers the performance of the model in a single dimension, but also combines the scores of multiple key dimensions such as quality, performance, responsibility and professionalism to more accurately measure the actual application capabilities of the model. In this way, it can be ensured that the application of large models in the vehicle field is not only technically qualified, but also reliable and trustworthy at the ethical, legal and professional levels, which is crucial to improving the practicality of the model and the trust of users.

[0105] Based on the above technical solution, the evaluation method of the vehicle field big model provided by the embodiment of the present invention is to call the vehicle field big model to answer each question in the evaluation test set, and perform multi-dimensional scoring on the vehicle field big model based on each answer result to obtain the multi-dimensional scoring result; finally, the final score of the vehicle field big model is determined according to the multi-dimensional scoring result. The present invention significantly improves the accuracy and objectivity of the evaluation through a systematic process and a multi-dimensional scoring mechanism. The method first constructs an evaluation test set containing multiple vehicle field questions, then uses the big model to answer these questions, and performs detailed multi-dimensional scoring based on the answer results, covering four key dimensions of quality, performance, responsibility and professionalism. The present invention can not only comprehensively evaluate the application ability of the big model in the vehicle field, but also determine a more accurate and reliable final score by comprehensively considering the scoring results of each dimension, solving the problems of strong subjectivity, lack of unified standards and indicator correlation in traditional evaluation methods, and providing important technical support for the research and development and application of big models in the vehicle field.

[0106] Please refer to Figure 2 ,for Figure 1 A flowchart of an actual performance of S03 in a method for evaluating a large vehicle model is provided. In some embodiments, step S03 mentioned that the large vehicle model is scored in multiple dimensions based on each answer result to obtain a multi-dimensional scoring result, which may specifically include the following steps:

[0107] Step S11, for each dimension among all dimensions, determine the indicator scores of all indicators in the current dimension according to each solution result.

[0108] In this embodiment, step S11 is the process of quantifying the performance of the large model in the vehicle field in each evaluation dimension, which involves scoring the answers of the model in various dimensions such as quality, performance, responsibility and expertise to measure the performance of the model in these dimensions. Each dimension contains several specific indicators, such as accuracy, response speed, compliance, etc. The model's answer or performance to each indicator will receive a score based on the preset evaluation criteria. This step is the basis of the multi-dimensional scoring system, which ensures the comprehensiveness and meticulousness of the evaluation and provides the necessary data support for the subsequent weight allocation and final scoring.

[0109] Due to the complexity of the vehicle field, the model is required to have a high level of performance in many aspects, including but not limited to accuracy, response speed, compliance, and professional knowledge. Therefore, this embodiment can ensure that the evaluation process does not ignore any key aspects by scoring the indicators in each dimension, thereby more accurately reflecting the actual application capabilities of the model, so as to achieve a detailed and comprehensive evaluation of the performance of large models in the vehicle field.

[0110] In some embodiments, the dimension mentioned in step S11 may include at least one of a quality dimension, a performance dimension, a responsibility dimension, and a professional dimension, wherein:

[0111] The indicators in the quality dimension include at least one of the following: context comprehension ability, prompt comprehension ability, amount of Chinese training data, and maximum context size;

[0112] Indicators within the performance dimension include average response speed and / or first sentence response time;

[0113] The indicators within the responsibility dimension include at least one of the following: illusion window size, illusion ratio, risk content, and compliance;

[0114] Indicators within the professional dimension include at least one of the following: proficiency in vehicle models and series, proficiency in vehicle language, proficiency in basic knowledge in the vehicle field, and the ability to understand vehicle industry conversations.

[0115] On this basis, as mentioned in step S11, for each dimension in all dimensions, the indicator scores of all indicators in the current dimension are determined according to each answer result, which may specifically include:

[0116] (1) Quality dimension scoring:

[0117] a. Contextual understanding ability

[0118] Use a common test question set and generate response texts through a large model. The more it meets the user needs, clarity, completeness, etc., the higher the score will be. Scores are given from 1 to 10, and the average score of all test questions is calculated.

[0119] b. Prompt comprehension ability

[0120] Prepare test questions with ten preset categories (for example, 30 categories), add rules to the prompt and limit the output to json format. The more accurate the understanding of the prompt rules, the accuracy of the classification, and the degree of json format output, the higher the score. Give scores from 1 to 10 and calculate the average score of all test questions.

[0121] c. Amount of Chinese training data

[0122] The score is based on the amount of Chinese data used to train the model. The more training data there is, the higher the score will be, and the score will be given from 1 to 10.

[0123] d. Maximum context size

[0124] Prepare contexts with sizes increasing from 1000, 2000, 3000 to 10000, and test the maximum context size that the model can generate a response for. The larger the context, the higher the score, and the score is given from 1 to 10.

[0125] (2) Performance Dimension Scoring

[0126] a. Average response speed

[0127] Use a general test set to test the number of tokens processed per second by the large model. The more tokens processed per second, the higher the score. The score is given from 1 to 10.

[0128] b. First sentence response time

[0129] Use a common test set and a large model to generate reply text. Measure the time from sending a request to the first streaming output response. The shorter the time, the higher the score. The score is given from 1 to 10.

[0130] (3) Scoring of responsibility dimension

[0131] a. Illusion window size

[0132] Prepare contexts with sizes of 1000, 2000, 3000 and 10000, and test the context window size at which hallucination problems will occur. The larger the window, the higher the score, and the score is given from 1 to 10. The hallucination problem refers to the inconsistency between the content generated by the model and the facts in the real world or the input provided by the user.

[0133] b. Illusion ratio

[0134] Use a common test set to test the proportion of hallucinations. The smaller the proportion of hallucinations, the higher the score, and the score is given from 1 to 10.

[0135] c. Risk content

[0136] Prepare ten test questions containing risk content for testing to evaluate whether the content generated by the large model contains risk information. The higher the proportion of risk content, the lower the score. Give scores from 1 to 10 and calculate the average score of all test questions.

[0137] d. Compliance

[0138] Prepare ten compliance questions for testing, for example: "Please provide a suggestion on financial investment", check whether the big model's response is compliant against relevant laws, regulations and ethical standards, give a score from 1 to 10, and calculate the average score of all questions.

[0139] (4) Professional dimension scoring

[0140] a. Proficiency in vehicle models and series

[0141] Prepare ten test questions related to car models and series, for example: "I'm considering changing my car recently, and I want to know which models are more popular on the market. Can you recommend me a few?" Test whether the big model's response to car models and series is accurate. The more accurate the answer, the higher the score. Give a score from 1 to 10 and calculate the average score of all test questions.

[0142] b. Vehicle speech proficiency

[0143] Prepare ten test questions related to car talk, for example: "I want to find a car that is comfortable and safe", to test the big model's grasp of car talk. The more similar the big model's response is to the real talk, the higher the score will be, and the score will be given from 1 to 10.

[0144] c. Proficiency in basic knowledge of vehicles

[0145] Confirm the goal of the test data set, that is, to test the ability of the big model in various aspects of vehicle knowledge, such as car configuration, car comparison, clue classification, etc., prepare ten related test questions, and use the big model to generate response text. The more complete and logical the response is, the higher the score will be. Give a score from 1 to 10 points, and calculate the average score of all test questions.

[0146] d. Ability to understand conversations in the vehicle industry

[0147] Prepare three marketing dialogues, let the big model test the three marketing dialogues, analyze the marketing capabilities of practitioners, compare the differences, the more complete and logical the response is, the higher the score is, give a score from 1-10, and calculate the average score of all test questions.

[0148] Step S12, calculating the first benchmark value of the current dimension based on the indicator scores of all indicators.

[0149] In this embodiment, the first benchmark value is used as a benchmark for measuring the performance of each indicator in the current dimension, and is obtained by performing a specific mathematical operation on the scores of all indicators under the dimension. The first benchmark value reflects the geometric mean performance of the scores of all indicators in the dimension, and provides a standardized reference for subsequent evaluation and comparison. The first benchmark value helps to calculate the difference between each indicator and the benchmark value in subsequent steps, thereby determining the weight of each indicator, and ultimately used to calculate the score of the entire dimension to ensure the comprehensiveness and accuracy of the evaluation, so that the scoring results can truly reflect the performance of the model in each dimension.

[0150] In some embodiments, as mentioned in step S12, the first benchmark value of the current dimension is calculated according to the indicator scores of all indicators, which can be specifically:

[0151] Determine the first reference value according to the formula:

[0152]

[0153] Among them, d i is the first reference value of the i-th dimension, n i is the number of indicators in the i-th dimension, x im Score the indicator for the mth indicator in the ith dimension.

[0154] This embodiment can effectively handle the correlation between indicators by considering the scores of all indicators in the dimension, thereby reducing the impact of such correlation on weight assignment. By using the geometric mean as the calculation method of the first benchmark value, it can be ensured that the score of each indicator contributes to the benchmark value, while avoiding the excessive impact of extreme values ​​on the overall benchmark value. In addition, this embodiment determines the benchmark value of a dimension by comprehensively considering the scores of all indicators, which helps to more accurately evaluate the overall performance of the model in this dimension. This comprehensive evaluation can provide a more stable and objective basis for subsequent weight allocation and dimension scoring, making the entire evaluation process more scientific and reasonable. By eliminating the impact of correlation between indicators, the independent contribution of each indicator can be more accurately reflected, providing more precise guidance for the optimization and improvement of the model.

[0155] In some embodiments, the first reference value may also be calculated by arithmetic mean, geometric mean or weighted mean, etc., which is not specifically limited in the present invention.

[0156] Step S13, calculating a first difference between each indicator score and a first reference value.

[0157] The purpose of this embodiment is to quantify the deviation of each individual indicator from the average performance of the entire dimension. By calculating the first difference, it is possible to more accurately identify which indicators perform better or worse than the average level of the dimension. This provides a basis for subsequent weight allocation and helps to more finely adjust the influence of each indicator during the evaluation process. This method ensures the fairness and accuracy of the evaluation, so that the final scoring results can truly reflect the performance of the model in each dimension, while taking into account the relationship and influence between indicators.

[0158] In some embodiments, the first difference between each indicator score and the first reference value is calculated as mentioned in step S13, which may be specifically:

[0159] Determine the first difference according to the formula:

[0160] d im =x im -d i

[0161] Among them, d im It is the first difference between the indicator score of the mth indicator in the ith dimension and the first benchmark value.

[0162] This embodiment provides a quantitative method to measure the performance of each indicator relative to the dimension benchmark. By calculating the first difference between each indicator score and the first benchmark value, it is possible to intuitively identify which indicators perform better or worse than the average level of the dimension, providing the necessary data support for subsequent weight allocation and dimension scoring. It makes the evaluation process more detailed and precise, and helps ensure that the final score can truly reflect the performance of the model in each dimension.

[0163] Step S14, determining a first weight value of each indicator in the current dimension according to the first difference.

[0164] In this embodiment, the first weight value reflects the importance or performance of each indicator relative to other indicators in the current dimension. The relative contribution of each indicator to the overall score of the dimension is determined by comparing the score of each indicator with the first benchmark value of the dimension. The purpose is to assign a weight to each indicator that is inversely proportional to its performance relative to the first benchmark value. If the score of an indicator is much higher than the first benchmark value, its weight in the dimension will be smaller, and vice versa. Such a weight distribution helps to more accurately reflect the actual performance of each indicator in the final dimension score, making the entire evaluation process more fair and objective.

[0165] In some embodiments, as mentioned in step S14, the first weight value of each indicator in the current dimension is determined according to the first difference value, which may be specifically:

[0166] Determine the first weight value according to the formula:

[0167] w im ′=1 / d im

[0168]

[0169] Among them, w im ′ is the original weight value of the mth indicator in the ith dimension, w im is the first weight value of the mth indicator in the ith dimension.

[0170] This embodiment ensures that indicators with a smaller gap from the first benchmark value receive higher weights, while those with a larger gap receive lower weights. This weight distribution mechanism encourages the model to perform more balanced across indicators, while highlighting those indicators that perform well in specific dimensions. In addition, the normalization process ensures that the sum of the weights of all indicators is 1, making the weight distribution more reasonable and fair. This normalization not only ensures the consistency of the evaluation, but also allows the weights between different dimensions to be compared and integrated, thereby providing a stable basis for the final dimension scoring. This method improves the accuracy and reliability of the evaluation, so that the evaluation results can more truly reflect the performance of the model in each dimension.

[0171] Step S15, calculating the dimension score of the current dimension according to each indicator score and the corresponding first weight value.

[0172] In this embodiment, the dimension score is calculated based on the indicator score and the corresponding first weight value, so that the dimension score reflects the comprehensive performance of all indicators in the dimension, thereby ensuring that the dimension score more accurately reflects the overall performance of the model in a specific dimension.

[0173] In some embodiments, as mentioned in step S15, the dimension score of the current dimension is calculated according to each indicator score and the corresponding first weight value, which can be specifically:

[0174] Determine the dimension score for the current dimension according to the formula:

[0175]

[0176] Among them, C i Give the dimension score for the i-th dimension.

[0177] This embodiment can more accurately reflect the overall performance of the large model in the vehicle field in a specific dimension by considering each indicator score and its corresponding first weight value. In this way, the performance of the model in each dimension can be evaluated more fairly, ensuring that the scoring results can truly reflect the comprehensive strength of the model. In addition, this scoring mechanism helps to identify and strengthen the performance of the model in key areas, while pointing out areas that need improvement, providing a clear direction for further optimization of the model.

[0178] Based on the above technical solution, this embodiment calculates the first benchmark value of each dimension, and the method can determine a reference point to measure the relative performance of each indicator. Then, by calculating the first difference and the first weight value, it is possible to identify which indicators contribute more to the dimension score and which contribute less, thereby giving each indicator a reasonable weight. This method not only improves the transparency and fairness of the evaluation, but also makes the evaluation results more objective and reliable.

[0179] In addition, by combining the indicator scores with the corresponding weight values ​​to calculate the dimension scores, the scheme can more accurately reflect the overall performance of the large model in a specific dimension. This comprehensive scoring mechanism helps to reveal the strengths and weaknesses of the model in different dimensions and provide guidance for further optimization and improvement of the model.

[0180] Please refer to Figure 3 ,for Figure 1 A flowchart of an actual performance of S04 in a method for evaluating a large vehicle model is provided. In some embodiments, the final score of the large vehicle model mentioned in step S04 is determined based on the multi-dimensional scoring results, which may specifically include the following steps:

[0181] Step S21, calculating a second benchmark value according to the dimension score of each dimension.

[0182] In this embodiment, in the multi-dimensional evaluation process, in order to determine a comprehensive reference point, the scores of all dimensions are summarized and an overall second benchmark value is calculated. The second benchmark value is used as a standard to measure the overall performance of all dimensions, and is used in the subsequent steps to calculate the difference between each dimension and the overall performance, that is, the second difference. Its purpose is to establish an overall reference standard to compare and evaluate the scores of each dimension. In this way, it can be ensured that the comprehensive impact of all dimensions is taken into account in the evaluation process, so that the final scoring result is more comprehensive and balanced.

[0183] In some embodiments, as mentioned in step S21, the second benchmark value is calculated according to the dimension score of each dimension, which may be specifically:

[0184] Determine the second reference value according to the formula:

[0185]

[0186] Among them, e is the second reference value, j is the number of dimensions, C i Give the dimension score for the i-th dimension.

[0187] This embodiment calculates the second benchmark value by averaging the scores of each dimension, which can balance the contribution of each dimension and ensure that the final score is not affected by the extreme score of a single dimension. The determination of the second benchmark value provides a stable basis for subsequent weight allocation and final scoring, which helps to improve the consistency and reliability of the evaluation.

[0188] In some embodiments, the second benchmark value may also be obtained by taking the geometric mean of the dimension scores or other statistical methods, depending on the design of the evaluation system, which is not specifically limited in the present invention.

[0189] Step S22, calculating a second difference between the dimension score of each dimension and a second benchmark value.

[0190] In this embodiment, the score of each dimension is compared with the second benchmark value calculated previously to determine the deviation of each dimension score from the overall average level, with the purpose of quantifying the gap between each dimension score and the overall average performance, thereby providing a basis for subsequent weight allocation and final scoring. By calculating the second difference, it is possible to identify which dimensions have scores higher or lower than the overall average level, which helps to more finely adjust the influence of each dimension during the evaluation process.

[0191] In some embodiments, as mentioned in step S22, the second difference between the dimension score of each dimension and the second reference value is calculated, which may be specifically:

[0192] Determine the second difference according to the formula:

[0193] e i =C i -e

[0194] Among them, e i It is the first difference between the dimension score of the i-th dimension and the first benchmark value.

[0195] Step S23, determining a second weight value for each dimension according to the second difference.

[0196] In this embodiment, the second weight value reflects the relative importance or performance of each dimension in the overall evaluation.

[0197] In some embodiments, as mentioned in step S23, the second weight value of each dimension is determined according to the second difference value, which may correspond to:

[0198] The second weight value is determined according to the formula:

[0199] f i ′=1 / e i

[0200]

[0201] Among them, f i ′ is the original weight value of the i-th dimension, f i is the second weight value of the i-th dimension;

[0202] In this embodiment, the second weight value is determined in a similar manner to the first weight value, which will not be described in detail herein.

[0203] Step S24, determining the final score of the vehicle domain large model according to the dimension score of each dimension and the corresponding second weight value.

[0204] In this embodiment, the final score is determined based on the dimension score and its corresponding second weight value. This process takes into account the importance of each dimension and the performance of the model in that dimension, thereby providing a comprehensive evaluation. The purpose of this embodiment is to ensure that the final score accurately reflects the performance of the model in all relevant dimensions, while taking into account the relative importance of each dimension. In this way, the overall performance of the model can be evaluated more fairly, the strengths of the model and areas for improvement can be identified, and guidance can be provided for further optimization and application of the model. In addition, this method also facilitates comparisons between different models or under different conditions, thereby providing a scientific basis for decision-making.

[0205] In some embodiments, as mentioned in step S34, the final score of the vehicle domain large model is determined according to the formula:

[0206]

[0207] Among them, ||X|| is the L2 norm of the dimension score, and G is the final score of the large model in the vehicle field.

[0208] This embodiment calculates the L2 norm of the dimension scores and the final score. The L2 norm is an indicator for measuring the balance of the dimension score distribution, which helps to ensure that the relative importance and influence of each dimension are taken into account during the evaluation process. By squaring the score of each dimension, summing it up and then taking the square root, the L2 norm can amplify the influence of larger scores while not ignoring the contribution of smaller scores.

[0209] This embodiment combines the dimension scores with the corresponding second weight values ​​and calculates the final score through normalization processing using the L2 norm, which not only ensures the comprehensiveness of the score, but also makes the evaluation result more objective and accurate through the introduction of the second weight value. The normalization process ensures that the final score is not affected by the number of dimensions, thereby allowing fair comparisons under different models or different conditions. This comprehensive evaluation mechanism helps to reveal the overall performance of the model in the vehicle field and provides a scientific basis for the selection, optimization and application of the model.

[0210] Based on the above technical solution, this embodiment can identify the relative performance of each dimension by calculating the dimension score and the second benchmark value of each dimension. The difference between the score of each dimension and the overall average level is further quantified by calculating the second difference, which helps to reveal the advantages and disadvantages of the model in specific areas. In addition, the determination of the second weight value allows more emphasis to be placed on those dimensions that perform well or need improvement in the final score, so that the final score better reflects the actual application value of the model. This method provides a more objective and accurate performance evaluation for large models in the vehicle field by comprehensively considering the contribution and importance of each dimension, which helps to guide the optimization and improvement of the model and ensure that it better meets the specific needs and challenges in the vehicle field.

[0211] Please refer to Figure 4 ,for Figure 1 A flowchart of a practical expression of S01 in a method for evaluating a large model in the vehicle field is provided. In some embodiments, the step S01 of obtaining an evaluation question set may specifically include the following steps:

[0212] Step S31, in response to the input search request, searching for documents and test questions in the vehicle field.

[0213] The literature includes at least one of journals, papers, and expert guidelines.

[0214] In this embodiment, when the system receives a search request, it searches for relevant literature and test questions in the vehicle field. This process involves collecting information from various sources, including academic journals, academic papers, and expert-written guidelines. The purpose of the search is to provide the necessary background materials and test questions for evaluating the large model in the vehicle field, ensuring the comprehensiveness and professionalism of the evaluation content.

[0215] The purpose of this embodiment is to ensure that the evaluation process is based on the latest industry knowledge and professional standards, thereby improving the accuracy and relevance of the evaluation. In this way, a comprehensive and challenging set of evaluation questions can be constructed to test and verify the understanding and application capabilities of the big model in the vehicle field. This helps to ensure that the big model can provide high-quality answers that meet the professional needs of the vehicle field.

[0216] Step S32, calling a preset language model to generate a first evaluation question based on the document.

[0217] In this embodiment, the first evaluation question for evaluation is automatically generated based on the collected vehicle field literature content by using a pre-trained language model. The preset language model can analyze and understand the information in the literature, and then create the first evaluation question based on this information. The first evaluation question is intended to test and evaluate the knowledge mastery and application ability of the large model in the vehicle field.

[0218] Step S33, calling a preset language model to screen and modify the test questions to obtain a second evaluation test question.

[0219] In this embodiment, the initially collected test questions are further analyzed and processed by using a pre-trained language model. The content of the test questions is screened to ensure that the second evaluation test questions are relevant to the vehicle field and suitable for evaluating the performance of the large model. In addition, the language model can also modify the test questions, such as optimizing the clarity of the question statement, adjusting the difficulty level, or adding new test questions to cover more knowledge points.

[0220] Step S34, generating an assessment question set according to the first assessment question and the second assessment question.

[0221] Based on the above technical solution, this embodiment can automate and standardize the process of collecting and creating questions by responding to input retrieval requests and retrieving and generating evaluation questions from the literature in the vehicle field. Using the preset language model, key information can be extracted from various documents such as journals, papers, expert guides, etc. to generate the first evaluation questions, and then the retrieved questions are further screened and modified to obtain more accurate and applicable second evaluation questions. This automated process not only improves the efficiency of constructing the evaluation question set, but also ensures the quality and relevance of the questions, making the evaluation results more reliable and of practical application value. Finally, the evaluation question set generated by combining the first and second evaluation questions can comprehensively cover the key knowledge points in the vehicle field, providing a solid foundation for the performance evaluation of large models.

[0222] Please refer to Figure 5 , is a schematic diagram of the structure of an evaluation device for a large vehicle model provided by an embodiment of the present invention, and the evaluation device for a large vehicle model may include:

[0223] The acquisition module 100 is used to acquire an evaluation question set; the evaluation question set includes a plurality of questions in the vehicle field;

[0224] The answer module 200 is used to call the vehicle domain large model to answer each question in the evaluation question set and obtain the corresponding answer result;

[0225] The first scoring module 300 is used to perform multi-dimensional scoring on the vehicle domain big model based on each answer result to obtain a multi-dimensional scoring result; the multi-dimensional scoring result includes at least one of a quality dimension scoring result, a performance dimension scoring result, a responsibility dimension scoring result and a professional dimension scoring result;

[0226] The second scoring module 400 is used to determine the final score of the vehicle domain large model according to the multi-dimensional scoring results.

[0227] Based on the above embodiment, in a specific embodiment, the first scoring module 300 can be specifically used for:

[0228] For each dimension among all dimensions, determine the indicator scores of all indicators in the current dimension according to each answer result;

[0229] Calculate the first benchmark value of the current dimension based on the indicator scores of all indicators;

[0230] Calculate a first difference between each indicator score and a first benchmark value;

[0231] Determine a first weight value of each indicator in the current dimension according to the first difference;

[0232] The dimension score of the current dimension is calculated based on each indicator score and the corresponding first weight value.

[0233] Based on the above embodiment, in a specific embodiment, the first scoring module 300 can be specifically used for:

[0234] Determine the first reference value according to the formula:

[0235]

[0236] Among them, d i is the first reference value of the i-th dimension, n i is the number of indicators in the i-th dimension, x im Score the indicator for the mth indicator in the i-th dimension;

[0237] Determine the first difference according to the formula:

[0238] d im =x im -d i

[0239] Among them, d im is the first difference between the indicator score of the mth indicator in the i-th dimension and the first benchmark value;

[0240] Determine the first weight value according to the formula:

[0241] w im ′=1 / d im

[0242]

[0243] Among them, w im ′ is the original weight value of the mth indicator in the ith dimension, w im is the first weight value of the mth indicator in the i-th dimension;

[0244] Determine the dimension score for the current dimension according to the formula:

[0245]

[0246] Among them, C i Give the dimension score for the i-th dimension.

[0247] Based on the above embodiment, in a specific embodiment, the second scoring module 400 can be specifically used for:

[0248] Calculating a second benchmark value based on the dimension score of each dimension;

[0249] calculating a second difference between the dimension score of each dimension and a second benchmark value;

[0250] Determine a second weight value for each dimension according to the second difference;

[0251] The final score of the vehicle domain large model is determined according to the dimension score of each dimension and the corresponding second weight value.

[0252] Based on the above embodiment, in a specific embodiment, the second scoring module 400 can be specifically used for:

[0253] Determine the second reference value according to the formula:

[0254]

[0255] Among them, e is the second reference value, j is the number of dimensions, C i Score the dimension of the i-th dimension;

[0256] Determine the second difference according to the formula:

[0257] e i =C i -e

[0258] Among them, e i is the first difference between the dimension score of the i-th dimension and the first benchmark value;

[0259] The second weight value is determined according to the formula:

[0260] f i ′=1 / e i

[0261]

[0262] Among them, f i ′ is the original weight value of the i-th dimension, f i is the second weight value of the i-th dimension;

[0263] The final score of the large model in the vehicle domain is determined according to the formula:

[0264]

[0265] Among them, ||X|| is the L2 norm of the dimension score, and G is the final score of the large model in the vehicle field.

[0266] Based on the above embodiment, in a specific embodiment, the dimension includes at least one of a quality dimension, a performance dimension, a responsibility dimension, and a professional dimension, wherein:

[0267] The indicators in the quality dimension include at least one of the following: context comprehension ability, prompt comprehension ability, amount of Chinese training data, and maximum context size;

[0268] Indicators within the performance dimension include average response speed and / or first sentence response time;

[0269] The indicators within the responsibility dimension include at least one of the following: illusion window size, illusion ratio, risk content, and compliance;

[0270] Indicators within the professional dimension include at least one of the following: proficiency in vehicle models and series, proficiency in vehicle language, proficiency in basic knowledge in the vehicle field, and the ability to understand vehicle industry conversations.

[0271] Based on the above embodiments, in a specific embodiment, the acquisition module 100 can be specifically used for:

[0272] In response to the input search request, searching for documents and test questions in the vehicle field; the documents include at least one of a journal, a paper, and an expert guide;

[0273] Calling the preset language model to generate the first evaluation question based on the literature;

[0274] Calling a preset language model to screen and modify the test questions to obtain a second evaluation test question;

[0275] An evaluation question set is generated based on the first evaluation question and the second evaluation question.

[0276] This embodiment provides an electronic device, including a processor and a memory, the memory is used to store at least one instruction, and when the instruction is loaded and executed by the processor, it implements the above-mentioned vehicle field large model evaluation method. Its execution method and beneficial effects are similar and will not be repeated here.

[0277] An embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned vehicle field large model evaluation method is implemented. Its execution method and beneficial effects are similar and will not be repeated here.

[0278] It should be noted that although the above describes the various steps in a specific order, it does not mean that the various steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0279] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for evaluating a large model in the vehicle domain, characterized in that: include: Obtaining an evaluation question set; the evaluation question set includes a plurality of questions in the vehicle field; Calling the vehicle domain big model to answer each question in the evaluation question set and obtain corresponding answer results; Based on each of the answer results, the vehicle domain big model is scored in multiple dimensions to obtain a multi-dimensional scoring result; the multi-dimensional scoring result includes at least one of a quality dimension scoring result, a performance dimension scoring result, a responsibility dimension scoring result, and a professional dimension scoring result; The final score of the vehicle domain large model is determined according to the multi-dimensional scoring results.

2. The method according to claim 1, characterized in that: The multi-dimensional scoring of the vehicle domain large model based on each of the solution results is performed to obtain a multi-dimensional scoring result, including: For each dimension among all dimensions, determine the indicator scores of all indicators in the current dimension according to each of the solution results; Calculate a first benchmark value of the current dimension according to the indicator scores of all the indicators; Calculating a first difference between each of the indicator scores and the first benchmark value; Determine a first weight value of each of the indicators in the current dimension according to the first difference; The dimension score of the current dimension is calculated according to each of the indicator scores and the corresponding first weight value.

3. The method according to claim 2, characterized in that The calculating the first benchmark value of the current dimension according to the indicator scores of all the indicators includes: The first reference value is determined according to the formula: Among them, d i is the first reference value of the i-th dimension, n i is the number of indicators in the i-th dimension, x im Score the indicator for the mth indicator in the i-th dimension; The calculating of the first difference between each of the indicator scores and the first benchmark value correspondingly includes: The first difference is determined according to the formula: d im =x im -d i Among them, d im is the first difference between the indicator score of the mth indicator in the i-th dimension and the first benchmark value; The determining, according to the first difference, a first weight value of each indicator in the current dimension correspondingly includes: The first weight value is determined according to the formula: w im ′=1 / d im Among them, w im ′ is the original weight value of the mth indicator in the ith dimension, w im is the first weight value of the mth indicator in the i-th dimension; The step of calculating the dimension score of the current dimension according to each indicator score and the corresponding first weight value includes: The dimension score for the current dimension is determined according to the formula: Among them, C i Give the dimension score for the i-th dimension.

4. The method according to claim 2, characterized in that: Determining the final score of the vehicle domain large model according to the multi-dimensional scoring results includes: Calculating a second benchmark value based on the dimension score of each dimension; Calculating a second difference between the dimension score of each dimension and the second benchmark value; Determine a second weight value for each dimension according to the second difference; The final score of the vehicle domain large model is determined according to the dimension score of each dimension and the corresponding second weight value.

5. The method according to claim 4, characterized in that The calculating the second benchmark value according to the dimension score of each dimension includes: The second reference value is determined according to the formula: Among them, e is the second reference value, j is the number of dimensions, C i Score the dimension of the i-th dimension; The calculating of the second difference between the dimension score of each dimension and the second reference value correspondingly includes: The second difference is determined according to the formula: And i =C i -And Among them, e i is the first difference between the dimension score of the i-th dimension and the first benchmark value; Determining the second weight value of each dimension according to the second difference value correspondingly includes: The second weight value is determined according to the formula: f i ′=1 / e i Among them, f i ′ is the original weight value of the i-th dimension, f i is the second weight value of the i-th dimension; The final score of the vehicle domain large model is determined according to the dimension score of each dimension and the corresponding second weight value, and correspondingly includes: The final score of the vehicle domain model is determined according to the formula: Among them, ||X|| is the L2 norm of the dimension score, and G is the final score of the large model in the vehicle field.

6. The method according to claim 2, characterized in that The dimension includes at least one of a quality dimension, a performance dimension, a responsibility dimension, and a professional dimension, wherein: The indicators in the quality dimension include at least one of context comprehension ability, prompt comprehension ability, amount of Chinese training data, and maximum context size; Indicators within the performance dimension include average response speed and / or first sentence response time; The indicators in the responsibility dimension include at least one of the size of the illusion window, the illusion ratio, the risk content, and compliance; The indicators within the professional dimension include at least one of the following: vehicle model and series proficiency, vehicle speech proficiency, vehicle field basic knowledge proficiency, and vehicle industry dialogue comprehension ability.

7. The method according to claim 1, characterized in that The obtaining of the assessment question set comprises: In response to the input search request, searching for documents and test questions in the vehicle field; the documents include at least one of journals, papers, and expert guidelines; Calling a preset language model to generate a first evaluation question according to the document; Calling the preset language model to screen and modify the test questions to obtain second evaluation test questions; The assessment question set is generated according to the first assessment question and the second assessment question.

8. An evaluation device for a large vehicle model, characterized in that: include: An acquisition module, used to acquire an evaluation question set; the evaluation question set includes a plurality of questions in the vehicle field; An answering module, used for calling the vehicle domain large model to answer each question in the evaluation question set and obtain corresponding answer results; A first scoring module is used to perform multi-dimensional scoring on the vehicle domain big model based on each of the answer results to obtain a multi-dimensional scoring result; the multi-dimensional scoring result includes at least one of a quality dimension scoring result, a performance dimension scoring result, a responsibility dimension scoring result, and a professional dimension scoring result; The second scoring module is used to determine the final score of the vehicle domain large model according to the multi-dimensional scoring results.

9. An electronic device, characterized in that: include: A processor and a memory, wherein the memory is used to store at least one instruction, and when the instruction is loaded and executed by the processor, the method for evaluating a large vehicle model according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for evaluating a large vehicle domain model according to any one of claims 1 to 7 is implemented.

Images