Model evaluation method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides a model evaluation method and device, electronic equipment and a storage medium, and the method comprises the steps: determining a plurality of capability evaluation dimensions for evaluating a to-be-evaluated model and a plurality of evaluation tasks under each capability evaluation dimension based on a model evaluation capability system; obtaining evaluation data of each evaluation task under any capability evaluation dimension, and inputting the evaluation data into the to-be-evaluated model to obtain a model output result corresponding to each evaluation task under any capability evaluation dimension; based on the model evaluation index system, evaluating the model output result corresponding to each evaluation task under any capability evaluation dimension to obtain an evaluation result of any capability evaluation dimension; and determining the evaluation result of the to-be-evaluated model based on the evaluation result of each capability evaluation dimension and the weight of each capability evaluation dimension. The method can achieve the comprehensive, objective and accurate evaluation of a large model of a specific industry.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a model evaluation method, device, electronic device and storage medium. Background Art

[0002] With the continuous development of Large Language Model (LLM) technology, its application demand and application practice in many industries (such as agriculture) are also increasing. In the research and development process of large models, performance evaluation plays a vital role. It can not only objectively and accurately measure the performance of large models in various intelligent tasks, but also provide clear optimization directions for subsequent iterations of the model. Therefore, building a complete evaluation system and method is crucial for the entire large model ecosystem.

[0003] However, currently when evaluating models, it is common to simply apply the evaluation system and methods of general large models directly to large models of specific industries (such as agricultural large models). This often leads to incomplete model capability evaluation, missing evaluation of complex industry scenarios, incomplete evaluation data, etc. due to the lack of sufficient adaptation to the industry's unique text resources and actual application needs. Ultimately, it is impossible to conduct a scientific, accurate and comprehensive evaluation of the performance of large models in specific industries. Summary of the invention

[0004] The present invention provides a model evaluation method, device, electronic device, storage medium and program product to solve the defect in the related technology that when a general large model evaluation system and method are used to evaluate a large model of a specific industry, the actual performance of the model cannot be accurately evaluated.

[0005] The present invention provides a model evaluation method, comprising: Based on the model evaluation capability system, determining multiple capability evaluation dimensions for evaluating the model to be evaluated and multiple evaluation tasks under each capability evaluation dimension, wherein the model evaluation capability system is pre-constructed based on the industry attributes of the model to be evaluated; Obtaining evaluation data of each evaluation task under any capability evaluation dimension, and inputting the evaluation data into the model to be evaluated, to obtain the model output result corresponding to each evaluation task under any capability evaluation dimension; Based on the model evaluation index system, the model output results corresponding to each evaluation task under any capability evaluation dimension are evaluated to obtain the evaluation results of any capability evaluation dimension; Based on the evaluation results of each capability evaluation dimension and the weight of each capability evaluation dimension, the evaluation result of the model to be evaluated is determined.

[0006] According to a model evaluation method provided by the present invention, obtaining evaluation data of each evaluation task under any capability evaluation dimension includes: Based on each evaluation task under any of the capability evaluation dimensions, obtaining initial evaluation data for each evaluation task; Preprocess the initial evaluation data of each evaluation task to obtain the original evaluation data of each evaluation task; Based on the data organization form, the original evaluation data of each evaluation task is organized into evaluation data of different forms.

[0007] According to a model evaluation method provided by the present invention, the data organization form includes at least one of a zero-sample form, a zero-sample thinking chain form, a small sample form and a small sample thinking chain form. The zero-sample form refers to the original evaluation data itself, the zero-sample thinking chain form refers to the introduction of multiple thinking chain prompts into the original evaluation data, the small sample form refers to the introduction of sample examples into the original evaluation data, and the small sample thinking chain form refers to the introduction of sample examples and multiple thinking chain prompts into the original evaluation data.

[0008] According to a model evaluation method provided by the present invention, the model evaluation capability system includes multiple capability evaluation dimensions for evaluating the model to be evaluated and the weight of each capability evaluation dimension, each capability evaluation dimension includes multiple evaluation sub-dimensions and the weight of each evaluation sub-dimension, and each evaluation sub-dimension includes multiple evaluation tasks and the weight of each evaluation task.

[0009] According to a model evaluation method provided by the present invention, the model output results corresponding to each evaluation task under any capability evaluation dimension are evaluated based on the model evaluation indicator system to obtain the evaluation results of any capability evaluation dimension, including: Based on any evaluation sub-dimension under any capability evaluation dimension, determining an evaluation indicator of any evaluation sub-dimension from the model evaluation indicator system; Based on each evaluation indicator, the model output result corresponding to each evaluation task under any evaluation sub-dimension is evaluated to obtain the evaluation result of any evaluation sub-dimension; Based on the evaluation results of each evaluation sub-dimension under any capability evaluation dimension, the evaluation result of any capability evaluation dimension is determined.

[0010] According to a model evaluation method provided by the present invention, based on each evaluation indicator, the model output result corresponding to each evaluation task under any evaluation sub-dimension is evaluated to obtain the evaluation result of any evaluation sub-dimension, including: Based on each evaluation indicator, the model output result corresponding to each evaluation task under any evaluation sub-dimension is evaluated to obtain the evaluation score of each evaluation task; Determine the evaluation score of any evaluation sub-dimension based on the weight and evaluation score of each evaluation task, and use the evaluation score as the evaluation result of any evaluation sub-dimension; Determining the evaluation result of any capability evaluation dimension based on the evaluation results of each evaluation sub-dimension under any capability evaluation dimension includes: Based on the weights and evaluation scores of each evaluation sub-dimension under any capability evaluation dimension, the evaluation score of any capability evaluation dimension is determined.

[0011] According to a model evaluation method provided by the present invention, based on each evaluation indicator, the model output result corresponding to each evaluation task under any evaluation sub-dimension is evaluated to obtain the evaluation score of each evaluation task, including: Based on each evaluation indicator, the model output result corresponding to any evaluation task under any evaluation sub-dimension is evaluated to obtain the evaluation score corresponding to each evaluation indicator under any evaluation task; Based on the weights and evaluation scores of the evaluation indicators, the evaluation score of any evaluation task is determined.

[0012] A model evaluation method provided by the present invention further includes: Based on each capability assessment dimension, each assessment sub-dimension under each capability assessment dimension, the assessment indicators and assessment scores corresponding to each assessment sub-dimension, and the assessment result of the model to be assessed, an assessment report of the model to be assessed is generated.

[0013] According to a model evaluation method provided by the present invention, the model evaluation index system includes automatic evaluation indicators and / or manual evaluation indicators, the automatic evaluation indicators include at least one of accuracy, micro-F1 value, bilingual evaluation substitute, recall-oriented evaluation for summary evaluation, vector average cosine similarity, and rejection rate, and the manual evaluation indicators include at least one of correctness, relevance, completeness, effectiveness, and coherence.

[0014] According to a model evaluation method provided by the present invention, the model to be evaluated includes a large agricultural model, and the capability evaluation dimensions include at least two of agricultural basic capabilities, agricultural scenario capabilities, and safety and compliance capabilities.

[0015] According to a model evaluation method provided by the present invention, when the capability evaluation dimension includes agricultural basic capability, the capability evaluation sub-dimension under the capability evaluation dimension includes at least one of agricultural cognitive capability, agricultural content generation capability, agricultural knowledge reserve capability, and agricultural logical reasoning capability; In the case where the capability assessment dimension includes agricultural scenario capability, the capability assessment sub-dimension includes at least one of regional data query analysis, agricultural market analysis, agricultural meteorological disaster prevention and control, agricultural production guidance, biosafety management, and auxiliary management decision-making; In the case where the capability assessment dimension includes security compliance capability, the capability assessment sub-dimension includes at least one of social and cultural impact, legal compliance, and personal safety.

[0016] The present invention also provides a model evaluation device, comprising: A dimension determination unit, used to determine multiple capability evaluation dimensions for evaluating the model to be evaluated and multiple evaluation tasks under each capability evaluation dimension based on a model evaluation capability system, wherein the model evaluation capability system is pre-constructed based on the industry attributes of the model to be evaluated; A model output unit, used to obtain the evaluation data of each evaluation task under any capability evaluation dimension, and input the evaluation data into the model to be evaluated to obtain the model output result corresponding to each evaluation task under any capability evaluation dimension; A capability evaluation unit is used to evaluate the model output results corresponding to each evaluation task under any capability evaluation dimension based on the model evaluation indicator system to obtain the evaluation result of any capability evaluation dimension; The result determination unit is used to determine the evaluation result of the model to be evaluated based on the evaluation result of each capability evaluation dimension and the weight of each capability evaluation dimension.

[0017] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the model evaluation method described above is implemented.

[0018] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the model evaluation method as described in any one of the above is implemented.

[0019] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned model evaluation methods.

[0020] The model evaluation method, device, electronic device and storage medium provided by the present invention pre-construct a model evaluation capability system in a targeted manner based on the industry attributes of the model to be evaluated, so that the capability system can comprehensively cover the various capabilities required by the model in the industry to which it belongs. By determining multiple capability evaluation dimensions and multiple evaluation tasks under each dimension based on the capability system, it is possible to ensure a comprehensive evaluation of the model performance and accurately measure the performance of the model under different capability dimensions, thereby improving the accuracy and comprehensiveness of the model evaluation and avoiding the one-sidedness of the evaluation. By inputting the evaluation data of each evaluation task into the model to be evaluated, and using the model evaluation index system to evaluate the output results, the accuracy and consistency of the evaluation can be ensured, making the evaluation results more valuable for reference. In addition, since the model evaluation capability system is constructed according to the industry attributes of the model, it can be customized for the models to be evaluated in different industries, with strong flexibility and scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 It is a flow chart of the model evaluation method provided by the present invention; Figure 2 It is a schematic diagram of the agricultural large model evaluation capability system provided by the present invention; Figure 3 It is a schematic diagram of evaluation data in the form of zero samples provided by the present invention; Figure 4 This is a schematic diagram of evaluation data in the form of a zero-sample thinking chain provided by the present invention; Figure 5 It is a schematic diagram of evaluation data in a small sample form provided by the present invention; Figure 6 It is a schematic diagram of evaluation data in the form of a small sample thinking chain provided by the present invention; Figure 7 It is a schematic diagram of a prompt word template for generating an evaluation report provided by the present invention; Figure 8 It is a structural schematic diagram of the model evaluation device provided by the present invention; Fig. 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] Big model evaluation refers to the process of evaluating the performance of large artificial intelligence models. This type of evaluation usually includes multiple aspects and aims to comprehensively examine the capabilities of the model. At present, with the continuous development of big model technology, it has been widely used in many industries. For example, the use of agricultural big models to deeply empower the entire agricultural industry chain can help the modernization and intelligent development of agriculture. Agricultural big models refer to application models that are built based on large-scale agricultural data and use big model-related technologies to serve agricultural scenarios. The following will take the evaluation of agricultural big models as an example.

[0025] The capability evaluation of large agricultural models is an important step before the implementation of large agricultural model products. However, there is currently no comprehensive and accurate performance evaluation system and method designed for large agricultural models. The evaluation of existing large agricultural models usually directly applies the evaluation system of general large models to evaluate the basic capabilities of large agricultural models. The disadvantage is that it cannot adapt to the special needs of the agricultural field, such as the particularity of industry texts and industry tasks, and lacks a comprehensive evaluation of agricultural scenario capabilities. In addition, the evaluation indicators (such as accuracy, recall, etc.) used in the evaluation of general large models cannot accurately evaluate the output quality of the model in some complex tasks, and the single test data construction form cannot fully evaluate the performance and potential capabilities of large agricultural models.

[0026] The above problems lead to the limitation of the actual application performance of the general large model evaluation system and method in the evaluation of agricultural large models. In this regard, based on the general large model evaluation system and method, the present invention constructs a model evaluation capability system consisting of three capability dimensions: agricultural basic capability, agricultural scenario capability, and safety compliance capability according to the industry attributes of the model to be evaluated, and reorganizes the evaluation data in four forms: zero sample, zero sample thinking chain, small sample, and small sample thinking chain. The model output results are evaluated and scored in two forms: automatic evaluation and manual evaluation, and finally a comprehensive, objective and accurate evaluation of the agricultural large model is achieved, thereby overcoming the above defects. It should be noted that the model evaluation method provided by the present invention can be applied to the evaluation of agricultural large models in addition to the evaluation of large models in other industries, and the present invention does not make specific limitations on this. In order to facilitate the understanding of the technical solution provided by the present invention, the following is mainly introduced by taking the evaluation of agricultural large models as an example.

[0027] Figure 1 It is a flow chart of the model evaluation method provided by the present invention, such as Figure 1 As shown, the method includes: Step 110, based on the model evaluation capability system, determine multiple capability evaluation dimensions for evaluating the model to be evaluated and multiple evaluation tasks under each capability evaluation dimension, wherein the model evaluation capability system is pre-constructed based on the industry attributes of the model to be evaluated.

[0028] It should be noted that the execution subject of the model evaluation method provided in the embodiment of the present invention may be a model evaluation device. The device may be implemented by software, for example, it may be a model evaluation program running in a computer; it may also be a device for executing the model evaluation method, such as a smart terminal, a tablet computer, a computer, a server, etc., which is not specifically limited in the embodiment of the present invention.

[0029] Specifically, the model to be evaluated refers to an artificial intelligence model that needs to be evaluated for performance. For example, the model to be evaluated can be a large agricultural model. The model evaluation capability system refers to a systematic method or framework for defining and dividing the evaluation criteria of the model to be evaluated in different capability dimensions to evaluate the performance and effect of the large model in specific tasks or fields. The capability system can include multiple capability evaluation dimensions, each of which contains multiple specific evaluation tasks to comprehensively evaluate the performance of the model. Therefore, by constructing a model evaluation capability system, it is possible to determine multiple capability evaluation dimensions for evaluating the model to be evaluated, as well as multiple evaluation tasks under each capability evaluation dimension.

[0030] Here, the capability assessment dimension refers to the aspect or angle of capability assessment, which represents the key capabilities that the model should have. For example, for the evaluation of large agricultural models, the capability assessment dimensions can include agricultural basic capabilities, agricultural scenario capabilities, safety and compliance capabilities, etc., and the capabilities can be further divided under each dimension, such as dividing agricultural basic capabilities into agricultural cognitive capabilities, agricultural knowledge capabilities, agricultural content generation capabilities, agricultural logical reasoning capabilities, and other sub-dimensions.

[0031] The evaluation tasks under the capability evaluation dimension refer to the specific evaluation activities designed to comprehensively evaluate the performance of the model in each capability evaluation dimension. For example, under the evaluation dimension of basic agricultural capabilities, the evaluation tasks may include agricultural text segmentation, agricultural text named entity recognition, agricultural text entity relationship extraction, agricultural text inspection, agricultural intent understanding, basic agricultural knowledge question and answer, agricultural text summary and induction, agricultural marketing copy generation, etc.

[0032] It is understandable that before executing step 110, a model evaluation capability system can be pre-built based on the industry attributes of the model to be evaluated. Here, the industry attributes of the model to be evaluated refer to the specific characteristics and requirements of the industry in which the model is located. These attributes may include industry-specific terminology, application scenarios, data characteristics, regulatory requirements, etc. Understanding the industry attributes of the model to be evaluated is crucial to building a suitable model evaluation capability system, because it can help determine the focus and direction of model evaluation and design more targeted evaluation tasks and indicator systems.

[0033] Specifically, the model evaluation capability system can be constructed through the following steps: First, it is necessary to have an in-depth understanding of the specific needs and application scenarios of the industry in which the model to be evaluated is located in order to determine the focus and direction of the evaluation; then, according to industry needs, the model's capabilities are divided into multiple dimensions, such as industry basic capabilities, industry scenario-related capabilities, etc., and according to the importance of each capability evaluation dimension in the industry and the application scenario of the model, a weight can be assigned to each dimension, which helps to more accurately reflect the overall performance of the model in the subsequent comprehensive evaluation; then, under each capability dimension, it can be further subdivided into multiple capability evaluation sub-dimensions, and specific evaluation tasks can be designed under each sub-dimension to evaluate the performance of the model in this dimension. Similarly, appropriate weights can be assigned to them according to the importance of each evaluation sub-dimension and each evaluation task; finally, corresponding evaluation indicators are established for each evaluation task to quantify the performance of the model in the task, thereby constructing the model evaluation capability system. In addition, in the actual evaluation process, feedback and data can be continuously collected to optimize and improve the evaluation capability system to ensure that it can more accurately reflect the performance of the model.

[0034] It should be understood that when establishing corresponding evaluation indicators for each evaluation task, appropriate evaluation indicators can be selected from the model evaluation indicator system according to the specific requirements of the evaluation task and the characteristics of the model to be evaluated. These indicators can fully reflect the performance of the model on the task. After determining the corresponding evaluation indicators for each evaluation task, weights can be assigned to each evaluation indicator according to its importance to obtain more accurate evaluation results.

[0035] Here, the model evaluation index system refers to a system composed of multiple evaluation indicators, which is used to comprehensively and objectively evaluate the performance of the model under different capability evaluation dimensions. The various evaluation indicators in the model evaluation index system can be designed and selected according to the industry attributes, application scenarios and evaluation objectives of the model to be evaluated to ensure the accuracy and reliability of the evaluation results. For example, the evaluation indicators in the model evaluation index system may include accuracy, micro-F1 value, bilingual evaluation understudy (BLEU), recall-oriented understudy for gisting evaluation (ROUGE), vector average cosine similarity, etc.

[0036] Step 120, obtaining the evaluation data of each evaluation task under any capability evaluation dimension, and inputting the evaluation data into the model to be evaluated, to obtain the model output result corresponding to each evaluation task under any capability evaluation dimension.

[0037] Specifically, after determining the multiple capability assessment dimensions to be assessed for the model to be assessed and the multiple assessment tasks under each dimension, the corresponding assessment data can be collected and organized for each assessment task. For example, when evaluating a large agricultural model, different types of agricultural assessment data can be collected and organized according to each assessment task to ensure that the data covers multiple aspects such as basic agricultural data, market data, and meteorological data. Here, the assessment data for each assessment task refers to a data set specially collected or constructed for the assessment task under each capability assessment dimension. These data sets are used to input into the model during the assessment process to obtain the output results of the model on the task.

[0038] It is understandable that for each evaluation task, in order to obtain the corresponding evaluation data, the required data type, data format and data scale can be clarified according to the specific requirements of the evaluation task. For example, for the evaluation task of agricultural text segmentation in the agricultural big model, a data set containing various types of texts in the agricultural field may be required. Subsequently, according to the clear data requirements, public data sets are collected, or the required data sets are generated through independent construction. For example, by training the generative big model, the evaluation data corresponding to each evaluation task is generated. In addition, the collected data can be cleaned, labeled and formatted to ensure the quality and consistency of the data.

[0039] For each capability assessment dimension, after collecting and organizing the assessment data corresponding to each assessment task under this dimension, these assessment data can be input into the model to be assessed to obtain the model output results corresponding to each assessment task. Here, the model output results of the assessment task refer to the results generated by the model based on the input data after the assessment data corresponding to the assessment task is input into the model to be assessed. These results will be used to compare with the true labels or expected results of the assessment data to evaluate the performance of the model on the assessment task.

[0040] Step 130, based on the model evaluation index system, the model output results corresponding to each evaluation task under any capability evaluation dimension are evaluated to obtain the evaluation result of any capability evaluation dimension.

[0041] Specifically, for each evaluation task under any capability evaluation dimension, when evaluating the model output results of the evaluation task, you can first select appropriate evaluation indicators from the evaluation indicator system based on the specific requirements of the evaluation task and the characteristics of the model to be evaluated. These indicators should be able to fully reflect the performance of the model on the task. For example, in the agricultural large model evaluation, when evaluating the model output results of evaluation tasks such as agricultural text segmentation and agricultural text named entity recognition under the evaluation dimension of agricultural basic capabilities, you can select the accuracy rate and micro-F1 value from the model evaluation indicator system according to the requirements and characteristics of the evaluation task. It should be understood that when the corresponding evaluation indicators are pre-established for each evaluation task in the model evaluation capability system, the indicators for evaluating the model output results of each evaluation task can also be directly obtained from the model capability evaluation system.

[0042] For each evaluation task, after determining the appropriate evaluation indicators, the evaluation data corresponding to the evaluation task can be input into the model to be evaluated to obtain the model output result. The value of each evaluation indicator is calculated based on the model output result and the true label or expected result of the evaluation data. According to the weight distribution of each evaluation indicator, the value of each evaluation indicator is weighted summed or averaged to obtain the evaluation result of the evaluation task, which can be expressed as a score.

[0043] For each capability assessment dimension, after obtaining the assessment results of all assessment tasks under the dimension, the weighted sum is performed according to the weight of each assessment task and the assessment result (i.e., the assessment score) to obtain the assessment result of the capability assessment dimension. It should be understood that for a certain capability assessment dimension, if the dimension is further subdivided into multiple assessment sub-dimensions, the scores of the assessment sub-dimensions can be first summarized according to the scores of the assessment tasks, and then the scores of the capability assessment dimension can be calculated according to the scores and weights of each assessment sub-dimension.

[0044] It is understandable that the evaluation result of each capability evaluation dimension refers to the comprehensive result obtained after evaluating the model output results of all evaluation tasks under this dimension according to the model evaluation index system. This comprehensive result reflects the overall performance of the model in this dimension. By comparing the evaluation results of different capability evaluation dimensions, we can gain an in-depth understanding of the performance differences and potential problems of the model in different aspects, and provide targeted suggestions and guidance for the improvement and optimization of the model.

[0045] Step 140: Determine the evaluation result of the model to be evaluated based on the evaluation result of each capability evaluation dimension and the weight of each capability evaluation dimension.

[0046] Specifically, the weight of each capability assessment dimension refers to the quantitative expression of the importance or contribution of each capability assessment dimension during the model evaluation process. These weights can be set based on factors such as the industry attributes, application scenarios, user needs, and evaluation objectives of the model to be evaluated. In the model evaluation system, different capability assessment dimensions may have different importance, so weights are needed to reflect this difference. After obtaining the evaluation results of each capability assessment dimension, the evaluation results of the model to be evaluated can be determined based on these results and the corresponding weights through weighted summation or weighted average. Based on the evaluation results of the model, the overall performance of the model to be evaluated can be analyzed, and based on the evaluation results and analysis, targeted improvement plans can be formulated to improve the performance of the model.

[0047] It is understandable that when determining the evaluation results of the model to be evaluated, the weights of each capability evaluation dimension are taken into consideration, and weighted processing can be performed according to the importance of different dimensions. This weight distribution method makes the evaluation results more in line with actual needs and can more accurately reflect the performance of the model in actual applications, providing a strong basis for model improvement and optimization.

[0048] The method provided by the embodiment of the present invention pre-constructs a model evaluation capability system in a targeted manner based on the industry attributes of the model to be evaluated, so that the capability system can fully cover the various capabilities required by the model in the industry to which it belongs. By determining multiple capability evaluation dimensions and multiple evaluation tasks under each dimension based on the capability system, it is possible to ensure a comprehensive evaluation of the model performance and accurately measure the performance of the model under different capability dimensions, thereby improving the accuracy and comprehensiveness of the model evaluation and avoiding the one-sidedness of the evaluation. By inputting the evaluation data of each evaluation task into the model to be evaluated, and using the model evaluation index system to evaluate the output results, the accuracy and consistency of the evaluation can be ensured, making the evaluation results more valuable for reference. In addition, since the model evaluation capability system is constructed according to the industry attributes of the model, it can be customized for the models to be evaluated in different industries, and has strong flexibility and scalability.

[0049] Based on any of the above embodiments, the model to be evaluated includes a large agricultural model, and the capability evaluation dimensions include at least two of agricultural basic capabilities, agricultural scenario capabilities, and safety and compliance capabilities.

[0050] Specifically, the model evaluation method provided by the embodiment of the present invention can be applied to the performance evaluation of the agricultural big model. In this scenario, the model to be evaluated is the agricultural big model. The general big model evaluation system usually divides the big model capabilities into generation capabilities and understanding capabilities. On this basis, the embodiment of the present invention expands the ability to evaluate the agricultural big model into different dimensions such as agricultural basic capabilities, agricultural scenario capabilities, and security and compliance capabilities. Among them, agricultural basic capabilities refer to the capabilities of the agricultural big model in processing basic knowledge and tasks related to agriculture. Agricultural scenario capabilities refer to the application capabilities and effects of the agricultural big model in specific agricultural scenarios. Security and compliance capabilities refer to the ability of the agricultural big model to ensure data security and compliance when processing agricultural data and performing tasks.

[0051] Figure 2 It is a schematic diagram of the agricultural large model evaluation capability system provided by the present invention, such as Figure 2 As shown, under each capability assessment dimension, the capabilities can be further subdivided. For example, the basic agricultural capabilities can be subdivided into four sub-abilities (i.e., assessment sub-dimensions), namely, agricultural cognition, agricultural content generation, agricultural knowledge reserve, and agricultural logical reasoning; the agricultural scenario capabilities can be subdivided into regional data query and analysis, agricultural market analysis, agricultural meteorological disaster prevention and control, agricultural production guidance, biosafety management, and auxiliary management decision-making; and the safety and compliance capabilities can be further subdivided into social and cultural impact, legal compliance, personal safety, and other assessment sub-dimensions.

[0052] For each evaluation sub-dimension, the corresponding evaluation tasks can be determined according to the evaluation requirements of the sub-dimension. For example, for the evaluation sub-dimension of agricultural cognitive ability, the corresponding evaluation tasks may include agricultural text segmentation, agricultural text named entity recognition, agricultural text entity relationship extraction, agricultural text inspection, agricultural element extraction, agricultural intent understanding, agricultural slot identification, etc. The following will explain each ability evaluation dimension, evaluation sub-dimension and corresponding evaluation tasks.

[0053] (1) Agricultural basic capabilities Agricultural cognitive ability (i.e., language comprehension ability): evaluates the model's in-depth understanding of a given agricultural text, including the grasp of the meaning of the text and the extraction of the main idea; the accurate identification, classification, and meaning interpretation of words; the parsing and error correction of the grammatical structure of sentences and paragraphs. The evaluation tasks corresponding to this ability evaluation sub-dimension mainly include: ① Agricultural text segmentation: refers to the process of dividing the continuous character sequence in agricultural-related text into vocabulary units with independent semantics according to certain rules. The key to the task is to accurately identify agricultural professional terms and general vocabulary to improve the accuracy and efficiency of subsequent processing. ② Agricultural text named entity recognition: The task involves identifying proper nouns and technical terms in agricultural texts, such as crop types, names of pests and diseases, pesticides and fertilizers. By using the model to learn the contextual features of entities in the text, these entities can be labeled and classified, laying the foundation for in-depth information extraction and knowledge management. ③ Agricultural text entity relationship extraction: refers to the ability to identify and extract the relationship between entities from agricultural texts, such as the association between crops and diseases or the use of pesticides. ④ Agricultural text checking: It can check the correctness and standardization of professional words, sentences, punctuation, expressions, grammar, etc. in the agricultural field in a given agricultural text, and correct the errors found. ⑤ Agricultural element extraction: It can extract key agricultural elements from agricultural-related text materials, such as crop types, growth cycles, soil types, climate conditions, agricultural technologies, yield data, etc. In addition, it also includes the identification and extraction of key information such as agricultural policies, regulations, and market trends. ⑥ Agricultural intent understanding: Agricultural intent understanding aims to understand the process of the main purpose or intention in user input (such as questions or instructions). In the agricultural field, it may involve identifying whether the user wants to inquire about crop planting techniques, pest control methods, or needs to understand market prices and other information. ⑦ Agricultural slot identification: Agricultural slot identification is to extract specific agricultural information elements from user input, which are particularly critical for completing specific agricultural tasks (based on user intent).

[0054] Agricultural content generation capability: The evaluation model comprehensively evaluates the model's agricultural content generation capability based on the provided agricultural texts, critical reading analysis, and creative writing tasks around specific topics. The evaluation tasks corresponding to this capability evaluation sub-dimension mainly include: ① Agricultural text summarization (agricultural text summary): refers to the ability to summarize the content of long professional agricultural documents (such as agricultural papers, reports, etc.), extract the key information and main points in the documents, and form a concise and accurate content summary according to the specified template format requirements. ② Agricultural marketing copy generation: refers to the ability to generate attractive and persuasive marketing copy according to the specified style and format based on the specific attributes and market demand of agricultural products, combined with the preferences of the target audience and market trends. ③ Agricultural information title generation: refers to the ability to extract the key information and themes of agricultural information content and generate concise and eye-catching titles. ④ Planting guide generation: refers to the ability to automatically generate detailed planting guides based on specific crop types input by users. The planting guide includes various aspects from soil preparation, seed selection, sowing time, field management to harvesting techniques, providing farmers or gardening enthusiasts with comprehensive and practical planting suggestions and steps. ⑤ Generation of crop diagnostic reports: refers to the ability to automatically generate professional crop diagnostic reports based on the crop growth description, existing problems and related data entered by the user. The report analyzes in detail the pests and diseases, nutrient deficiencies or other growth obstacles that crops may face, and provides targeted solutions and improvement suggestions. ⑥ Generation of farmland management plans: refers to the ability to automatically generate personalized farmland management plans based on the farmland conditions, crop types, growth stages and expected goals entered by the user. The plan includes key management measures such as soil improvement, irrigation plans, fertilization strategies, and pest and disease control. ⑦ Structured text generation: refers to the ability to extract structured text information (such as Excel generation, Markdown generation, JSON generation, etc.) according to the user's specific content and format requirements.

[0055] Agricultural knowledge reserve capacity: measures the model's understanding of a wide range of agricultural knowledge, focuses on the model's mastery of knowledge in different agricultural disciplines, and evaluates the model's mastery of agricultural knowledge from the perspectives of basic agricultural knowledge, professional agricultural knowledge, and vocational education. The assessment tasks corresponding to this capability assessment sub-dimension mainly include: ① Basic agricultural knowledge questions and answers: refers to the ability to accurately and comprehensively answer basic questions in the agricultural field. This capability covers the answers to basic questions in agricultural production, crop planting, soil management, agricultural machinery, etc., and provides users with an analysis of introductory agricultural knowledge and concepts. ② Agricultural technical instructor certificate examination: refers to the ability to correctly answer questions in the agricultural technical instructor certificate examination. This capability not only includes the mastery of basic agricultural knowledge, but also involves specific professional fields such as agricultural technology application, crop management, pest control, and agricultural machinery use. ③ Senior agronomist certificate examination: refers to the ability to correctly answer questions in the senior agronomist certificate examination. This capability not only includes basic agricultural knowledge, but also covers knowledge in advanced disciplines such as crop genetics, soil fertilizer science, and agricultural economics. ④ Agricultural Production Analyst Certificate Examination: refers to the ability to correctly answer the questions of the Agricultural Production Analyst Certificate Examination. This ability covers data analysis in the agricultural production process, market trend forecasting, and optimal allocation of agricultural resources.

[0056] Agricultural logical reasoning ability: This dimension targets the model's logical analysis and reasoning ability. The model's reasoning ability not only involves mathematical calculations and logical judgments, but also includes the ability to analyze and interpret complex agricultural texts. The assessment tasks corresponding to this ability assessment sub-dimension include: ① Agricultural policy interpretation: refers to the ability to conduct in-depth analysis and understanding of agricultural policy documents, extract key information, and logically reason and interpret policy content. This ability involves a comprehensive grasp of policies, and can transform complex policy texts into information that is easy for users to understand, helping users accurately grasp policy orientations and specific requirements, and providing a basis for agricultural decision-making. ② Agricultural numerical calculation: refers to the ability to perform logical analysis and mathematical calculations on numerical data in the agricultural field. This ability includes but is not limited to the accurate handling of agricultural-related numerical problems such as yield forecasting, cost-benefit analysis, and resource allocation.

[0057] (2) Agricultural scenario capabilities Regional data query and analysis: Based on the given agricultural historical data, it can correctly analyze the various contents of the data and realize the historical information query function. The assessment tasks corresponding to this capability assessment sub-dimension mainly include: ① Historical information query: It can automatically extract and generate the corresponding historical information query results according to the specified format requirements in combination with the relevant agricultural historical data accumulated in different divided regions. ② Regional data analysis: It can accurately divide and classify the location, weather, soil, crops, market, policy and other data in the given agricultural historical data, and extract and analyze the key information therein to provide support for all aspects of agricultural production.

[0058] Agricultural market analysis: By integrating and analyzing market information, it can provide the latest market trends and trend forecasts, and help formulate scientific response strategies. The assessment tasks corresponding to this capability assessment sub-dimension mainly include: ① Market information integration: Ability to collect and integrate market information such as agricultural product prices, supply and demand relationships, and inventory status from different channels. Establish a market information database, classify and organize market information, and provide a data basis for subsequent analysis and forecasting. ② Market trend analysis and forecasting: Ability to analyze historical market data and analyze and identify changing trends in market prices, supply and demand relationships, etc. Predict future market conditions, provide information such as agricultural product price trend forecasts and market demand changes, and help intuitively understand market dynamics. ③ Response strategy planning: Ability to formulate scientific planting and sales strategies based on the results of market trend analysis, and provide risk management suggestions to help cope with risks such as market price fluctuations and changes in supply and demand, and adjust planting structures and sales plans according to policy changes and market demand to optimize agricultural production and management.

[0059] Agricultural meteorological disaster prevention and control: Ability to use meteorological data to predict the possibility of extreme weather events such as heavy rain, drought, frost, and predict the time and scope of occurrence of meteorological disasters. The assessment tasks corresponding to this capability assessment sub-dimension mainly include: ① Meteorological disaster warning: Ability to use meteorological data to predict the possibility of extreme weather events such as heavy rain, drought, frost, and predict the time and scope of occurrence of meteorological disasters. ② Post-disaster remedial measures: Ability to formulate post-disaster recovery plans based on disaster assessment results, including field management, pest and disease prevention and control, water and fertilizer control, soil remediation and other measures, and provide post-disaster subsidies and insurance claims support to reduce economic losses. Summarize the lessons learned from disaster response and recovery, optimize the meteorological disaster management system, and strengthen disaster prevention and response capacity building to improve the resilience of agricultural production.

[0060] Agricultural production guidance: It can provide comprehensive agricultural knowledge questions and answers, professional vocabulary explanations and guidance program generation to help improve production technology levels and achieve scientific planting and efficient management. The assessment tasks corresponding to this capability assessment sub-dimension mainly include: ① Professional vocabulary explanation: It can accurately understand professional vocabulary such as crop varieties, pests and diseases, agricultural equipment, technical terms, etc. in the agricultural field, and provide detailed definitions, application scenarios and learning resources such as relevant pictures or videos for each professional vocabulary. ② Agricultural knowledge questions and answers: It can establish an agricultural knowledge base covering crop cultivation, soil science, agricultural ecology, plant pathology, agricultural entomology, meteorology, etc. to answer common questions, and support multiple query methods such as text input and voice input. At the same time, according to user feedback, it optimizes and expands the entries and answers in the knowledge base. ③ Agricultural technical guidance suggestions: It can generate personalized agricultural production guidance plans in terms of planting planning, fertilization suggestions, pest and disease control, irrigation management, etc. based on specific needs and regional agricultural data, and dynamically optimize and improve the guidance plan based on feedback and actual results to improve practicality and effectiveness.

[0061] Biosafety management: Able to provide effective farmland ecological protection measures and emergency response measures to ensure biosafety in agricultural production, reduce the harm of diseases and natural disasters to crops, and improve the safety and stability of agricultural production. The assessment tasks corresponding to this capability assessment sub-dimension mainly include: ① Farmland ecological protection: Able to provide biological control technologies such as natural enemy insects, microbial preparations, and biological pesticides to replace chemical pesticides, and promote ecological agricultural models such as mixed cropping, crop rotation, and intercropping to reduce the occurrence of pests and diseases and improve the sustainability of agricultural production. ② Emergency response: Able to formulate detailed emergency plans and measures including event monitoring, early warning release, emergency response, and post-event evaluation for different types of natural disaster emergencies to ensure effective response to emergencies.

[0062] Assisted management decision-making: It can provide scientific management decision support for agricultural production through the fusion of multivariate data and the application of agricultural knowledge, and improve the management efficiency and benefits of agricultural production. The assessment tasks corresponding to this capability assessment sub-dimension mainly include: ① Multi-source data fusion: It can collect and integrate meteorological, soil, crop, market and other data from different sources, and make intelligent predictions such as crop yield prediction and market price prediction in agricultural production based on historical data and real-time data, produce prediction results and suggestions, and help formulate scientific production and management decision-making plans. ② Comprehensive application of agricultural knowledge: It can establish an agricultural knowledge base covering all aspects of agricultural production, including planting technology, pest and disease control, soil management, market analysis, etc., and intelligently recommend relevant agricultural knowledge and technical solutions from the knowledge base based on specific needs and production conditions. ③ Related variety recommendation: It can intelligently recommend seed varieties, fertilizers, pesticides, agricultural machinery and other related agricultural products according to the planting area and requirements, and provide detailed product information, usage methods, user evaluation and other usage guidance and technical support, and collect and analyze user feedback on the use of recommended products, evaluate the actual effect of products and optimize product recommendations.

[0063] (3) Security and compliance capabilities Social and cultural impact: The assessment tasks corresponding to this sub-dimension mainly include: ① Sensitive topics: refuse to answer all sensitive topics (such as false information, input content that may disrupt economic and social order, etc.). ② Exclude prejudice: refuse to answer or guide correct ideas for inputs containing discriminatory content such as region, gender, age, and occupation. ③ Negative value: refute negative value content in the input or guide positive value.

[0064] Legal compliance: The assessment tasks corresponding to this sub-dimension mainly include: ① Rights infringement: Refuse to answer or refute the relevant infringing content in the input that infringes on the legitimate interests of others, harms the physical and mental health of others. ② Malicious criticism: Prompt malicious criticism and other content in the input, and refuse to answer in serious cases. ③ Illegal and disorderly: Refuse to answer or warn if the input contains illegal and disorderly content.

[0065] Personal safety: The evaluation tasks corresponding to this evaluation sub-dimension mainly include: ① Privacy and safety: Refuse to answer questions related to privacy and safety in the input. ② Personal harm: For inputs containing personal harm and other content, directly refuse to answer or guide the correct value orientation. ③ Mental health: For inputs containing content that affects mental health, output correct value-oriented content.

[0066] Based on the above embodiment, in step 120, obtaining the evaluation data of each evaluation task under any capability evaluation dimension includes: Step 121, based on each evaluation task under any capability evaluation dimension, obtain initial evaluation data of each evaluation task.

[0067] Specifically, for each assessment task under any capability assessment dimension, when obtaining the corresponding assessment data for the assessment task, the initial assessment data of the assessment task can be obtained from the public data set according to the specific requirements of the assessment task and the data input characteristics of the model to be assessed. The initial assessment data of the assessment task can also be generated by self-construction. Here, the initial assessment data refers to the most original and unprocessed data collected for each assessment task under a certain capability assessment dimension.

[0068] Step 122, pre-processing the initial evaluation data of each evaluation task to obtain the original evaluation data of each evaluation task.

[0069] Specifically, for each evaluation task, after obtaining the corresponding initial evaluation data, the tree pool evaluation data can be preprocessed to improve the quality and usability of the data, thereby obtaining the original evaluation data of the evaluation task. For example, the initial evaluation data can be preprocessed by data cleaning (removing invalid, repeated or erroneous data), labeling, formatting (converting the data into a format suitable for analysis) and other preprocessing operations. It should be understood that the original evaluation data refers to the evaluation data that has been preprocessed for further analysis and evaluation, and the invalid and erroneous information has been removed and converted into a format suitable for analysis.

[0070] Step 123 , based on the data organization form, the original evaluation data of each evaluation task is organized into evaluation data in different forms.

[0071] It should be noted that, considering that evaluation data in a single organizational form cannot comprehensively evaluate the performance and potential capabilities of large agricultural models, in order to overcome this defect, the embodiment of the present invention introduces a variety of data organization forms. By organizing the original evaluation data of each evaluation task into evaluation data in different forms, the performance and potential capabilities of the model can be evaluated more comprehensively.

[0072] Specifically, the data organization form refers to organizing and presenting the data in a specific way for subsequent analysis and evaluation. Here, for the evaluation data of each evaluation task, its data organization form may include one or more of the zero-sample form, the zero-sample thinking chain form, the small sample form, and the small sample thinking chain form. These forms are intended to present data in different ways in order to more comprehensively evaluate the performance and potential capabilities of the model. For example, the small sample form can evaluate the generalization ability of the model with limited training data; while the thinking chain form can evaluate the logical reasoning ability of the model in solving complex problems. By combining different forms of evaluation data, the performance and potential capabilities of the model can be more accurately evaluated, and strong support can be provided for the improvement and optimization of the model.

[0073] It is understandable that for each evaluation task, when reorganizing the corresponding original evaluation data into different forms of evaluation data, the required data organization form can be determined according to the characteristics and requirements of the evaluation task, and then the original evaluation data can be processed accordingly according to the selected form. For example, in the zero-sample thinking chain form, multiple thinking chain prompts can be introduced to enrich the data; in the small sample form, sample examples can be introduced to assist in evaluation. Finally, the processed data is organized and presented in the selected form to obtain the evaluation data of the evaluation task.

[0074] Based on any of the above embodiments, the data organization form includes at least one of a zero-sample form, a zero-sample thinking chain form, a small sample form and a small-sample thinking chain form. The zero-sample form refers to the original evaluation data itself, the zero-sample thinking chain form refers to the introduction of multiple thinking chain prompts into the original evaluation data, the small-sample form refers to the introduction of sample examples into the original evaluation data, and the small-sample thinking chain form refers to the introduction of sample examples and multiple thinking chain prompts into the original evaluation data.

[0075] Specifically, in order to comprehensively evaluate the performance and potential capabilities of large agricultural models, the original evaluation data can be reorganized into different forms of evaluation data. Among them, the zero-sample form refers to directly using the original evaluation data itself for evaluation without introducing any additional information or examples. In this form, the model needs to understand and process the input data based on its learned knowledge and capabilities, and generate corresponding outputs. Figure 3 is a schematic diagram of the evaluation data in the form of zero samples provided by the present invention, such as Figure 3 As shown, the evaluation data includes input data and labels corresponding to the input data. Figure 3 The content of the output part is the annotation content (also called label) corresponding to the input data.

[0076] The zero-sample thinking chain form introduces a multi-chain-of-thought (McoT) setting based on the zero-sample form. Chain-of-thought (CoT) is a prompt that prompts the large model to generate a text sequence of the reasoning process and the final answer, which has shown great success in reasoning tasks. Based on the chain of thought, the embodiment of the present invention designs a more complex multi-chain of thought prompt, which aims to guide the large model to generate multiple reasoning processes and keep the reasoning results consistent. This prompt setting is conducive to allowing the agricultural large model to perform more complex reasoning tasks. Here, multi-chain of thought prompts refer to a series of logically interrelated and step-by-step derived prompts, which are intended to guide the model to process input data according to a specific way of thinking or path. In this way, it can be evaluated whether the model can reason and judge according to the expected way of thinking when understanding and processing complex problems. Figure 4 is a schematic diagram of the evaluation data in the form of a zero-sample thinking chain provided by the present invention, such as Figure 4 As shown in the figure, the content of the output part is the annotation content of the input data. Multi-thinking chain prompts are introduced in this part to help the model better understand and deal with complex problems.

[0077] Small sample form refers to the introduction of a small number of sample examples into the original evaluation data to assist in evaluation. These sample examples are usually related to the evaluation task and contain input data and corresponding output data. Here, sample examples refer to the input data and corresponding output data pairs used to train or evaluate the model. Different from the data organization form of zero samples, data in small sample form is more conducive to evaluating the potential of large agricultural models. Figure 5 is a schematic diagram of the evaluation data in the form of a small sample provided by the present invention, such as Figure 5 Shown is a large agricultural model evaluated in a two-sample setting, where the input data contains sample examples of two inputs and outputs.

[0078] The small sample thinking chain form introduces multiple thinking chain prompts to assist evaluation based on the small sample form. By introducing sample examples and multiple thinking chain prompts, the model can be guided to process input data in the expected way of thinking and generate corresponding output. Figure 6 This is a schematic diagram of evaluation data in the form of a small sample thinking chain provided by the present invention, such as Figure 6 As shown in the figure, two input and output sample examples are introduced in the input data part, and the output content in the sample examples is organized in the form of multi-thinking chains. This form is more conducive to further exploring the potential capabilities of large agricultural models by integrating small samples with data in the form of multi-thinking chains.

[0079] Based on any of the above embodiments, the model evaluation index system includes automatic evaluation indicators and / or manual evaluation indicators, the automatic evaluation indicators include at least one of accuracy, micro-F1 value, bilingual evaluation substitute, recall-oriented evaluation for summary evaluation, vector average cosine similarity, and rejection rate, and the manual evaluation indicators include at least one of correctness, relevance, completeness, effectiveness, and coherence.

[0080] Specifically, for each evaluation task, after collecting and collating the corresponding evaluation data, the evaluation data can be input into the model to be evaluated to obtain and save the output result of the model. Subsequently, the output result of the model can be evaluated. In the embodiment of the present invention, in order to achieve a more accurate evaluation of the agricultural large model, a set of agricultural large model evaluation index system consisting of automatic evaluation index + manual evaluation index is constructed.

[0081] Among them, the automatic evaluation indicators include: 1) Accuracy: Accuracy is the ratio between the number of correctly classified samples and the total number of samples. The accuracy can be calculated by the following formula: : In the above formula, It refers to the number of correctly predicted positive samples, that is, the number of samples predicted to be positive and actually positive; It refers to the number of samples predicted to be positive but actually negative; It refers to the number of samples predicted as negative but actually positive; It refers to the number of negative samples predicted correctly, that is, the number of samples predicted to be negative and actually negative.

[0082] 2) micro-F1 value: micro-F1 value is an accuracy evaluation indicator for common machine learning tasks such as classification, extraction, and error correction. It is calculated jointly by precision P and recall R. The F1 value can be calculated using the following formula: 3) BLEU: The full name is Bilingual Evaluation Understudy, which means bilingual evaluation understudy. It is an evaluation indicator for machine translation tasks, and its overall idea is accuracy. The BLEU score is mainly calculated by comparing the overlap between the text output by the model and one or more reference texts. The overlapping words are calculated as n-grams (i.e., phrases of length n). The calculation formula of BLEU-n is as follows: Among them, BLEU-n represents an evaluation indicator based on n-gram accuracy, which is used to evaluate the quality of machine translation. N Represents the maximum length of the n-gram considered. BP is a penalty factor used to penalize short translations. The calculation formula of BP is as follows: Where c is the length of the model output and r is the length of the reference text.

[0083] It is used to calculate the precision of n-gram, that is, the proportion of the generated n-gram in the reference text. The calculation formula is: 4) ROUGE: The full name is Recall-Oriented Understudy for Gisting Evaluation, which is a recall-oriented evaluation for summary evaluation. It is an evaluation indicator that automatically evaluates the quality of text summarization based on recall. The calculation formula of ROUGE-n is as follows: in, refers to the number of n-grams that match between the model-generated content and the reference text. Refers to the total number of n-grams that appear in the reference text. Refers to multiple reference texts.

[0084] 5) Vector average cosine similarity: Based on the above-mentioned BLEU, ROUGE and other indicators, the embodiment of the present invention introduces the text embedding model BGE, embeds the model output and the standard output in the data set into the vector space, and measures the similarity between the model output and the expected output through the vector cosine similarity. This approach is more conducive to measuring the quality of the model output at the semantic level. The calculation formula is as follows: Among them, the model output is A, the expected output is B, and both A and B are of dimension n A high-dimensional vector, , Refers to the first i dimensional component, this formula is used to calculate the cosine similarity between the model output A and the expected output B.

[0085] The manual evaluation indicators are as follows: 1) Correctness Correctness refers to the degree to which the response content generated by the agricultural model is true and correct. The evaluation score of correctness is divided into 6 levels, with a full score of 20 points. The specific classification rules are shown in Table 1 below: Table 1 Correctness grading rules

[0086] 2) Relevance Relevance refers to the degree of association between the answer of the agricultural model and a series of user input information. The evaluation score of relevance is divided into 6 levels, with a full score of 20 points. The specific classification rules are shown in Table 2 below: Table 2 Relevance classification rules

[0087] 3) Completeness Completeness refers to whether the answer generated by the agricultural model contains any missing information. The completeness evaluation score is divided into 6 levels, with a full score of 20 points. The specific grading rules are shown in Table 3 below: Table 3 Completeness classification rules

[0088] 4) Effectiveness Effectiveness refers to the usefulness of the answers generated by the agricultural big model. The evaluation score of effectiveness is divided into 6 levels, with a full score of 20 points. The specific classification rules are shown in Table 4 below: Table 4 Effectiveness grading rules

[0089] 5) Coherence Coherence refers to whether the answers generated by the agricultural model conform to the dialogue process. The evaluation score of coherence is divided into 6 levels, with a full score of 20 points. The specific grading rules are shown in Table 5 below: Table 5 Coherence grading rules

[0090] Based on any of the above embodiments, the model assessment capability system includes multiple capability assessment dimensions for evaluating the model to be evaluated and the weight of each capability assessment dimension, each capability assessment dimension includes multiple assessment sub-dimensions and the weight of each assessment sub-dimension, and each assessment sub-dimension includes multiple assessment tasks and the weight of each assessment task.

[0091] Specifically, in the model evaluation capability system, each capability evaluation dimension is further subdivided into multiple more specific evaluation aspects, which are called evaluation sub-dimensions. Evaluation sub-dimensions are further refinements and concretizations of capability evaluation dimensions in order to more comprehensively evaluate the performance of the model in different aspects. Since different evaluation sub-dimensions may have different impacts on the overall performance of the model, a weight can be assigned to each evaluation sub-dimension to reflect its importance in the overall evaluation.

[0092] Under each evaluation sub-dimension, a series of specific evaluation tasks are designed to actually perform the model evaluation work. These evaluation tasks are specific operations or experiments of the model evaluation, which are used to collect data and evaluate the performance of the model under this evaluation sub-dimension. Similarly, since different evaluation tasks may have different degrees of influence on the evaluation sub-dimension, a weight can be assigned to each evaluation task to reflect its importance in this evaluation sub-dimension.

[0093] According to the specific needs and characteristics of each evaluation sub-dimension, appropriate evaluation indicators can be selected from the model evaluation indicator system to accurately evaluate the various evaluation tasks under the evaluation sub-dimension. In the model evaluation capability system, the weights of each capability evaluation dimension, the evaluation sub-dimensions and their weights under each capability evaluation dimension, the evaluation tasks and their weights, and the evaluation indicators and their weights are as follows: 1) Agricultural basic capabilities (weight 40%) Table 6 Agricultural basic capability system

[0094] 2) Agricultural scenario capabilities (weight 40%) Table 7 Agricultural scenario capability system

[0095] 3) Security and compliance capabilities (weight 20%)

[0096] In the embodiment of the present invention, by dividing multiple capability assessment dimensions, assessment sub-dimensions and assessment tasks, a comprehensive evaluation of model performance can be ensured, covering multiple key aspects. By setting weights for each dimension, sub-dimension and task, the key points and important aspects in the model assessment can be highlighted, making the assessment results more targeted and instructive.

[0097] Based on any of the above embodiments, step 130 specifically includes: Step 131, based on any evaluation sub-dimension under any capability evaluation dimension, determining an evaluation index of any evaluation sub-dimension from the model evaluation index system; Step 132, based on each evaluation indicator, the model output result corresponding to each evaluation task under any evaluation sub-dimension is evaluated to obtain the evaluation result of any evaluation sub-dimension; Step 133: Determine the evaluation result of any capability evaluation dimension based on the evaluation results of each evaluation sub-dimension under any capability evaluation dimension.

[0098] Specifically, for each evaluation sub-dimension under each capability evaluation dimension, the evaluation indicators of the evaluation sub-dimension can be determined from the model evaluation indicator system according to the evaluation requirements and characteristics of the sub-dimension. In addition, since the model evaluation capability system also includes evaluation indicators corresponding to each evaluation sub-dimension, the evaluation indicators corresponding to each evaluation sub-dimension can also be directly determined from the system. Subsequently, for each evaluation task under any evaluation sub-dimension, after inputting the evaluation data corresponding to each evaluation task into the model to be evaluated and obtaining the model output result, the model output result of each evaluation task can be evaluated and scored according to the determined evaluation indicators.

[0099] Exemplarily, for the agricultural text segmentation task under the evaluation sub-dimension of agricultural cognitive ability, after obtaining the model output results of the task, the accuracy and micro-F1 value are calculated according to each output result, and the score of each evaluation indicator is determined. The score of the evaluation task can be obtained by weighted summing the score and weight of each evaluation indicator. After obtaining the scores of all evaluation tasks under agricultural cognitive ability, the score of the evaluation sub-dimension (i.e., agricultural cognitive ability) can be determined by calculating according to the score of each evaluation task and the corresponding weight. After obtaining the scores of all evaluation sub-dimensions under agricultural basic ability, the score of each evaluation sub-dimension and the corresponding weight can be weighted to obtain the score of the ability evaluation dimension (i.e., agricultural basic ability), which can be used as the evaluation result.

[0100] Based on any of the above embodiments, step 132 specifically includes: Step 1321, based on each evaluation indicator, evaluate the model output result corresponding to each evaluation task under any evaluation sub-dimension to obtain an evaluation score of each evaluation task; Step 1322, based on the weight and evaluation score of each evaluation task, determine the evaluation score of any evaluation sub-dimension, and use the evaluation score as the evaluation result of any evaluation sub-dimension; Accordingly, step 133 specifically includes: Step 1331, based on the weights and evaluation scores of each evaluation sub-dimension under any capability evaluation dimension, determine the evaluation score of any capability evaluation dimension.

[0101] Specifically, for each evaluation task, prepare the corresponding evaluation data, run the agricultural model to be evaluated, and obtain the model output results. According to the model output results and the true labels corresponding to the evaluation data, calculate the score of each evaluation task on each evaluation indicator. According to the weight and evaluation score of each evaluation task, the score of the evaluation sub-dimension can be calculated using the weighted summation method. According to the weight and evaluation score of each evaluation sub-dimension, the score of the ability evaluation dimension is also calculated using the weighted summation method.

[0102] For example, taking the evaluation sub-dimension of agricultural cognitive ability as an example, the evaluation score of each evaluation task can be calculated according to the evaluation indicators and their weights. Then, the evaluation scores of all evaluation tasks under agricultural cognitive ability are weighted and summed to obtain the score of the evaluation sub-dimension of agricultural cognitive ability. Subsequently, the scores of other evaluation sub-dimensions are calculated in the same way, and finally the weighted sum is obtained to obtain the score of the agricultural basic ability evaluation dimension. Finally, the scores of other ability evaluation dimensions are calculated in the same way, and the scores of each ability evaluation dimension are weighted and summed to obtain the comprehensive score of the model to be evaluated.

[0103] Based on any of the above embodiments, step 1321 specifically includes: Based on each evaluation indicator, the model output result corresponding to any evaluation task under any evaluation sub-dimension is evaluated to obtain the evaluation score corresponding to each evaluation indicator under any evaluation task; Based on the weights and evaluation scores of the evaluation indicators, the evaluation score of any evaluation task is determined.

[0104] Specifically, for each evaluation task, the performance of the model on the evaluation task can be quantitatively evaluated according to the determined evaluation indicators and their weights, so as to obtain the evaluation score of the evaluation task. For example, taking the evaluation task of agricultural text segmentation as an example, after collecting or generating the evaluation data for the agricultural text segmentation task, these data are input into the agricultural large model to obtain the segmentation result of the model on the evaluation data, and the accuracy is calculated according to the segmentation result of the model and the real segmentation result in the evaluation data. The micro-F1 value is also calculated based on the segmentation result of the model and the real segmentation result. Subsequently, the evaluation scores of the two evaluation indicators of "accuracy" and "micro-F1 value" are weighted and summed using the pre-assigned weights to obtain the evaluation score of the evaluation task of agricultural text segmentation. Assuming that the weight of the accuracy is 0.5, the weight of the micro-F1 value is 0.5, the accuracy of the model on the agricultural text segmentation task is 0.85, and the micro-F1 value is 0.80, then the score of the evaluation task is 0.825.

[0105] Based on any of the above embodiments, after step 140, the method further includes: Based on each capability assessment dimension, each assessment sub-dimension under each capability assessment dimension, the assessment indicators and assessment scores corresponding to each assessment sub-dimension, and the assessment result of the model to be assessed, an assessment report of the model to be assessed is generated.

[0106] Specifically, Figure 7 Schematic diagram of the prompt word template generated by the evaluation report provided by the present invention, such as Figure 7 As shown in the figure, according to the evaluation scores and evaluation indicators of each capability evaluation dimension and capability evaluation sub-dimension, the general large model can use prompts to output an evaluation summary report, including evaluation results, problem analysis, model optimization suggestions, etc., and provide a reference for future model iterations to promote continuous optimization of the model.

[0107] Based on any of the above embodiments, the embodiment of the present invention proposes a multi-level, multi-dimensional professional evaluation system for large agricultural models, which provides a reliable solution for the objective and accurate evaluation of large agricultural models. The method includes: Step S1, data preparation Construct a set of agricultural large model evaluation capability system consisting of three levels: agricultural basic capability, agricultural scenario capability, and safety and compliance capability, and further subdivide it under each capability dimension. For example, the agricultural basic capability is divided into four sub-capabilities: agricultural cognition, agricultural knowledge, agricultural content generation, and agricultural logical reasoning. Define the evaluation tasks of the agricultural large model under each sub-capability, collect and organize different types of agricultural evaluation data according to the specific needs of the evaluation task, ensure that the data covers multiple aspects such as agricultural basic data, market data, meteorological data, etc., and clean, annotate and format the evaluation data to obtain the original evaluation data. Furthermore, the original evaluation data is reorganized in four forms: zero sample, small sample, zero sample thinking chain, and small sample thinking chain.

[0108] Step S2: Capability Assessment The evaluation data of each evaluation task in the capability system is input step by step into the agricultural model to be tested, and the output results of the agricultural model are obtained and saved.

[0109] Step S3, model scoring Using the agricultural large model evaluation index system, the model output is weighted and scored through both automatic and manual evaluation. Specifically, the performance of the model is calculated using automatic evaluation indicators such as accuracy, micro-F1 value, BLEU, ROUGE, and vector average cosine similarity, and a team of agricultural industry experts is organized to manually evaluate the content generated by the model to assess its quality, coherence, and suitability. The scores of each model capability and the overall score of the model are calculated based on the scoring rules and weight ratios in the evaluation system.

[0110] Step S4: Summarize feedback Finally, the evaluation results of various basic capabilities, scenario capabilities, and security capabilities are scored and integrated and input into the general large model, and the final evaluation summary report is output through prompt word engineering.

[0111] The model evaluation device provided by the present invention is described below. The model evaluation device described below and the model evaluation method described above can be referenced to each other.

[0112] Based on any of the above embodiments, Figure 8 is a schematic diagram of the structure of the model evaluation device provided by the present invention, such as Figure 8 As shown, the device comprises: A dimension determination unit 810 is used to determine a plurality of capability evaluation dimensions for evaluating the model to be evaluated and a plurality of evaluation tasks under each capability evaluation dimension based on a model evaluation capability system, wherein the model evaluation capability system is pre-constructed based on the industry attributes of the model to be evaluated; The model output unit 820 is used to obtain the evaluation data of each evaluation task under any capability evaluation dimension, and input the evaluation data into the model to be evaluated to obtain the model output result corresponding to each evaluation task under any capability evaluation dimension; A capability evaluation unit 830 is used to evaluate the model output results corresponding to each evaluation task under any capability evaluation dimension based on the model evaluation indicator system to obtain the evaluation result of any capability evaluation dimension; The result determination unit 840 is used to determine the evaluation result of the model to be evaluated based on the evaluation result of each capability evaluation dimension and the weight of each capability evaluation dimension.

[0113] The device provided by the embodiment of the present invention pre-constructs a model evaluation capability system in a targeted manner based on the industry attributes of the model to be evaluated, so that the capability system can fully cover the various capabilities required by the model in the industry to which it belongs. By determining multiple capability evaluation dimensions and multiple evaluation tasks under each dimension based on the capability system, it is possible to ensure a comprehensive evaluation of the model performance and accurately measure the performance of the model under different capability dimensions, thereby improving the accuracy and comprehensiveness of the model evaluation and avoiding the one-sidedness of the evaluation. By inputting the evaluation data of each evaluation task into the model to be evaluated and using the model evaluation index system to evaluate the output results, the accuracy and consistency of the evaluation can be ensured, making the evaluation results more valuable for reference. In addition, since the model evaluation capability system is constructed according to the industry attributes of the model, it can be customized for the models to be evaluated in different industries, and has strong flexibility and scalability.

[0114] Based on any of the above embodiments, the model output unit 820 includes a data acquisition subunit, and the data acquisition subunit is used to: Based on each evaluation task under any of the capability evaluation dimensions, obtaining initial evaluation data for each evaluation task; Preprocess the initial evaluation data of each evaluation task to obtain the original evaluation data of each evaluation task; Based on the data organization form, the original evaluation data of each evaluation task is organized into evaluation data of different forms.

[0115] Based on any of the above embodiments, the data organization form includes at least one of a zero-sample form, a zero-sample thinking chain form, a small sample form and a small sample thinking chain form. The zero-sample form refers to the original evaluation data itself, the zero-sample thinking chain form refers to the introduction of multiple thinking chain prompts into the original evaluation data, the small sample form refers to the introduction of sample examples into the original evaluation data, and the small sample thinking chain form refers to the introduction of sample examples and multiple thinking chain prompts into the original evaluation data.

[0116] Based on any of the above embodiments, the model assessment capability system includes multiple capability assessment dimensions for evaluating the model to be evaluated and the weight of each capability assessment dimension, each capability assessment dimension includes multiple assessment sub-dimensions and the weight of each assessment sub-dimension, and each assessment sub-dimension includes multiple assessment tasks and the weight of each assessment task.

[0117] Based on any of the above embodiments, the capability evaluation unit 830 includes: An indicator determination subunit, configured to determine an evaluation indicator of any evaluation sub-dimension from the model evaluation indicator system based on any evaluation sub-dimension under any capability evaluation dimension; The dimension evaluation subunit is used to evaluate the model output results corresponding to each evaluation task under any evaluation sub-dimension based on each evaluation indicator to obtain the evaluation result of any evaluation sub-dimension; The result determination subunit is used to determine the evaluation result of any capability evaluation dimension based on the evaluation results of each evaluation sub-dimension under any capability evaluation dimension.

[0118] Based on any of the above embodiments, the dimension evaluation subunit includes: The task evaluation subunit is used to evaluate the model output results corresponding to each evaluation task under any evaluation sub-dimension based on each evaluation indicator to obtain an evaluation score for each evaluation task; A score determination subunit, used to determine the evaluation score of any evaluation sub-dimension based on the weight and evaluation score of each evaluation task, and use the evaluation score as the evaluation result of any evaluation sub-dimension; Accordingly, the result determination subunit is specifically used for: Based on the weights and evaluation scores of each evaluation sub-dimension under any capability evaluation dimension, the evaluation score of any capability evaluation dimension is determined.

[0119] Based on any of the above embodiments, the task evaluation subunit is specifically used for: Based on each evaluation indicator, the model output result corresponding to any evaluation task under any evaluation sub-dimension is evaluated to obtain the evaluation score corresponding to each evaluation indicator under any evaluation task; Based on the weights and evaluation scores of the evaluation indicators, the evaluation score of any evaluation task is determined.

[0120] Based on any of the above embodiments, the device further includes a report generating unit, and the report generating unit is used to: Based on each capability assessment dimension, each assessment sub-dimension under each capability assessment dimension, the assessment indicators and assessment scores corresponding to each assessment sub-dimension, and the assessment result of the model to be assessed, an assessment report of the model to be assessed is generated.

[0121] Based on any of the above embodiments, the model evaluation index system includes automatic evaluation indicators and / or manual evaluation indicators, the automatic evaluation indicators include at least one of accuracy, micro-F1 value, bilingual evaluation substitute, recall-oriented evaluation for summary evaluation, vector average cosine similarity, and rejection rate, and the manual evaluation indicators include at least one of correctness, relevance, completeness, effectiveness, and coherence.

[0122] Based on any of the above embodiments, the model to be evaluated includes a large agricultural model, and the capability evaluation dimensions include at least two of agricultural basic capabilities, agricultural scenario capabilities, and safety and compliance capabilities.

[0123] Based on any of the above embodiments, when the capability assessment dimension includes agricultural basic capability, the capability assessment sub-dimension under the capability assessment dimension includes at least one of agricultural cognitive capability, agricultural content generation capability, agricultural knowledge reserve capability, and agricultural logical reasoning capability; In the case where the capability assessment dimension includes agricultural scenario capability, the capability assessment sub-dimension includes at least one of regional data query analysis, agricultural market analysis, agricultural meteorological disaster prevention and control, agricultural production guidance, biosafety management, and auxiliary management decision-making; In the case where the capability assessment dimension includes security compliance capability, the capability assessment sub-dimension includes at least one of social and cultural impact, legal compliance, and personal safety.

[0124] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9 As shown, the electronic device may include: a processor 910 , a communication interface 920 , a memory 930 and a communication bus 940 , wherein the processor 910 , the communication interface 920 , and the memory 930 communicate with each other via the communication bus 940 . The processor 910 can call the logic instructions in the memory 930 to execute the model evaluation method, which includes: based on the model evaluation capability system, determining multiple capability evaluation dimensions and multiple evaluation tasks under each capability evaluation dimension to be evaluated for the model to be evaluated, and the model evaluation capability system is pre-constructed based on the industry attributes of the model to be evaluated; obtaining evaluation data of each evaluation task under any capability evaluation dimension, and inputting the evaluation data into the model to be evaluated to obtain the model output result corresponding to each evaluation task under any capability evaluation dimension; based on the model evaluation indicator system, evaluating the model output result corresponding to each evaluation task under any capability evaluation dimension to obtain the evaluation result of any capability evaluation dimension; based on the evaluation results of each capability evaluation dimension and the weight of each capability evaluation dimension, determining the evaluation result of the model to be evaluated.

[0125] In addition, the logic instructions in the above-mentioned memory 1030 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0126] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the model evaluation method provided by the above methods, which includes: based on a model evaluation capability system, determining multiple capability evaluation dimensions and multiple evaluation tasks under each capability evaluation dimension to be evaluated for the model to be evaluated, and the model evaluation capability system is pre-constructed based on the industry attributes of the model to be evaluated; obtaining evaluation data for each evaluation task under any capability evaluation dimension, and inputting the evaluation data into the model to be evaluated to obtain the model output result corresponding to each evaluation task under any capability evaluation dimension; based on a model evaluation indicator system, evaluating the model output result corresponding to each evaluation task under any capability evaluation dimension to obtain the evaluation result of any capability evaluation dimension; determining the evaluation result of the model to be evaluated based on the evaluation results of each capability evaluation dimension and the weight of each capability evaluation dimension.

[0127] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the model evaluation method provided by the above-mentioned methods, the method comprising: based on a model evaluation capability system, determining multiple capability evaluation dimensions for evaluating the model to be evaluated and multiple evaluation tasks under each capability evaluation dimension, wherein the model evaluation capability system is pre-constructed based on the industry attributes of the model to be evaluated; obtaining evaluation data for each evaluation task under any capability evaluation dimension, and inputting the evaluation data into the model to be evaluated to obtain a model output result corresponding to each evaluation task under any capability evaluation dimension; based on a model evaluation indicator system, evaluating the model output result corresponding to each evaluation task under any capability evaluation dimension to obtain an evaluation result of any capability evaluation dimension; determining the evaluation result of the model to be evaluated based on the evaluation results of each capability evaluation dimension and the weight of each capability evaluation dimension.

[0128] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0129] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A model evaluation method, characterized in that: include: Based on the model evaluation capability system, determining multiple capability evaluation dimensions for evaluating the model to be evaluated and multiple evaluation tasks under each capability evaluation dimension, wherein the model evaluation capability system is pre-constructed based on the industry attributes of the model to be evaluated; Obtaining evaluation data of each evaluation task under any capability evaluation dimension, and inputting the evaluation data into the model to be evaluated, to obtain the model output result corresponding to each evaluation task under any capability evaluation dimension; Based on the model evaluation index system, the model output results corresponding to each evaluation task under any capability evaluation dimension are evaluated to obtain the evaluation results of any capability evaluation dimension; Based on the evaluation results of each capability evaluation dimension and the weight of each capability evaluation dimension, the evaluation result of the model to be evaluated is determined.

2. The model evaluation method according to claim 1, characterized in that: The obtaining of evaluation data of each evaluation task under any capability evaluation dimension includes: Based on each evaluation task under any of the capability evaluation dimensions, obtaining initial evaluation data for each evaluation task; Preprocess the initial evaluation data of each evaluation task to obtain the original evaluation data of each evaluation task; Based on the data organization form, the original evaluation data of each evaluation task is organized into evaluation data of different forms.

3. The model evaluation method according to claim 2, characterized in that: The data organization form includes at least one of a zero-sample form, a zero-sample thinking chain form, a small-sample form and a small-sample thinking chain form. The zero-sample form refers to the original evaluation data itself, the zero-sample thinking chain form refers to the introduction of multiple thinking chain prompts into the original evaluation data, the small-sample form refers to the introduction of sample examples into the original evaluation data, and the small-sample thinking chain form refers to the introduction of sample examples and multiple thinking chain prompts into the original evaluation data.

4. The model evaluation method according to claim 1, characterized in that: The model evaluation capability system includes multiple capability evaluation dimensions for evaluating the model to be evaluated and the weight of each capability evaluation dimension. Each capability evaluation dimension includes multiple evaluation sub-dimensions and the weight of each evaluation sub-dimension. Each evaluation sub-dimension includes multiple evaluation tasks and the weight of each evaluation task.

5. The model evaluation method according to claim 4, characterized in that: The model-based evaluation index system evaluates the model output results corresponding to each evaluation task under any capability evaluation dimension to obtain the evaluation results of any capability evaluation dimension, including: Based on any evaluation sub-dimension under any capability evaluation dimension, determining an evaluation indicator of any evaluation sub-dimension from the model evaluation indicator system; Based on each evaluation indicator, the model output result corresponding to each evaluation task under any evaluation sub-dimension is evaluated to obtain the evaluation result of any evaluation sub-dimension; Based on the evaluation results of each evaluation sub-dimension under any capability evaluation dimension, the evaluation result of any capability evaluation dimension is determined.

6. The model evaluation method according to claim 5, characterized in that: The step of evaluating the model output results corresponding to each evaluation task under any evaluation sub-dimension based on each evaluation indicator to obtain the evaluation result of any evaluation sub-dimension includes: Based on each evaluation indicator, the model output result corresponding to each evaluation task under any evaluation sub-dimension is evaluated to obtain the evaluation score of each evaluation task; Determine the evaluation score of any evaluation sub-dimension based on the weight and evaluation score of each evaluation task, and use the evaluation score as the evaluation result of any evaluation sub-dimension; Determining the evaluation result of any capability evaluation dimension based on the evaluation results of each evaluation sub-dimension under any capability evaluation dimension includes: Based on the weights and evaluation scores of each evaluation sub-dimension under any capability evaluation dimension, the evaluation score of any capability evaluation dimension is determined.

7. The model evaluation method according to claim 6, characterized in that: Based on each evaluation indicator, the model output result corresponding to each evaluation task under any evaluation sub-dimension is evaluated to obtain the evaluation score of each evaluation task, including: Based on each evaluation indicator, the model output result corresponding to any evaluation task under any evaluation sub-dimension is evaluated to obtain the evaluation score corresponding to each evaluation indicator under any evaluation task; Based on the weights and evaluation scores of the evaluation indicators, the evaluation score of any evaluation task is determined.

8. The model evaluation method according to claim 1, characterized in that: Also includes: Based on each capability assessment dimension, each assessment sub-dimension under each capability assessment dimension, the assessment indicators and assessment scores corresponding to each assessment sub-dimension, and the assessment result of the model to be assessed, an assessment report of the model to be assessed is generated.

9. The model evaluation method according to any one of claims 1 to 8, characterized in that: The model evaluation index system includes automatic evaluation indicators and / or manual evaluation indicators. The automatic evaluation indicators include at least one of accuracy, micro-F1 value, bilingual evaluation substitute, recall-oriented evaluation for summary evaluation, vector average cosine similarity, and rejection rate. The manual evaluation indicators include at least one of correctness, relevance, completeness, effectiveness, and coherence.

10. The model evaluation method according to any one of claims 1 to 8, characterized in that: The model to be evaluated includes a large agricultural model, and the capability evaluation dimensions include at least two of agricultural basic capabilities, agricultural scenario capabilities, and safety and compliance capabilities.

11. The model evaluation method according to claim 10, characterized in that: In the case where the capability assessment dimension includes agricultural basic capability, the capability assessment sub-dimensions under the capability assessment dimension include at least one of agricultural cognitive capability, agricultural content generation capability, agricultural knowledge reserve capability, and agricultural logical reasoning capability; In the case where the capability assessment dimension includes agricultural scenario capability, the capability assessment sub-dimension includes at least one of regional data query analysis, agricultural market analysis, agricultural meteorological disaster prevention and control, agricultural production guidance, biosafety management, and auxiliary management decision-making; In the case where the capability assessment dimension includes security compliance capability, the capability assessment sub-dimension includes at least one of social and cultural impact, legal compliance, and personal safety.

12. A model evaluation device, characterized in that: include: A dimension determination unit, used to determine multiple capability evaluation dimensions for evaluating the model to be evaluated and multiple evaluation tasks under each capability evaluation dimension based on a model evaluation capability system, wherein the model evaluation capability system is pre-constructed based on the industry attributes of the model to be evaluated; A model output unit, used to obtain the evaluation data of each evaluation task under any capability evaluation dimension, and input the evaluation data into the model to be evaluated to obtain the model output result corresponding to each evaluation task under any capability evaluation dimension; A capability evaluation unit is used to evaluate the model output results corresponding to each evaluation task under any capability evaluation dimension based on the model evaluation indicator system to obtain the evaluation result of any capability evaluation dimension; The result determination unit is used to determine the evaluation result of the model to be evaluated based on the evaluation result of each capability evaluation dimension and the weight of each capability evaluation dimension.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the model evaluation method according to any one of claims 1 to 11 is implemented.

14. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the model evaluation method according to any one of claims 1 to 11 is implemented.

Images