Large model evaluation method and device

Abstract

The invention provides a large model evaluation method and device, and belongs to the field of artificial intelligence. The method comprises the steps of selecting a to-be-evaluated model based on an interactive interface, and receiving text input of evaluation indication content; selecting a plurality of target indexes from an index database according to the text input; determining an evaluation automation mode corresponding to the target index, and generating an evaluation task according to the evaluation automation mode; monitoring a plurality of evaluation tasks in real time, responding to a target index information modification instruction of any target evaluation task, adjusting a target index corresponding to the target evaluation task, calculating an index value corresponding to a newly added target index based on an evaluation result before receiving the information modification instruction, and modifying and updating an evaluation automation mode based on the changed target index; and continuing to evaluate the target evaluation task by using the updated automation mode and the adjusted target index. According to the method and the device provided by the invention, the problems that the traditional large model evaluation is fixed, the flexibility is poor and the dynamic demand cannot be adapted can be solved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and apparatus for evaluating large models. Background Technology

[0002] With the rapid evolution of artificial intelligence technology, large-scale models are being applied more and more widely in various fields, and the demand for accurate performance evaluation of large-scale models is also continuously growing. Currently, when conducting large-scale model evaluations in the industry, a technical framework with a pre-set fixed process is generally adopted. That is, the evaluation system pre-fixes the set of evaluation indicators (such as only including general indicators such as recognition accuracy and inference time). Users need to determine all evaluation dimensions before starting the task, and after the evaluation task is generated, it is mostly executed in a serial manner. The next indicator can only be started after the evaluation of one indicator is completed. At the same time, the automated processing links mostly adopt a single mode (such as all manual review or full system automatic calculation), without combining the differentiating adaptation of indicator characteristics.

[0003] Existing assessment technologies struggle to meet dynamic and personalized assessment needs, primarily due to a severe lack of flexibility. Because indicators are fixed in advance, users cannot easily convey their personalized needs through natural language input, nor can they add or adjust indicators based on changing scenarios. This results in assessment solutions becoming disconnected from actual application requirements and failing to accurately match the specific assessment needs of different fields.

[0004] Furthermore, existing technologies suffer from inefficiency and a lack of dynamic adjustment capabilities. The serial execution of tasks significantly extends the evaluation cycle for multiple indicators, leading to extremely low resource utilization, especially in complex, multi-dimensional scenarios. Simultaneously, a single automation model either over-relies on simple indicators, resulting in wasted costs, or over-automates complex indicators, leading to biased results. More critically, once an evaluation task is started, it cannot be adjusted midway. Modifying indicator parameters or adding evaluation dimensions requires terminating the current task, deleting historical data, and reconfiguring and restarting, resulting in significant computational and time losses. This rigid process severely hinders evaluation progress, particularly in scenarios with challenging sample collection (such as power equipment defect samples). Summary of the Invention

[0005] In view of this, this application provides a large model evaluation method and apparatus to solve the problems of rigid evaluation process, insufficient flexibility and inability to adapt to dynamic changes in requirements in traditional large model evaluation methods.

[0006] Specifically, this application is implemented through the following technical solution:

[0007] The first aspect of this application provides a method for evaluating large models, the method comprising:

[0008] Select the model to be evaluated based on the interactive interface and receive text input of the evaluation instructions;

[0009] Select multiple target metrics from the metric database based on the text input;

[0010] Determine the automated assessment mode corresponding to the target indicator, and generate assessment tasks according to the automated assessment mode, wherein multiple assessment tasks are performed in parallel;

[0011] The system monitors multiple assessment tasks in real time, responds to any target assessment task's target indicator information modification instruction, adjusts the target indicator corresponding to the target assessment task, calculates the indicator value corresponding to the newly added target indicator based on the assessment results that received the information modification instruction earlier, and modifies and updates the assessment automation mode based on the changed target indicator. The automation mode is different for each target indicator.

[0012] The evaluation of the target assessment task will continue with the updated automation mode and adjusted target metrics.

[0013] A second aspect of this application provides a large model evaluation device, the device comprising a receiving module, a selection module, a determination module, a monitoring module, and a processing module;

[0014] The receiving module is used to select the model to be evaluated based on the interactive interface and receive text input of the evaluation instructions.

[0015] The selection module is used to select multiple target indicators from the indicator database based on the text input;

[0016] The determining module is used to determine the evaluation automation mode corresponding to the target indicator, and generate evaluation tasks according to the evaluation automation mode, wherein multiple evaluation tasks are performed in parallel.

[0017] The monitoring module is used to monitor multiple assessment tasks in real time, respond to any target assessment task's target indicator information modification instruction, adjust the target indicator corresponding to the target assessment task, calculate the indicator value corresponding to the newly added target indicator based on the assessment results that received the information modification instruction earlier, and modify and update the assessment automation mode based on the changed target indicator. The automation mode is different for each target indicator.

[0018] The processing module is used to continue the evaluation of the target evaluation task in an updated automated mode and with adjusted target indicators.

[0019] The large-scale model evaluation method and apparatus provided in this application offer a platform for large-scale model evaluation through software interaction, enabling intervention, real-time configuration adjustment, and adaptive calculation during the evaluation process, thus expanding the applicable scenarios of the large-scale model evaluation method. Through a complete large-scale model evaluation process involving requirement input, indicator matching, and subsequent dynamic adjustment of indicators and evaluation modes during the evaluation process, the flexibility, efficiency, and adaptability of the evaluation process are improved, providing dynamic and personalized solutions for large-scale model evaluation in different scenarios. By monitoring multiple evaluation tasks to complete indicator modifications and corresponding automated evaluation mode updates, users can flexibly initiate indicator adjustments during evaluation without interrupting the current task; furthermore, the system reuses evaluation results before receiving modification instructions to calculate new indicator values, avoiding redundant data collection and significantly reducing computational and time consumption. Multiple target indicators can be selected from the indicator database through text input. When users input personalized requirements using natural language, the system can automatically match corresponding target indicators from the indicator database, eliminating the need for users to search or manually configure them one by one. This allows the evaluation to accurately meet the specific needs of different fields and users, improving the relevance and adaptability of the evaluation. Furthermore, by adapting to multi-task parallelism and automation modes, on the one hand, the evaluation tasks of multiple target indicators can be carried out in parallel without waiting for one task to be completed before starting another, which greatly shortens the overall evaluation cycle; on the other hand, each indicator is matched with a dedicated automation mode, avoiding the efficiency loss caused by over-reliance on manual or single automation modes, which can further improve evaluation efficiency and result accuracy. Attached Figure Description

[0020] Figure 1 A flowchart of an embodiment of the large model evaluation method provided in this application;

[0021] Figure 2 This is a schematic diagram of the structure of Embodiment 2 of the large model evaluation device provided in this application. Detailed Implementation

[0022] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0023] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0024] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0025] The following specific embodiments are given to illustrate the technical solution of this application in detail.

[0026] Example 1

[0027] Figure 1 The flowchart for Example 1 of the large model evaluation method provided in this application is shown below. Please refer to... Figure 1 The method provided in this embodiment may include:

[0028] S101. Select the model to be evaluated based on the interactive interface and receive text input of the evaluation instructions.

[0029] It should be noted that the interactive interface is the operating platform through which users interact with the large model evaluation system. It can be presented through a graphical interface (such as a web interface, client software interface, etc.) and provides entry points for functions such as model selection and text input. The evaluation instructions are text information entered by the user to clarify the evaluation requirements of the large model.

[0030] Specifically, users select the desired large model from the model list displayed on the interactive interface; then, they enter text specifying their evaluation requirements in the text input box. During this process, the large model evaluation system pre-builds the functional logic of the interactive interface, including the interaction logic for model selection and the logic for receiving text input. When users interact with the interface, the system's front-end translates the user's selections and inputs into specific instructions, which are then passed to the back-end for processing. The back-end module receives this information, performs initial parsing and storage, and then uses it in subsequent processes.

[0031] In addition to the aforementioned method of allowing users to manually select models and input text through a graphical interface, a configuration file import method can also be used. This involves users pre-writing the identification information of the model to be evaluated and the evaluation instructions into a configuration file of a specific format. Then, through the file import function of the interactive interface, the configuration file is uploaded to the system. The system reads the configuration file content and parses out the model to be evaluated and the evaluation instructions. Compared to the previous method, this approach is suitable for scenarios involving batch evaluation tasks and can improve the efficiency of initiating multiple tasks.

[0032] S102. Select multiple target indicators from the indicator database based on the text input.

[0033] It should be noted that target metrics refer to metrics selected from the metric database that match the evaluation instructions input by the user and are used to conduct large-scale model evaluations. Each metric includes attribute information such as metric name, calculation rules, domain, and applicable scenarios, facilitating system retrieval and matching based on requirements.

[0034] Specifically, the target indicators stored in the indicator database are classified according to the evaluation scenario. In the power equipment inspection scenario, typical target indicators include equipment defect identification accuracy (evaluating the accuracy of defect identification), single sample inference time (evaluating the model running efficiency), defect area positioning accuracy (evaluating the defect location positioning capability), and small defect identification recall rate (evaluating the coverage capability of minor defects).

[0035] It should be noted that the system performs natural language processing on the input evaluation instruction text to extract key information, such as the evaluation objects (e.g., transformers, insulators) and evaluation dimensions (e.g., recognition accuracy, inference speed). Then, based on this key information, it searches and matches in the indicator database, selecting multiple indicators highly relevant to the text content as target indicators. Specifically, natural language processing technology is used to perform semantic analysis on the evaluation instruction text, transforming the natural language descriptions in the text into keywords or semantic vectors that the system can understand. Simultaneously, each indicator in the indicator database has a corresponding keyword tag or semantic feature vector. By calculating the similarity between the text semantics and the indicator semantic features, indicators with similarity higher than a preset threshold are selected as target indicators.

[0036] Optionally, in addition to semantic matching based on natural language processing, template matching can also be used. The system predefines templates for various common assessment needs, and each template corresponds to a fixed set of target indicators. When the assessment instruction text is received, the system matches the text with these templates. If the text matches the characteristics of a certain template (such as containing a specific combination of keywords), the target indicator corresponding to that template is directly selected.

[0037] In this way, based on the text input by the user, the system automatically selects multiple target indicators from the indicator database, eliminating the need for manual searching and filtering of indicators one by one. This reduces the cost of manual operation and the probability of errors, automates and automates indicator selection, improves the efficiency and accuracy of indicator matching, and can quickly respond to different evaluation needs.

[0038] Furthermore, after selecting multiple target indicators from the indicator database based on the text input, the process includes:

[0039] (1) Display the selected target indicators and default attribute parameters in the interactive interface.

[0040] It's important to note that after selecting a target indicator from the indicator database, the interface will display information in a structured format (such as a list or card), including the name and default attribute parameters of each target indicator. These default attribute parameters are pre-defined general configurations from the indicator database, defining the calculation method of the target indicator and the rules for determining whether the target indicator meets the criteria, covering calculation rules, weight coefficients, and threshold ranges. In practice, front-end rendering technology is used to transform the indicator data transmitted from the back-end into visual interface elements, ensuring that users can intuitively understand the basic configuration of the current indicator.

[0041] (2) Receive adjustment parameters input by the user through the interactive interface.

[0042] It should be noted that for each indicator attribute displayed on the interface (such as calculation rules, weights, and thresholds), the system provides interactive input components (such as text boxes, sliders, and drop-down menus). Users can input adjustment values ​​according to actual evaluation needs (such as industry standards and business priorities). For example, in a power inspection scenario, if a user believes that the default acceptable threshold (300ms) for single-sample inference time is too wide, they can adjust it to 200ms using the slider. Furthermore, the system will perform validity checks on the input parameters (such as format and value range) to ensure that the adjustment is effective before temporarily saving the data.

[0043] (3) Update the adjusted parameters to the new configuration of the corresponding target indicators and store them.

[0044] In this step, after selecting the target indicator, the default attribute parameters are first displayed, and then the user's adjusted parameters are received and the storage is updated. This realizes flexible pre-configuration of indicator attribute parameters, meets the personalized needs of users in different evaluation scenarios, and makes the indicator values ​​more in line with the actual evaluation goals, laying the foundation for subsequent generation of evaluation tasks and accurate calculation.

[0045] S103. Determine the evaluation automation mode corresponding to the target indicator, and generate evaluation tasks according to the evaluation automation mode, wherein multiple evaluation tasks are performed in parallel.

[0046] It should be noted that the automated assessment mode refers to the type of automated process for completing the assessment of target indicators. Each mode corresponds to a different human-machine collaboration method, adapting to indicator assessment scenarios with varying complexity and human intervention requirements. An assessment task is an independent execution unit generated according to its matched automated mode for a single target indicator, containing complete process logic such as data collection, calculation, and result output. Multiple assessment tasks can run simultaneously.

[0047] Specifically, the system matches the most suitable type from multiple patterns based on the attributes of each target indicator (such as computational logic complexity and whether manual review is required). Pattern matching is based on a preset rule engine. The system has a built-in pattern matching rule library and automatically completes pattern allocation by comparing indicator attributes with the rule library.

[0048] Specifically, multiple assessment tasks need to be run, and their execution order can be scheduled. This means that some assessment tasks run simultaneously, while others run sequentially. Preferably, the method further includes: determining the correlation between the various assessment tasks; dividing the assessment task groups based on the correlation, with each group containing multiple assessment tasks. Each group runs independently and simultaneously, and the multiple assessment tasks within a group run sequentially; for each assessment task group, determining the target indicators, determining the dependent variables of the target indicators based on their calculation methods, and generating the time sequence of each assessment task within the assessment task group based on the dependent variables. Dependent variables refer to variables that need to be known in the calculation method of the target indicators but are calculated by other assessment tasks. When dividing the groups, all dependent variables for each target indicator are determined based on their calculation methods, and assessment tasks with the same dependent variables or whose dependent variables are related are grouped together. After dividing the group, within a group, a dependency sequence chain of dependent variables is established based on the calculation method of each target indicator. Each chain is merged to obtain a dependency graph of all variables within the group. Each variable in the dependency graph corresponds to an assessment task, and an assessment task can correspond to multiple variables. The timing sequence of the corresponding assessment task is generated based on the timing sequence of the variables in the dependency graph.

[0049] In this way, on the one hand, it ensures that data-dependent indicators are calculated in the correct order, guaranteeing the accuracy of the evaluation results; on the other hand, it executes task groups without dependencies in parallel, maximizing the use of system resources, shortening the overall evaluation time, and solving the problems that full parallelism may lead to calculation errors and full serialism may lead to inefficiency.

[0050] It should be noted that the automated assessment modes provided in this application include a fully automated mode, a human-assisted mode, and a human-led mode. The fully automated mode requires no manual input, automatically collects data from the assessment data source using a preset algorithm, completes indicator calculations according to built-in calculation rules, and directly outputs the final indicator value. The human-assisted mode automatically collects data from the assessment data source and completes preliminary calculations, pushes the preliminary calculation results to the interactive interface, and generates the final indicator value after receiving a manual review and confirmation instruction. The human-led mode receives manually input calculation parameters through the interactive interface, automatically collects matching data from the assessment data source based on the calculation parameters, completes the calculations, and directly outputs the final indicator value.

[0051] Specifically, the fully automated mode requires no manual operation and relies on pre-set system logic to complete the entire process from data collection to result output. It is suitable for indicator evaluations with clear calculation rules, standardized data sources, and no need for manual judgment. Specifically, the system automatically retrieves the raw data required for the indicator (e.g., processing time data for each sample is required for the single-sample inference time indicator) from a specified evaluation data source (e.g., a power equipment inspection sample library) using pre-set algorithms (e.g., API calls, batch reading). Then, based on the calculation rules built into the indicator database (e.g., single-sample inference time = total processing time / total number of samples), the system automatically performs calculations on the collected data without requiring any manual parameter input. Finally, after the calculation is completed, the system directly outputs the final indicator value (e.g., 280ms / sample), without manual review or confirmation; the result is directly used for the evaluation.

[0052] The manual-assisted mode involves the system completing data collection and preliminary calculations, but the final result requires manual review and confirmation. This mode is suitable for indicator evaluations where the calculation process can be automated, but the accuracy of the results needs manual verification (such as scenarios involving sample label accuracy and subjective judgment). Specifically, consistent with the fully automated mode, the system automatically collects data from the evaluation data source and completes preliminary calculations according to built-in calculation rules to obtain preliminary results. The system then pushes the preliminary calculation results and associated supporting data (such as screenshots of incorrectly identified samples and label information) to the interactive interface, prompting the user to review them. The user views the preliminary results and supporting data through the interactive interface to confirm the reasonableness of the results (e.g., determining whether incorrectly identified samples are due to labeling errors rather than model problems). If everything is correct, the user clicks "Approval," and the system generates the final indicator value. If problems exist (such as label errors), corrections can be made, the calculation can be retried, and the result can be confirmed again.

[0053] The human-driven mode centers on manually input calculation parameters. The system completes data collection and calculation based on these parameters and supports real-time manual parameter adjustment. It is suitable for indicator assessments with highly personalized needs and flexible parameter configuration requirements (e.g., different users may have different calculation standards for the same indicator). Specifically, users input the indicator calculation parameters through an interactive interface. Parameter types include weight coefficients, threshold ranges, and sample selection conditions. The system then receives the parameters, automatically collects matching data from the assessment data source, completes the indicator calculation according to the configured rules, and directly outputs the final result. If the user is dissatisfied with the result (e.g., believing the weight configuration is unreasonable), they can modify the calculation parameters in real-time through the interactive interface. The system immediately re-collects data, recalculates, and updates the result without terminating the entire assessment process.

[0054] Based on the preceding descriptions, for example, single-sample inference time calculation logic is simple and requires no manual intervention, matching the fully automated mode; equipment defect identification accuracy requires manual confirmation of sample labels, matching the manual assistance mode; and custom risk scoring requires user input of weight parameters, matching the manual-led mode.

[0055] This step matches a dedicated automation mode for each indicator, generates independent tasks, and executes them in parallel. The parallel execution of multiple tasks reduces the overall evaluation time, which is especially suitable for complex scenarios with multiple indicators. By matching modes according to indicator characteristics, the efficiency of automation and the need for manual intervention are balanced, and the independent operation of each task increases the flexibility of the evaluation.

[0056] S104. Monitor multiple assessment tasks in real time, respond to any target assessment task's target indicator information modification instruction, adjust the target indicator corresponding to the target assessment task, calculate the indicator value corresponding to the newly added target indicator based on the assessment results that received the information modification instruction, and modify and update the assessment automation mode based on the changed target indicator. The automation mode corresponding to each target indicator is different.

[0057] It should be noted that the target indicator information modification command refers to the adjustment command initiated by the user through the interactive interface during the evaluation task, targeting a specific evaluation task. This mainly includes two categories: adding a new target indicator and modifying an existing target indicator. The evaluation results preceding the receiving modification command refer to the evaluation data already generated for that target evaluation task before the user initiated the modification command, including basic sample data, intermediate results (such as the identification results of some samples), and output indicator values.

[0058] Specifically, the large-scale model evaluation system monitors the normal operation of each evaluation task by capturing its runtime logs in real time (such as data acquisition progress, computation node status, and result output). Simultaneously, it listens for modification commands for target metrics from the interactive interface. This can be achieved using a dual mechanism of timed polling and event listening. Timed polling of task runtime data ensures real-time monitoring of task progress; while listening for command-triggered events from the interactive interface (such as button clicks and form submissions) immediately triggers subsequent adjustment processes upon receiving a modification command, avoiding delays. The monitored content includes task runtime status, resource usage, whether any exceptions have been triggered, and whether any user-initiated modification commands have been received.

[0059] Once a modification instruction for any target assessment task is detected, the instruction is first parsed to determine the modification type and extract key information. For example, if the instruction is to add a new target indicator, the indicator name and indicator attributes can be extracted; if the instruction is to modify the attributes of an existing indicator, the indicator ID to be modified and the content to be modified can be extracted. Through instruction parsing, the objects and content to be adjusted in the future are determined.

[0060] Furthermore, different adjustment logics are executed for the two scenarios of adding new indicators and modifying attributes. In the case of adding a new target indicator, the system calls the evaluation results that received the modification instruction first, parses the calculation logic of the new indicator, and determines whether the required data is complete. If the data is complete, historical data is directly called, and the initial indicator value is calculated according to the calculation rules for the new indicator. If the data is incomplete, missing data is collected based on existing intermediate results, and then the initial indicator value is calculated. Then, based on the attributes of the new indicator (such as computational complexity and whether manual intervention is required), a specific mode is matched from fully automated mode, manually assisted mode, and manually-led mode. Finally, the new indicator, its initial indicator value, and the matched automated mode are added to the indicator set of the current target evaluation task and executed in parallel with the original indicators without affecting the operation of other tasks.

[0061] In the case of attribute modification, for the modified attribute (which includes at least the calculation rule, weight coefficient, and threshold range; here, the attribute refers to the modification range of the indicator attribute), historical evaluation results (such as original sample data and intermediate calculated values) are called to recalculate. If the calculation rule or weight coefficient is modified, the indicator value can be recalculated according to the new rule based on historical sample data; if the threshold range is modified, there is no need to recalculate the indicator value. The automated verification process is directly triggered to compare the original indicator value with the new threshold and generate the adjusted pass / fail judgment result.

[0062] In specific implementation, further explanation is provided for the two scenarios mentioned above. The step of calculating the indicator value corresponding to the newly added target indicator based on the evaluation results prior to receiving the information modification instruction, and modifying and updating the automated evaluation mode based on the changed target indicator, includes:

[0063] (1) Receive input information for newly added target indicators, the input information including indicator name and indicator attributes.

[0064] It's important to note that the indicator name uniquely identifies a new indicator, while the indicator attributes determine the core configuration for the indicator's calculation logic and mode adaptation. These attributes include calculation rules, data dimensions, and the need for manual intervention (here, indicator attributes refer to the indicator attributes themselves). Specifically, the system receives user input for new indicator information through an interactive interface. This interface provides a structured input form (with dropdown menus for selecting data dimensions and text boxes for entering calculation rules), and the system performs validity checks on the input information (e.g., avoiding duplicate indicator names and ensuring the calculation rules are formatted correctly). This ensures that the information can be directly used in subsequent steps.

[0065] (2) Analyze the calculation logic of the newly added target indicators and determine the required basic parameters and intermediate results.

[0066] It should be noted that the system performs semantic parsing on the calculation rules of newly added indicators, extracting the "basic parameters" required for calculation (such as defect sample coordinate data and pass / fail deviation thresholds needed to calculate positioning accuracy) and intermediate results (such as the total number of defect samples calculated in previous assessments and the sample ID association table). Based on a pre-defined indicator calculation logic knowledge base, the system can transform the calculation rules described in natural language (such as the number of qualified samples / total number of samples) into parameter-operation logic mapping relationships that the system can recognize (such as parameter A = number of qualified samples, parameter B = total number of samples, operation logic = A / B), thus clarifying the data requirement list. Through this process, the calculation dependencies of new indicators can be broken down, determining whether historical data can be reused and avoiding the waste of resources from repeated data collection.

[0067] (3) When the required basic parameters or intermediate results are already included in the evaluation results before receiving the new input, the data is directly called to generate the initial indicator value of the new target indicator.

[0068] Comparing the data requirement list determined in step (2) with the historical evaluation result database, if the basic parameters and intermediate results already exist in the historical data, these data can be directly called, and the initial indicator values ​​can be automatically calculated according to the calculation rules of the newly added indicators. Here, the system can establish an indicator-data association index, and the historical evaluation results can be stored according to indicator ID and data type. When comparing, the required data can be quickly located through the index, avoiding a full scan of the database. In the calculation process, the preset indicator calculation engine is called to perform the calculation according to the parsed calculation logic.

[0069] (4) When the required basic parameters or intermediate calculation results are not fully included, the missing parameters are supplemented based on the existing intermediate results, and the initial index value of the new target index is generated; according to the index attribute, the target evaluation automation mode corresponding to the new target index is matched from the evaluation automation mode, and the new target index and the initial index value of the new target index are replaced into the current target index set.

[0070] Specifically, this process can be executed in two phases. The first phase involves supplementing the calculation of missing parameters. If historical data only contains some basic parameters / intermediate results (such as missing sample shooting angle data), the system supplements the missing data from the evaluation data source based on existing intermediate results (such as sample IDs) (e.g., capturing shooting angles associated with sample IDs), and then generates initial indicator values ​​according to the calculation rules. The second phase involves matching the mode and adding it to the task. Based on the attributes of the new indicator, the system matches the target mode from three types of automated modes, and finally adds it to the current target indicator set, sharing computing power and executing in parallel with the original indicators. During data supplementation, an incremental collection strategy can be used, collecting only the missing data instead of re-collecting the entire dataset.

[0071] In specific implementation, for attribute modification scenarios, after receiving the attribute modification instruction, the following is included:

[0072] (1) When the calculation rules or weight coefficients are modified, the calculation logic corresponding to the target indicator is called, the indicator value of the target indicator is recalculated based on the existing evaluation data and intermediate results, and the updated indicator value is used to replace the target indicator in the current target indicator set.

[0073] Specifically, upon receiving an instruction to modify the calculation rules or weighting coefficients, the system executes three steps: invoking the new calculation logic for the indicator, retrieving existing evaluation data (historical evaluation data) and intermediate results, recalculating the indicator value according to the new calculation logic, and replacing the old values ​​in the current target indicator set with the updated values. The system retains historical attribute versions of the indicator, automatically triggering a recalculation of the indicator value after modification, prioritizing the reuse of historical data during recalculation (to avoid duplicate data collection), and recording a comparison of the indicator values ​​before and after the recalculation.

[0074] (2) When the threshold range is modified, the automated verification process of the target indicator is retried, and the adjusted evaluation result is generated based on the current evaluation data.

[0075] It should be noted that, unlike after the calculation rules or weighting coefficients are modified, after the threshold is modified, a verification is performed instead of a recalculation. In this case, the indicator value is not recalculated; instead, the system re-triggers the automated verification process for that indicator, compares the current indicator value with the new threshold, generates an adjusted evaluation result (e.g., the original unqualified judgment is updated to qualified), and synchronizes it to the interactive interface and the result database.

[0076] (3) After the attribute modification is completed, the corresponding automated assessment mode is re-matched according to the modified indicator attribute.

[0077] It should be noted that modifications to indicator attributes may alter the need for manual intervention, requiring simultaneous updates to the automation mode to prevent anomalies in the evaluation process.

[0078] In addition, it should be noted that besides adding new target metrics and modifying the attributes of existing target metrics, the target metric information modification command also includes the operation of deleting target metrics. During the execution of an assessment task, users can initiate a metric deletion command for a specific target assessment task through the interactive interface. In the current set of target metrics displayed on the interactive interface, the user selects the metric to be deleted, clicks the delete button, and confirms. The system receives the deletion command and simultaneously extracts the metric ID to be deleted and the identifier of the target assessment task to which it belongs. At the same time, the system will perform a legality check on the deletion command. For example, it checks whether the metric to be deleted belongs to the currently running target assessment task and whether there are any dependencies (if the metric is the basis for the calculation of other metrics, it will prompt "Dependency exists, cannot be deleted") to avoid abnormal assessment processes caused by accidental deletion. After the verification is successful, the subsequent deletion process is triggered.

[0079] Furthermore, based on the metric ID to be deleted in the deletion command, the system removes the metric from the target metric set and terminates the corresponding evaluation process. Specifically, in the metric set of the current target evaluation task, all configuration information of the metric (including metric name, attributes, matching automation mode, and generated metric values) is deleted. Simultaneously, the metric display list on the interactive interface is updated to no longer show the deleted metric. At the same time, the evaluation task thread corresponding to the metric is stopped, releasing the computing power, memory, and other system resources it occupies, ensuring that resources can be allocated to other running metric tasks. It should be noted that if there are incomplete evaluation operations for the metric (such as some sample data being collected or preliminary calculation results awaiting review), the system will first terminate these incomplete operations to avoid generating invalid data. Simultaneously, the deletion time, operator, and historical evaluation results of the metric are recorded before the process is completely terminated.

[0080] S105. Continue the evaluation of the target evaluation task using the updated automation mode and adjusted target indicators.

[0081] It should be noted that the specific evaluation process after the update is described above and will not be repeated here. In this step, the evaluation of the target evaluation task continues in the updated automation mode and with the adjusted target indicators without interrupting the overall task. Unmodified indicators can continue to run in the original mode, avoiding the entire process from being halted due to partial adjustments and significantly improving evaluation efficiency. Secondly, this step also ensures the consistency of evaluation results. Historical incomplete samples are reprocessed with the new configuration, and new samples are directly adapted to the new rules, ensuring that all samples follow the same evaluation standard and avoiding the incomparability of results between old and new samples due to configuration differences. In addition, it can maximize resource reuse, without the need to recollect historical data or rebuild the task environment. Only incremental adjustments are made to the changed parts, significantly reducing system computing power and time costs.

[0082] It should also be noted that after continuing the evaluation of the target assessment task with the updated automation mode and adjusted target indicators, the process includes: real-time collection of evaluation process data and final result data for each target indicator; generating visualization charts according to the type and time dimension of the target indicators, marking the time nodes of indicator adjustments and corresponding mode changes in the charts; and displaying the visualization charts on the interactive interface.

[0083] Specifically, the visualization process requires acquiring two types of data for each target indicator: first, assessment process data, covering intermediate results of indicator calculations and records of automation mode switching; and second, final result data, including the final values ​​calculated for each indicator according to the current configuration and the pass / fail judgment results. During data acquisition, information is categorized and stored according to ID, timestamp, and data type. Next, visualization charts are generated based on indicator type and time dimension. Indicators that need to reflect changes in value over time are categorized as numerical indicators, while indicators that need to reflect differences between different categories of data are categorized as categorical indicators. Then, appropriate chart types are generated accordingly. For numerical indicators, a line chart is generated with assessment time as the horizontal axis and indicator value as the vertical axis, clearly showing the real-time trend of indicator changes as the assessment progresses. For categorical indicators, a bar chart is generated with indicator category as the horizontal axis and corresponding indicator value as the vertical axis, intuitively presenting the distribution of indicators under each category.

[0084] It should also be noted that, after continuing the evaluation of the target assessment task with the updated automation mode and adjusted target metrics, the following is also included:

[0085] (1) Based on the real-time evaluation results of each target indicator, automatically identify erroneous samples whose deviation from the preset standard exceeds the threshold.

[0086] It should be noted that, based on the real-time evaluation results of each target indicator, erroneous samples that exceed the preset standard deviation threshold are automatically identified, including:

[0087] (i) Call the preset standard database corresponding to the target indicator to obtain the historical deviation threshold and sample label standard of the target indicator in the same evaluation scenario.

[0088] It should be noted that the system first calls the preset standard database corresponding to the current target indicator. This database stores historical data of each indicator in similar evaluation scenarios, as well as sample label standards. The obtained historical deviation threshold is used as a quantitative standard to judge whether the sample is biased. The sample label standards are used to compare the accuracy of the sample prediction results in the future to ensure that the identification basis complies with the general specifications of the industry or scenario.

[0089] (ii) Compare the predicted results of each evaluation sample with the actual labels and calculate the deviation.

[0090] For each evaluation sample, the system compares the prediction results output by the large model with the actual labels of the sample dimension by dimension, and calculates the deviation between the two using a preset algorithm. For example, for classification indicators (such as defect type identification), the deviation is calculated using category matching degree (100% deviation if there is no match), while for numerical indicators (such as positioning accuracy), the deviation is calculated using absolute error / relative error (e.g., if the positioning deviation is 8 pixels and the preset allowable deviation is 5 pixels, then the deviation is 60%).

[0091] (iii) When the deviation exceeds the historical deviation threshold of the target indicator, it is marked as a suspected erroneous sample.

[0092] The calculated sample deviation is compared with the historical deviation threshold obtained in step (i). If the deviation exceeds the threshold (e.g., a sample with a deviation of 8% exceeds the "±5%" threshold), the sample is marked as a suspected erroneous sample. This step, through quantitative threshold screening, can initially exclude normal samples with deviations within a reasonable range, focus on abnormal samples that require further verification, and reduce the workload of subsequent verification.

[0093] (iiii) The suspected erroneous samples are verified a second time by searching the sample feature library.

[0094] To avoid misjudgments caused by non-model factors such as incorrect labeling or abnormal sample data, the system searches the sample feature library (which stores typical features of various types of samples) to perform a secondary verification on suspected misjudgments. If it is confirmed that the misjudgment is due to the model rather than a labeling error, the sample passes the secondary verification; if the search reveals that the labeling is incorrect, the sample is excluded and not included in the range of misjudgments.

[0095] Furthermore, recording the identifier, target index, deviation degree, and characteristic parameters of suspected erroneous samples that pass the secondary verification can provide a basis for subsequent analysis of the causes of deviations and positioning parameter issues.

[0096] (2) For the identified erroneous samples, analyze the calculation logic and parameter configuration of the target indicators associated with the erroneous samples, and locate the fine-tuning parameter items that cause the deviation.

[0097] It should be noted that the system first obtains the complete configuration of the target indicator to which the erroneous samples belong, covering calculation rules, parameter configurations (weight coefficients, threshold ranges, sample selection conditions, etc.), and parameters associated with the automated mode, ensuring coverage of key nodes in the entire evaluation process. For example, the configuration of the equipment defect identification accuracy indicator includes calculation rules (number of correctly identified samples / total number of valid samples), sample selection conditions (pixel percentage of defective area ≥ 5%), and a sample review ratio ≥ 30% under manual assistance mode. For the scenario-specific deviations of erroneous samples, the system compares the indicator configurations one by one to find directly related parameters. If the erroneous samples are concentrated in small-pixel defect samples (e.g., pixel percentage of defective area 3%-4%), comparing the sample selection conditions reveals that the threshold of ≥ 5% pixel percentage of defective area in the indicator configuration causes these samples to be excluded from effective calculations. The model does not include their identification results in the accuracy statistics, thus generating deviation. In this case, the pixel percentage of defective area selection threshold is the fine-tuning parameter that causes the deviation. Preferably, as an optional embodiment, for the identified erroneous samples, the model parameter combination for erroneous sample identification is obtained, the calculation logic and parameter configuration of the target index associated with the erroneous samples are determined, the parameter combination with the highest influence in the model parameter combination is determined according to the calculation logic, and the parameter combination is used as the fine-tuning parameter item.

[0098] (3) Generate parameter correction suggestions, recalculate the target index value corresponding to the erroneous sample based on the corrected parameters, and update the evaluation results.

[0099] Specifically, the recommendations for parameter correction need to be determined based on the actual situation, and can be found in the descriptions of relevant technologies, which will not be elaborated here.

[0100] The method provided in this embodiment achieves flexible adjustment of indicators and automation modes during the evaluation process by monitoring the evaluation task and responding to indicator modification instructions in real time. Users can add target indicators, modify existing indicator attributes, or delete unnecessary indicators during task execution. The system can reuse historical evaluation results to calculate new indicator values ​​and re-match automation modes based on modified attributes, without terminating and restarting the entire process. This allows the evaluation to dynamically adapt to changing needs, significantly improving its flexibility and scenario adaptability. Through the design of automatic indicator selection via text input and parallel execution of multiple tasks, users only need to input natural language evaluation requirements through the interactive interface. The system can automatically match target indicators from the indicator database based on the text, eliminating the need for manual screening and configuration. Simultaneously, evaluation tasks corresponding to multiple target indicators are executed in parallel according to their respective automation modes, avoiding the time-consuming problems of traditional serial evaluation. Especially in complex scenarios with multiple indicators, this significantly shortens the overall evaluation cycle and improves resource utilization. Furthermore, by automating processes to replace manual operations and reusing historical data, the core processes such as indicator selection, task generation, and pattern matching are all completed automatically by the system, reducing the operational costs and error probability of manual configuration. When adjusting indicators, the evaluation results before receiving modification instructions can be reused, avoiding repeated data collection and rebuilding of the task environment, which greatly reduces the system's computing power and time consumption, allowing the evaluation to proceed efficiently while achieving optimal resource utilization.

[0101] Example 2

[0102] Corresponding to the aforementioned embodiment of a large model evaluation method, this application also provides an embodiment of a large model evaluation device.

[0103] Figure 2 This is a schematic diagram of the structure of Embodiment 2 of the large model evaluation device provided in this application. Please refer to... Figure 2 The device provided in this embodiment includes a receiving module 210, a selection module 220, a determination module 230, a monitoring module 240, and a processing module 250.

[0104] The receiving module 210 is used to select the model to be evaluated based on the interactive interface and receive text input of the evaluation instruction content;

[0105] The selection module 220 is used to select multiple target indicators from the indicator database based on the text input;

[0106] The determining module 230 is used to determine the evaluation automation mode corresponding to the target indicator, and generate evaluation tasks according to the evaluation automation mode, wherein multiple evaluation tasks are performed in parallel.

[0107] The monitoring module 240 is used to monitor multiple assessment tasks in real time, respond to any target assessment task target indicator information modification instruction, adjust the target indicator corresponding to the target assessment task, calculate the indicator value corresponding to the newly added target indicator based on the assessment results that received the information modification instruction in advance, and modify and update the assessment automation mode based on the changed target indicator. The automation mode corresponding to each target indicator is different.

[0108] The processing module 250 is used to continue the evaluation of the target evaluation task in an updated automation mode and with adjusted target indicators.

[0109] The apparatus of this embodiment can be used to perform... Figure 1 The steps of the method embodiment shown are similar in principle and process, and will not be repeated here.

[0110] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0111] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0112] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A large model evaluation method, characterized in that, The method comprises: selecting an evaluation model to be evaluated based on an interactive interface, receiving text input of evaluation instruction content; selecting a plurality of target indicators from an indicator database according to the text input; determining an evaluation automation mode corresponding to the target indicators, and generating an evaluation task according to the evaluation automation mode, wherein a plurality of evaluation tasks are performed in parallel; real-time monitoring of a plurality of evaluation tasks, in response to any target evaluation task target indicator information modification instruction, adjusting the target indicator corresponding to the target evaluation task, calculating the indicator value corresponding to the newly added target indicator based on the evaluation result received before the information modification instruction, modifying and updating the evaluation automation mode based on the changed target indicator, and the automation mode corresponding to each target indicator is different; continuing the evaluation of the target evaluation task in the updated automation mode and the adjusted target indicator.

2. The large model evaluation method of claim 1, wherein, The evaluation automation mode includes full automation mode, artificial assistance mode and artificial dominant mode; wherein the full automation mode does not require manual input, automatically collects data from the evaluation data source through a preset algorithm, completes indicator calculation according to the built-in calculation rule, and directly outputs the final indicator value; the artificial assistance mode automatically collects data from the evaluation data source and completes preliminary calculation, pushes the preliminary calculation result to the interactive interface, and generates the final indicator value after receiving the artificial review confirmation instruction; the artificial dominant mode receives the calculation parameters input by the human through the interactive interface, automatically collects matching data from the evaluation data source based on the calculation parameters and completes the calculation, and directly outputs the final indicator value.

3. The large model evaluation method of claim 1, wherein, The calculation of the indicator value corresponding to the newly added target indicator based on the evaluation result received before the information modification instruction, and the modification and updating of the evaluation automation mode based on the changed target indicator, comprise: receiving input information of the newly added target indicator, the input information including indicator name and indicator attribute; analyzing the calculation logic of the newly added target indicator to determine the required basic parameters and intermediate results; when the required basic parameters or intermediate results are already included in the evaluation result before receiving the new input, directly call the data to generate the initial indicator value of the newly added target indicator; when the required basic parameters or intermediate results are not completely included, supplement the missing parameters based on the existing intermediate results, and generate the initial indicator value of the newly added target indicator; according to the indicator attribute, match the target evaluation automation mode corresponding to the newly added target indicator from the evaluation automation mode, and replace the newly added target indicator and the initial indicator value of the newly added target indicator into the current target indicator set.

4. The large model evaluation method of claim 1, wherein, The adjustment of the target indicator corresponding to the target evaluation task includes attribute modification, and the attribute at least includes calculation rule, weight coefficient and threshold range; wherein after receiving the attribute modification instruction, when the calculation rule or the weight coefficient is modified, the calculation logic corresponding to the target indicator is called, the indicator value of the target indicator is recalculated based on the existing evaluation data and intermediate results, and the updated indicator value is replaced into the current target indicator set; when the threshold range is modified, the automatic verification process of the target indicator is retriggered, and the adjusted evaluation result is generated based on the current evaluation data; After the attribute modification is completed, the corresponding evaluation automation mode is re-matched according to the modified index attribute.

5. The large model evaluation method of claim 1, wherein, After the plurality of target indexes are selected from the index database according to the text input, the method comprises the following steps: The selected plurality of target indexes and default attribute parameters are displayed on the interactive interface. The adjustment parameters input by the user through the interactive interface are received. The adjusted parameters are updated as new configurations of the corresponding target indexes and stored.

6. The large model evaluation method of claim 1, wherein, After the target evaluation task is continued to be evaluated in the updated automation mode and the adjusted target indexes, the method comprises the following steps: Real-time evaluation process data and final result data of each target index are collected. Visual charts are generated according to the types and time dimensions of the target indexes, and the time nodes of index adjustment and corresponding mode changes are marked in the charts. The visual charts are displayed on the interactive interface.

7. The large model evaluation method of claim 1, wherein, After the target evaluation task is continued to be evaluated in the updated automation mode and the adjusted target indexes, the method further comprises the following steps: Based on the real-time evaluation results of each target index, mis-samples whose deviations from preset standards exceed a threshold value are automatically identified. For the identified mis-samples, the calculation logic and parameter configuration of the target indexes associated with the mis-samples are analyzed to locate the fine-tuning parameter items that cause the deviations. Parameter correction suggestions are generated, and the target index values corresponding to the mis-samples are recalculated based on the corrected parameters, and the evaluation results are updated.

8. The large model evaluation method of claim 7, wherein, The method of automatically identifying mis-samples whose deviations from preset standards exceed a threshold value based on the real-time evaluation results of each target index comprises the following steps: A preset standard database corresponding to the target index is called to obtain historical deviation thresholds and sample label standards of the target index in the same type of evaluation scene. The prediction results of the evaluation samples are compared with the actual labels one by one, and the deviation degrees are calculated. When the deviation degree exceeds the historical deviation threshold of the target index, the mis-sample is marked as a suspected mis-sample. The suspected mis-sample is verified again by searching a sample feature library.

9. The large model evaluation method of claim 1, wherein, The method of generating an evaluation task according to the evaluation automation mode comprises the following steps: The relevance between each evaluation task is determined. Based on the relevance, the evaluation task groups are divided, each group including a plurality of evaluation tasks, each group being independent and running simultaneously, and the plurality of evaluation tasks in one group being run in time sequence. For each evaluation task group, the target indexes of each evaluation task group are determined, the dependent variables of the target indexes are determined according to the calculation methods in the target index attributes, and the time sequence of each evaluation task in the evaluation task group is generated according to the dependent variables. 10.A large model evaluation device, characterized by comprising: The device comprises a receiving module, a selection module, a determination module, a monitoring module, and a processing module. The receiving module is configured to select a model to be evaluated based on an interactive interface and receive a text input of evaluation instruction content. The selection module is configured to select a plurality of target indexes from an index database according to the text input. The determination module is configured to determine an evaluation automation mode corresponding to the target indexes and generate an evaluation task according to the evaluation automation mode, wherein a plurality of evaluation tasks are performed in parallel. The monitoring module is configured to monitor a plurality of evaluation tasks in real time, adjust a target index corresponding to a target evaluation task in response to a target evaluation task target index information modification instruction, calculate an index value corresponding to a newly added target index based on an evaluation result received before the information modification instruction, modify and update an evaluation automation mode based on the changed target index, and make the automation mode corresponding to each target index different; The processing module is configured to continue the evaluation of the target evaluation task in the updated automation mode and the adjusted target index.

Images