Object evaluation method and system, data processing method

Abstract

The embodiments of this specification provide an object evaluation method and system, and a data processing method, which include: obtaining an object evaluation task, and determining the evaluation object and target evaluation data in a target format based on the object evaluation task, wherein the target evaluation data in the target format is used to evaluate the evaluation object, and the target evaluation data includes a parameter value corresponding to a target input field and a reference parameter value corresponding to a reference field; through a parameter mapping script, the parameter value of the target input field in the target evaluation data is determined as the input parameter of the evaluation object, and the input parameter is processed using the evaluation object to obtain a target output parameter; based on the target output parameter, the reference parameter value in the target evaluation data, and the evaluation index, the evaluation result of the evaluation object on the target evaluation data is determined; the addition and integration of the evaluation object can be completed quickly, thereby enhancing the applicability of the method to a diverse user group.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and in particular, to an object evaluation method. One or more embodiments of this specification also relate to an object evaluation system, a computing device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Traditional model evaluation systems target open-source common evaluation sets and language models, providing a set of evaluation rankings to compare the capabilities of various language models. However, when it comes to evaluating user-developed models in professional fields, traditional evaluation systems have significant design limitations, resulting in users facing high technical barriers and operational costs when performing independent evaluation tasks.

[0003] Specifically, traditional evaluation systems often require users to complete complex configuration and development processes when adding new participants. This process places high demands on users' code development capabilities, making it difficult for users without technical backgrounds to independently add participants. This high barrier to entry not only reduces the evaluation system's ease of use but also weakens its applicability to a diverse user base. Summary of the Invention

[0004] In view of this, embodiments of this specification provide an object evaluation method. One or more embodiments of this specification also relate to an object evaluation method, a data processing method, a computing device, a computer-readable storage medium, and a computer program product to address technical deficiencies in the prior art.

[0005] According to a first aspect of an embodiment of this specification, there is provided an object evaluation method, which is applied to an evaluation platform and includes:

[0006] Obtaining an object evaluation task, and determining an evaluation object and target evaluation data in a target format according to the object evaluation task, wherein the target evaluation data in the target format is used to evaluate the evaluation object, and the target evaluation data includes a parameter value corresponding to a target input field and a reference parameter value corresponding to a reference field;

[0007] By means of a parameter mapping script, the parameter value of the target input field in the target evaluation data is determined as the input parameter of the evaluation object, and the input parameter is processed by the evaluation object to obtain the target output parameter;

[0008] An evaluation result of the evaluation object on the target evaluation data is determined based on the target output parameter, the reference parameter value in the target evaluation data, and the evaluation index.

[0009] According to a second aspect of the embodiments of this specification, there is provided an object evaluation system, which is applied to an evaluation platform and includes:

[0010] a task acquisition unit, configured to acquire an object evaluation task and determine, based on the object evaluation task, an evaluation object and target evaluation data in a target format, wherein the target evaluation data in the target format is used to evaluate the evaluation object, and the target evaluation data includes a parameter value corresponding to a target input field and a reference parameter value corresponding to a reference field;

[0011] a parameter mapping unit, configured to determine, through a parameter mapping script, the parameter value of the target input field in the target evaluation data as the input parameter of the evaluation object, and process the input parameter using the evaluation object to obtain a target output parameter;

[0012] The result determination unit is used to determine the evaluation result of the evaluation object on the target evaluation data according to the target output parameter, the reference parameter value in the target evaluation data and the evaluation index.

[0013] According to a third aspect of the embodiments of this specification, a data processing method is provided, which is applied to a client and includes:

[0014] In response to an object evaluation task creation operation triggered by a user, creating an object evaluation task;

[0015] Submit the object evaluation task to the evaluation platform, obtain and display the evaluation results, wherein the evaluation results are obtained by the evaluation platform through the above-mentioned object evaluation method.

[0016] According to a fourth aspect of the embodiments of this specification, a computing device is provided, including:

[0017] memory and processor;

[0018] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the above method are implemented.

[0019] According to a fifth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores a computer program / instruction, and the computer program / instruction implements the steps of the above method when executed by a processor.

[0020] According to a sixth aspect of the embodiments of this specification, a computer program product is provided, comprising a computer program / instruction, which implements the steps of the above method when executed by a processor.

[0021] An embodiment of the present specification provides an object evaluation method applied to an evaluation platform. When obtaining an object evaluation task, the evaluation object and the target evaluation data in a target format used to evaluate the evaluation object can be determined according to the object evaluation task. The target evaluation data in the target format include parameter values ​​corresponding to the target input field and reference parameter values ​​corresponding to the reference field. When the target evaluation data in the target format is used to evaluate the evaluation object, the input specification of the evaluation object is defined using the target input field based on the target evaluation data in a unified target format, thereby reducing the user's dependence on complex development work during docking. Specifically, through the parameter mapping script, the parameter value of the target input field can be determined as the input parameter of the evaluation object, thereby realizing the docking of the unified target input field to the input parameter of the evaluation object. , when users can complete the mapping between the input parameters of the evaluation object and the unified target input field parameter values ​​by writing a lightweight and easy-to-learn parameter mapping script, the configuration complexity can be significantly reduced while maintaining flexibility; the input parameters are processed by the evaluation object to obtain the target output parameters, and the evaluation results of the evaluation object on the target evaluation data are determined according to the target output parameters, the reference parameter values ​​in the target evaluation data and the evaluation indicators. With the help of the unified target input field and mapping the parameter values ​​of the target input field to the input parameters of the evaluation object, users do not need to have a deep code development background to quickly complete the addition and integration of evaluation objects. The whole process is more intuitive and efficient, suitable for both technical and non-technical users, and enhances the applicability of the object evaluation method to diverse user groups. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a schematic diagram of a scenario of an object evaluation method provided by an embodiment of this specification;

[0023] Figure 2 This is a flow chart of an object evaluation method provided by one embodiment of this specification;

[0024] Figure 3a This is a schematic diagram of the processing process of an object evaluation method provided by an embodiment of this specification;

[0025] Figure 3b This is a data diagram of a built-in evaluation set provided by an embodiment of this specification;

[0026] Figure 4 This is a schematic diagram of the structure of an object evaluation system provided by one embodiment of this specification;

[0027] Figure 5 This is a structural block diagram of a computing device provided by one embodiment of this specification. DETAILED DESCRIPTION

[0028] The following description sets forth many specific details to facilitate a thorough understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0029] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0030] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0031] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0032] First, the terms involved in one or more embodiments of this specification are explained.

[0033] Large Language Model (LLM) refers to a natural language processing model based on deep learning technology. It typically has a large number of parameters and powerful language generation and comprehension capabilities. Trained on large amounts of text data, these models can perform a variety of complex language tasks, such as text generation, translation, question answering, and summarization.

[0034] Model evaluation: Model evaluation is the process of comprehensively assessing the performance of a large language model, aiming to measure its performance in specific tasks or scenarios. This evaluation helps developers understand the strengths and weaknesses of the model and provides a basis for model optimization.

[0035] Evaluation task template: An evaluation task template is a standardized input and output format designed for model evaluation tasks, which is used to quickly copy and initiate evaluation tasks.

[0036] Evaluation Dataset: An evaluation dataset is a set of data specifically used to evaluate model performance. It contains input samples and corresponding correct answers or reference answers. The evaluation dataset should be designed to cover a variety of scenarios to fully reflect the model's capabilities.

[0037] Participant: The participant refers to the specific model or system participating in the evaluation. It can be a single language model, an integrated system, or a combination of multiple models.

[0038] Question type: This refers to the specific type or format of the task in the evaluation set, reflecting the type of problem the model needs to solve. Different question types correspond to different task requirements and difficulty.

[0039] Evaluation Metrics: Evaluation metrics are standards or metrics used to quantify model performance. Different tasks may require different evaluation metrics to accurately reflect the performance of the model.

[0040] Conversion script: Code that converts the format of the original evaluation data into the standard format required by the evaluation task template.

[0041] Traditional evaluation systems typically require users to complete complex configuration and development processes when adding new participants. This process places high demands on users' code development capabilities, making it difficult for users without technical backgrounds to independently add participants. This high barrier to entry not only limits the evaluation system's ease of use but also reduces its applicability to diverse user groups.

[0042] In the embodiments of this specification, in order to simplify the process of adding new evaluation objects and lower the usage threshold, a set of concise and clear standardized input and output formats are introduced, and a lightweight parameter mapping script is combined to achieve efficient configuration.

[0043] In this specification, an object evaluation method is provided. This specification also involves an object evaluation system, a data processing method, a computing device, a computer-readable storage medium and a computer program product, which are described in detail one by one in the following embodiments.

[0044] See also Figure 1 , Figure 1A schematic diagram of a scenario of an object evaluation method provided according to an embodiment of this specification is shown.

[0045] The object evaluation method is applied to a data processing system, which includes an end-side device 102 and an evaluation platform 104, wherein the end-side device 102 creates an object evaluation task in response to an object evaluation task creation operation triggered by a user, and submits the object evaluation task to the evaluation platform 104.

[0046] The evaluation platform 104 obtains an object evaluation task, and determines the evaluation object and the target evaluation data in the target format according to the object evaluation task, wherein the target evaluation data in the target format is used to evaluate the evaluation object, and the target evaluation data includes a parameter value corresponding to a target input field and a reference parameter value corresponding to a reference field; through a parameter mapping script, the parameter value of the target input field in the target evaluation data is determined as the input parameter of the evaluation object, and the input parameter is processed using the evaluation object to obtain a target output parameter; based on the target output parameter, the reference parameter value in the target evaluation data and the evaluation index, the evaluation result of the evaluation object on the target evaluation data is determined.

[0047] The end-side device 102 may include a browser, an application (APP), or a web application such as an H5 (Hypertext Markup Language 5) application, a lightweight application (also known as a mini-program, a type of lightweight application), or a cloud application. The end-side device may be developed based on a software development kit (SDK) for the corresponding service provided by the server, such as a real-time communication (RTC) SDK. The end-side device may be deployed in an electronic device and may rely on the device or certain apps in the device to operate. The electronic device may have a display and support information browsing, such as a personal mobile terminal such as a mobile phone, tablet computer, or personal computer. Various other types of applications may also be configured in the electronic device, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0048] The object evaluation method provided in the embodiment of this specification can determine the evaluation object and the target evaluation data in the target format for evaluating the evaluation object according to the object evaluation task when obtaining the object evaluation task. The target evaluation data in the target format includes the parameter value corresponding to the target input field and the reference parameter value corresponding to the reference field. When the target evaluation data in the target format is used to evaluate the evaluation object, the input specification of the evaluation object is defined by the target input field based on the target evaluation data in a unified target format, thereby reducing the dependence on complex development work during the user docking process. Specifically, through the parameter mapping script, the parameter value of the target input field can be determined as the input parameter of the evaluation object, thereby realizing the docking of the unified target input field to the input parameter of the evaluation object. By writing a lightweight and easy-to-learn parameter mapping script, the mapping between the input parameters of the evaluation object and the unified target input field parameter values ​​can be completed, which can significantly reduce the configuration complexity while maintaining flexibility; the input parameters are processed by the evaluation object to obtain the target output parameters, and the evaluation results of the evaluation object on the target evaluation data are determined based on the target output parameters, the reference parameter values ​​in the target evaluation data and the evaluation indicators. With the help of a unified target input field and mapping the parameter values ​​of the target input field to the input parameters of the evaluation object, users do not need to have a deep code development background to quickly complete the addition and integration of evaluation objects. The whole process is more intuitive and efficient, suitable for both technical and non-technical users, and enhances the applicability of this object evaluation method to diverse user groups.

[0049] See also Figure 2 , Figure 2 A flowchart of an object evaluation method provided by an embodiment of this specification is shown, which specifically includes the following steps.

[0050] Step 202: Obtain an object evaluation task, and determine the evaluation object and target evaluation data in a target format based on the object evaluation task, wherein the target evaluation data in the target format is used to evaluate the evaluation object, and the target evaluation data includes a parameter value corresponding to a target input field and a reference parameter value corresponding to a reference field.

[0051] The object evaluation task can be understood as the task of evaluating the participating objects to determine their performance. Specifically, the participating object can be understood as the specific object being evaluated. For example, the participating object can be a single machine learning model, an integrated system, or a combination of multiple models.

[0052] When evaluating the participants, it is necessary to determine the evaluation set of the evaluation participants. Usually, the evaluation set contains a large amount of evaluation data, and the evaluation data consists of input samples and corresponding correct answers or reference answers, where the input sample corresponds to the input field (for example, the input field is the text field), that is, the input sample is the parameter value of the input field, and the correct answer or reference answer corresponds to the reference field (for example, the reference field is the answer field), that is, the correct answer or reference answer is the parameter value of the reference field.

[0053] In different evaluation sets, the formats of the evaluation data in the evaluation sets may be different. The different formats can be understood as containing the same semantics but using different names, or as different field types. For example, the text field of the evaluation data in evaluation set a represents the input field, and the query field of the evaluation data in evaluation set b represents the input field. The field type of the text field of the evaluation data in evaluation set a is String, and the field type of the query field of the evaluation data in evaluation set b is list.

[0054] In an embodiment of the present specification, evaluation data of different formats are formatted uniformly to determine target evaluation data in a target format, wherein the target format can be understood as a predefined standard format. When evaluating an evaluation object, the evaluation object is evaluated based on the target evaluation data in the standard format. The standard format indicates that the target evaluation data uses a unified standard field name to represent parameter values ​​with the same semantics, and the field types of the fields are also consistent. For example, the target evaluation data in the target format contains a unified standard target input field parameter value (indicating content input to the evaluation object) and a reference parameter value of a reference field (indicating a correct answer or a reference answer). See Table 1. Table 1 shows an example of a target format. For example, different target evaluation data all use a text field to represent text content (i.e., input sample) input to the evaluation object, and the field type of the text field is uniformly defined as a String type.

[0055]

[0056] Table 1

[0057] Specifically, when an object evaluation task is obtained, the object to be evaluated and the target evaluation data used to evaluate the object can be determined according to the object evaluation task. When the target evaluation data is in a target format, these target evaluation data include parameter values ​​of target input fields of unified standards and reference parameter values ​​of reference fields.

[0058] In fact, in an evaluation system for evaluating evaluation objects, the following processes are usually included: obtaining an evaluation set and evaluation objects, inputting input samples in the evaluation set into the evaluation objects, obtaining predicted results of the evaluation objects' outputs, and determining the evaluation objects' performance on the evaluation set based on the evaluation indicators, predicted results, and the correct answers or reference answers of the input samples in the evaluation set; in terms of evaluation sets, the evaluation data in different evaluation sets may not have uniform formats, and the input / output fields corresponding to different evaluation objects may also not be uniform. Therefore, when calculating the evaluation scores based on the determined evaluation indicators and their calculation logic, they need to be configured according to the specific output fields of the evaluation objects and the reference fields corresponding to the correct answers or reference answers in the evaluation set before they can be calculated; in the evaluation set, the evaluation data in different evaluation sets may not have uniform formats, and the input / output fields corresponding to different evaluation objects may not be uniform. Therefore, when calculating the evaluation scores based on the determined evaluation indicators and their calculation logic, they need to be configured according to the specific output fields of the evaluation objects and the reference fields corresponding to the correct answers or reference answers in the evaluation set before they can be calculated; The strong coupling between objects and evaluation indicators requires complex configuration upload when users upload customized evaluation sets in order to connect with the input of subsequent evaluation objects and the calculation of evaluation indicators. When adding new evaluation objects, complex configuration and development processes are also required. Moreover, when the evaluation indicators and their calculation logic are deeply bound to the fixed evaluation sets, the indicator fields of the calculation functions corresponding to the evaluation indicators correspond to the fields of the fixed evaluation sets. The evaluation sets uploaded by users themselves cannot flexibly select evaluation indicators. If users want to configure or expand evaluation indicators for customized evaluation sets, they need to pay high technical costs, including redevelopment or adjustment of indicator calculation logic. Therefore, the design method of traditional evaluation systems not only limits flexibility, but also increases the user's usage threshold.

[0059] Therefore, in order to solve the above problems, a standardized design is adopted to obtain target evaluation data in a unified target format, define unified input and output specifications for evaluation objects, and then, through modular design, realize the decoupling of evaluation indicator calculation logic from evaluation data in the evaluation center. That is, on the basis of the unified target format of target evaluation data and the consistent output format of evaluation objects, the evaluation indicator calculation code can directly reference the fields in these standardized formats without the need for additional adaptation or conversion.

[0060] In order to implement the above-mentioned standardized design, it is necessary to first complete the data configuration task on the evaluation platform, and then design and complete the evaluation task template based on the data configuration task. The subsequent standardized evaluation task template can lower the threshold for using the evaluation system, improve flexibility and efficiency, support multi-scenario applications, and be suitable for technical and non-technical users. It also lays the foundation for future expansion and has broad application prospects.

[0061] In one or more embodiments of this specification, a data configuration task is obtained, and through the data configuration task, a target conversion script for format conversion between the initial format and the target format can be determined, and a unified input and output specification for the evaluation object is defined, that is, the input parameters all correspond to the target input fields, the parameter values ​​corresponding to the target output fields are the output parameters of the final unified format, and the target calculation function corresponding to the evaluation indicator includes the target output fields and the reference fields defined in the target format. The specific implementation method is as follows:

[0062] Before obtaining the object evaluation task, the method further includes:

[0063] Obtain a data configuration task, and determine at least one target conversion script, the target input field and target output field corresponding to the evaluation object, and the target calculation function corresponding to the evaluation indicator based on the data configuration task, wherein the at least one target conversion script is constructed based on the target format and at least one initial format, one target conversion script corresponds to one initial format, and the target calculation function includes the target output field and the reference field defined in the target format.

[0064] Among them, the data configuration task can be understood as a standardized design task before executing the object evaluation task, which is used to design a set of evaluation task templates. The evaluation task template includes a conversion script for converting data sets in different initial formats into target data sets in the target format, unified input and output specifications for the participating objects, and modular design of evaluation indicators.

[0065] The conversion script can be understood as the code that converts the format of the original evaluation set (i.e., the initial format) into the standard format (i.e., the target format) required by the evaluation task template.

[0066] Specifically, the data configuration task can be used to standardize the target format of a data set, that is, to determine at least one target conversion script based on the data configuration task. When one target conversion script corresponds to one initial format, data in any initial format can be converted into data in the target format. By determining the target conversion script between data in different initial formats and data in the target format through the data configuration task, the target format of the target data set is defined in a standardized manner.

[0067] When the target input fields and target output fields corresponding to the evaluation objects can be determined through data configuration tasks, unified input and output formats can be defined between different evaluation objects to clarify the structure and format of data interaction. Through standardized design, the dependence of users on complex development work can be reduced during docking. For example, if the target input field is represented by a text field, mapping can be performed between the text field and the input field defined when designing the evaluation object, so that the input fields of different evaluation objects can be adapted through a unified target input field.

[0068] The data configuration task can also be used to define the target calculation function corresponding to the evaluation indicator. When the unified evaluation object output format and evaluation set standard format are defined as above, the structures between the two are clear and consistent. Therefore, the target calculation function can be defined to include the target output field and the reference field defined in the target format. When the target calculation function is used to calculate the evaluation indicators in the future, the evaluation indicator calculation logic can be decoupled from the evaluation set data. In this way, when the user uploads a customized evaluation set or adds a new evaluation object, there is no need to perform additional adaptation or conversion on the evaluation indicator calculation logic.

[0069] The object evaluation method provided in the embodiments of this specification can design an evaluation task template based on the data configuration task. Subsequently, the target conversion script based on the evaluation task template can convert the initial format data set into a standard format to obtain the target data set in the target format. The user can write a lightweight and easy-to-learn parameter mapping script based on the unified input and output specifications of the evaluation objects in the evaluation task template, thereby quickly completing the addition and integration of evaluation objects. The standardized output format and modular design provide good scalability for future new evaluation indicators. By defining the target calculation function, custom evaluation indicators can be easily added according to needs.

[0070] In one or more embodiments of this specification, users can submit and upload a customized initial dataset, or specify the platform dataset of the evaluation platform in the object evaluation task, that is, specify to use the platform dataset built into the evaluation platform to evaluate the participating objects. Of course, it is also possible to use customized initial datasets and platform datasets to evaluate the participating objects. The specific implementation is as follows:

[0071] The step of determining the evaluation object and the target evaluation data in the target format according to the object evaluation task includes:

[0072] In the case that the object evaluation task carries initial construction data in an initial format and / or a platform dataset identifier, the evaluation object is determined according to the object evaluation task, and the target evaluation data in the target format is determined according to the initial construction data and / or the platform dataset identifier.

[0073] Among them, the initial construction data is the evaluation data in the initial format that is custom-constructed by the user. The platform dataset identifier can be understood as the identifier corresponding to the platform dataset built into the evaluation platform. The evaluation platform can automatically call the corresponding platform dataset as the target dataset of the evaluation participant through the platform dataset identifier.

[0074] In actual applications, the initial format can understand the question type format. For example, question types can include single-choice questions, multiple-choice questions, factual questions, security questions, reading comprehension-multiple-choice questions, reading comprehension-question questions, programming questions, instruction following questions, multi-round dialogue questions, etc., and each question type corresponds to a question type format (that is, the initial format). Each question type format is unified. When the target conversion script is determined through the data configuration task to convert the initial format into the target format, each question type corresponds to a unique and standard question type format. Only then can the initial format corresponding to the question type be converted into the target format according to the target conversion script, ensuring that the initial construction data based on the unified initial format can be converted into the target construction data in the target format.

[0075] See Table 2 and Table 3, which respectively show examples of question formats corresponding to multiple-choice questions and factual questions.

[0076]

[0077] Table 2

[0078]

[0079] Table 3

[0080] Specifically, when the user chooses to upload customized initial construction data, the user first needs to select a question format that matches the task type, and then construct an initial data set in the corresponding question format based on the question format, or configure and upload the existing evaluation data according to the question format, so that the object evaluation task carries an initial data set in the initial format.

[0081] Of course, users can also choose to specify the platform dataset built into the evaluation platform. At this time, the object evaluation task will carry the platform dataset identifier corresponding to the platform dataset. For example, the platform dataset identifier is represented by the dataset name of the platform dataset. There is no limitation here. The corresponding platform dataset can be determined by the platform dataset identifier.

[0082] The target evaluation data in the target format actually used to evaluate the evaluation object is determined through the initial construction data and / or platform dataset identifier carried by the object evaluation task.

[0083] The object evaluation method provided in the embodiments of this specification designs a set of clear and unified standard question formats, and multiple question types can cover a variety of common evaluation scenarios. Through standardized format definitions, users can create and upload evaluation data based on a more intuitive and easy-to-understand question format, and based on the unified standard question format and the target format of the target evaluation data, can accurately use the target conversion script to perform format conversion; in the case where the target evaluation data can also be determined by using the evaluation data in the platform data set, the target evaluation data can be determined through multiple data sources, which increases the flexibility of the target evaluation data.

[0084] In one or more embodiments of this specification, the target construction data in the target format is obtained by format conversion of the initial construction data in the initial format; and when the user specifies the platform data set, the target platform data in the platform data set is in the target format, so the target evaluation data in the target format can be determined by the target construction data and / or the target platform data in the target format. The specific implementation is as follows:

[0085] Determining the target evaluation data in the target format according to the initial construction data and / or the platform dataset identifier includes:

[0086] Converting the initial construction data in the initial format to obtain target construction data in the target format, and / or

[0087] Determining a platform dataset according to the platform dataset identifier, wherein the platform dataset is a dataset preset by the evaluation platform, and the platform dataset includes target platform data in the target format, and the target platform data in the target format is obtained by format conversion of initial platform data in an initial format;

[0088] The target construction data and / or the target platform data are determined as target evaluation data in the target format.

[0089] Specifically, converting the format of the initial construction data in the initial format to obtain the target construction data in the target format includes:

[0090] A target conversion script corresponding to the initial format is determined, and the initial construction data in the initial format is converted into the target construction data in the target format according to the target conversion script.

[0091] The target platform data in the target format is also obtained by performing format conversion on the initial platform data in the initial format based on the target conversion script corresponding to the initial format.

[0092] When the user constructs the initial data set according to the question format, the initially constructed data is the data constructed by the user. When the user configures and uploads the evaluation data he has according to the question format, the initially constructed data is the evaluation data in the question format obtained after the configuration.

[0093] The evaluation platform can obtain open source data sets from different data sources, configure and upload these data sets according to the question format, and thus obtain initial platform data in the initial format.

[0094] These initial build data and / or initial platform data in the initial format can be converted into target build data and / or target platform data in the target format using target conversion scripts built into the evaluation platform. After converting the initial platform data into target platform data in the target format, the evaluation platform organizes the target platform data into a pre-configured dataset for users to select when creating object evaluation tasks.

[0095] Since different question types have different formats, the conversion scripts for converting different question types into target formats are different. Only by determining the initial format (i.e., question type format) corresponding to the target conversion script can the data in the initial format be accurately converted into data in the target format.

[0096] The object evaluation method provided in the embodiments of this specification is compatible with user-defined initial construction data in the object evaluation task, and uses the target conversion script to convert the initial construction data into target construction data. It can also use the standardized platform data set preset by the evaluation platform, based on the target construction data in the target format and / or the target platform data in the target format in the platform data set, to ensure the consistency and scalability of the evaluation.

[0097] Step 204: Using a parameter mapping script, the parameter value of the target input field in the target evaluation data is determined as the input parameter of the evaluation object, and the input parameter is processed using the evaluation object to obtain the target output parameter.

[0098] Among them, the parameter mapping script can be understood as a script written in JQ (JSON query language), which can be used to extract or convert the value of a specific field from the target evaluation data (the target data set used for evaluation, usually in JSON format) and map it to the input parameters of the evaluation object.

[0099] In the embodiment of this specification, since the target evaluation data has a unified target format and contains parameter values ​​of the target input fields, the parameter mapping script is used to extract the parameter values ​​of the target input fields in the target evaluation data and map them to the input parameters of the evaluation object.

[0100] The target input field is a field in the target evaluation data that is used to represent the content input to the evaluation object. For example, in the target evaluation data, the target input field is represented by a text field. At this time, the value of the text field (parameter value) will be used as the input parameter of the evaluation object, that is, the value of the text field will be determined as the input data of the evaluation object.

[0101] When the evaluation object makes inferences based on the input parameters, the target output parameters are obtained. The target output parameters will be used as the prediction results and the reference parameter values ​​(correct answers or reference answers) of the reference fields in the target evaluation data for calculations related to the evaluation indicators.

[0102] In fact, when the parameter mapping script is a JQ script, JQ, as a lightweight and easy-to-learn tool, can significantly reduce the configuration complexity while maintaining flexibility.

[0103] In one or more embodiments of this specification, the parameter mapping script includes an input parameter mapping script, which is used to process the mapping relationship of input parameters in the evaluation object. Specifically, based on the determination of the evaluation object, the object input field of the evaluation object (the input parameter name defined when designing the evaluation object) can be determined. The input parameter mapping script can be used to map the value of the target input field to the object input field, thereby determining the parameter value of the target input field as the input parameter of the evaluation object. The specific implementation method is as follows:

[0104] The step of determining the parameter value of the target input field in the target evaluation data as the input parameter of the evaluation object through a parameter mapping script includes:

[0105] Determine the object input field of the evaluation object and the parameter mapping script according to the object evaluation task, wherein the parameter mapping script includes an input parameter mapping script;

[0106] Establishing an input mapping relationship between the target input field and the object input field through the input parameter mapping script;

[0107] According to the input mapping relationship, the parameter value of the target input field in the target evaluation data is determined as the input parameter of the object input field corresponding to the evaluation object, wherein the input parameter is used to call the evaluation object.

[0108] Specifically, the object input fields of different participating objects are inconsistent. For example, the object input field of participating object A is content, and the object input field of participating object B is input_text. In order to enable the target participating data to be applied to different participating objects, that is, to use the target participating data to evaluate different participating objects, when the target participating data contains target input fields in a unified format, the input mapping relationship between the target input field and the object input field can be established through the input parameter mapping script, thereby realizing the dynamic adaptation of the input parameter names (that is, object input fields) of different participating objects using the parameter mapping script.

[0109] Still taking the above example, when the target input field is a text field, the input parameter mapping script corresponding to the evaluation object A includes content:.text, and the input parameter mapping script corresponding to the evaluation object B includes input_text:.text, so that the reference value of the target input field is extracted from the target evaluation data through the input parameter mapping script and filled into the input parameter of the evaluation object, that is, the parameter value of the target input field is determined as the input parameter of the object input field corresponding to the evaluation object, thereby realizing the compatibility evaluation of the same target evaluation data for multiple heterogeneous evaluation objects.

[0110] Semantic unification is achieved through standardization of field names in target evaluation data, and the underlying field association logic is mapped through parameter mapping scripts. Dynamic parameter binding is achieved through unified target input fields in target evaluation data and declarative parameter mapping scripts. By dynamically adapting the object input fields of different evaluation objects, the evaluation system can adapt to the input requirements of different evaluation objects, solving the adaptation problem between heterogeneous data (different field names or structures) and evaluation object inputs, and improving the versatility of the evaluation system.

[0111] In fact, the parameters required when calling the evaluation object usually include "messages": [{"role":"user","content":"text content"}], where the content is the object input field. Only after obtaining the input parameters corresponding to the object input field can the call to the evaluation object be realized. Therefore, the parameter mapping script is used to determine the parameter value of the target input field in the target evaluation data as the input parameter of the evaluation object, and the call to the evaluation object is realized based on the input parameters, and then the evaluation of the evaluation object is performed.

[0112] The object evaluation method provided in the embodiments of this specification dynamically establishes the correspondence between the target input fields and the input parameters of the participating objects through a declarative mapping script, automatically generates compliant input parameters, and thus realizes a flexible and low-coupled evaluation process. The parameter values ​​of the target input fields in the target participating data will be directly passed to the downstream participating objects, thereby realizing the full process automation from data uploading to evaluation execution.

[0113] In one or more embodiments of this specification, the parameter mapping script further includes an output parameter mapping script; when the evaluation object obtains input parameters, the input parameters are inferred and the initial output parameters are obtained. The output parameter conversion script maps the parameter value of the object output field in the initial output parameter to the target output parameter of the target output field. The specific implementation is as follows:

[0114] The step of processing the input parameters using the evaluation object to obtain target output parameters includes:

[0115] Calling the evaluation object according to the input parameters, processing the input parameters using the evaluation object to obtain initial output parameters of the evaluation object, wherein the initial output parameters are obtained by the evaluation object through reasoning on the input parameters;

[0116] The target output parameters are determined according to the output parameter conversion script and the initial output parameters.

[0117] Specifically, determining the target output parameters according to the output parameter conversion script and the initial output parameters includes:

[0118] Determine the parameter value of the object output field corresponding to the evaluation object according to the initial output parameter;

[0119] Establishing an output mapping relationship between the object output field and the target output field through the output parameter mapping script;

[0120] According to the output mapping relationship, the parameter value of the object output field in the initial output parameter is determined as the target output parameter of the target output field.

[0121] The initial output parameters can be understood as the raw output of the evaluation object, whose format may not meet the requirements of the evaluation system. The output parameter conversion script is used to convert the initial output parameters into the standard output format corresponding to the evaluation object, ensuring that the evaluation system can uniformly process the output of different evaluation objects. The target output parameters can be understood as the standardized output after conversion and can be directly used in the calculation of evaluation indicators.

[0122] Specifically, the input parameters are passed to the participating object, the participating object performs reasoning / calculation, and returns the original output (initial output parameter). The initial output parameter contains the parameter value of the object output field in the participating object. For example, the initial output parameter is {"sentiment":"positive","time":0.92}. The sentiment can be determined as the object output field. When the unified standard target output field is represented by content, the output parameter conversion script can be content:.sentiment, thereby obtaining the target output parameter {"content":"positive"}.

[0123] That is, the output formats of different evaluation objects are different (for example, evaluation object A returns label, and evaluation object B returns class), which need to be converted into a unified target output field to facilitate the calculation of subsequent evaluation indicators.

[0124] In fact, through the above-mentioned input parameter conversion script and output parameter conversion script, it is possible to dynamically bind standard input parameters (target input fields) and standard output parameters (target output fields) to the object input fields and object output fields of the evaluation object, thereby achieving adaptive conversion of data formats.

[0125] That is, the input and output formats of different evaluation objects may vary, but the evaluation system needs to have a unified standard. For example, the input requirement for evaluation object A is {"prompt":"user question"} (prompt is the object input field of evaluation object A), and the input requirement for evaluation object B is {"messages":[{"role":"user","content":"user question"}]} (messages.content is the object input field of evaluation object B); while the target input field of the evaluation standard is {"text":"user question"}

[0126] When directly calling different evaluation objects, the unified target input field must be converted to the input format of the different evaluation objects; otherwise, compatibility will not occur. By establishing field associations through mapping scripts, automatic data flow conversion can be achieved. That is, regardless of the data format required by the evaluation object, the value extracted from the text field and filled into the object input field can meet the input requirements of the evaluation object.

[0127] Through the above-mentioned input parameter conversion script and output parameter conversion script, the evaluation system provided in the embodiment of this specification only needs to process fixed formats (such as text input / content output), and does not need to care about the internal parameter names of different participating objects. When the participating objects are upgraded or replaced, only the parameter mapping script needs to be adjusted, and the evaluation logic does not need to be modified; through standardized design, it supports the access of diverse participating objects while ensuring the compatibility and scalability of the evaluation system. The lightweight parameter mapping script and automated processing mechanism significantly shorten the configuration time and improve the overall work efficiency.

[0128] Step 206: Determine the evaluation result of the evaluation object on the target evaluation data according to the target output parameter, the reference parameter value in the target evaluation data, and the evaluation index.

[0129] Specifically, evaluation metrics are quantitative standards for measuring the performance of the evaluation object. Different evaluation metrics can be specified for target evaluation data of different types or fields. Of course, multiple evaluation metrics can also be applied to the target evaluation data in a dataset to determine the performance of the evaluation object under different measurement criteria on the target evaluation data. For example, evaluation metrics include but are not limited to ACC, rouge1, rouge2, rougeL, etc., which are not limited here.

[0130] The target output parameter is the predicted result of the participant on the target evaluation data, and the reference parameter value of the reference field in the target evaluation data is the correct answer or reference answer (i.e., the true value) in the target data set. The evaluation result of the participant on the target data set can be determined through the evaluation indicators, target output parameters, and reference parameter values ​​of the reference field.

[0131] In one or more embodiments of this specification, an evaluation result can be generated by dynamically selecting an evaluation indicator and calculating an evaluation score based on the target output parameter (prediction result) and the reference parameter value (true value) of the reference field. Specific implementations are as follows:

[0132] Determining the evaluation result of the evaluation object on the target data set according to the target output parameter, the reference parameter value of the reference field in the target evaluation data, and the evaluation index includes:

[0133] Determine the evaluation indicator according to the object evaluation task, and calculate the target output parameter and the reference parameter value of the reference field according to the target calculation function corresponding to the evaluation indicator to obtain an evaluation score, wherein the evaluation score is used to quantitatively evaluate the performance of the participating object;

[0134] The evaluation object, the target data set, the evaluation indicators and the evaluation scores are structurally integrated to obtain the evaluation results of the evaluation object on the target evaluation data.

[0135] In fact, different evaluation indicators can be determined according to different evaluation task types. In the embodiment of this specification, when the user uploads the evaluation set based on the question type format, different question types can correspond to different evaluation indicators. For example, for reading comprehension-question-answer questions, the optional evaluation indicators include but are not limited to summary questions ACC, reading comprehension question-answer questions ACC, rouge1, rouge2, rougeL, F1, etc.

[0136] The target calculation function is a mathematical formula or algorithm for implementing the evaluation indicator. By calling the target calculation function corresponding to the evaluation indicator and calculating the target output parameters and the reference parameter values ​​of the reference field, the evaluation score of the evaluation object based on the measurement standard of the evaluation indicator and the target evaluation data can be obtained.

[0137] After obtaining the evaluation scores, since multiple evaluation indicators can be specified, multiple evaluation scores can be determined. By structurally organizing and integrating the evaluation objects, the target data set, the evaluation indicators and the evaluation scores, a structured evaluation result including the evaluation objects, the target data set, the evaluation indicators and the evaluation scores can be obtained.

[0138] In fact, ability dimension attribute information can be added to different target evaluation data, so as to facilitate the subsequent statistics of the evaluation scores of the evaluation objects based on different ability dimensions, and to sort different evaluation objects based on the same ability dimension.

[0139] For example, the evaluation result is {“evaluation object”: “sv-tp6emn78-5-1”, “ability dimension”: “knowledge”, “dataset”: “AGIEval”, “question type”: “single choice question”, “evaluation indicator”: “ACC”, “evaluation score”: 0.46444444444444444, “range”: “[0,1]”, “type”: “discrete”}.

[0140] In one or more embodiments of this specification, the target output parameter is the parameter value of the target output field corresponding to the evaluation object, and the target calculation function corresponding to the evaluation indicator includes the target output field and the reference field; by unifying the target output field, reference field, and target calculation function, the decoupling of the evaluation indicator calculation logic and the evaluation set data is achieved. The specific implementation method is as follows:

[0141] The step of calculating the target output parameter and the reference parameter value of the reference field according to the target calculation function corresponding to the evaluation indicator to obtain an evaluation score includes:

[0142] According to the target calculation function, extracting the parameter value of the target output field as the target output parameter and extracting the parameter value of the reference field as the reference parameter value;

[0143] The target output parameter and the reference parameter value are calculated according to the target calculation function to obtain the evaluation score.

[0144] Specifically, the target calculation function is a mathematical formula or algorithm for realizing the evaluation index. Its input needs to clearly define which fields it depends on. When the target calculation function includes the target output field and the reference field, and the target output field and the reference field are standardized fields, by configuring rather than hard-coding the calculation function and the data field, the decoupling of the evaluation index calculation logic and the evaluation set data can be achieved based on a standardized and modular design.

[0145] In practical applications, a unified output format for evaluation objects and a standard format for evaluation sets are defined to ensure clear and consistent data structures. Downstream metric calculation code (including target calculation functions) can directly reference fields in these standardized formats without additional adaptation or conversion. Users simply specify the desired metrics for the evaluation set through a simple declaration, and the system automatically calculates the relevant metrics. This mechanism significantly reduces the complexity of user metric configuration. Furthermore, this evaluation system pre-packages the following 12 common metrics, covering the needs of most market scenarios: text generation quality evaluation metrics: F1, BLEU, Rouge-1, Rouge-2, Rouge-L; programming question evaluation metric: Pass@K; question type accuracy evaluation metrics: Single-choice ACC, Multiple-choice ACC, Factual Q&A ACC, Summary ACC, Reading Comprehension Q&A ACC, and Instruction Following ACC. The standardized output format and modular metric calculation framework provide excellent scalability for future additions of new evaluation metrics, allowing users to easily add custom metrics based on their needs.

[0146] The object evaluation method provided in the embodiments of this specification decouples evaluation indicators from evaluation sets through a standardized output format, thereby improving the flexibility of the system. The declarative indicator configuration method allows users to quickly complete the setting of evaluation indicators without having to deeply understand the underlying logic, thereby reducing configuration costs. The evaluation platform has 12 built-in common evaluation indicators, which can meet the needs of most scenarios and reduce the workload of users in repeated development. The modular design supports users to flexibly expand new evaluation indicators according to actual needs and adapt to diverse evaluation scenarios.

[0147] The object evaluation method provided in the embodiments of this specification covers a variety of common evaluation scenarios by designing a set of clear and unified question type standard formats and evaluation set standard formats. Through standardized format definitions, users can create and upload evaluation data based on more intuitive and easy-to-understand question type templates; the converted evaluation set data will be directly passed to the downstream participating objects for generating evaluation tasks or questions, thereby realizing full process automation from data upload to evaluation execution; define unified participant input and output specifications, and clarify the structure and format of data interaction. Through standardized design, the user's dependence on complex development work during docking is reduced; users only need to write a short JQ script to complete the mapping between the input parameters of the participating objects and the output parameters required by the evaluation system, which can significantly reduce the configuration complexity while maintaining flexibility; and by defining a unified participant output format and evaluation set standard format, ensure that the data structures of the two are clear and consistent; the downstream indicator calculation code can directly reference the fields in these standardized formats without the need for additional adaptation or conversion.

[0148] See also Figure 3a , Figure 3a A schematic diagram of the processing process of an object evaluation method provided by an embodiment of this specification is shown.

[0149] Specifically, the user needs to select a question format that matches the task type, and upload the evaluation data in accordance with the question format; refer to the calling parameters (i.e., the object input field in the above embodiment) and model output parameters (i.e., the object output field in the above embodiment) of the large language model to be evaluated (i.e., the evaluation object in the above embodiment), and construct a JQ script (parameter mapping script) in accordance with the standard parameter format required by the system; and specify the evaluation indicators corresponding to each evaluation set (built-in in the platform) so that the evaluation system can automatically calculate the indicators based on the parameter mapping relationship.

[0150] It should be noted that users can refer to the target conversion script from the question format to the standard format (i.e. target format) evaluation set to independently complete the production of the standard format evaluation set; they can also upload the evaluation data they have according to the question format configuration, and the evaluation platform will automatically call the target conversion script to achieve the output of the standard format evaluation set. Of course, you can also choose the built-in evaluation set of the evaluation platform, see Figure 3b , Figure 3b The following is a data diagram of a built-in evaluation set provided by an embodiment of this specification. The built-in evaluation set is divided into 10 categories according to the task type: knowledge, reading comprehension, multi-round dialogue, summary, long text, reasoning, mathematics, programming, instruction following, and security. The evaluation sets included in each category can be found in Figure 3bFor example, the knowledge type includes evaluation sets such as MMLU (Massive Multitask Language Understanding) and CMMLU (Chinese Massive Multitask Language Understanding). The reading comprehension type includes evaluation sets such as RACE (Reading Assessment of Comprehension by English Learners) and TriviaQA (Trivia Question Answering). The mathematics type includes evaluation sets such as GSM8K (Grade School Math 8K) and MATH (Mathematics Question Bank). In other words, users can choose to initiate an evaluation using a built-in evaluation set or independently construct an evaluation set based on the question format and then initiate an evaluation. There are no restrictions here.

[0151] In reality, traditional evaluation systems suffer from the limitation of fixed evaluation sets, resulting in high technical barriers and operational costs for users to upload their own evaluation sets. This fixed design not only limits the flexibility of evaluation sets but also increases the time and resource investment in data preparation, affecting the usability and scalability of the evaluation system.

[0152] Therefore, in the embodiments of this specification, a set of clear and unified standard question format and evaluation set standard format (target format) is designed to cover a variety of common evaluation scenarios. Through standardized format definitions, users can create and upload evaluation data based on more intuitive and easy-to-understand question templates; when designing question types, the main types used in the scenarios are covered, so that different evaluation needs can be met, and an automated data processing process is introduced to automatically convert the data in the question format uploaded by the user into evaluation set data in the target format that meets the requirements of the evaluation system according to the built-in conversion script (i.e., the target conversion script in the above embodiment). This mechanism effectively reduces the workload of users manually adjusting the data format and ensures the consistency and compatibility of the data; the evaluation set data after format conversion will be directly passed to downstream participants for generating evaluation tasks or questions, thereby realizing full process automation from data upload to evaluation execution.

[0153] Standardized question formats, target formats, and automated conversions significantly reduce the technical difficulty of users uploading evaluation sets. It also supports diverse question design to meet evaluation needs in different scenarios. The automated process reduces manual intervention, shortens data preparation time, and improves efficiency. The unified data format ensures that evaluation sets can seamlessly adapt to downstream systems, enhancing the stability and scalability of the overall evaluation system.

[0154] Traditional evaluation systems typically require users to complete complex configuration and development processes when adding new participants. This process places high demands on user code development skills, making it difficult for users without technical backgrounds to independently add participants. This high barrier to entry not only limits the evaluation system's ease of use but also reduces its applicability to diverse user groups.

[0155] In the embodiments of this specification, unified input and output specifications for participating objects are defined, and the structure and format of data interaction are clarified. Through standardized design, the dependence of users on complex development work during docking is reduced. Users can complete the mapping between the object input / output fields in the participating objects and the target input / output fields required by the evaluation system by writing short JQ scripts; with the help of the above-mentioned standardized interface (defining standard target input fields and target output fields) and mapping mechanism, users do not need to have a deep code development background to quickly complete the addition and integration of participating objects. The entire process is more intuitive and efficient, and is suitable for both technical and non-technical users.

[0156] Through standardized interfaces and JQ script mapping, the requirements for user code development capabilities are greatly reduced. The simplified configuration process makes the new operations of the participating objects more intuitive, allowing users to get started quickly, lowering the technical threshold and improving ease of use.

[0157] Traditional evaluation systems often tightly bind evaluation metrics and their calculation logic to fixed evaluation sets, making it difficult to flexibly select evaluation metrics for user-uploaded evaluation sets. If users wish to configure or expand evaluation metrics for custom evaluation sets, they incur significant technical costs, including redevelopment or adjustments to the metric calculation logic. This tightly coupled design not only limits the flexibility of the evaluation system but also increases the user experience.

[0158] In the embodiments of this specification, the decoupling of the evaluation indicator calculation logic and the evaluation set data is achieved through standardized design and modular encapsulation, which greatly improves the flexibility and ease of use of the system. Specifically, a unified output format for the evaluation object and the standard format for the evaluation set are defined to ensure that the data structures of the two are clear and consistent. The downstream indicator calculation code can directly reference the fields in these standardized formats (target output fields and reference fields) without the need for additional adaptation or conversion. The user specifies the type of indicator to be calculated for the evaluation set through a simple declaration, and the system can automatically complete the calculation of the relevant indicators. This mechanism significantly reduces the complexity of user configuration of evaluation indicators; and commonly used evaluation indicators are built into the evaluation platform. The standardized output format and modular indicator calculation framework provide good scalability for future new evaluation indicators, and users can easily add custom indicators according to their needs.

[0159] When implementing indicator calculation, capability dimension attribute information can be added to different evaluation sets, so as to facilitate the subsequent statistics of the evaluation scores of the participants based on different capability dimensions, and to sort different participants based on the same capability dimension, thereby generating an evaluation report.

[0160] Standardized output formats decouple evaluation metrics from test sets, enhancing system flexibility. Declarative metric configuration allows users to quickly set up metrics without requiring a deep understanding of the underlying logic. Twelve commonly used metrics are built-in to meet the needs of most scenarios, reducing repetitive development workload. The modular design allows users to flexibly add new metrics based on actual needs, adapting to diverse evaluation scenarios.

[0161] The embodiments of this specification also provide a data processing method applied to a client, including:

[0162] In response to an object evaluation task creation operation triggered by a user, creating an object evaluation task;

[0163] Submit the object evaluation task to the evaluation platform, obtain and display the evaluation results, wherein the evaluation results are obtained by the evaluation platform through the above-mentioned object evaluation method.

[0164] Specifically, users can trigger the object evaluation task creation operation through the front-end interface (such as a web page), such as clicking the "New Evaluation Task" button on the AI ​​model evaluation platform; and users need to fill in or select the following information: evaluation object, target dataset (such as the platform's preset dataset (ImageNet-Val) or custom data uploaded by the user), evaluation indicators (such as accuracy, F1 value) and parameter mapping script (used to specify the mapping script of input / output fields, such as the parameter mapping script is a JQ script).

[0165] After the user configuration is completed, the front-end encapsulates the task parameters into a structured request (such as JSON) and submits it to the evaluation platform; the evaluation platform then uses the above-mentioned object evaluation method to obtain the evaluation results and returns the evaluation results to the client, which will receive and display the evaluation results.

[0166] The specific implementation method can be found in the above embodiments and will not be described in detail here.

[0167] Corresponding to the above method embodiment, this specification also provides an object evaluation system embodiment, Figure 4 FIG1 shows a schematic diagram of the structure of an object evaluation system provided by an embodiment of this specification. Figure 4 As shown in the figure, the system is applied to the evaluation platform, including:

[0168] The task acquisition unit 402 is used to acquire an object evaluation task and determine the evaluation object and target evaluation data in a target format according to the object evaluation task, wherein the target evaluation data in the target format is used to evaluate the evaluation object, and the target evaluation data includes a parameter value corresponding to a target input field and a reference parameter value corresponding to a reference field;

[0169] A parameter mapping unit 404 is configured to determine, through a parameter mapping script, the parameter value of the target input field in the target evaluation data as the input parameter of the evaluation object, and process the input parameter using the evaluation object to obtain a target output parameter;

[0170] The result determination unit 406 is configured to determine the evaluation result of the evaluation object on the target evaluation data according to the target output parameter, the reference parameter value in the target evaluation data, and the evaluation index.

[0171] Optionally, the task acquisition unit 402 is further configured to:

[0172] In the case that the object evaluation task carries initial construction data in an initial format and / or a platform dataset identifier, the evaluation object is determined according to the object evaluation task, and the target evaluation data in the target format is determined according to the initial construction data and / or the platform dataset identifier.

[0173] Optionally, the task acquisition unit 402 is further configured to:

[0174] Converting the initial construction data in the initial format to obtain target construction data in the target format, and / or

[0175] Determining a platform dataset according to the platform dataset identifier, wherein the platform dataset is a dataset preset by the evaluation platform, and the platform dataset includes target platform data in the target format, and the target platform data in the target format is obtained by format conversion of initial platform data in an initial format;

[0176] The target construction data and / or the target platform data are determined as target evaluation data in the target format.

[0177] Optionally, the task acquisition unit 402 is further configured to:

[0178] A target conversion script corresponding to the initial format is determined, and the initial construction data in the initial format is converted into the target construction data in the target format according to the target conversion script.

[0179] Optionally, the parameter mapping unit 404 is further configured to:

[0180] Determine the object input field of the evaluation object and the parameter mapping script according to the object evaluation task, wherein the parameter mapping script includes an input parameter mapping script;

[0181] Establishing an input mapping relationship between the target input field and the object input field through the input parameter mapping script;

[0182] According to the input mapping relationship, the parameter value of the target input field in the target evaluation data is determined as the input parameter of the object input field corresponding to the evaluation object, wherein the input parameter is used to call the evaluation object.

[0183] Optionally, the parameter mapping unit 404 is further configured to:

[0184] Calling the evaluation object according to the input parameters, processing the input parameters using the evaluation object to obtain initial output parameters of the evaluation object, wherein the initial output parameters are obtained by the evaluation object through reasoning on the input parameters;

[0185] Determine the target output parameters according to the output parameter conversion script and the initial output parameters.

[0186] Optionally, the parameter mapping unit 404 is further configured to:

[0187] Determine the parameter value of the object output field corresponding to the evaluation object according to the initial output parameter;

[0188] Establishing an output mapping relationship between the object output field and the target output field through the output parameter mapping script;

[0189] According to the output mapping relationship, the parameter value of the object output field in the initial output parameter is determined as the target output parameter of the target output field.

[0190] Optionally, the result determination unit 406 is further configured to:

[0191] Determine the evaluation indicator according to the object evaluation task, and calculate the target output parameter and the reference parameter value of the reference field according to the target calculation function corresponding to the evaluation indicator to obtain an evaluation score, wherein the evaluation score is used to quantitatively evaluate the performance of the participating object;

[0192] The evaluation object, the target evaluation data, the evaluation indicators and the evaluation scores are structurally integrated to obtain the evaluation results of the evaluation object on the target evaluation data.

[0193] Optionally, the result determination unit 406 is further configured to:

[0194] According to the target calculation function, extracting the parameter value of the target output field as the target output parameter and extracting the parameter value of the reference field as the reference parameter value;

[0195] The target output parameter and the reference parameter value are calculated according to the target calculation function to obtain the evaluation score.

[0196] The system further includes:

[0197] A data configuration unit is used to obtain a data configuration task, and determine at least one target conversion script, a target input field and a target output field corresponding to the evaluation object, and a target calculation function corresponding to the evaluation indicator based on the data configuration task, wherein the at least one target conversion script is constructed based on the target format and at least one initial format, one target conversion script corresponds to one initial format, and the target calculation function includes the target output field and the reference field defined in the target format.

[0198] An embodiment of the present specification provides an object evaluation system applied to an evaluation platform. When obtaining an object evaluation task, the evaluation object and the target evaluation data in the target format used to evaluate the evaluation object can be determined according to the object evaluation task. The target evaluation data in the target format include parameter values ​​corresponding to the target input field and reference parameter values ​​corresponding to the reference field. When the target evaluation data in the target format is used to evaluate the evaluation object, the input specification of the evaluation object is defined using the target input field based on the target evaluation data in a unified target format, thereby reducing the user's dependence on complex development work during docking. Specifically, through the parameter mapping script, the parameter value of the target input field can be determined as the input parameter of the evaluation object, thereby realizing the docking of the unified target input field to the input parameter of the evaluation object. , when users can complete the mapping between the input parameters of the evaluation object and the unified target input field parameter values ​​by writing a lightweight and easy-to-learn parameter mapping script, the configuration complexity can be significantly reduced while maintaining flexibility; the input parameters are processed by the evaluation object to obtain the target output parameters, and the evaluation results of the evaluation object on the target evaluation data are determined according to the target output parameters, the reference parameter values ​​in the target evaluation data and the evaluation indicators. With the help of the unified target input field and mapping the parameter values ​​of the target input field to the input parameters of the evaluation object, users do not need to have a deep code development background to quickly complete the addition and integration of evaluation objects. The whole process is more intuitive and efficient, suitable for both technical and non-technical users, and enhances the applicability of the object evaluation method to diverse user groups.

[0199] The above is a schematic diagram of an object evaluation system according to this embodiment. It should be noted that the technical solution of this object evaluation system and the technical solution of the object evaluation method described above are based on the same concept. For details not described in detail in the technical solution of the object evaluation system, please refer to the description of the technical solution of the object evaluation method described above.

[0200] Figure 5 The block diagram of a computing device 500 according to one embodiment of the present disclosure is shown. Components of the computing device 500 include, but are not limited to, a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and a database 550 is used to store data.

[0201] The computing device 500 also includes an access device 540 that enables the computing device 500 to communicate via one or more networks 560. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 540 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, or a near field communication (NFC) interface.

[0202] In one embodiment of the present specification, the above components of the computing device 500 and Figure 5 Other components not shown in the figure may also be connected to each other, for example, via a bus. Figure 5 The computing device structure block diagram shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art may add or replace other components as needed.

[0203] Computing device 500 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or personal computer (PC). Computing device 500 can also be a mobile or stationary server.

[0204] The processor 520 is configured to execute the following computer program / instructions, which, when executed by the processor, implement the steps of the above-mentioned object evaluation method and data processing method.

[0205] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the computing device embodiment is generally similar to the object evaluation method and data processing method embodiments, so the description is relatively simple. For relevant portions, refer to the description of the object evaluation method and data processing method embodiments.

[0206] An embodiment of the present specification further provides a computer-readable storage medium storing a computer program / instruction, which implements the steps of the above-mentioned object evaluation method and data processing method when executed by a processor.

[0207] Each embodiment in this specification is described in a progressive manner. Similar portions between embodiments can be referenced to each other. Each embodiment focuses on the differences from other embodiments. In particular, the computer-readable storage medium embodiment is generally similar to the object evaluation method and data processing method embodiments, so its description is relatively simple. For relevant portions, refer to the description of the object evaluation method and data processing method embodiments.

[0208] An embodiment of the present specification further provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned object evaluation method and data processing method when executed by a processor.

[0209] The above is a schematic diagram of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product is based on the same concept as the technical solutions of the aforementioned object evaluation method and data processing method. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solutions of the aforementioned object evaluation method and data processing method.

[0210] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0211] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content of the computer-readable medium may be appropriately increased or decreased based on the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0212] It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0213] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0214] The preferred embodiments disclosed above are intended only to help illustrate this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made based on the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that those skilled in the art can better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. An object evaluation method, applied to an evaluation platform, comprising: Obtaining an object evaluation task, and determining an evaluation object and target evaluation data in a target format according to the object evaluation task, wherein the target evaluation data in the target format is used to evaluate the evaluation object, and the target evaluation data includes a parameter value corresponding to a target input field and a reference parameter value corresponding to a reference field; Through the parameter mapping script, the parameter value of the target input field in the target evaluation data is determined as the input parameter of the evaluation object, and the input parameter is processed by the evaluation object to obtain the target output parameter. The object input fields of different evaluation objects are different, and the parameter mapping script is used to dynamically adapt the object input fields of different evaluation objects to achieve compatibility evaluation of multiple different evaluation objects with the same target evaluation data, and the parameter mapping script is used to make the target output parameters of the different evaluation objects correspond to a unified target output field, and the object input field is the input parameter name defined when designing the evaluation object; An evaluation result of the evaluation object on the target evaluation data is determined based on the target output parameter, the reference parameter value in the target evaluation data, and the evaluation index.

2. The object evaluation method according to claim 1, wherein determining the evaluation object and target evaluation data in a target format according to the object evaluation task comprises: In the case that the object evaluation task carries initial construction data in an initial format and / or a platform dataset identifier, the evaluation object is determined according to the object evaluation task, and the target evaluation data in the target format is determined according to the initial construction data and / or the platform dataset identifier.

3. The object evaluation method according to claim 2, wherein determining the target evaluation data in the target format based on the initial construction data and / or the platform dataset identifier comprises: Converting the initial construction data in the initial format to obtain target construction data in the target format, and / or Determining a platform dataset according to the platform dataset identifier, wherein the platform dataset is a dataset preset by the evaluation platform, and the platform dataset includes target platform data in the target format, and the target platform data in the target format is obtained by format conversion of initial platform data in an initial format; The target construction data and / or the target platform data are determined as target evaluation data in the target format.

4. The object evaluation method according to claim 3, wherein the step of converting the initial construction data in the initial format to obtain the target construction data in the target format comprises: A target conversion script corresponding to the initial format is determined, and the initial construction data in the initial format is converted into the target construction data in the target format according to the target conversion script.

5. The object evaluation method according to any one of claims 1 to 4, wherein determining the parameter value of the target input field in the target evaluation data as the input parameter of the evaluation object through a parameter mapping script comprises: Determine the object input field of the evaluation object and the parameter mapping script according to the object evaluation task, wherein the parameter mapping script includes an input parameter mapping script; Establishing an input mapping relationship between the target input field and the object input field through the input parameter mapping script; According to the input mapping relationship, the parameter value of the target input field in the target evaluation data is determined as the input parameter of the object input field corresponding to the evaluation object, wherein the input parameter is used to call the evaluation object.

6. The object evaluation method according to claim 1, wherein the parameter mapping script further comprises an output parameter mapping script; The step of processing the input parameters using the evaluation object to obtain target output parameters includes: Calling the evaluation object according to the input parameters, processing the input parameters using the evaluation object to obtain initial output parameters of the evaluation object, wherein the initial output parameters are obtained by the evaluation object through reasoning on the input parameters; Determine the target output parameters according to the output parameter mapping script and the initial output parameters.

7. The object evaluation method according to claim 6, wherein determining the target output parameter according to the output parameter mapping script and the initial output parameter comprises: Determine the parameter value of the object output field corresponding to the evaluation object according to the initial output parameter; Establishing an output mapping relationship between the object output field and the target output field through the output parameter mapping script; According to the output mapping relationship, the parameter value of the object output field in the initial output parameter is determined as the target output parameter of the target output field.

8. The object evaluation method according to any one of claims 1 to 4, 6 to 7, wherein determining the evaluation result of the evaluation object on the target evaluation data based on the target output parameter, the reference parameter value in the target evaluation data, and the evaluation index comprises: Determining the evaluation indicator according to the object evaluation task, and calculating the target output parameter and the reference parameter value according to the target calculation function corresponding to the evaluation indicator to obtain an evaluation score, wherein the evaluation score is used to quantitatively evaluate the performance of the participating object; The evaluation object, the target evaluation data, the evaluation indicators and the evaluation scores are structurally integrated to obtain the evaluation results of the evaluation object on the target evaluation data.

9. The object evaluation method according to claim 8, wherein the target output parameter is a parameter value of a target output field corresponding to the evaluation object, and the target calculation function corresponding to the evaluation indicator includes the target output field and the reference field; The step of calculating the target output parameter and the reference parameter value of the reference field according to the target calculation function corresponding to the evaluation indicator to obtain an evaluation score includes: According to the target calculation function, extracting the parameter value of the target output field as the target output parameter and extracting the parameter value of the reference field as the reference parameter value; The target output parameter and the reference parameter value are calculated according to the target calculation function to obtain the evaluation score.

10. The object evaluation method according to any one of claims 1-4, 6-7, and 9, before obtaining the object evaluation task, further comprising: Obtain a data configuration task, and determine at least one target conversion script, the target input field and target output field corresponding to the evaluation object, and the target calculation function corresponding to the evaluation indicator based on the data configuration task, wherein the at least one target conversion script is constructed based on the target format and at least one initial format, one target conversion script corresponds to one initial format, and the target calculation function includes the target output field and the reference field defined in the target format.

11. A data processing method, applied to a client, comprising: In response to an object evaluation task creation operation triggered by a user, creating an object evaluation task; Submit the object evaluation task to the evaluation platform, obtain and display the evaluation results, wherein the evaluation results are obtained by the evaluation platform through the object evaluation method according to any one of claims 1 to 10.

12. An object evaluation system, applied to an evaluation platform, comprising: a task acquisition unit, configured to acquire an object evaluation task and determine, based on the object evaluation task, an evaluation object and target evaluation data in a target format, wherein the target evaluation data in the target format is used to evaluate the evaluation object, and the target evaluation data includes a parameter value corresponding to a target input field and a reference parameter value corresponding to a reference field; a parameter mapping unit, configured to determine, through a parameter mapping script, the parameter value of the target input field in the target evaluation data as the input parameter of the evaluation object, and process the input parameter using the evaluation object to obtain a target output parameter, wherein different evaluation objects have different object input fields, and the parameter mapping script is used to dynamically adapt the object input fields of different evaluation objects to achieve compatibility evaluation of multiple different evaluation objects with the same target evaluation data, and the parameter mapping script is used to make the target output parameters of the different evaluation objects correspond to a unified target output field, wherein the object input field is the input parameter name defined when the evaluation object is designed; The result determination unit is used to determine the evaluation result of the evaluation object on the target evaluation data according to the target output parameter, the reference parameter value in the target evaluation data and the evaluation index.

13. A computing device comprising: memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the method according to any one of claims 1 to 11 are implemented.

14. A computer-readable storage medium storing a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the steps of the method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program / instruction, which implements the steps of the method according to any one of claims 1 to 11 when executed by a processor.

Images