Model evaluation task processing method and device

Abstract

The present application discloses a method and device for processing a model evaluation task, and relates to the field of artificial intelligence technology. The present application can obtain the evaluation task corresponding to the model to be evaluated, and perform streaming processing on the subtasks based on the multiple substeps contained in each subtask; in the process of processing the subtasks, the task status of the subtask is determined based on the execution data corresponding to each substep; the execution progress of the evaluation task is calculated based on the number of subtasks whose task status is completed; if the execution progress reaches the preset progress, a backup task that is the same as the candidate task is established and executed; if the task status of all subtasks in the evaluation task is completed, the related tasks of the evaluation task are terminated. When the evaluation task is executed to a certain progress, the unfinished subtasks are backed up and executed. When the candidate task has problems and is executed slowly, the backup task can also be executed normally to speed up the efficiency of the subtasks, thereby shortening the time consumption of the entire model evaluation.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method and device for processing a model evaluation task. Background Art

[0002] Model evaluation refers to the performance evaluation of a trained model to understand how the model performs on new data that it has not seen. This process involves a comprehensive and detailed consideration of the model in order to accurately understand its capabilities in practical applications. In model evaluation, a series of carefully designed indicators are usually used to quantify the performance of the model in multiple dimensions, such as predictive ability, generalization ability, etc.

[0003] In order to achieve a comprehensive evaluation of the model, a variety of different data are usually used to evaluate the model's capabilities. However, this approach also makes the evaluation process time-consuming. Therefore, there is an urgent need for a way to shorten the time required for model evaluation. Summary of the invention

[0004] The embodiments of the present application provide a method and device for processing a model evaluation task, which can effectively shorten the time consumption of model evaluation.

[0005] The present application embodiment provides a method for processing a model evaluation task, including:

[0006] Obtaining an evaluation task corresponding to the model to be evaluated, wherein the evaluation task includes multiple subtasks, and the subtask includes multiple substeps arranged in an execution order;

[0007] Based on the multiple sub-steps included in each of the sub-tasks, stream processing is performed on the multiple sub-tasks;

[0008] In the process of processing the subtask, based on the execution data corresponding to each substep, determining the task status of the subtask, the task status including a completed state and an uncompleted state;

[0009] Calculating the execution progress of the evaluation task based on the number of subtasks whose task status is completed;

[0010] If the execution progress of the evaluation task reaches the preset progress, a backup task identical to the candidate task is established and executed, and the candidate task is a subtask whose task status is in an unfinished state;

[0011] If the task statuses of all subtasks in the evaluation task are in the completed state, the related tasks of the evaluation task are terminated, and the related tasks include the candidate task and the backup task.

[0012] The present application also provides a model evaluation task processing device, including:

[0013] An acquisition module, used to acquire an evaluation task corresponding to the model to be evaluated, wherein the evaluation task includes multiple subtasks, and the subtask includes multiple substeps arranged in an execution order;

[0014] A processing module, configured to perform stream processing on the multiple subtasks based on the multiple substeps included in each of the subtasks;

[0015] A status module, used to determine the task status of the subtask based on the execution data corresponding to each substep during the processing of the subtask, wherein the task status includes a completed status and an uncompleted status;

[0016] A calculation module, used for calculating the execution progress of the evaluation task based on the number of subtasks whose task status is completed;

[0017] A backup module, configured to create and execute a backup task that is the same as a candidate task if the execution progress of the evaluation task reaches a preset progress, wherein the candidate task is a subtask whose task status is an unfinished state;

[0018] The ending module is used to end the related tasks of the evaluation task if the task status of all subtasks in the evaluation task are in the completion status, and the related tasks include the candidate task and the backup task.

[0019] An embodiment of the present application also provides an electronic device, including a processor and a memory, wherein the memory stores multiple instructions; the processor loads instructions from the memory to execute the steps in any model evaluation task processing method provided in the embodiment of the present application.

[0020] An embodiment of the present application also provides a computer-readable storage medium, which stores multiple instructions, and the instructions are suitable for a processor to load to execute the steps in any model evaluation task processing method provided in the embodiment of the present application.

[0021] An embodiment of the present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in any one of the model evaluation task processing methods provided in the embodiments of the present application.

[0022] The embodiment of the present application can obtain the evaluation task corresponding to the model to be evaluated. The evaluation task includes multiple subtasks. Based on the multiple substeps in the subtasks, the multiple subtasks can be stream-processed. In the process of processing the subtasks, the task status of the subtasks is determined based on the execution data of each substep; the execution progress of the evaluation task is calculated based on the number of subtasks in the completed state; if the execution progress reaches the preset progress, a backup task that is the same as the candidate task is established and executed; when all subtasks in the evaluation task are in the completed state, the execution of the candidate task and the backup task is terminated. When the evaluation task is executed to a certain progress, the unfinished subtasks are backed up and executed. When the candidate task has problems and executes slowly, the backup task can also be executed normally to speed up the efficiency of the subtasks, thereby shortening the time consumption of the entire model evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 It is a flowchart of a model evaluation task processing method provided in an embodiment of the present application;

[0025] Figure 2 It is a schematic diagram of multiple sub-steps provided in the embodiment of the present application;

[0026] Figure 3 is a schematic diagram of determining the state of a step provided in an embodiment of the present application;

[0027] Figure 4 It is a schematic diagram of the architecture of the model evaluation task processing method provided in the embodiment of the present application;

[0028] Figure 5 is a structural diagram of a model evaluation task processing device provided in an embodiment of the present application;

[0029] Figure 6 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0031] With the rapid development of artificial intelligence technology, models trained based on artificial intelligence technology have been widely used in various fields. The trained neural network model needs to clarify its actual performance on unknown new data. Through model evaluation, the model can be comprehensively evaluated in order to accurately understand its capabilities in practical applications. Since the performance of the model in multiple dimensions needs to be quantified in model evaluation, a complete model evaluation task can be divided into multiple subtasks. Since the model evaluation processes a large amount of data, it takes a long time. The inventor found in the study that when the model evaluation task is executed to a certain progress, a small part of the subtasks will be left to execute very slowly, thereby lengthening the duration of the entire model evaluation. Therefore, the inventor proposed a model evaluation task processing method to shorten the execution time of the model evaluation task.

[0032] It is understandable that in the specific implementation of the present application, relevant data such as user information needs to obtain user permission or consent, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0033] In this embodiment, a model evaluation task processing method is provided, such as Figure 1 As shown, the specific process of the model evaluation task processing method can be as follows:

[0034] S110: Obtain the evaluation task corresponding to the model to be evaluated.

[0035] The model to be evaluated refers to the neural network model that needs to be evaluated, and the evaluation task refers to the task that needs to be performed when evaluating the model to be evaluated. Usually, after fine-tuning and training, a large number of evaluations are required to clarify the performance and effect of the neural network model. For example, the model's generalization ability in different scenarios, its ability to understand and reason about natural language, and its ability to filter unsafe content can be evaluated. Therefore, the evaluation task can include multiple subtasks.

[0036] For each subtask, it is necessary to call the model to be evaluated to process the corresponding evaluation data and give the corresponding processing results to realize the execution of the subtask. Among them, different subtasks use different evaluation data, and each subtask can be divided into multiple sub-steps with an execution order, that is, only the multiple sub-steps need to be executed in sequence according to the execution order to be considered to have been executed. Among them, the sub-steps included in the subtask can be set according to actual needs, and the multiple sub-steps included in all subtasks can be the same or different, and no specific limitation is made here.

[0037] Optionally, the evaluation task of the model to be evaluated can be initiated by the user. For example, a management platform can be provided, and the user can pre-configure the data and parameters required for model evaluation on the management platform, and then the user actively initiates the evaluation task of the model to be evaluated. Optionally, the evaluation task can also be automatically triggered after the model fine-tuning training is completed. For example, after the model fine-tuning training is completed, the training completion information can be obtained, and when the training completion information is obtained, the evaluation task can be automatically started.

[0038] S120: Based on the multiple sub-steps included in each of the sub-tasks, perform stream processing on the multiple sub-tasks.

[0039] Stream processing is a data processing mode that can be used to process data streams generated in real time. In stream processing, the data source continuously generates data, which needs to be processed in real time and generate corresponding output. Subtasks can be considered as data sources. Since an evaluation task contains multiple subtasks, when the user starts to execute the evaluation task, these subtasks can be continuously obtained, and then the server cluster executes the subtasks.

[0040] When a server cluster runs a large number of subtasks at the same time, the cluster may be overloaded, which may lead to cluster crash, failure to start some containers, etc. In order to avoid cluster overload, as an implementation method, based on the multiple substeps contained in the subtask, when multiple subtasks are stream-processed, for each subtask, a resource request corresponding to the subtask is obtained; when the resource request is passed, the execution environment of the subtask is initialized, and the task identifier corresponding to the subtask is determined; according to the task identifier, the evaluation type and evaluation data of the subtask are obtained; the evaluation data is processed using the model to be evaluated to obtain a processing result; based on the evaluation type, the processing result is aggregated to obtain the evaluation result of the model to be evaluated under the evaluation type.

[0041] Among them, multiple subtasks can be stored in a queue, and corresponding resource requests are initiated in the order of the subtasks in the queue, that is, each subtask corresponds to a resource request, and the resource request includes the resource information required to execute the subtask. Based on the resource information carried therein and the available resources of the server cluster, it is determined whether there are sufficient resources to run the subtask. If the current server cluster has sufficient resources to run the subtask, it can be determined that the resource request is passed, and the resources can be locked for the subtask so that the subtask can be executed normally. If there are not enough resources to run the subtask, it can be determined that the resource request is not passed. The subtask can continue to initiate resource requests after waiting for a certain period of time. During the waiting process, it can receive resource requests corresponding to other subtasks for processing. Only when the subtask seizes the resources can it start execution, which can avoid cluster overload.

[0042] Optionally, in the embodiment of the present application, each subtask corresponds to the same multiple substeps, which may include container initialization, pulling evaluation data, evaluating the model, scoring, and reporting the results. For example, see Figure 2 , showing a schematic diagram of multiple sub-steps, which will be combined below Figure 2 , detailing the execution process of the subtask. Among them, container initialization can initialize the execution environment of the subtask and make basic preparations for the execution of the subtask. Optionally, the process of initializing the execution environment may include loading the required software packages, libraries, and dependencies, etc., to ensure that the subtask can run in the correct computing environment. At the same time, during the environment initialization process, a unique task identifier can be assigned or determined for the subtask to identify the current subtask. The task identifier plays an important identification and tracking role in the entire evaluation process, which can help identify and manage different subtasks. The container initialization substep can generate a task identifier, which can be passed into the pull evaluation data substep.

[0043] Pulling evaluation data refers to the data set required to execute the subtask. The task identifier can be used to obtain the evaluation type and evaluation data of the subtask. Each subtask can be pre-set with configuration information, which may include the evaluation type of the subtask, the acquisition address of the evaluation data, and the batch of evaluation data. After querying the configuration information, the acquisition address of the evaluation data can be used to download the data, and then the corresponding batch of data can be intercepted from the data based on the batch of evaluation data to obtain the evaluation data of the subtask. Optionally, the evaluation data can be an evaluation test set. The evaluation type characterizes the purpose of the current subtask, for example, whether it is to evaluate generalization ability, language comprehension ability, or the ability to filter unsafe content.

[0044] Then the obtained evaluation type and evaluation data can be passed to the next sub-step for use. That is, when evaluating the model, the model to be evaluated can be loaded, so that the model to be evaluated can understand the evaluation questions and give answers. For example, the evaluation question set may include multiple-choice questions, true-or-false questions, etc. For multiple-choice questions, the model to be evaluated needs to give the options it infers, and for true-or-false questions, the model to be evaluated needs to give correct or wrong answers. That is, this sub-step can obtain the answers inferred by the model to be evaluated, and these answers and the evaluation type are passed to the next sub-step, that is, the "scoring" sub-step. Then the answer can be compared with the correct answer to calculate the accuracy of the model to be evaluated, and then the accuracy, total number of questions and evaluation type are used as the processing results and passed to the next sub-step, that is, the "result reporting" sub-step. In the result reporting sub-step, the processing results can be reported to the streaming engine, and finally the processing results are aggregated according to the evaluation type to obtain the evaluation results of the model to be evaluated.

[0045] S130. In the process of executing the subtask, determine the task status of the subtask based on the execution data corresponding to each substep.

[0046] Among them, the subtask can be executed in the container, and executing the subtask means executing each substep contained in the subtask. For each substep, when executing the substep, the execution data of the substep can be obtained, wherein the execution data may include the process data of executing the substep and the execution time of the substep, etc. The task status of the subtask may include a completed status and an uncompleted status. The completed status means that all substeps in the subtask have been executed, while the uncompleted status means that there are still substeps in the subtask being executed.

[0047] As an implementation mode, when determining the execution status of a subtask, for each substep corresponding to each subtask, in response to the start execution information of the substep, the process data and execution time of the substep are obtained; the exception level of the substep is determined in combination with the execution time and the process data; based on the exception level and the end execution information of the substep, the step status corresponding to the substep is determined, the step status including a success status and a running status; if the step status corresponding to all substeps in the subtask are success status, the task status of the subtask is determined to be a completed status; if there is any substep in the subtask whose corresponding step status is a running status, the task status of the subtask is determined to be an uncompleted status.

[0048] To determine the task status of a subtask, you need to first determine the step status of each substep in the subtask, which can include a success status and a running status. If the step status of all substeps in the subtask is a success status, the task status of the subtask can be determined to be a completed status. If the step status of a substep in the subtask is a running status, the task status of the subtask can be determined to be an uncompleted status.

[0049] In some embodiments, the step status may also include an abnormal state, see Figure 3 , showing a schematic diagram of determining the state of the step, the following will be combined Figure 3 Detailed description. When each sub-step starts to execute, the corresponding start execution information can be obtained, and when the sub-step ends to execute, the corresponding end execution information can be obtained. The execution time refers to the time consumed by the execution of the sub-step. When the sub-step start execution information is obtained, a timer can be started based on the start execution information to detect the execution time of the sub-step.

[0050] Process data refers to detailed operation data generated when executing a sub-step, which may include exceptions encountered during execution, etc. Process data can be obtained at multiple time points, for example, it can be obtained at fixed intervals, when an exception is encountered during the execution of a sub-step, or when the sub-step is completed.

[0051] Combined with the execution time and the process data of the sub-step, it can be determined whether the sub-step encounters an exception during execution, and then the exception level of the sub-step can be determined. Among them, the exception level can be divided into level 1 exception, level 2 exception and level 3 exception. Level 1 exception can be considered as no exception, level 2 exception can be considered as an exception that can be resolved by retrying, and level 3 exception is a more serious exception that can only be resolved after staff intervention.

[0052] As an implementation mode, when determining the abnormality level of a sub-step, an abnormality analysis may be performed on the process data to determine the abnormality type of the sub-step, wherein the abnormality type includes no abnormality, system abnormality, and business abnormality; if the execution time is not greater than the preset time, and the abnormality type is no abnormality, the abnormality level of the sub-step is determined to be a first-level abnormality; if the execution time is greater than the preset time, or the abnormality type is a system abnormality, the abnormality level of the sub-step is determined to be a second-level abnormality; if the abnormality type is a business abnormality, the abnormality level of the sub-step is determined to be a third-level abnormality.

[0053] By performing an exception analysis on the process data, it is possible to determine whether an exception occurs during the execution of the sub-step, and based on the process data, the exception type of the sub-step can be determined. Among them, the exception type of the sub-step may include no exception, system exception, and business exception. No exception means that there is no data related to the exception in the process data of the sub-step, system exceptions may include local file IO exceptions, download resource exceptions, etc., and business exceptions may include parameter setting exceptions, exceeding parameter ranges, etc.

[0054] The preset duration refers to the maximum execution duration of the sub-step that is preset in advance, wherein the preset duration of each sub-step can be set to the same value, or can be set to different values ​​based on actual circumstances.

[0055] The execution duration can be recorded by a timer. If the information indicating the end of the execution of the sub-step is received before the timer reaches the preset duration, it can be determined that the execution duration of the sub-step is not greater than the preset duration. If the information indicating the end of the execution of the sub-step is not received before the timer reaches the preset duration, it can be determined that the execution duration of the sub-step is greater than the preset duration.

[0056] If the execution time is not greater than the preset time, and the exception type is no exception, the exception level of the sub-step can be determined as a level 1 exception, that is, no exception occurs during the execution of the sub-step. If the execution time is greater than the preset time, or the exception type is a system exception, the exception level of the sub-step can be determined as a level 2 exception. If the exception type is a business exception, the exception level of the sub-step can be directly determined as a level 3 exception.

[0057] Then, the step status of the sub-step can be determined based on the exception level and the end execution information of the sub-step, wherein the step status of the sub-step can include a success status, a running status and an exception status, the success status means that the sub-step is executed normally and completed, the running status means that the sub-step is being executed, and the exception status means that an exception occurs in the execution of the sub-step.

[0058] Optionally, if the exception level is a level one exception and the end execution information of the sub-step is obtained, the step status of the sub-step is determined to be a success status; if the exception level is a level one exception and the end execution information of the sub-step is not obtained, the step status of the sub-step is determined to be a running status; if the exception level is a level two exception, the designated task is retried, and the step status of the sub-step is determined based on the result of the retry process, and the designated task is the sub-task corresponding to the sub-step; if the exception level is a level three exception, the step status of the sub-step is determined to be an abnormal status, and the evaluation task is terminated.

[0059] The end execution information of the sub-step can be obtained when the sub-step is executed, which also indicates that the sub-step has been executed. If the exception level is a level 1 exception and the end execution information of the sub-step is obtained, it can be determined that the sub-step is successfully executed, and the step status is a success status. If the exception level is a level 1 exception but the end execution information is not obtained, it can be determined that the sub-step is still being executed, and the step status is a running status.

[0060] If the exception level is level 2, the sub-step needs to be retried. When retrying, the sub-task corresponding to the sub-step is used as the designated task, and then the designated task is re-established to achieve retry, and the step status of the sub-step is determined based on the retry result. If the exception level is level 3, it can be considered that a major exception has been encountered during the execution of the sub-step, which requires manual processing. The step status can be determined to be an abnormal state, and the evaluation task can be terminated directly.

[0061] Optionally, performing a retry process and determining the step status based on the result of the retry process may include the following steps: obtaining the number of retries corresponding to the specified task and updating it to obtain a new number of retries; if the new number of retries is not greater than a preset number, establishing a retry task that is the same as the specified task; executing the retry task to determine the step status corresponding to the retry sub-step in the retry task, the retry sub-step being the same as the sub-step; using the step status corresponding to the retry sub-step as the step status of the sub-step; if the new number of retries is greater than the preset number, determining that the step status of the sub-step is an abnormal state, and terminating the evaluation task.

[0062] The number of retries is the number of times the specified task is retried. The initial value of the number of retries for each subtask can be set to 0 by default. When the specified task enters the retry process, the number of retries can be obtained and updated first, that is, 1 is added to the obtained number of retries to obtain a new number of retries. If the new number of retries is not greater than the preset number, it can be determined that the specified task can enter the retry process; if the new number of retries is greater than the preset number, it can be determined that the specified task can no longer enter the retry process.

[0063] The preset number of times may be a preset maximum number of retries for a specified task, and the number of retries may be set according to actual needs and is not specifically limited here.

[0064] Optionally, the preset number of times can be set separately according to the triggering conditions of the retry process. For example, based on the above description, it can be known that there are two triggering conditions for retrying the specified task. Trigger condition 1 is that the execution time of the sub-step is greater than the preset time, and trigger condition 2 is that the exception type of the sub-step is a system exception. Then, for different triggering conditions, corresponding preset durations can be set. Of course, in this case, when obtaining the number of retries, it is also necessary to obtain it for the specific triggering conditions to correctly determine whether to enter the retry process. For example, the preset number of times for the above two triggering conditions are both set to 3, the number of retries for the specified task under trigger condition 1 is 1, and the number of retries under trigger condition 2 is 3. When the specified task is retried due to trigger condition 1, the number of retries obtained should be 1, and when the specified task is retried due to trigger condition 2, the number of retries obtained should be 3.

[0065] When the new number of retries is not greater than the preset number of times, a retry task identical to the specified task can be established and then executed. Since the retry task and the specified task are identical, the retry task can include a retry substep identical to the substep. The step state of the retry substep can be determined based on the same method as the aforementioned determination of the step state of the substep, wherein the step state of the retry substep is the step state of the substep. If the new number of retries is greater than the preset number of times, the step state of the substep can be directly determined to be an abnormal state.

[0066] If the abnormality type is a business abnormality, the step status of the sub-step can be determined to be an abnormal state, and the evaluation task can be terminated. That is, when the step status of the sub-step is an abnormal state, the entire evaluation task can be terminated directly, and the evaluation task can be performed again after the staff makes adjustments.

[0067] Based on the above process, the step status corresponding to each sub-step can be determined, and the step status of the sub-step can include abnormal status, successful status, and running status. When the step status of any sub-step in the subtask is abnormal, the entire evaluation task will be terminated directly.

[0068] If the step status of all sub-steps in a sub-task is in the successful state, the task status of the sub-task can be determined to be completed. If the step status of a sub-step in a sub-task is in the running state, the task status of the sub-task can be determined to be unfinished. In the process of determining the task status of a sub-task, the latest execution progress and related data can be obtained in real time, and when an exception occurs in the execution of a sub-step, a corresponding fault-tolerant mechanism is introduced to improve the success rate of task execution.

[0069] S140: Calculate the execution progress of the evaluation task based on the number of subtasks whose task status is completed.

[0070] Based on the contents of S110 to S130 above, it can be known that the task status of a subtask can include two types, one is a completed state and the other is an uncompleted state. The number of subtasks with a completed state can be counted, and then the execution progress of the evaluation task can be calculated using the number.

[0071] The number of subtasks with a completed task status may be taken as the first number, the total number of subtasks in the evaluation task may be taken as the second number, and then the execution progress of the evaluation task may be obtained by dividing the first number by the second number.

[0072] S150: If the execution progress of the evaluation task reaches a preset progress, create and execute a backup task that is the same as the candidate task.

[0073] The preset progress is a progress set according to actual needs, which can be obtained through a large amount of data analysis. For example, the historical data corresponding to the historical evaluation task can be obtained, and the occurrence point of the long tail situation can be determined by analyzing the historical data. The long tail situation refers to the situation where most of the subtasks are completed, but the execution time of the remaining small part of the subtasks is too long, resulting in a long time for the entire evaluation task. Through the analysis of historical data, most of the subtasks can be completed in a relatively short time, but the remaining small part of the subtasks takes longer to complete, resulting in a long overall time for the model evaluation task.

[0074] In order to speed up the execution efficiency of the evaluation task, when the execution progress of the evaluation task reaches the preset progress, a backup task identical to the candidate task can be established and executed. The candidate task refers to a subtask whose task status is unfinished. For the candidate task, the candidate task can be backed up, that is, a backup task identical to the candidate task can be established and executed.

[0075] In some implementations, each executed task has a corresponding task identifier. To avoid identifier conflicts, the task identifier of the backup task can be converted from the task identifier of the corresponding candidate task. The specific conversion rules can be set according to actual needs. Of course, based on the conversion rules, the task identifier of the backup task can also be converted into the task identifier of the candidate task. For example, if the task identifier of a candidate task is x, the task identifier of the corresponding backup task can be y(x).

[0076] S160: If the task statuses of all subtasks in the evaluation task are completed, terminate the related tasks of the evaluation task.

[0077] The evaluation task includes multiple subtasks. When the task status of all subtasks in the evaluation task is completed, it can be determined that the evaluation task is completed, and the related tasks of the evaluation task can be terminated. The related tasks may include candidate tasks and backup tasks.

[0078] Since the candidate task is a subtask that is still being executed, and the task status of the remaining subtasks are all in the completed state, in order to reduce the amount of data processing, it is possible to determine whether the evaluation task has been completed based on the candidate task. For example, it can be to obtain the task identifier corresponding to the candidate task and obtain the pending identification group; detect the task status of the candidate task and the backup task; if the task status of any one of the candidate task or the backup task is in the completed state, obtain the task identifier corresponding to the task with the completed state as a specified identifier; delete the specified identifier from the pending identification group; if the pending identification group is empty, close the candidate task or the backup task that is being executed.

[0079] Candidate tasks are subtasks that have not yet been completed. Each candidate task will be assigned a task ID after execution. The task ID of the candidate task can be directly obtained to obtain the pending ID group. It is understandable that in addition to the candidate task being executed, the newly created backup task also starts to execute, so the task status of the candidate task and the backup task can be detected.

[0080] The target task refers to a candidate task or backup task whose task status is completed. After the target task is detected, the task ID of the target task can be obtained as the target ID. Then the target ID can be deleted from the pending ID group. If the pending ID group is empty, it can be determined that all subtasks in the evaluation task have been completed, and the related tasks of the evaluation task can be terminated, that is, the backup task and candidate task that are being executed can be closed.

[0081] The task identifier corresponding to the candidate task in the to-be-completed identifier group is a task identifier. Since the target task may be a backup task, when deleting the target identifier from the to-be-completed identifier group, the target identifier can be searched in the to-be-completed identifier group; if the target identifier is found in the to-be-completed identifier group, the target identifier is deleted from the to-be-completed identifier group; if the target identifier is not found in the to-be-completed identifier group, the target identifier is converted to obtain a conversion identifier; and the conversion identifier is deleted from the to-be-completed identifier group.

[0082] The target identifier can be searched in the to-be-completed identifier group. If the target identifier is found, it is directly deleted from the to-be-completed identifier group. If the target identifier is not found in the to-be-completed identifier group, it indicates that the target identifier may be the task identifier of the backup task. The task identifier of the backup task and the task identifier of the corresponding candidate task can be converted to each other based on the conversion rule. Therefore, the target identifier can be converted based on the corresponding conversion rule to obtain the identifier of the corresponding candidate task, which is recorded as the conversion identifier, and then the conversion identifier is found from the to-be-completed identifier group and deleted.

[0083] When all the identifiers in the pending identifier group are deleted, it can be determined that the candidate tasks have been completed. At this time, it can be determined that the model evaluation task has been completed, so all the subtasks being executed in the model evaluation task can be closed. For example, the candidate tasks include A, B, C, and D, and the backup tasks are A1, B1, C1, and D1 respectively. If A1 and B1 are completed, and C and D are completed, then the subtasks that need to be ended are A, B, C1, and D1.

[0084] For a more detailed description of the evaluation task processing method, see Figure 4, showing an architectural diagram of the model evaluation task processing method. Among them, multiple subtasks can be run in the cluster, each subtask contains multiple substeps, and each substep will report corresponding data to the streaming computing engine when it starts and ends execution. The streaming computing engine can obtain the execution status of each substep and the staged processing results, and then perform statistics, etc., to master the relevant data of the evaluation task. During the execution of the subtasks, they are processed according to the contents of the above steps S110 to S160. Finally, when the evaluation task is completed, the corresponding evaluation results can be sent to the management platform for display.

[0085] The model evaluation task processing scheme provided in the embodiment of the present application can be applied to various scenarios for evaluating trained neural network models. For example, taking the vertical model evaluation in the health care field as an example, the scheme provided in the embodiment of the present application can split the entire evaluation task into multiple subtasks when evaluating the capabilities of the vertical model, and when the execution progress of the evaluation task reaches the preset progress, the execution efficiency of the evaluation task is accelerated through the backup task, further shortening the time consumption of the model evaluation.

[0086] The method provided in the embodiment of the present application can split the model evaluation task into multiple subtasks, each of which may include multiple sub-steps. The subtasks are stream-processed through multiple sub-steps to avoid excessive cluster load. A complete fault-tolerant mechanism is introduced during execution to increase the probability of successful execution of the subtasks. When the execution progress of the entire model evaluation task reaches the preset progress, the subtask being executed is backed up. Even if the subtask being executed is slow, the backup task is still executed, which increases the possibility of completing the subtask, thereby effectively shortening the execution time of the model evaluation task.

[0087] In order to better implement the above method, the embodiment of the present application also provides a model evaluation task processing device, which can be integrated in an electronic device, and the electronic device can be a terminal, a server, etc. Among them, the terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a laptop, a personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers.

[0088] For example, in this embodiment, the method of the embodiment of the present application will be described in detail by taking the specific integration of the model evaluation task processing device in the server as an example.

[0089] For example, Figure 5 As shown, the model evaluation task processing device may include an acquisition module 210 , a processing module 220 , a state module 230 , a calculation module 240 , a backup module 250 and an end module 260 .

[0090] An acquisition module 210 is used to acquire an evaluation task corresponding to the model to be evaluated, wherein the evaluation task includes multiple subtasks, and the subtask includes multiple substeps arranged in an execution order;

[0091] A processing module 220, configured to perform stream processing on the multiple subtasks based on the multiple substeps included in each of the subtasks;

[0092] A status module 230 is used to determine the task status of the subtask based on the execution data corresponding to each substep during the processing of the subtask, wherein the task status includes a completed state and an uncompleted state;

[0093] A calculation module 240, configured to calculate the execution progress of the evaluation task based on the number of subtasks whose task status is completed;

[0094] A backup module 250 is used to create and execute a backup task that is the same as a candidate task if the execution progress of the evaluation task reaches a preset progress, and the candidate task is a subtask whose task status is an unfinished state;

[0095] The ending module 260 is used to end the related tasks of the evaluation task if the task status of all subtasks in the evaluation task are in the completed status, and the related tasks include the candidate task and the backup task.

[0096] In some embodiments, the execution data includes execution duration and process data, and the state module 230 is specifically used to:

[0097] For each of the sub-steps corresponding to each of the sub-tasks, in response to the start execution information of the sub-steps, obtaining the process data and execution duration of the sub-steps;

[0098] Determine the abnormality level of the sub-step in combination with the execution time and the process data;

[0099] Based on the abnormal level and the end execution information of the sub-step, determining the step status corresponding to the sub-step, the step status including a success status and a running status;

[0100] If the step statuses corresponding to all sub-steps in the sub-task are all in the success status, determining that the task status of the sub-task is the completion status;

[0101] If the step status corresponding to any sub-step in the sub-task is in the running status, it is determined that the task status of the sub-task is in the unfinished status.

[0102] In some embodiments, the status module 230 is specifically configured to:

[0103] Performing an abnormality analysis on the process data to determine the abnormality type of the sub-step, wherein the abnormality type includes no abnormality, system abnormality, and business abnormality;

[0104] If the execution time is not greater than the preset time, and the exception type is no exception, determining that the exception level of the sub-step is a first-level exception;

[0105] If the execution time is longer than the preset time, or the exception type is a system exception, determining that the exception level of the sub-step is a secondary exception;

[0106] If the exception type is a business exception, determine that the exception level of the sub-step is a level three exception.

[0107] In some embodiments, the step status also includes an abnormal state, and the status module is specifically used to:

[0108] If the exception level is a level one exception and the end execution information of the sub-step is obtained, it is determined that the step status of the sub-step is a success status;

[0109] If the abnormal level is a level one abnormality and the end execution information of the sub-step is not obtained, determining that the step state of the sub-step is a running state;

[0110] If the abnormality level is a level 2 abnormality, retry the designated task, and determine the step status of the sub-step based on the result of the retry process, and the designated task is the sub-task corresponding to the sub-step;

[0111] If the abnormality level is level three, the step status of the sub-step is determined to be an abnormal state, and the evaluation task is terminated.

[0112] In some embodiments, the status module 230 is specifically configured to:

[0113] Obtain the number of retries corresponding to the specified task and update it to obtain a new number of retries;

[0114] If the new retry number is not greater than the preset number, creating a retry task that is the same as the specified task;

[0115] Execute the retry task to determine a step status corresponding to a retry sub-step in the retry task, the retry sub-step being the same as the sub-step;

[0116] Using the step status corresponding to the retry sub-step as the step status of the sub-step;

[0117] If the new number of retries is greater than the preset number of times, the step status of the sub-step is determined to be an abnormal state, and the evaluation task is terminated.

[0118] In some embodiments, the processing module 220 is specifically configured to:

[0119] For each of the subtasks, obtaining a resource request corresponding to the subtask;

[0120] When the resource request is passed, initializing the execution environment of the subtask and determining the task identifier corresponding to the subtask;

[0121] According to the task identifier, obtaining the evaluation type and evaluation data of the subtask;

[0122] Processing the evaluation data using the model to be evaluated to obtain a processing result;

[0123] Based on the evaluation type, the processing results are aggregated to obtain an evaluation result of the model to be evaluated under the evaluation type.

[0124] In some embodiments, the calculation module 240 is specifically used to:

[0125] Obtaining the number of subtasks whose task status is completed to obtain a first number;

[0126] Obtaining the total number of the subtasks in the evaluation task to obtain a second number;

[0127] The ratio of the first number to the second number is calculated to obtain the execution progress of the evaluation task.

[0128] In some embodiments, the task identifier of the backup task is obtained by converting the task identifier of the candidate task, and the ending module 260 is specifically used to:

[0129] Obtaining the task identifier corresponding to the candidate task to obtain a to-be-completed identifier group;

[0130] Detecting the task status of the candidate task and the backup task;

[0131] If a target task is detected, the task identifier of the target task is used as the target identifier, and the target task is a candidate task or a backup task whose task status is a completed status;

[0132] Deleting the target identifier from the to-be-completed identifier group;

[0133] If the to-be-completed identification group is empty, the candidate task or the backup task being executed is closed.

[0134] In some embodiments, the ending module 260 is specifically used to:

[0135] Searching for the target identifier in the to-be-completed identifier group;

[0136] If the target identifier is found in the to-be-completed identifier group, the target identifier is deleted from the to-be-completed identifier group;

[0137] If the target identifier is not found in the to-be-completed identifier group, converting the target identifier to obtain a conversion identifier;

[0138] The conversion flag is deleted from the to-be-completed flag group.

[0139] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined and implemented as the same or several entities. The specific implementation of the above modules can be found in the previous method embodiments, which will not be repeated here.

[0140] From the above, it can be seen that the model evaluation task processing device of this embodiment can split the model evaluation task into multiple subtasks, each of which may include multiple sub-steps. The subtasks are stream-processed through multiple sub-steps to avoid excessive cluster load, and a complete fault-tolerant mechanism is introduced during execution to increase the probability of successful execution of subtasks. When the execution progress of the entire model evaluation task reaches the preset progress, the subtask being executed is backed up. Even if the subtask being executed is slow, the backup task is still executed, which increases the possibility of completing the subtask, thereby effectively shortening the execution time of the model evaluation task.

[0141] The embodiment of the present application also provides an electronic device, which can be a terminal, a server, etc. The terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a laptop, a personal computer, etc. The server can be a single server or a server cluster composed of multiple servers, etc.

[0142] In some embodiments, the model evaluation task processing device can also be integrated into multiple electronic devices. For example, the model evaluation task processing device can be integrated into multiple servers, and the model evaluation task processing method of the present application can be implemented by multiple servers.

[0143] In this embodiment, the electronic device of this embodiment is a server as an example for detailed description, for example, Figure 6 As shown, it shows a schematic diagram of the structure of the electronic device involved in the embodiment of the present application, specifically:

[0144] The electronic device may include components such as a processor 310 with one or more processing cores, a memory 320 with one or more computer-readable storage media, a power supply 330, an input module 340, and a communication module 350. Those skilled in the art will appreciate that Figure 6The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0145] The processor 310 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 320, and calling data stored in the memory 320. In some embodiments, the processor 310 may include one or more processing cores; in some embodiments, the processor 310 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 310.

[0146] The memory 320 can be used to store software programs and modules. The processor 310 executes various functional applications and data processing by running the software programs and modules stored in the memory 320. The memory 320 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 320 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 320 may also include a memory controller to provide the processor 310 with access to the memory 320.

[0147] The electronic device also includes a power supply 330 for supplying power to various components. In some embodiments, the power supply 330 can be logically connected to the processor 310 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 330 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0148] The electronic device may further include an input module 340, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0149] The electronic device may further include a communication module 350. In some embodiments, the communication module 350 may include a wireless module. The electronic device may perform short-range wireless transmission through the wireless module of the communication module 350, thereby providing the user with wireless broadband Internet access. For example, the communication module 350 may be used to help the user send and receive emails, browse web pages, and access streaming media.

[0150] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 310 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 320 according to the following instructions, and the processor 310 will run the application programs stored in the memory 320, thereby implementing the steps in the methods of the embodiments of the present application.

[0151] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0152] From the above, it can be seen that the embodiment of the present application can split the model evaluation task into multiple subtasks, each of which may include multiple sub-steps. The subtasks are stream-processed through multiple sub-steps to avoid excessive cluster load, and a complete fault-tolerant mechanism is introduced during execution to increase the probability of successful execution of the subtasks. When the execution progress of the entire model evaluation task reaches the preset progress, the subtask being executed is backed up. Even if the subtask being executed is slow, the backup task is still executing, which increases the possibility of completing the subtask, thereby effectively shortening the execution time of the model evaluation task.

[0153] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0154] To this end, an embodiment of the present application provides a computer-readable storage medium, in which multiple instructions are stored, and the instructions can be loaded by a processor to execute the steps in any model evaluation task processing method provided in the embodiment of the present application.

[0155] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0156] According to one aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program including a computer program / instruction, the computer program / instruction being stored in a computer-readable storage medium. A processor of an electronic device reads the computer program / instruction from the computer-readable storage medium, and the processor executes the computer program / instruction, so that the electronic device executes the method provided in various optional implementations of the model evaluation task processing provided in the above embodiments.

[0157] Since the instructions stored in the storage medium can execute the steps in any model evaluation task processing method provided in the embodiments of the present application, the beneficial effects that can be achieved by any model evaluation task processing method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0158] The above is a detailed introduction to a model evaluation task processing method and device provided in an embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for processing a model evaluation task, characterized in that: The method comprises: Obtaining an evaluation task corresponding to the model to be evaluated, wherein the evaluation task includes multiple subtasks, and the subtask includes multiple substeps arranged in an execution order; Based on the multiple sub-steps included in each of the sub-tasks, stream processing is performed on the multiple sub-tasks; In the process of processing the subtask, based on the execution data corresponding to each substep, determining the task status of the subtask, the task status including a completed state and an uncompleted state; Calculating the execution progress of the evaluation task based on the number of subtasks whose task status is completed; If the execution progress of the evaluation task reaches the preset progress, a backup task identical to the candidate task is established and executed, and the candidate task is a subtask whose task status is in an unfinished state; If the task status of all subtasks in the evaluation task is completed, terminate the related tasks of the evaluation task, wherein the related tasks include the candidate task and the backup task; The execution data includes execution time and process data. In the process of processing the subtask, determining the task status of the subtask based on the execution data corresponding to each substep includes: For each of the sub-steps corresponding to each of the sub-tasks, in response to the start execution information of the sub-step, the process data and execution time of the sub-step are obtained; the process data is analyzed for exceptions to determine the exception type of the sub-step, and the exception type includes no exception, system exception and business exception; if the execution time is not greater than the preset time, and the exception type is no exception, the exception level of the sub-step is determined to be a first-level exception; if the execution time is greater than the preset time, or the exception type is a system exception, the exception level of the sub-step is determined to be a second-level exception; if the exception type is a business exception, the exception level of the sub-step is determined to be a third-level exception; based on the exception level and the end execution information of the sub-step, the step status corresponding to the sub-step is determined, and the step status includes a success status and a running status; if the step status corresponding to all the sub-steps in the sub-task are all success status, the task status of the sub-task is determined to be a completed status; if the step status corresponding to any sub-step in the sub-task is a running status, the task status of the sub-task is determined to be an unfinished status.

2. The method according to claim 1, characterized in that The step status also includes an abnormal state, and determining the step status corresponding to the sub-step based on the abnormal level and the end execution information of the sub-step includes: If the exception level is a level one exception and the end execution information of the sub-step is obtained, it is determined that the step status of the sub-step is a success status; If the abnormal level is a level one abnormality and the end execution information of the sub-step is not obtained, determining that the step state of the sub-step is a running state; If the abnormality level is a level 2 abnormality, retry the designated task, and determine the step status of the sub-step based on the result of the retry process, and the designated task is the sub-task corresponding to the sub-step; If the abnormality level is level three, the step status of the sub-step is determined to be an abnormal state, and the evaluation task is terminated.

3. The method according to claim 2, characterized in that The retrying of the designated task and determining the step status of the sub-step based on the result of the retrying process include: Obtain the number of retries corresponding to the specified task and update it to obtain a new number of retries; If the new retry number is not greater than the preset number, creating a retry task that is the same as the specified task; Execute the retry task to determine a step status corresponding to a retry sub-step in the retry task, the retry sub-step being the same as the sub-step; Using the step status corresponding to the retry sub-step as the step status of the sub-step; If the new number of retries is greater than the preset number of times, the step status of the sub-step is determined to be an abnormal state, and the evaluation task is terminated.

4. The method according to claim 1, characterized in that: The step of performing stream processing on the multiple subtasks based on the multiple substeps included in each of the subtasks includes: For each of the subtasks, obtaining a resource request corresponding to the subtask; When the resource request is passed, initializing the execution environment of the subtask and determining the task identifier corresponding to the subtask; According to the task identifier, obtaining the evaluation type and evaluation data of the subtask; Processing the evaluation data using the model to be evaluated to obtain a processing result; Based on the evaluation type, the processing results are aggregated to obtain an evaluation result of the model to be evaluated under the evaluation type.

5. The method according to claim 1, characterized in that The calculating the execution progress of the evaluation task based on the number of subtasks whose task status is completed includes: Obtaining the number of subtasks whose task status is completed to obtain a first number; Obtaining the total number of the subtasks in the evaluation task to obtain a second number; The ratio of the first number to the second number is calculated to obtain the execution progress of the evaluation task.

6. The method according to claim 1, characterized in that The task identifier of the backup task is obtained by converting the task identifier of the candidate task, and if the task statuses of multiple subtasks in the evaluation task are all in the completed status, ending the related tasks of the evaluation task includes: Obtaining the task identifier corresponding to the candidate task to obtain a to-be-completed identifier group; Detecting the task status of the candidate task and the backup task; If a target task is detected, the task identifier of the target task is used as the target identifier, and the target task is a candidate task or a backup task whose task status is a completed status; Deleting the target identifier from the to-be-completed identifier group; If the to-be-completed identification group is empty, the candidate task or the backup task being executed is closed.

7. The method according to claim 6, characterized in that The step of deleting the target identifier from the to-be-completed identifier group includes: Searching for the target identifier in the to-be-completed identifier group; If the target identifier is found in the to-be-completed identifier group, the target identifier is deleted from the to-be-completed identifier group; If the target identifier is not found in the to-be-completed identifier group, converting the target identifier to obtain a conversion identifier; The conversion flag is deleted from the to-be-completed flag group.

8. A model evaluation task processing device, characterized in that: The device comprises: An acquisition module, used to acquire an evaluation task corresponding to the model to be evaluated, wherein the evaluation task includes multiple subtasks, and the subtask includes multiple substeps arranged in an execution order; A processing module, configured to perform stream processing on the multiple subtasks based on the multiple substeps included in each of the subtasks; A status module, used to determine the task status of the subtask based on the execution data corresponding to each substep during the processing of the subtask, wherein the task status includes a completed status and an uncompleted status; A calculation module, used for calculating the execution progress of the evaluation task based on the number of subtasks whose task status is completed; A backup module, configured to create and execute a backup task that is the same as a candidate task if the execution progress of the evaluation task reaches a preset progress, wherein the candidate task is a subtask whose task status is an unfinished state; An ending module, configured to end the related tasks of the evaluation task if the task status of all subtasks in the evaluation task are in the completed status, wherein the related tasks include the candidate task and the backup task; The execution data includes execution time and process data. In the process of processing the subtask, determining the task status of the subtask based on the execution data corresponding to each substep includes: For each of the sub-steps corresponding to each of the sub-tasks, in response to the start execution information of the sub-step, the process data and execution time of the sub-step are obtained; the process data is analyzed for exceptions to determine the exception type of the sub-step, and the exception type includes no exception, system exception and business exception; if the execution time is not greater than the preset time, and the exception type is no exception, the exception level of the sub-step is determined to be a first-level exception; if the execution time is greater than the preset time, or the exception type is a system exception, the exception level of the sub-step is determined to be a second-level exception; if the exception type is a business exception, the exception level of the sub-step is determined to be a third-level exception; based on the exception level and the end execution information of the sub-step, the step status corresponding to the sub-step is determined, and the step status includes a success status and a running status; if the step status corresponding to all the sub-steps in the sub-task are all success status, the task status of the sub-task is determined to be a completed status; if the step status corresponding to any sub-step in the sub-task is a running status, the task status of the sub-task is determined to be an unfinished status.

Images