Resource scheduling methods, apparatus, and computer equipment for model evaluation tasks

Abstract

This disclosure provides a resource scheduling method for model evaluation tasks, relating to the field of computer technology. The implementation scheme is as follows: acquiring risk data of the target model in the scenario to be evaluated, whereby the risk data indicates the degree of risk in the scenario to be evaluated; inputting the risk data into a preset model for risk classification to obtain the risk level corresponding to the scenario to be evaluated; matching the corresponding test strategy template from a preset strategy template library including one or more test strategy templates based on the risk level; constructing an evaluation dataset adapted to the risk level according to the dataset coverage rules in the test strategy template; and allocating computing resources to the model evaluation task according to the resource scheduling strategy in the test strategy template. This method can quantify risk and thus solve the problem of risk and resource mismatch in model evaluation tasks.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, particularly to the fields of algorithm model evaluation and task scheduling, and especially to resource scheduling methods, apparatus, computer equipment, computer-readable storage media, and computer program products for model evaluation tasks. Background Technology

[0002] Existing algorithm evaluation / continuous integration (CI) testing / computing scheduling platforms typically: On the evaluation side, they focus on key metrics such as average precision, recall, F1 score, and area under the curve (AUC), with fixed dataset partitioning and sampling, and multiple metrics only used for supplementary reports; On the scheduling side, they target utilization, throughput, queuing time, and cost, using static priority or first-in-first-out (FIFO) / round-robin, with resource allocation weakly tied to scenario risks; On the verification side, they mostly use key metric thresholds for blocking, and high-risk scenarios mainly rely on manual experience for additional testing, lacking calculable and verifiable strategies. Summary of the Invention

[0003] This disclosure provides a resource scheduling method, apparatus, computer device, computer-readable storage medium, and computer program product for model evaluation tasks.

[0004] According to one aspect of this disclosure, a resource scheduling method for a model evaluation task is provided. The model evaluation task includes a task for evaluating a target model in an evaluation scenario. The method includes: acquiring risk data of the target model in the evaluation scenario, wherein the risk data indicates the degree of risk in the evaluation scenario; inputting the risk data into a preset model for risk classification to obtain a risk level corresponding to the evaluation scenario, wherein the model for risk classification is constructed to output a risk level, a risk score, and a risk composition explanation based on the given risk data; and matching a corresponding test strategy template from a preset strategy template library including one or more test strategy templates based on the risk level. Each test strategy template includes a multi-dimensional indicator set associated with the risk level, dataset coverage rules, and resource scheduling strategies. Each indicator in the multi-dimensional indicator set has a corresponding threshold. Based on the dataset coverage rules in the test strategy template, an assessment dataset adapted to the risk level is constructed. Constructing the assessment dataset includes: dynamically adjusting the sample ratio and sample size of each data slice in the assessment dataset under the scenario to be assessed according to the risk weights associated with the risk level; and allocating computing resources to the model assessment task according to the resource scheduling strategies in the test strategy template. Allocating computing resources to the model assessment task includes: determining the resource allocation share of the model assessment task based on the risk weights.

[0005] In some embodiments, the method further includes: performing a model evaluation task on the target model using a constructed evaluation dataset on allocated computing resources to obtain multi-dimensional indicator evaluation results; verifying the multi-dimensional indicator evaluation results according to the threshold corresponding to each indicator in the multi-dimensional indicator set in the test strategy template to obtain verification results; and determining whether to increase testing on the target model based on the verification results.

[0006] In some embodiments, the test strategy template also includes statistical confidence requirements associated with the risk level and the number and order of verification checkpoints. Verifying the multi-dimensional indicator evaluation results to obtain verification results further includes: verifying whether the multi-dimensional indicator evaluation results meet the statistical confidence requirements; and verifying the multi-dimensional indicator evaluation results according to the number and order of verification checkpoints to obtain verification results.

[0007] In some embodiments, the dataset coverage rule is used to constrain the list of data slice dimensions to be covered for the scenario to be evaluated, as well as the minimum sample size and minimum coverage rate of each data slice. The dynamic adjustment of the sample ratio and sample size of each data slice further includes: under the premise of meeting the minimum sample size and minimum coverage rate, making the sample ratio positively correlated with the risk weight, so as to increase the sample ratio of the data slice corresponding to the target risk level whose risk level is higher than the preset target threshold.

[0008] In some embodiments, allocating computing resources for the model evaluation task further includes: allocating dedicated computing resources or stable node resources to the model evaluation task in response to a risk level higher than a first preset threshold; and allocating preemptible resources or elastic nodes to the model evaluation task in response to a risk level lower than a second preset threshold.

[0009] In some embodiments, the method further includes: feeding back historical data from the model evaluation task to a library of models and strategy templates for risk grading, in order to update the parameters of the models used for risk grading or optimize the configuration of the test strategy templates.

[0010] In some embodiments, risk data includes one or more of the following: scenario impact scope data, misjudgment cost data, compliance and regulatory intensity data, scenario sensitivity data, and historical incident frequency and severity data.

[0011] In some embodiments, the multi-dimensional metric set includes one or more of the following metrics: accuracy, recall, false positive rate, false negative rate, fairness metric, robustness metric, latency metric, and resource consumption metric.

[0012] According to another aspect of this disclosure, a resource scheduling apparatus for a model evaluation task is provided. The model evaluation task includes a task for evaluating a target model in a scenario to be evaluated. The apparatus includes: an acquisition module configured to acquire risk data of the target model in the scenario to be evaluated, wherein the risk data indicates the degree of risk of the scenario to be evaluated; a risk grading module configured to input the risk data into a preset model for risk grading to obtain a risk level corresponding to the scenario to be evaluated, wherein the model for risk grading is constructed to output a risk level, a risk score, and a risk composition explanation item based on the given risk data; and a strategy matching module configured to match a corresponding test strategy template from a preset strategy template library including one or more test strategy templates based on the risk level. The system comprises a test strategy template, each including a multi-dimensional indicator set associated with risk level, dataset coverage rules, and resource scheduling strategy. Each indicator in the multi-dimensional indicator set has a corresponding threshold. A dataset construction module is configured to construct an assessment dataset adapted to the risk level based on the dataset coverage rules in the test strategy template. Constructing the assessment dataset includes dynamically adjusting the sample ratio and sample size of each data slice in the assessment dataset for the scenario to be assessed based on the risk weights associated with the risk level. A resource allocation module is configured to allocate computing resources to the model assessment task based on the resource scheduling strategy in the test strategy template. Allocating computing resources to the model assessment task includes determining the resource allocation share for the model assessment task based on the risk weights.

[0013] According to another aspect of this disclosure, a computer device is provided, comprising: at least one processor; and a memory having a computer program stored thereon, wherein the computer program, when executed by the at least one processor, causes the at least one processor to perform the methods provided above in this disclosure.

[0014] According to another aspect of this disclosure, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, causes the processor to perform the methods provided above in this disclosure.

[0015] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, causes the processor to perform the methods provided above in this disclosure.

[0016] According to one or more embodiments of this disclosure, the risk level of a scenario is bound to the test strategy, dataset construction, and resource scheduling, thereby achieving adaptive matching of the test coverage and resource allocation in high-risk scenarios. This fundamentally solves the technical problems of risk and resource mismatch and unreproducible fluctuations in evaluation results in model evaluation.

[0017] These and other aspects of this disclosure will be apparent from the embodiments described below, and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0018] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of this disclosure. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0019] Figure 1 This is a flowchart illustrating a resource scheduling method for a model evaluation task according to an exemplary embodiment.

[0020] Figure 2 This is a flowchart illustrating a portion of the process of a resource scheduling method for a model evaluation task according to an exemplary embodiment.

[0021] Figure 3 This is a flowchart illustrating a portion of the process of a resource scheduling method for a model evaluation task according to an exemplary embodiment.

[0022] Figure 4 This is a flowchart illustrating a portion of the process of a resource scheduling method for a model evaluation task according to an exemplary embodiment.

[0023] Figure 5 This is a block diagram illustrating a resource scheduling apparatus for a model evaluation task according to an exemplary embodiment.

[0024] Figure 6 An example computer device is shown in which any of the embodiments described herein may be implemented. Detailed Implementation

[0025] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0026] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0027] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. As used herein, the term "multiple" means two or more, and the term "based on" should be interpreted as "at least partially based on". Furthermore, the terms "and / or" and "at least one of..." cover any one of the listed items and all possible combinations thereof.

[0028] The current existing technologies and algorithms for evaluation mainly suffer from the following systemic defects: (1) Lack of risk modeling: The scenario risk is not taken as the core input variable. The same evaluation standard is used for scenarios with different consequence levels, resulting in insufficient constraints for high-risk scenarios and waste of resources for low-risk scenarios; (2) Rigid indicator thresholds: A unified set of indicators and fixed thresholds are used. The key indicators such as false alarm rate, fairness, and confidence required for high-risk scenarios are missing. The "average value meets the standard" problem masks the failure of key slices; (3) Data set coverage blind spots: Fixed sampling and fixed division are used. The sample ratio is not adjusted with the risk level, and the sample size of key slices is insufficient. 1. Low statistical confidence, which can easily lead to “overall good results but local collapse”; (4) Decoupling of resource scheduling and risk: With utilization and throughput as optimization targets, FIFO or static priority is adopted. High-risk tasks cannot obtain stable resources and redundancy guarantees, and the evaluation results fluctuate greatly and regression is difficult to locate; (5) Verification depends on human experience: There is a lack of calculable and auditable risk-driven strategies. High-risk scenarios rely on manual testing, which is highly subjective and easy to miss; (6) Modules are fragmented and there is no closed loop: Evaluation, dataset, scheduling and verification are independent of each other. There is a lack of “risk-coverage-resource” linkage mechanism, which makes it impossible to form a continuous improvement iterative optimization capability.

[0029] The embodiments of this disclosure provide a resource scheduling method for model evaluation tasks, which binds the risk level of a scenario with test strategies, dataset construction, and resource scheduling, and achieves adaptive matching of test coverage and resource allocation in high-risk scenarios. This fundamentally solves the technical problems of risk and resource mismatch and unreproducible fluctuations in evaluation results in model evaluation.

[0030] Figure 1 This is a flowchart illustrating a resource scheduling method 100 for a model evaluation task according to an exemplary embodiment.

[0031] like Figure 1As shown, this disclosure proposes a resource scheduling method 100 for a model evaluation task. The model evaluation task includes evaluating a target model in a scenario to be evaluated. The method 100 may include the following steps: S102, obtaining risk data of the target model in the scenario to be evaluated, wherein the risk data indicates the degree of risk in the scenario to be evaluated; S104, inputting the risk data into a preset model for risk classification to obtain the risk level corresponding to the scenario to be evaluated, wherein the model for risk classification is constructed to output a risk level, a risk score, and a risk composition explanation based on the given risk data; S106, matching the corresponding test strategy from a preset strategy template library including one or more test strategy templates based on the risk level. The test strategy template includes a multi-dimensional indicator set associated with the risk level, a dataset coverage rule, and a resource scheduling strategy. Each indicator in the multi-dimensional indicator set has a corresponding threshold. S108: Based on the dataset coverage rule in the test strategy template, construct an assessment dataset adapted to the risk level. Constructing the assessment dataset includes dynamically adjusting the sample ratio and sample size of each data slice in the assessment dataset under the scenario to be assessed based on the risk weight associated with the risk level. S110: Based on the resource scheduling strategy in the test strategy template, allocate computational resources to the model assessment task. Allocating computational resources to the model assessment task includes determining the resource allocation share of the model assessment task based on the risk weight.

[0032] In step S102, risk data of the target model in the scenario to be evaluated is obtained. The risk data indicates the degree of risk of the scenario to be evaluated.

[0033] In the example, the scenario to be evaluated can be a scenario in which the model is applied, such as financial risk control, facial recognition authentication, content moderation, medical diagnosis assistance, autonomous driving perception, content preference tag prediction, internal testing environment, etc. The target model can be a financial risk control model, a facial recognition model, an autonomous driving model, and a user preference recommendation model, etc., without any restrictions.

[0034] In the examples, different scenarios under evaluation have varying degrees of risk. For instance, in a medical auxiliary diagnosis scenario, missed diagnoses lead to delayed treatment, while misdiagnoses result in incorrect medication and medical accidents. Similarly, in a facial recognition authentication scenario, misidentification can lead to identity theft or prevent legitimate users from accessing the system, thus impairing the user experience. Therefore, these scenarios are considered high-risk. Conversely, in a content preference tag prediction scenario, even if incorrect tags cause slight deviations in recommendations, there is no substantial impact; these scenarios are considered low-risk.

[0035] In the example, risk data can be the basic input information used to calculate and determine the risk level of the scenario to be assessed.

[0036] In step S104, the risk data is input into a preset model for risk classification to obtain the risk level corresponding to the scenario to be evaluated. The model for risk classification is constructed to output the risk level, risk score and risk composition explanation based on the given risk data.

[0037] In the example, the risk level can be divided into five levels, R1-R5, where R1 is low risk, R2-R3 are medium risk, and R4-R5 are high risk. The risk score can be a score representing the degree of risk within a preset score range (e.g., 1-100). The risk composition explanation term can be used to describe why the model used for risk classification gives such a risk level and risk score.

[0038] Taking the autonomous driving pedestrian perception model as an example, risk data from autonomous driving scenarios, such as the user impact range (sales of the vehicle model are 100,000 units, and the daily mileage is 5 million kilometers), misjudgment cost (a single missed pedestrian detection can lead to injury or death, with an estimated compensation standard of 2 million yuan), regulatory constraints (involving ISO 26262 functional safety certification and national standard GB / T41798-2022), historical accidents (two pedestrian recognition delay records occurred during the testing phase), and scenario sensitivity (involving life safety, children, the elderly, and other vulnerable road users), are input into a pre-set risk classification model. The model can output: risk level R5 (highest risk), risk score of 96 points, and risk composition explanation items as "life safety weight 50%, regulatory compliance weight 25%, user exposure range 15%, historical defect weight 10%, the core reason for judging it as a high-risk scenario is 'missed pedestrian detection directly endangers life'". This quantitative and traceable risk decision-making link provides a clear quantitative basis for subsequent allocation of dedicated computing power resources and data slicing tests of datasets under extreme weather conditions.

[0039] This transforms risk assessments that previously relied on human experience into quantifiable risk levels, risk scores, and risk composition explanations, thus providing data support for decision-making traceability.

[0040] In step S106, based on the risk level, a corresponding test strategy template is matched from a preset strategy template library that includes one or more test strategy templates. Each test strategy template includes a multi-dimensional indicator set associated with the risk level, a dataset coverage rule, and a resource scheduling strategy. Each indicator in the multi-dimensional indicator set has a corresponding threshold.

[0041] Continuing with the example of the autonomous driving pedestrian perception model, based on the autonomous driving pedestrian perception model being rated R5 (the highest risk level), the corresponding R5-autonomous driving high-risk template is matched from the preset strategy template library. The multi-dimensional indicator set in this R5-autonomous driving high-risk template may include: recall rate (pedestrian detection rate ≥ 99.9%), false alarm rate (false alarm ≤ 0.01%), response latency (≤ 50ms), fairness indicator (recall rate ≥ 99.5% for sub-scenes such as children / elderly / night / rainy days), and robustness indicator (recall rate in adversarial occlusion scenarios). ≥98%); the dataset coverage rules can be that it must cover 12 lighting conditions such as sunny / rainy / snowy / night / backlight, 8 pedestrian forms such as children / adults / elderly / groups / non-standard postures, and 6 road scenes such as urban / highway / rural / tunnel, and the minimum sample size of each subdivided dataset slice is not less than 5000 frames; the resource scheduling strategy can be to allocate a dedicated GPU cluster (non-preemptive resources), enable dual-run consistency verification (two independent environments running synchronously), set the highest priority queue, and reserve 20% redundant computing power to prepare for retesting.

[0042] Taking the daily regression test of the user preference tag prediction model as an example, this scenario is rated as the lowest risk level of R1 (because it is only used for internal interest tagging, does not affect the core recommendation ranking, and misjudgments do not cause substantial harm). A corresponding R1-low-risk lightweight template can be matched from the preset strategy template library. The multi-dimensional indicator set in this R1-low-risk lightweight template can be: accuracy ≥85%, recall ≥80%, training latency ≤30 minutes, without needing to focus on fairness or robustness indicators; the dataset coverage rule can be to cover only four coarse-grained content categories: "movies / sports / games / food," with each slice containing at least 1000 samples, without needing to be subdivided into specific tags or demographic attributes, and allowing the use of public datasets to replace real-world scenario data; the resource scheduling strategy can be to allocate preemptible elastic nodes (elastic nodes can be reclaimed when resources are scarce), no need for repeated verification per run, use of the lowest priority queue, and no reserved redundant resources, and allow execution during off-peak hours at night. This lightweight configuration minimizes the resource consumption of low-risk tasks, reserving computing power for autonomous driving waiting evaluation scenarios with a risk level of R5.

[0043] Therefore, the statistical reliability of key scenarios can be improved by refining the slices of risk-driven datasets and increasing the sample ratio of key slices.

[0044] In step S108, an assessment dataset adapted to the risk level is constructed according to the dataset coverage rules in the test strategy template. The construction of the assessment dataset includes: dynamically adjusting the sample ratio and sample size of each data slice of the assessment dataset under the scenario to be assessed according to the risk weight associated with the risk level.

[0045] In some examples, taking an autonomous driving pedestrian perception model with a risk level of R5 as an example, its risk weight can be set to 1.0 (the highest weight). Based on its dataset coverage rules, the following steps can be taken when constructing the evaluation dataset: Divide the dataset into slices according to the risk weight (e.g., 1.0), covering 12 lighting conditions such as "sunny / rainy / snowy / night / backlight," 8 pedestrian morphologies such as "children / elderly / non-standard postures," and 6 road scenarios such as "city / highway / tunnel," totaling 576 sub-slices; increase the proportion of high-risk slices in the dataset, for example, increasing the sample size of high-risk slices (such as "night + children + rainy") to three times the mean, ensuring that each high-risk slice has no less than 5000 frames, and controlling the total sample size to 2 million frames, with high-risk slices accounting for 60% of the total sample, to ensure statistical confidence under extreme weather conditions.

[0046] In some examples, for the R1 user preference label prediction model, the risk weight is set to 0.1 (minimum weight). When constructing the evaluation dataset according to the dataset coverage rules, the following operations are performed: Based on a risk weight of 0.1, the dataset is sliced, covering only four coarse-grained content categories: "Film / Sports / Games / Food," without further subdivision of user attributes or scene conditions, resulting in four slices. Samples can be evenly distributed across the four coarse-grained content categories; for example, samples are evenly distributed across each slice without bias towards any slice, only requiring a minimum sample size of 1000 samples per category, for a total sample size of only 5000 samples. Furthermore, the use of public datasets to replace real-world scene data is permitted, and the sample size for high-risk slices can be set to 0.

[0047] In step S110, computing resources are allocated to the model evaluation task according to the resource scheduling strategy in the test strategy template. The allocation of computing resources to the model evaluation task includes: determining the resource allocation share of the model evaluation task based on the risk weight.

[0048] In the example, risk weights directly determine resource allocation. For instance, with a total of 66 GPUs of idle computing resources, an autonomous driving pedestrian perception model with a risk weight of 1.0 can be allocated 60 GPUs, while a user preference label prediction model with a risk weight of 0.1 can obtain 6 GPUs. This allocation mechanism, where "higher risk, more resources," ensures that high-risk models receive sufficient and stable computing power to meet their testing needs, while low-risk assessment tasks consume only the minimum necessary resources, achieving optimal risk allocation for limited computing power.

[0049] This achieves an adaptive match between the test coverage of high-risk scenarios, the intensity of resource investment, and the rigor of verification, fundamentally solving the technical problems of risk and resource mismatch, missed testing of key slices, and unreproducible fluctuations in evaluation results in model evaluation.

[0050] Figure 2 This is a flowchart illustrating a portion of the process of a resource scheduling method 100 for a model evaluation task according to an exemplary embodiment.

[0051] In some embodiments, method 100 may further include: S112, performing a model evaluation task on the target model using the constructed evaluation dataset on the allocated computing resources to obtain multi-dimensional indicator evaluation results; S114, verifying the multi-dimensional indicator evaluation results according to the threshold corresponding to each indicator in the multi-dimensional indicator set in the test strategy template to obtain a verification result; and S116, determining whether to increase the testing of the target model based on the verification result.

[0052] In step S112, on the allocated computing resources, the constructed evaluation dataset is used to perform a model evaluation task on the target model to obtain multi-dimensional index evaluation results.

[0053] In some examples, on 60 allocated GPUs, a dual-run parallel evaluation task was performed on an autonomous driving pedestrian perception model with risk level R5 using an evaluation dataset containing 576 fine-grained slices and 2 million frames of images. The final output was multi-dimensional evaluation results: overall pedestrian recall rate of 99.92% (threshold ≥ 99.9%), false alarm rate of 0.008% (threshold ≤ 0.01%), and latency of 48ms (threshold ≤ 50ms).

[0054] In step S114, the evaluation results of the multi-dimensional indicators are verified according to the threshold corresponding to each indicator in the multi-dimensional indicator set in the test strategy template to obtain the verification results.

[0055] In some examples, the multi-dimensional evaluation results of the autonomous driving pedestrian perception model for risk level R5 showed that the recall rate on the high-risk slice of "backlight + tunnel entrance" was only 97.85%, which did not meet the mandatory threshold requirement of 98.0%, and the verification result was unqualified.

[0056] In step S114, it is determined whether to add testing to the target model based on the verification results.

[0057] In some examples, additional tests are added to the target model for cases where the verification result is unqualified (i.e., the threshold requirement of the corresponding indicator is not met).

[0058] In other examples, if the validation result is unsatisfactory, the model release can be stopped directly or the validation can be switched to manual verification.

[0059] Figure 3 This is a flowchart illustrating a portion of the process of a resource scheduling method 100 for a model evaluation task according to an exemplary embodiment.

[0060] In some embodiments, the test strategy template may also include statistical confidence requirements associated with risk levels and the number and order of verification checkpoints. Step S114 may further include: S302, verifying whether the evaluation results of multi-dimensional indicators meet the statistical confidence requirements; and S304, verifying the evaluation results of multi-dimensional indicators according to the number and order of verification checkpoints to obtain verification results.

[0061] In step S302, verify whether the evaluation results of the multi-dimensional indicators meet the statistical confidence requirements.

[0062] In the example, after the evaluation of the autonomous driving pedestrian perception model with a risk level of R5 is completed, the evaluation results of the multi-dimensional indicators are checked to see if they meet the statistical confidence requirements. For example, the statistical confidence of the recall rate of the "backlight + tunnel entrance" slice is checked in the multi-dimensional indicator evaluation results: the sample size of this slice is 5230 frames, and the system calculates its 95% confidence interval as [97.32%, 98.38%]. Since the lower limit of the confidence interval of 97.32% is lower than the mandatory threshold of 98.0%, and the significance level p=0.03<0.05, it can be determined that the indicator does not meet the statistical confidence requirements.

[0063] In step S302, the evaluation results of the multi-dimensional indicators are verified according to the number and order of the verification checkpoints to obtain the verification results.

[0064] In the example, taking an autonomous driving pedestrian perception model with a risk level of R5 as an example, the verification can be performed step by step according to the number and order of verification checkpoints bound in the R5-autonomous driving high-risk template to obtain the verification results. The verification checkpoints bound in the high-risk template can include offline unit testing, offline regression comparison testing, grayscale simulation testing, adversarial testing, etc. If the test in the verification is passed, the verification result is qualified; if the test in the verification is failed, the verification result is unqualified.

[0065] In some embodiments, the dataset coverage rule is used to constrain the list of data slice dimensions to be covered for the scenario to be evaluated, as well as the minimum sample size and minimum coverage rate of each data slice. The dynamic adjustment of the sample ratio and sample size of each data slice further includes: under the premise of meeting the minimum sample size and minimum coverage rate, making the sample ratio positively correlated with the risk weight, so as to increase the sample ratio of the data slice corresponding to the target risk level whose risk level is higher than the preset target threshold.

[0066] In the example, based on the mandatory 120 data slice dimensions (6 lighting conditions × 5 pedestrian morphologies × 4 road scenes) and the hard constraints of a minimum sample size of 5,000 frames and a minimum coverage of 100% for each slice in the dataset coverage rules, a dynamic allocation of sample ratios positively correlated with risk weights can be performed within a budget of 2 million frames. The sample size of high-risk slices with a risk weight ≥ 0.8 (such as "night + children + tunnel entrance", weight 1.0) is increased to 25,000 frames (while low-risk slices such as "sunny day + adults + city" have only 7,500 frames). This allows high-risk slices to obtain a 3.5-fold sample skew while meeting the minimum threshold, ultimately achieving a situation where high-risk slices (accounting for 25% of the total number) occupy 62% of the total sample size, ensuring that the statistical confidence of high-risk scenes is significantly higher than that of regular scenes.

[0067] In the example, redundant checks can also be performed on assessment tasks in high-risk scenarios to reduce misjudgments caused by fluctuations in risk data.

[0068] Figure 4 This is a flowchart illustrating a portion of the process of a resource scheduling method 100 for a model evaluation task according to an exemplary embodiment.

[0069] In some embodiments, step S110 may further include: S402, in response to a risk level higher than a first preset threshold, allocating dedicated computing resources or stable node resources to the model evaluation task; and S404, in response to a risk level lower than a second preset threshold, allocating preemptible resources or elastic nodes to the model evaluation task.

[0070] In step S402, in response to the risk level being higher than the first preset threshold, dedicated computing resources or stable node resources are allocated to the model evaluation task.

[0071] In the example, in the autonomous driving pedestrian perception model evaluation task with a risk level of R5, in response to its risk level (R5) being higher than a first preset threshold (e.g., R4), a dedicated A100 computing cluster is automatically isolated and allocated from the shared resource pool to ensure exclusive access to stable node resources throughout the evaluation process. Stable node resources refer to computing resources that are available for a long time, have predictable performance, and are not affected by other tasks. They are usually physical servers or dedicated cloud instances to ensure the consistency and reliability of the evaluation environment, which is suitable for the stringent testing requirements of high-risk scenarios.

[0072] In step S404, in response to the risk level being lower than the second preset threshold, preemptible resources or elastic nodes are allocated to the model evaluation task.

[0073] In the example, during the daily regression test of a user preference recommendation model with a risk level of R1, in response to its risk level (R1) being lower than a second preset threshold (e.g., R3), it is automatically scheduled to a preemptible EC2 auction instance pool, allowing it to be interrupted and reclaimed by high-risk tasks when resources are scarce; at the same time, an elastic node strategy is adopted, which refers to computing resources that can be dynamically created, released, or scaled up or down according to the actual load. For example, the evaluation can be performed only during idle hours at night, and the resources can be released immediately after the run is completed, minimizing the testing cost.

[0074] In some embodiments, method 100 may further include: feeding back historical data from the model evaluation task to a library of models and strategy templates for risk grading, in order to update the parameters of the model used for risk grading or optimize the configuration of the test strategy templates.

[0075] In the example, after the autonomous driving pedestrian perception model with a risk level of R5 had been running online for 3 months, two complaints about delayed pedestrian perception at tunnel entrances at night and one online accident involving a child being misjudged and missed in rainy weather were collected. After feeding this historical data back to the model used for risk classification, the model can automatically adjust its parameters. For example, the risk weight of the "tunnel entrance" scenario can be increased from 0.8 to 1.0. At the same time, the policy template library can tighten the threshold of the "backlight + tunnel entrance" slice from 98.0% to 98.5% and increase the minimum sample size of the slice from 5,000 frames to 8,000 frames, making the risk classification of subsequent versions more accurate and the testing strategy more stringent, forming a closed-loop iteration capability, thereby reducing the probability of accidents in high-risk scenarios in the long term.

[0076] In some embodiments, risk data may include one or more of the following: scenario impact scope data, misjudgment cost data, compliance and regulatory intensity data, scenario sensitivity data, and historical incident frequency and severity data.

[0077] In some examples, scenario impact data can be the number of users of the target model in the scenario to be evaluated, the geographical scope involved, etc.; misjudgment cost data can be quantitative data of the loss caused by the target model misjudging in the scenario to be evaluated; compliance and regulatory intensity data can be data describing the intensity of compliance and regulation in the scenario to be evaluated (for example, data used to determine whether the scenario to be evaluated processes biometric information such as face / fingerprint / voiceprint or whether it involves non-compliant content); scenario sensitivity data can be data used to describe the scenario sensitivity of the scenario to be evaluated (for example, if the audience is mostly minors, the scenario sensitivity is high); and historical accident frequency and severity data can be data describing accidents that occurred within a historical period and their severity.

[0078] In some embodiments, the multi-dimensional metric set may include one or more of the following metrics: accuracy, recall, false positive rate, false negative rate, fairness metric, robustness metric, latency metric, and resource consumption metric. The multi-dimensional metric set may also include any other metrics used to evaluate the model, without limitation herein.

[0079] Figure 5 This is a block diagram illustrating a resource scheduling apparatus 500 for a model evaluation task according to an exemplary embodiment.

[0080] like Figure 5 As shown, in some embodiments, the model evaluation task includes a task for evaluating a target model in a scenario to be evaluated. The apparatus 500 may include: an acquisition module 510 configured to acquire risk data of the target model in the scenario to be evaluated, the risk data indicating the degree of risk in the scenario to be evaluated; a risk grading module 520 configured to input the risk data into a preset model for risk grading to obtain a risk level corresponding to the scenario to be evaluated, wherein the model for risk grading is constructed to output a risk level, a risk score, and a risk composition explanation based on the given risk data; and a strategy matching module 530 configured to match a corresponding test strategy template from a preset strategy template library including one or more test strategy templates based on the risk level. The test strategy template includes a multi-dimensional indicator set associated with risk levels, dataset coverage rules, and resource scheduling strategies. Each indicator in the multi-dimensional indicator set has a corresponding threshold. The dataset construction module 540 is configured to construct an assessment dataset adapted to the risk level according to the dataset coverage rules in the test strategy template. Constructing the assessment dataset includes dynamically adjusting the sample ratio and sample size of each data slice in the assessment dataset under the scenario to be assessed based on the risk weights associated with the risk level. The resource allocation module 550 is configured to allocate computing resources to the model assessment task according to the resource scheduling strategy in the test strategy template. Allocating computing resources to the model assessment task includes determining the resource allocation share of the model assessment task based on the risk weights.

[0081] In some embodiments, the apparatus 500 may further include: a task execution module configured to perform a model evaluation task on the target model using a constructed evaluation dataset on allocated computing resources to obtain multi-dimensional indicator evaluation results; a verification module configured to verify the multi-dimensional indicator evaluation results according to the threshold corresponding to each indicator in the multi-dimensional indicator set in the test strategy template to obtain verification results; and an additional testing module configured to determine whether to add testing to the target model based on the verification results.

[0082] In some embodiments, the test strategy template may further include statistical confidence requirements associated with risk levels and the number and order of verification checkpoints. The verification module may further include: a first verification submodule configured to verify whether the evaluation results of multi-dimensional indicators meet the statistical confidence requirements; and a second verification submodule configured to verify the evaluation results of multi-dimensional indicators according to the number and order of verification checkpoints to obtain verification results.

[0083] In some embodiments, the resource allocation module 550 may further include: a first resource allocation submodule configured to allocate dedicated computing resources or stable node resources to the model evaluation task in response to a risk level higher than a first preset threshold; and a second resource allocation submodule configured to allocate preemptible resources or elastic nodes to the model evaluation task in response to a risk level lower than a second preset threshold.

[0084] The operations of the aforementioned acquisition module 510, risk classification module 520, strategy matching module 530, dataset construction module 540, and resource allocation module 550 can be combined. Figure 1 The operations of steps S102, S104, S106, S108 and S110 are the same, so the details of each aspect will not be repeated here.

[0085] According to one aspect of this disclosure, a computer device is also provided, comprising: at least one processor; and a memory storing a computer program thereon, wherein the computer program, when executed by the at least one processor, causes the at least one processor to perform the steps of any of the method embodiments described above.

[0086] According to one aspect of this disclosure, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, causes the processor to perform the steps of any of the method embodiments described above.

[0087] According to one aspect of this disclosure, a computer program product is also provided, which includes a computer program that, when executed by a processor, implements the steps of any of the method embodiments described above.

[0088] Figure 6 An example computer device 600 is shown in which any of the embodiments described herein may be implemented. The computer device 600 may be used to implement one or more components of the systems and methods described above. The computer device 600 may include a bus 602 or other communication mechanism for communicating information, and one or more processors 604 coupled to the bus 602 for processing information. The processor 604 may be, for example, one or more general-purpose microprocessors.

[0089] Computer device 600 may also include main memory 606, such as random access memory (RAM), cache, and / or other dynamic storage devices, coupled to bus 602, for storing information and instructions to be executed by processor 604. Main memory 606 may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 604. Such instructions, when stored in a storage medium accessible to processor 604, can make computer device 600 a special-purpose machine customized to perform the operations specified in the instructions. Main memory 606 may include non-volatile media and / or volatile media. Non-volatile media may include, for example, optical discs or magnetic disks. Volatile media may include dynamic memory. Common media formats may include, for example, floppy disks, collapsible disks, hard disks, solid-state drives, magnetic tapes or any other magnetic data storage media, CD-ROMs (read-only optical disc drives), any other optical data storage media, any physical media with a perforated arrangement, RAM (random access memory), DRAM (dynamic random access memory), PROM (programmable read-only memory) and EPROM (erasable programmable read-only memory), FLASH-EPROM (fast erase programmable read-only memory), NVRAM (non-volatile random access memory), any other memory chips or tape cartridges, or network versions of the above.

[0090] Computer device 600 may implement the techniques described herein using custom hardwired logic, one or more ASICs (Application-Specific Integrated Circuits) or FPGAs (Field-Programmable Gate Arrays), firmware, and / or program logic, which, when combined with computer device 600, enable computer device 600 to become a special-purpose machine or to be programmed therein. According to one embodiment, the techniques herein are executed by computer device 600 in response to processor 604 executing one or more sequences of one or more instructions contained in main memory 606. Such instructions may be read into main memory 606 from another storage medium, such as storage device 608. Executing the sequence of instructions contained in main memory 606 causes processor 604 to perform the processing steps described herein. For example, the processes / methods disclosed herein may be implemented by computer program instructions stored in main memory 606. When these instructions are executed by processor 604, they may perform the steps shown in the corresponding figures and as described above. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0091] Computer device 600 also includes a network interface 610 coupled to bus 602. Network interface 610 can provide bidirectional data communication coupled to one or more network links connected to one or more networks. As another example, network interface 610 can be a local area network (LAN) card to provide data communication connectivity with a compatible LAN (or a WAN component communicating with a WAN (wide area network)). Wireless links can also be implemented.

[0092] The performance of certain operations can be distributed across processors, not just residing within a single machine, but deployed across many machines. In some exemplary embodiments, the processor or the processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other exemplary embodiments, the processor or the processor-implemented engine may be distributed across many geographic locations.

[0093] Each process, method, and algorithm described in the preceding sections can be embodied in a code module executed by one or more computer systems or computer processors including computer hardware, and can be fully or partially automated by them. These processes and algorithms can be implemented, in part or in whole, in a specific application circuit.

[0094] When the functions disclosed herein are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Specific technical solutions (all or part) disclosed herein, or aspects contributing to the prior art, can be embodied in the form of a software product. This software product can be stored in a storage medium and includes instructions to cause a computer device (which may be a personal computer, server, network device, etc.) to perform all or part of the steps of the methods described in the embodiments of this application. The storage medium may include a flash drive, a portable hard drive, ROM, RAM, a magnetic disk, an optical disk, another medium suitable for storing program code, or any combination thereof.

[0095] The embodiments disclosed herein can be implemented via a cloud platform, server, or group of servers that interact with a client. The client can be a terminal device or a client registered by a user on the platform, wherein the terminal device can be a mobile terminal, a personal computer (PC), or any device that can install platform applications.

[0096] The various features and processes described above can be used independently or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Furthermore, certain method or process blocks may be omitted in some embodiments. The methods and processes described herein are not limited to any particular order, and associated blocks or states may be executed in other suitable orders. For example, described blocks or states may be executed in a non-specifically disclosed order, or multiple blocks or states may be combined in a single block or state. Exemplary blocks or states may be executed serially, in parallel, or otherwise. Blocks or states may be added to or removed from the disclosed exemplary embodiments. The exemplary systems and components described herein may be configured differently from those described. For example, elements may be added, removed, or rearranged compared to the disclosed exemplary embodiments.

[0097] The various operations of the exemplary methods described herein can be performed at least in part by an algorithm. An algorithm may consist of program code or instructions stored in memory (such as the non-transitory computer-readable storage medium described above). Such an algorithm may include a machine learning algorithm. In some embodiments, the machine learning algorithm may not be explicitly programmed into the computer to perform the function, but may learn from training data to obtain a predictive model for performing that function.

[0098] The various operations of the exemplary methods described herein can be performed at least in part by one or more processors, which are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute the engine of a processor implementation whose operation is to perform one or more of the operations or functions described herein.

[0099] Similarly, the methods described herein can be implemented at least partially by a processor, where a specific processor or one or more processors are examples of hardware. For example, at least some operations of the methods can be performed by one or more processors or an engine implemented by a processor. Furthermore, one or more processors can also run in a “cloud computing” environment or as “Software as a Service” (SaaS) to support the execution of the relevant operations. For example, at least some operations can be performed by a group of computers (as an example of a machine including processors), which can be accessed via a network (e.g., the Internet) and through one or more appropriate interfaces (e.g., application programming interfaces (APIs)).

[0100] The performance of certain operations can be distributed across processors, not just residing within a single machine, but deployed across many machines. In some exemplary embodiments, the processor or the processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other exemplary embodiments, the processor or the processor-implemented engine may be distributed across many geographic locations.

[0101] In this specification, multiple instances may implement components, operations, or structures described as a single instance. Although individual operations of one or more methods are described and illustrated as independent operations, one or more individual operations may be performed concurrently, and these operations are not required to be performed in the order shown. Structures and functionalities presented as independent components in the example configuration may be implemented as combined structures or components. Similarly, structures and functionalities presented as individual components may be implemented as independent components. These and other variations, modifications, additions, and improvements are all within the scope of this document.

[0102] As used herein, “or” is inclusive rather than exclusive unless explicitly stated or indicated by context. Furthermore, “and” is both common and individual unless explicitly stated or indicated by context. Moreover, multiple instances may be provided for the resources, operations, or structures described herein as a single example. Furthermore, the boundaries between various resources, operations, engines, and data stores are somewhat arbitrary, and specific operations are illustrated within the context of a particular illustrative configuration. The allocation of other functionalities is conceivable and may fall within the scope of various embodiments of this disclosure. Generally, structures and functionalities presented as independent resources in example configurations may be implemented as combined structures or resources. Similarly, structures and functionalities presented as individual resources may be implemented as independent resources. These and other variations, modifications, additions, and improvements are all within the scope of embodiments of this disclosure. Therefore, this specification and accompanying drawings should be viewed in an illustrative rather than restrictive sense.

[0103] The terms “comprising” or “including” are used to indicate the presence of a subsequently stated feature, but do not preclude the addition of other features. Conditional language, in particular, such as “may,” “can,” or “may,” unless specifically stated or otherwise understood in the context of use, is generally intended to express that certain embodiments include certain features, elements, and / or steps, while other embodiments do not. Therefore, such conditional language generally does not imply that a feature, element, and / or step is necessary in any way for one or more embodiments, or that one or more embodiments must include logic that, with or without user input or prompting, determines whether such features, elements, and / or steps are included in any particular embodiment, or whether they are to be performed in any particular embodiment.

Claims

1. A resource scheduling method for model evaluation tasks, characterized in that, The model evaluation task includes a task for evaluating a target model in the scenario to be evaluated, and the method includes: Obtain risk data of the target model in the scenario to be evaluated, wherein the risk data indicates the degree of risk of the scenario to be evaluated; The risk data is input into a preset model for risk classification to obtain the risk level corresponding to the scenario to be evaluated. The model for risk classification is constructed to output the risk level, risk score and risk composition explanation based on the given risk data. Based on the risk level, a corresponding test strategy template is matched from a preset strategy template library that includes one or more test strategy templates. Each test strategy template includes a multi-dimensional indicator set, dataset coverage rules, and resource scheduling strategy associated with the risk level. Each indicator in the multi-dimensional indicator set has a corresponding threshold. Based on the dataset coverage rules in the test strategy template, an assessment dataset adapted to the risk level is constructed. Constructing the assessment dataset includes: dynamically adjusting the sample ratio and sample size of each data slice in the assessment dataset under the scenario to be assessed based on the risk weights associated with the risk level; and According to the resource scheduling strategy in the test strategy template, computing resources are allocated to the model evaluation task, wherein allocating computing resources to the model evaluation task includes: determining the resource allocation share of the model evaluation task according to the risk weight.

2. The method according to claim 1, characterized in that, The method further includes: On the allocated computing resources, the model evaluation task is performed on the target model using the constructed evaluation dataset to obtain multi-dimensional indicator evaluation results; Based on the threshold corresponding to each indicator in the multi-dimensional indicator set in the test strategy template, the evaluation results of the multi-dimensional indicators are verified to obtain a verification result; and Based on the verification results, determine whether to add further testing to the target model.

3. The method according to claim 2, characterized in that, The test strategy template also includes statistical confidence requirements associated with the risk level, as well as the number and order of verification checkpoints. The verification of the multi-dimensional indicator evaluation results to obtain the verification results further includes: Verify whether the evaluation results of the multi-dimensional indicators meet the statistical confidence requirements; and The evaluation results of the multi-dimensional indicators are verified according to the number and order of the verification checkpoints to obtain the verification results.

4. The method according to any one of claims 1 to 3, characterized in that, The dataset coverage rule is used to constrain the list of data slice dimensions to be covered for the scenario to be evaluated, as well as the minimum sample size and minimum coverage rate of each data slice. The dynamic adjustment of the sample ratio and sample size of each data slice further includes: Under the premise of satisfying the minimum sample size and the minimum coverage, the sample ratio is positively correlated with the risk weight, so as to increase the sample ratio of data slices corresponding to the target risk level whose risk level is higher than the preset target threshold.

5. The method according to any one of claims 1 to 3, characterized in that, The allocation of computing resources for the model evaluation task also includes: In response to the risk level exceeding a first preset threshold, dedicated computing resources or stable node resources are allocated to the model evaluation task; and In response to the risk level being lower than a second preset threshold, preemptible resources or elastic nodes are allocated to the model evaluation task.

6. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Historical data from the model evaluation task are fed back to the model used for risk classification and the strategy template library to update the parameters of the model used for risk classification or optimize the configuration of the test strategy template.

7. The method according to any one of claims 1 to 3, characterized in that, The risk data includes one or more of the following: scenario impact range data, misjudgment cost data, compliance and regulatory intensity data, scenario sensitivity data, and historical accident frequency and severity data.

8. The method according to any one of claims 1 to 3, characterized in that, The multi-dimensional indicator set includes one or more of the following indicators: accuracy, recall, false positive rate, false negative rate, fairness indicator, robustness indicator, latency indicator, and resource consumption indicator.

9. A resource scheduling device for model evaluation tasks, characterized in that, The model evaluation task includes a task for evaluating a target model in the scenario to be evaluated, and the apparatus includes: The acquisition module is configured to acquire risk data of the target model in the scenario to be evaluated, wherein the risk data indicates the degree of risk of the scenario to be evaluated; The risk classification module is configured to input the risk data into a preset model for risk classification to obtain the risk level corresponding to the scenario to be evaluated. The model for risk classification is constructed to output the risk level, risk score and risk composition explanation based on the given risk data. The strategy matching module is configured to match a corresponding test strategy template from a preset strategy template library that includes one or more test strategy templates based on the risk level. Each test strategy template includes a multi-dimensional indicator set, a dataset coverage rule, and a resource scheduling strategy associated with the risk level. Each indicator in the multi-dimensional indicator set has a corresponding threshold. The dataset construction module is configured to construct an assessment dataset adapted to the risk level according to the dataset coverage rules in the test strategy template. Constructing the assessment dataset includes: dynamically adjusting the sample ratio and sample size of each data slice of the assessment dataset under the scenario to be assessed based on the risk weights associated with the risk level; and The resource allocation module is configured to allocate computing resources to the model evaluation task according to the resource scheduling strategy in the test strategy template, wherein allocating the computing resources to the model evaluation task includes: determining the resource allocation share of the model evaluation task according to the risk weight.

10. A computer device, characterized in that, The computer device includes: At least one processor; A memory having a computer program stored thereon, wherein, when executed by the at least one processor, the computer program causes the at least one processor to perform the method of any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1-8.

12. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1-8.

Images