A model scheduling method and device based on task execution quality and equipment

Abstract

The application provides a model scheduling method and device based on task execution quality and equipment, comprising: obtaining task data submitted by a user; assigning any model in a preset model pool to the task according to a preset exploration rate and executing; monitoring the task execution process, triggering multi-level dynamic correction when the execution process deviates from the expected process, and obtaining the output after correction; when the output after correction still cannot meet the preset quality standard, switching the model to continue executing the current task and retaining the switching record; receiving feedback data and dynamically updating the credit score according to the feedback data; when the number of failures of any task type and model combination reaches a first threshold, adding the combination to a blacklist to obtain a blacklist state; and when the cumulative execution data volume of any task type and model combination reaches a second threshold, generating an optimal model scheduling routing strategy corresponding to the task type according to the credit score, the switching record and the blacklist state. The application can automatically match appropriate models for different tasks.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a model scheduling method, apparatus and equipment based on task execution quality. Background Technology

[0002] Currently, in the field of artificial intelligence (AI) model scheduling and intelligent agent application, mainstream AI model routing systems mostly adopt static preset rules, fixed task allocation, or scheduling methods based on prior parameters. They mainly rely on manual configuration of model priorities, task-model binding relationships, or external labels to achieve scheduling, which cannot adapt to the personalized task needs of different enterprises, departments, or job types, resulting in a low matching degree between models and tasks.

[0003] Existing AI products optimized by task type have fixed functions and scheduling strategies, making it difficult to cover the diverse needs of real-world work scenarios. This results in high user costs and unstable performance. At the same time, existing technologies only support single execution and simple feedback, lacking real-time monitoring and progressive correction mechanisms for the execution process. This makes it impossible to effectively correct deviations from expected model outputs. Furthermore, model switching is generally based on performance indicators such as response timeout or computing load, without considering task execution quality and failure count for adaptive decision-making. This may lead to the recurrence of similar errors and make it difficult to achieve self-optimization and long-term adaptation of the model in the right direction. Summary of the Invention

[0004] This invention provides a model scheduling method, apparatus, and device based on task execution quality, which solves the problems of existing AI model scheduling relying on factory rules, difficulty adapting to different task types, and difficulty in correcting task deviations, thereby improving the application efficiency of the model.

[0005] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0006] This invention provides a model scheduling method based on task execution quality, comprising:

[0007] Retrieve user-submitted task data;

[0008] Based on the preset exploration rate, assign any model from the preset model pool to each task in the task data and execute it;

[0009] The task execution process of the model is monitored, and multi-level dynamic correction is triggered when the execution process deviates from the expected process, and the corrected output is obtained.

[0010] If the corrected output still fails to meet the preset quality standard, switch models to continue executing the current task and retain the switching record;

[0011] The system receives feedback data from the model after the task is completed and dynamically updates the reputation score of each task type and model combination based on the feedback data.

[0012] When the number of failures of any task type and model combination reaches the first threshold, the combination is added to the blacklist, and the blacklist status is obtained.

[0013] When the cumulative execution data of any task type and model combination reaches the second threshold, the optimal model scheduling routing strategy corresponding to the task type is generated based on the reputation score, the switching record, and the blacklist status.

[0014] Optionally, based on a preset exploration rate, assign any model from a preset model pool to each task in the task data and execute it, including:

[0015] During the system startup phase, the exploration rate of all models in the preset model pool is set to the default exploration rate, and a model is selected from the preset model pool for each task and executed.

[0016] During system operation, the model's exploration rate is dynamically reduced based on the accumulated amount of executed data and the preset exploration rate threshold. For each task, a model is selected from the preset model pool and executed.

[0017] The exploration rate is used to balance the ratio between random exploration by the model and the use of reputation scoring.

[0018] Optionally, the task execution process of the model is monitored, and multi-level dynamic correction is triggered when the execution process deviates from the expected process, resulting in corrected output, including:

[0019] Real-time monitoring is performed on at least one of the following during the model execution process: task completion, logical coherence, and format standardization, to obtain execution path characteristics;

[0020] The execution path features are compared with the preset expected path features. When the deviation exceeds the preset deviation threshold, it is determined to be a deviation from the expected process. The type of deviation is automatically obtained and the number of corrections is accumulated.

[0021] Based on the type of deviation and the number of corrections accumulated, select the corresponding level of correction instruction from the preset multi-level correction strategy library;

[0022] The correction instructions are automatically injected into the model dialogue to obtain the corrected output.

[0023] Optionally, the multi-level correction strategy library includes at least:

[0024] The first level of correction generates a gentle, prompting correction instruction based on the first level of correction.

[0025] The second-level correction generates a serious warning-type corrective instruction based on the second-level correction.

[0026] The third level of correction generates simulated assessment pressure-type correction instructions based on the third level of correction.

[0027] Among them, the triggering conditions for each level of correction are positively correlated with the correction intensity, and when the same model deviates continuously in the same task, the invalid correction level that has been used is automatically skipped.

[0028] Optionally, if the corrected output still fails to meet the preset quality standards, switch models to continue executing the current task, and retain the switching record, including:

[0029] If any model fails to meet the preset quality standard after triggering correction in the current task, the number of failures for the corresponding combination is accumulated.

[0030] When the number of failures reaches the failure threshold, a model switch is triggered;

[0031] The model is reassigned from the model pool to take over the current task, the data of the model session is migrated to the standby model, and the switchover record is retained.

[0032] Optionally, the system receives feedback data from the model after the task execution is completed, and dynamically updates the reputation score for each task type and model combination based on the feedback data, including:

[0033] Obtain at least one of the following feedback data after the task execution is completed: success or failure status, task completion time, and number of corrections.

[0034] The reputation score of the task type and model combination is updated according to the preset score update rules.

[0035] When a task is successful, the reputation score increases by a first update amount; when a task fails, the reputation score decreases by a second update amount, and the second update amount is greater than the first update amount.

[0036] Optionally, when the cumulative execution data volume of any task type and model combination reaches a second threshold, an optimal model scheduling routing strategy corresponding to the task type is generated based on the reputation score, the switching records, and the blacklist status, including:

[0037] When the cumulative execution data of any task type and model combination reaches the second threshold, the models that have been marked as blacklisted are filtered out according to the blacklist status to obtain a candidate model set.

[0038] Based on the switching records, extract the total number of switching times for each candidate model under the corresponding task type;

[0039] Based on the reputation score and the total number of switching times, the comprehensive scheduling priority of each candidate model is obtained;

[0040] Based on the comprehensive scheduling priority sorted from high to low, the optimal model scheduling routing strategy corresponding to the task type is generated.

[0041] This invention also provides a model scheduling device based on task execution quality, comprising:

[0042] The acquisition module is used to acquire task data submitted by the user.

[0043] The processing module is used to allocate any model from a preset model pool to each task in the task data according to a preset exploration rate and execute it; monitor the task execution process of the model, trigger multi-level dynamic correction when the execution process deviates from the expected process, and obtain the corrected output; when the corrected output still fails to meet the preset quality standard, switch models to continue executing the current task and retain the switching record; receive feedback data given by the model after the task execution is completed, and dynamically update the reputation score of each task type and model combination according to the feedback data; when the number of failures of any task type and model combination reaches a first threshold, add the combination to the blacklist and obtain the blacklist status;

[0044] The generation module is used to generate the optimal model scheduling routing strategy corresponding to the task type based on the reputation score, the switching record and the blacklist status when the cumulative execution data of any task type and model combination reaches the second threshold.

[0045] This invention also provides a computing device, including: a processor and a memory storing a computer program, wherein the computer program, when run by the processor, executes the above-described method.

[0046] This invention also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the above-described method.

[0047] The technical solution of the present invention has at least the following effects:

[0048] The above-described solution of the present invention obtains user-submitted task data; allocates any model from a preset model pool to each task in the task data according to a preset exploration rate and executes it; monitors the task execution process of the model, triggering multi-level dynamic correction when the execution process deviates from the expected process, and obtains the corrected output; when the corrected output still fails to meet the preset quality standard, switches the model to continue executing the current task, and retains the switching record; receives feedback data given by the model after the task execution is completed, and dynamically updates the reputation score of each task type and model combination according to the feedback data; when the number of failures of any task type and model combination reaches a first threshold, the combination is added to a blacklist, obtaining a blacklist status; when the cumulative execution data of any task type and model combination reaches a second threshold, an optimal model scheduling routing strategy corresponding to the task type is generated according to the reputation score, the switching record, and the blacklist status. This allows the model to adapt to different task types, improves the success rate and reliability of task execution, realizes the classification scheduling of AI models, and improves the application efficiency of the models. Attached Figure Description

[0049] Figure 1 This is a flowchart of a model scheduling method based on task execution quality provided in an embodiment of the present invention;

[0050] Figure 2 This is a structural diagram of the model scheduling device based on task execution quality provided in an embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of the structure of the computing device provided in an embodiment of the present invention. Detailed Implementation

[0052] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0053] like Figure 1 As shown, an embodiment of the present invention proposes a model scheduling method based on task execution quality, including:

[0054] Step 11: Obtain the task data submitted by the user;

[0055] Step 12: Assign any model from the preset model pool to each task in the task data according to the preset exploration rate and execute it;

[0056] Step 13: Monitor the task execution process of the model. When the execution process deviates from the expected process, trigger multi-level dynamic correction to obtain the corrected output.

[0057] Step 14: If the corrected output still fails to meet the preset quality standard, switch models to continue executing the current task and retain the switching record;

[0058] Step 15: Receive feedback data from the model after the task is completed, and dynamically update the reputation score of each task type and model combination based on the feedback data.

[0059] Step 16: When the number of failures of any task type and model combination reaches the first threshold, add the combination to the blacklist to obtain the blacklist status;

[0060] Step 17: When the cumulative execution data of any task type and model combination reaches the second threshold, generate the optimal model scheduling routing strategy corresponding to the task type based on the reputation score, the switching record and the blacklist status.

[0061] The solution described in this embodiment achieves autonomous evolution of AI model routing strategies by constructing a closed-loop model adaptive scheduling mechanism, significantly improving task success rate and response reliability in multi-model scheduling scenarios with different task types. This solution eliminates the costs and biases caused by preset rules through zero-prior knowledge initialization design, enabling customized scheduling for different task types; it achieves a dynamic balance between exploration and utilization through dynamic adjustment of the exploration rate; it triggers different levels of correction instructions through a multi-level dynamic correction mechanism, automatically skipping invalid levels and enabling fine-grained intervention in model behavior; and it avoids frequent switching due to accidental fluctuations through a model adaptive switching mechanism, preventing difficulties in handling long-tail failure cases. This solution realizes a shift from conventional model scheduling relying on preset rules to specific scheduling based on accumulated experience, significantly improving the task success rate of AI models, effectively reducing trial-and-error costs, and providing a fully adaptable adaptive application solution for different business scenarios.

[0062] In an optional embodiment of the present invention, step 11, obtaining the task data submitted by the user, may include:

[0063] Step 111: Obtain and parse the task data submitted by the user to obtain the task type and task content data.

[0064] In this embodiment, in step 111, the task type includes text generation tasks, code writing tasks, data analysis tasks, image recognition tasks, and real-time information retrieval tasks; the task content data includes at least one of the following: the user's original input instructions, the data file to be processed, or the dialogue history.

[0065] In an optional embodiment of the present invention, step 12, which involves allocating any model from a preset model pool to each task in the task data according to a preset exploration rate and executing it, may include:

[0066] Step 121: During the system startup phase, set the exploration rate of all models in the preset model pool to the default exploration rate, and select and execute a model from the preset model pool for each task.

[0067] Step 122: During the system operation phase, based on the accumulated amount of executed data and the preset exploration rate threshold, the exploration rate of the model is dynamically reduced, and a model is selected from the preset model pool for each task and executed.

[0068] The exploration rate is used to balance the ratio between random exploration by the model and the use of reputation scoring.

[0069] In this embodiment, during the system startup phase in step 121, the model is in a state of zero prior knowledge, meaning there are no preset rules regarding the matching effect of all task types and model combinations. The closer the exploration rate value is to 1, the stronger the exploration tendency; the closer it is to 0, the stronger the exploitation tendency. During the system startup phase, a default exploration rate (high exploration rate, such as...) is used. ) run, that is, use for each model arrive The model is selected from the model pool for each task using a probability random allocation method, with only a 0% to 20% probability of selecting a model based on reputation score. After execution, performance data of each model on different tasks is collected. The preset model pool includes at least two different large language models or artificial intelligence models.

[0070] In step 122, the exploration rate threshold has different levels (e.g., a first exploration rate threshold of 100 and a second exploration rate threshold of 500). The exploration rate of the model is dynamically reduced from an initial high exploration rate through the exploration rate threshold: when the cumulative amount of executed data is less than the first exploration threshold, the exploration rate is maintained at a high level; when the cumulative amount of executed data reaches the first exploration threshold but is less than the second exploration threshold, the exploration rate is generally reduced to the second stage, and models with higher reputation scores are preferentially selected; when the cumulative amount of executed data reaches the second exploration threshold (the second exploration threshold is equal to the second threshold), the exploration rate is generally reduced to the third stage, at which point the model is selected according to the optimal model scheduling and routing strategy, thereby achieving a dynamic balance between model exploration and experience utilization through the preset exploration rate.

[0071] In an optional embodiment of the present invention, step 13, which involves monitoring the task execution process of the model and triggering multi-level dynamic correction when the execution process deviates from the expected process, to obtain the corrected output, may include:

[0072] Step 131: Monitor at least one of the following in real time during the model execution process: task completion, logical coherence, and format standardization, to obtain execution path characteristics;

[0073] Step 132: Compare the execution path features with the preset expected path features. If the deviation exceeds the preset deviation threshold, it is determined to be a deviation from the expected process. The type of deviation is automatically obtained and the number of corrections is accumulated.

[0074] Step 133: Select the corresponding level of correction instruction from the preset multi-level correction strategy library according to the type of deviation and the number of corrections accumulated;

[0075] Step 134: The correction instruction is automatically injected into the model dialogue to obtain the corrected output.

[0076] In this embodiment, step 131 involves real-time monitoring of the model execution process. The task completion rate is used to measure the proportion of completed task steps to the expected total steps, which is quantified by detecting key nodes in the model output. The specific process includes:

[0077] (1) Extract the key node sequence of the expected task steps

[0078] For the current task type, there is a preset standard task step sequence and key nodes corresponding to each step. The key nodes refer to the actions that must be completed or the key information that must be output during the task execution process.

[0079] (2) Real-time detection of the completion status of key nodes in the model output

[0080] The system performs streaming parsing of the model's step-by-step output, detecting whether each key node appears in the model output. Specific detection methods include: matching keyword patterns corresponding to key nodes using regular expressions (such as matching "read successful" or "loaded"); determining whether the model output expresses the same semantic content as the key node through semantic similarity calculation (such as "data has been cleaned" being semantically similar to "data cleaning completed"); and parsing the structured output to extract the values ​​of corresponding fields from the JSON or XML format output. When a key node is detected, the system marks that node as "completed".

[0081] (3) Calculate the proportion of completed critical nodes to expected critical nodes.

[0082] The number of currently detected key nodes is counted, and the ratio is calculated to the preset total number of expected key nodes, which is expressed as: ;in, This represents the completion rate of routine tasks. This represents the number of critical nodes that have been detected so far. This represents the expected total number of critical nodes.

[0083] (4) Weighted completion calculation considering the order of steps

[0084] Directly calculating the proportion of completed key nodes to expected key nodes as the task completion rate may lead to inaccurate completion assessments. To further improve the accuracy of completion assessments, a step-by-step order weight can be introduced. Key nodes completed strictly in sequence are given higher weights, while key nodes completed with skipped steps or out of order are given lower weights. The formula for calculating the weighted task completion rate is as follows: ;in, Weighted task completion rate; The preset weight for the g-th key node; is the order coefficient of the g-th critical node (takes 1 when the node is completed in the expected order, 0.5 when it is completed out of order, and 0.2 when it is completed with a skip step); g is the index; This represents the number of critical nodes that have been detected so far. This represents the expected total number of critical nodes.

[0085] (5) Completion threshold determination

[0086] The calculated task completion rate is compared with a preset completion rate threshold (e.g., 80%): when the completion rate is greater than or equal to the threshold, the task completion rate is determined to meet expectations; when the completion rate is less than the threshold, the task completion rate is determined to be insufficient, triggering a deviation from the expected process.

[0087] The logical coherence is used to detect whether there are logical problems in the model output, which is determined through natural language inference models or rule matching. The format conformity is used to detect whether the model output meets the expected format requirements, such as JSON format. By monitoring at least one of the above indicators, an execution path feature vector is generated to represent the current execution state of the model.

[0088] In step 132, the execution path features are compared with preset expected path features. The expected path features are standard execution trajectories pre-defined for the current task type. For example, for a data analysis task, the expected path features could be steps such as "data reading → data cleaning → statistical analysis → result visualization → conclusion generation". The deviation value between the execution path features and the expected path features is calculated. When the deviation value exceeds a preset deviation threshold, it is determined that the current execution process of the model has deviated from the expected process. The preset deviation threshold is configured according to the sensitivity of the task type; a lower preset deviation threshold is set for tasks with high precision requirements, and a higher preset deviation threshold is set for tasks with higher fault tolerance.

[0089] The calculation process for the deviation value is as follows:

[0090] (1) For each step in the actual execution path, match it with each expected step in the expected path in turn. The matching judgment conditions include the semantic similarity of the step name, the compatibility of the step type, and the feature consistency of the step output. When the comprehensive similarity between the actual step and an expected step in the above three dimensions exceeds the preset matching threshold, it is determined that the two are aligned. At this time, the actual step is mapped to the corresponding expected step.

[0091] (2) For aligned steps, according to This yields the step deviation for a single actual step; for redundant actual steps, a fixed penalty value is set. For missing actual steps, set a fixed penalty value. ;in, Let be the step deviation of the j-th actual step; i represents the index of the expected step; j represents the index of the actual step. Indicates the actual steps; Indicates the expected steps; Indicates whether the types of the actual steps and the expected steps match, with a value ranging from 0 to 1; This indicates the proportion of missing keywords in actual steps and expected steps, with a value ranging from 0 to 1. This indicates the degree of formatting violation between the actual and expected steps, with a value ranging from 0 to 1; where, Indicates the weighting coefficient for type matching; Weighting coefficients representing the proportion of missing items; The weighting coefficient indicating formatting violations; A fixed penalty value representing any unnecessary actual steps; This represents a fixed penalty value for missing actual steps.

[0092] (3) According to The step deviations are aggregated into a global deviation value; where, Indicates the global deviation value; It represents the characteristics of the execution path and is a collection of multiple actual steps; It represents the expected path characteristics and is a set of multiple expected steps; Indicates the expected total number of steps; Indicates the total number of actual steps; Let be the step deviation of the j-th actual step; i represents the index of the expected step; j represents the index of the actual step. This represents the alignment indicator function, which is 1 if the actual step j is aligned to a certain expected step, and 0 otherwise. This indicates a missing indicator function; it is 1 if step i is not expected to be matched by any actual step, and 0 otherwise. This represents the fixed penalty value for the i-th missing actual step.

[0093] (4) According to Normalize the global deviation value to The interval; where, This is the deviation value; This is the preset maximum allowable deviation; Indicates the global deviation value; This represents the function that takes the minimum value.

[0094] When the above deviation value is greater than the preset deviation threshold, it is determined to be a deviation from the expected process. At this time, the type of deviation is automatically obtained, including format mismatch deviation obtained after detecting the format, logic breakage deviation obtained after detecting the logic, and task target deviation caused by low detection similarity, etc.; and the number of corrections for the current task type and model combination is incremented by 1.

[0095] In step 133, based on the type of deviation and the accumulated number of corrections, a corresponding level of correction instruction is selected from a preset multi-level correction strategy library. This multi-level correction strategy library includes at least: a first-level correction, which generates mild reminder-type correction instructions; a second-level correction, which generates serious warning-type correction instructions; and a third-level correction, which generates simulated assessment pressure-type correction instructions. That is, the severity of the instructions generated by the second-level correction is greater than that generated by the first-level correction; and the severity of the instructions generated by the third-level correction is greater than that generated by the second-level correction. The triggering conditions for each level of correction are positively correlated with the correction intensity, and when the same model deviates consecutively in the same task, the already used invalid correction levels are automatically skipped.

[0096] Depending on the type of deviation, a severity weight can be preset. The higher the weight, the more likely a more severe correction instruction will be generated. The more corrections have been accumulated, the more likely a more severe correction instruction will be generated.

[0097] In step 134, a correction instruction is added to the input context of the model without modifying the model's own parameters. For example, when a format deviation is detected, the injected correction instruction can be "Please output strictly according to the JSON format", thus obtaining the corrected output.

[0098] In an optional embodiment of the present invention, step 14, where the corrected output still fails to meet the preset quality standard, involves switching models to continue executing the current task and retaining the switching record. This may include:

[0099] Step 141: When any model fails to meet the preset quality standard after triggering correction in the current task, accumulate the number of failures for the corresponding combination.

[0100] Step 142: When the number of failures reaches the failure threshold, a model switch is triggered;

[0101] Step 143: Reassign a model from the model pool to take over the current task, migrate the data of the model session to the standby model, and retain the switchover record.

[0102] In this embodiment, in step 141, if the preset quality standard cannot be met, it means that the result requirement is still not met after all levels of correction. At this time, the failure count of the current task type and the current model combination is incremented by 1.

[0103] In step 142, the accumulated number of failures is compared with a preset failure threshold. This failure threshold is a critical value that triggers model switching and can be configured differently based on the importance of the task type; that is, different failure thresholds can be set for different types of tasks. When the number of failures reaches the failure threshold, the system triggers the model switching process.

[0104] In step 143, a model is selected again from the model pool to take over the current task. When selecting a backup model, the currently executing model is excluded. After selecting a backup model, the data of the current model session is migrated to the backup model. The session data includes: historical dialogue context, intermediate results of executed steps, the current progress status of the task, and completed correction records, etc. After the migration is completed, the backup model will continue executing the task from the point of interruption in the current task.

[0105] Simultaneously, relevant information for this switching operation is recorded to generate a switching record. The switching record includes task type, original model identifier, backup model identifier, and switching timestamp. The task type is used to identify the category to which the current task belongs. The original model identifier is used to record the model that was replaced. The backup model identifier is used to record the model that was newly taken over. The switching timestamp is used to record the specific time point when the switching occurred.

[0106] In an optional embodiment of the present invention, step 15, receiving feedback data from the model after the task execution is completed, and dynamically updating the reputation score of each task type and model combination based on the feedback data, may include:

[0107] Step 151: Obtain at least one of the following feedback data after the task execution is completed: success or failure flag, task completion time, and number of corrections.

[0108] Step 152: Update the reputation score of the task type and model combination according to the preset score update rules;

[0109] When a task is successful, the reputation score increases by a first update amount; when a task fails, the reputation score decreases by a second update amount, and the second update amount is greater than the first update amount.

[0110] In this embodiment, in step 151, feedback data returned by the model is received after the task execution is completed. The feedback data includes at least one of the following: a success or failure indicator, a task completion time, and a number of corrections. The success or failure indicator is used to indicate whether the current task has been successfully completed. The task completion time is used to measure the efficiency of the model in executing the task, and the longer the time, the greater the negative impact on the reputation score. The number of corrections is used to measure the stability of the model execution process, and the more corrections, the more unstable the model's performance on the task, and the reputation score needs to be appropriately reduced.

[0111] In step 152, the reputation scores of the task type and model combination are dynamically updated according to the preset score update rules; when training is scheduled for the first time, the reputation scores of all combinations are set to the same default value.

[0112] The preset scoring update rules include update rules for successful tasks and update rules for failed tasks. The current value of the reputation score is S1. Each successful task increases the score by an initial update amount of S2. For the update rules for successful tasks: S2 is fine-tuned based on the task completion time and the number of corrections. Specifically, if the task completion time is greater than a preset time threshold, the weight coefficient decreases; conversely, if the number of corrections is greater than a preset correction threshold, the weight coefficient decreases; conversely, if the number of corrections is less than a preset correction threshold, the weight coefficient increases. The weight coefficient is multiplied by S2 to obtain the final update amount S3 for successful tasks. The current value of the reputation score is added to the final update amount to obtain the new reputation score. That is: according to... This yields the weighting coefficient for a successful task; where... Indicates the weighting coefficient when the task is successful; This represents the weighting coefficient for the time taken to complete the task; Indicates the time factor for task completion; This represents the corrective weight coefficient; This represents the correction factor. According to... The final update value when the task is successful is obtained; where S2 is the first update value added when the task is successful. This is the final update value when the task succeeds. This represents the weighting coefficient for a successful task. According to... Upon successful completion of the task, a new reputation score is obtained; among them, A new reputation score is assigned upon successful completion of the task; This represents a function that takes the minimum value. S1 represents the upper limit of the credit score; S1 is the current value of the credit score. This is the final update amount when the task is successful.

[0113] The current reputation score is S1. The second update amount, S4, decreases with each task failure. The update rules for task failure are as follows: S4 is fine-tuned based on task completion time and the number of corrections. If the task completion time exceeds a preset time threshold, the weight coefficient increases; conversely, if it exceeds a preset correction threshold, the weight coefficient decreases. The weight coefficient is multiplied by S4 to obtain the final update amount S5 for task failure. The current reputation score is subtracted from S5 to obtain the new reputation score. Generally, the first update amount S2 for a successful task is less than the second update amount S4 for a failed task, meaning the penalty for task failure is greater than the reward for success. That is: according to... This yields the weighting coefficients for task failure; where, Indicates the weighting coefficient when a task fails; This represents the weighting coefficient for the time taken to complete the task; Indicates the time factor for task completion; This represents the corrective weight coefficient; This represents the correction factor. According to... This yields the final update amount when the task fails; where S4 is the second update amount reduced when the task fails. This is the final update amount when the task fails. This represents the weighting coefficient when a task fails. According to... A new reputation score is obtained when the task fails; among them, A new reputation score is assigned when a task fails. This represents the function that takes the maximum value. S1 represents the lower limit of the credit score; S1 is the current value of the credit score. This is the final update amount when the task fails.

[0114] In an optional embodiment of the present invention, step 16, when the number of failures of any task type and model combination reaches a first threshold, adds the combination to the blacklist to obtain the blacklist status, may include:

[0115] Step 161: Monitor the number of failures for each task type and model combination in real time. When the number of failures reaches the preset first threshold, add the combination to the blacklist and update the blacklist status.

[0116] In this embodiment, when a model fails a certain number of times on a certain type of task, a model switch is triggered, and a different model continues to execute the current task. However, the model will be tried again in other tasks or subsequent tasks. When the number of failures accumulates to the first threshold, the model is determined to be unsuitable for that type of task. At this time, the combination of task type and model is added to the blacklist, and the model is marked as blacklisted for that type of task.

[0117] In an optional embodiment of the present invention, in step 17, when the cumulative execution data volume of any task type and model combination reaches a second threshold, an optimal model scheduling routing strategy corresponding to the task type is generated based on the reputation score, the switching record, and the blacklist status, which may include:

[0118] Step 171: When the cumulative execution data of any task type and model combination reaches the second threshold, filter out the models that have been marked as blacklisted according to the blacklist status to obtain a candidate model set.

[0119] Step 172: Based on the switching records, extract the total number of switching times for each candidate model under the corresponding task type;

[0120] Step 173: Based on the reputation score and the total number of switching times, obtain the comprehensive scheduling priority of each candidate model;

[0121] Step 174: Based on the comprehensive scheduling priority sorted from high to low, generate the optimal model scheduling routing strategy corresponding to the task type.

[0122] In this embodiment, in step 171, it is determined whether the cumulative execution data volume of the task type has reached a preset second threshold; the second threshold is the data volume threshold that triggers the generation of the optimal model scheduling routing strategy. At this time, based on the current blacklist status, all models in the model pool are filtered. That is, for the task type, each model in the model pool is traversed, and it is checked whether there is a blacklist status for the task type in that model, thereby filtering out unsuitable models and obtaining a candidate model set. Through blacklist filtering, it is ensured that models unsuitable for this type of task will not be included in the scope of the routing strategy, thereby avoiding invalid attempts.

[0123] In step 172, based on the saved model switching records, the total number of switching times for each candidate model under this task type is extracted; the total number of switching times represents the stability of the model on this type of task: the more switching times, the easier it is for the model to be replaced by other models, indicating that its performance on this type of task is more unstable, and it can only be given a lower priority in the routing strategy.

[0124] In step 173, when the cumulative execution data of any task type and model combination reaches the second threshold, a whitelist is maintained; when the number of consecutive successes of any task type and model combination reaches a preset condition, the combination is added to the whitelist; the model added to the whitelist enjoys priority scheduling rights on that type of task.

[0125] When a whitelist model exists for a certain task type, calculate the overall scheduling priority of each whitelist model and the overall scheduling priority of the non-whitelist model, and prioritize the whitelist model for execution; when there is no corresponding whitelist model for a certain task type, calculate the overall scheduling priority of each model to select the model.

[0126] Based on the dynamically updated reputation score and the extracted total number of switches, the comprehensive scheduling priority of each candidate model is calculated. The comprehensive scheduling priority is used to measure the overall suitability of each candidate model for the task type. The higher the comprehensive scheduling priority, the more suitable the model is for performing this type of task.

[0127] The formula for calculating the overall scheduling priority is expressed as follows: ;in, This represents the overall scheduling priority of the r-th candidate model; This represents the model's reputation score for that task type; This indicates the total number of times the model switches across this task type; This represents a normalization function, used to map numerical values ​​of different dimensions to the same scale interval; and The preset weighting coefficients represent the importance of reputation score and total number of switching times in priority calculation, respectively, satisfying the following conditions: .

[0128] when When the value is large, it relies more on the reputation score for decision-making, which is suitable for scenarios where there is high confidence in the model's historical performance; when When the value is large, more attention is paid to the stability of the model, which is suitable for scenarios with high requirements for task continuity.

[0129] In step 174, the whitelisted models and non-whitelisted models are classified and sorted. When there is only one whitelisted model, that model is executed first. When there are multiple whitelisted models, the whitelisted model with the highest overall scheduling priority is executed first. When there is only one non-whitelisted model, the non-whitelisted model with the highest overall scheduling priority is executed first, thereby generating the optimal model scheduling routing strategy.

[0130] Once an optimal model scheduling and routing strategy has been developed for a specific task type, it can be shared and used, including by promoting it to users in the same department or position, or by promoting it to the market.

[0131] The model scheduling method based on task execution quality proposed in this invention enables tasks to select the most suitable execution model through zero-experience adaptive routing and closed-loop feedback mechanism, preventing the model from continuously deviating and difficult to correct, realizing adaptive scheduling of multiple models, and improving the success rate of task execution.

[0132] like Figure 2 As shown, this embodiment of the invention also provides a model scheduling device 20 based on task execution quality, comprising:

[0133] Module 21 is used to obtain task data submitted by the user;

[0134] Processing module 22 is used to allocate any model from a preset model pool to each task in the task data according to a preset exploration rate and execute it; monitor the task execution process of the model, trigger multi-level dynamic correction when the execution process deviates from the expected process, and obtain the corrected output; when the corrected output still cannot meet the preset quality standard, switch the model to continue executing the current task and keep the switching record; receive feedback data given by the model after the task execution is completed, and dynamically update the reputation score of each task type and model combination according to the feedback data; when the number of failures of any task type and model combination reaches a first threshold, add the combination to the blacklist and obtain the blacklist status;

[0135] The generation module 23 is used to generate the optimal model scheduling routing strategy corresponding to the task type based on the reputation score, the switching record and the blacklist status when the cumulative execution data of any task type and model combination reaches the second threshold.

[0136] Optionally, processing module 22 is specifically used for:

[0137] During the system startup phase, the exploration rate of all models in the preset model pool is set to the default exploration rate, and a model is selected from the preset model pool for each task and executed.

[0138] During system operation, the model's exploration rate is dynamically reduced based on the accumulated amount of executed data and the preset exploration rate threshold. For each task, a model is selected from the preset model pool and executed.

[0139] The exploration rate is used to balance the ratio between random exploration by the model and the use of reputation scoring.

[0140] Optionally, processing module 22 is specifically used for:

[0141] Real-time monitoring is performed on at least one of the following during the model execution process: task completion, logical coherence, and format standardization, to obtain execution path characteristics;

[0142] The execution path features are compared with the preset expected path features. When the deviation exceeds the preset deviation threshold, it is determined to be a deviation from the expected process. The type of deviation is automatically obtained and the number of corrections is accumulated.

[0143] Based on the type of deviation and the number of corrections accumulated, select the corresponding level of correction instruction from the preset multi-level correction strategy library;

[0144] The correction instructions are automatically injected into the model dialogue to obtain the corrected output.

[0145] Optionally, the multi-level correction strategy library includes at least:

[0146] The first level of correction generates a gentle, prompting correction instruction based on the first level of correction.

[0147] The second-level correction generates a serious warning-type corrective instruction based on the second-level correction.

[0148] The third level of correction generates simulated assessment pressure-type correction instructions based on the third level of correction.

[0149] Among them, the triggering conditions for each level of correction are positively correlated with the correction intensity, and when the same model deviates continuously in the same task, the invalid correction level that has been used is automatically skipped.

[0150] Optionally, processing module 22 is specifically used for:

[0151] If any model fails to meet the preset quality standard after triggering correction in the current task, the number of failures for the corresponding combination is accumulated.

[0152] When the number of failures reaches the failure threshold, a model switch is triggered;

[0153] The model is reassigned from the model pool to take over the current task, the data of the model session is migrated to the standby model, and the switchover record is retained.

[0154] Optionally, processing module 22 is specifically used for:

[0155] Obtain at least one of the following feedback data after the task execution is completed: success or failure status, task completion time, and number of corrections.

[0156] The reputation score of the task type and model combination is updated according to the preset score update rules.

[0157] When a task is successful, the reputation score increases by a first update amount; when a task fails, the reputation score decreases by a second update amount, and the second update amount is greater than the first update amount.

[0158] Optionally, the generation module 23 is specifically used for:

[0159] When the cumulative execution data of any task type and model combination reaches the second threshold, the models that have been marked as blacklisted are filtered out according to the blacklist status to obtain a candidate model set.

[0160] Based on the switching records, extract the total number of switching times for each candidate model under the corresponding task type;

[0161] Based on the reputation score and the total number of switching times, the comprehensive scheduling priority of each candidate model is obtained;

[0162] Based on the comprehensive scheduling priority sorted from high to low, the optimal model scheduling routing strategy corresponding to the task type is generated.

[0163] It should be noted that this device is a device corresponding to the above method. All implementation methods in the above method embodiments are applicable to this embodiment and can achieve the same technical effect.

[0164] like Figure 3 As shown, this embodiment of the invention also provides a computing device 30, including a processor 31, a memory 32, and a program or instructions stored in the memory 32 and executable on the processor 31. When the program or instructions are executed by the processor 31, they implement the various processes of the above-described method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here. It should be noted that the computing device in this embodiment of the invention includes the aforementioned mobile electronic devices and non-mobile electronic devices.

[0165] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0166] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0167] In the embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0168] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0169] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0170] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0171] Furthermore, it should be noted that in the apparatus and method of the present invention, it is obvious that the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered equivalent solutions of the present invention. Moreover, the steps performing the above series of processes can naturally be executed in the order described, but are not necessarily required to be executed in chronological order; some steps can be executed in parallel or independently of each other. Those skilled in the art will understand that all or any step or component of the method and apparatus of the present invention can be implemented in any computing device (including processors, storage media, etc.) or network of computing devices, in hardware, firmware, software, or a combination thereof. This is something that those skilled in the art can achieve by using their basic programming skills after reading the description of the present invention.

[0172] Therefore, the object of the present invention can also be achieved by running a program or a set of programs on any computing device. The computing device can be a known general-purpose device. Therefore, the object of the present invention can also be achieved simply by providing a program product containing program code for implementing the method or apparatus. That is, such a program product also constitutes the present invention, and the storage medium storing such a program product also constitutes the present invention. Obviously, the storage medium can be any known storage medium or any storage medium developed in the future. It should also be noted that in the apparatus and method of the present invention, it is obvious that the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered equivalent to the present invention. Furthermore, the steps for performing the above series of processes can naturally be performed in the order described, but are not necessarily required to be performed in chronological order. Some steps can be performed in parallel or independently of each other.

[0173] The above are preferred embodiments of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A model scheduling method based on task execution quality, characterized in that, include: Retrieve user-submitted task data; Based on the preset exploration rate, assign any model from the preset model pool to each task in the task data and execute it; The task execution process of the model is monitored, and multi-level dynamic correction is triggered when the execution process deviates from the expected process, and the corrected output is obtained. If the corrected output still fails to meet the preset quality standard, switch models to continue executing the current task and retain the switching record; The system receives feedback data from the model after the task is completed and dynamically updates the reputation score of each task type and model combination based on the feedback data. When the number of failures of any task type and model combination reaches the first threshold, the combination is added to the blacklist, and the blacklist status is obtained. When the cumulative execution data of any task type and model combination reaches the second threshold, the optimal model scheduling routing strategy corresponding to the task type is generated based on the reputation score, the switching record and the blacklist status. This involves monitoring the model's task execution process, triggering multi-level dynamic correction when the execution deviates from the expected process, and obtaining the corrected output, including: Real-time monitoring is performed on at least one of the following during the model execution process: task completion, logical coherence, and format standardization, to obtain execution path characteristics; The execution path features are compared with the preset expected path features. When the deviation exceeds the preset deviation threshold, it is determined to be a deviation from the expected process. The type of deviation is automatically obtained and the number of corrections is accumulated. Based on the type of deviation and the number of corrections accumulated, select the corresponding level of correction instruction from the preset multi-level correction strategy library; The correction instructions are automatically injected into the model dialogue to obtain the corrected output. The multi-level correction strategy library includes at least the following: The first level of correction generates a gentle, prompting correction instruction based on the first level of correction. The second-level correction generates a serious warning-type corrective instruction based on the second-level correction. The third level of correction generates simulated assessment pressure-type correction instructions based on the third level of correction. Among them, the triggering conditions for each level of correction are positively correlated with the correction intensity, and when the same model deviates continuously in the same task, the invalid correction level that has been used is automatically skipped.

2. The task execution quality-based model scheduling method according to claim 1, characterized in that, Based on a preset exploration rate, each task in the task data is assigned any model from a preset model pool and executed, including: During the system startup phase, the exploration rate of all models in the preset model pool is set to the default exploration rate, and a model is selected from the preset model pool for each task and executed. During system operation, the model's exploration rate is dynamically reduced based on the accumulated amount of executed data and the preset exploration rate threshold. For each task, a model is selected from the preset model pool and executed. The exploration rate is used to balance the ratio between random exploration by the model and the use of reputation scoring.

3. The task execution quality-based model scheduling method according to claim 1, characterized in that, If the corrected output still fails to meet the preset quality standard, switch models to continue executing the current task, and retain the switching record, including: If any model fails to meet the preset quality standard after triggering correction in the current task, the number of failures for the corresponding combination is accumulated. When the number of failures reaches the failure threshold, a model switch is triggered; The model is reassigned from the model pool to take over the current task, the data of the model session is migrated to the standby model, and the switchover record is retained.

4. The task execution quality-based model scheduling method according to claim 1, characterized in that, The system receives feedback data from the model after task execution and dynamically updates the reputation score for each task type and model combination based on this feedback data, including: Obtain at least one of the following feedback data after the task execution is completed: success or failure status, task completion time, and number of corrections. The reputation score of the task type and model combination is updated according to the preset score update rules. When a task is successful, the reputation score increases by a first update amount; when a task fails, the reputation score decreases by a second update amount, and the second update amount is greater than the first update amount.

5. The task execution quality-based model scheduling method according to claim 1, characterized in that, When the cumulative execution data volume of any task type and model combination reaches a second threshold, an optimal model scheduling routing strategy corresponding to the task type is generated based on the reputation score, the switching records, and the blacklist status, including: When the cumulative execution data of any task type and model combination reaches the second threshold, the models that have been marked as blacklisted are filtered out according to the blacklist status to obtain a candidate model set. Based on the switching records, extract the total number of switching times for each candidate model under the corresponding task type; Based on the reputation score and the total number of switching times, the comprehensive scheduling priority of each candidate model is obtained; Based on the comprehensive scheduling priority sorted from high to low, the optimal model scheduling routing strategy corresponding to the task type is generated.

6. A model scheduling device based on task execution quality, characterized in that, include: The acquisition module is used to acquire task data submitted by the user. The processing module is used to allocate any model from a preset model pool to each task in the task data according to a preset exploration rate and execute it; monitor the task execution process of the model, trigger multi-level dynamic correction when the execution process deviates from the expected process, and obtain the corrected output; when the corrected output still fails to meet the preset quality standard, switch models to continue executing the current task and retain the switching record; receive feedback data given by the model after the task execution is completed, and dynamically update the reputation score of each task type and model combination according to the feedback data; when the number of failures of any task type and model combination reaches a first threshold, add the combination to the blacklist and obtain the blacklist status; The generation module is used to generate the optimal model scheduling routing strategy corresponding to the task type based on the reputation score, the switching record and the blacklist status when the cumulative execution data of any task type and model combination reaches the second threshold. This involves monitoring the model's task execution process, triggering multi-level dynamic correction when the execution deviates from the expected process, and obtaining the corrected output, including: Real-time monitoring is performed on at least one of the following during the model execution process: task completion, logical coherence, and format standardization, to obtain execution path characteristics; The execution path features are compared with the preset expected path features. When the deviation exceeds the preset deviation threshold, it is determined to be a deviation from the expected process. The type of deviation is automatically obtained and the number of corrections is accumulated. Based on the type of deviation and the number of corrections accumulated, select the corresponding level of correction instruction from the preset multi-level correction strategy library; The correction instructions are automatically injected into the model dialogue to obtain the corrected output. The multi-level correction strategy library includes at least the following: The first level of correction generates a gentle, prompting correction instruction based on the first level of correction. The second-level correction generates a serious warning-type corrective instruction based on the second-level correction. The third level of correction generates simulated assessment pressure-type correction instructions based on the third level of correction. Among them, the triggering conditions for each level of correction are positively correlated with the correction intensity, and when the same model deviates continuously in the same task, the invalid correction level that has been used is automatically skipped.

7. A computing device, characterized in that, include: A processor, a memory storing a computer program, wherein the computer program, when executed by the processor, performs the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The system stores instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 5.

Images