Task generation method and device, task deployment method and device, electronic equipment and storage medium
By generating tasks through a large language model and template engine, combined with reinforcement learning and phased release, the problems of low efficiency and accuracy in traditional task generation are solved, and intelligent task generation and deployment are achieved.
Patent Information
- Application Number
- CN202510741176.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-04
- Publication Date
- 2025-09-12
AI Technical Summary
Traditional task generation methods require developers to have programming skills and spend a lot of time writing and debugging DAG files. They are prone to errors, have high maintenance costs, and have a low level of intelligence.
By obtaining user task requirements, using a large language model to complete and extract structured data, generating tasks based on a template engine, combining abstract syntax trees and dependency graph verification, and using a reinforcement learning model and a grayscale release mechanism for task deployment.
It realizes the intelligence of task generation, reduces manual errors and costs, improves generation efficiency and accuracy, and ensures task safety and compliance.
Smart Images
Figure CN120631337A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of big data technology, and in particular to a task generation method, deployment method, device, electronic device and computer-readable storage medium. Background Art
[0002] Traditional task generation methods require developers to manually write DAG (Directed Acyclic Graph) files (Python scripts) to define the task's logic, dependencies, and scheduling rules. Consequently, these traditional task generation methods present the following technical challenges: ① They require developers to possess certain programming skills and consume a significant amount of time to write and debug DAG files, resulting in low development efficiency; ② Manually writing DAG files is prone to errors, especially when task dependencies are complex; and ③ When business requirements change, the corresponding DAG files need to be modified, resulting in high maintenance costs.
[0003] In summary, the task generation in the traditional task generation method has a low level of intelligence and relies heavily on manual labor. Therefore, the task generation cost is high, and the efficiency and accuracy are not high. Summary of the Invention
[0004] The technical problem to be solved by the present invention is to address the above-mentioned deficiencies in the prior art and to provide a task generation method, deployment method, device, electronic device and computer-readable storage medium, which can realize intelligent task generation.
[0005] In a first aspect, the present invention provides a task generation method, comprising: obtaining a user's task requirements; inputting the task requirements into a large language model to complete and extract the task requirements, and outputting structured data corresponding to the task requirements; and generating a task corresponding to the task requirements based on a template engine and the structured data corresponding to the task requirements.
[0006] Preferably, the structured data includes task name, task type, task parameters, task dependencies and task scheduling strategy. Based on the structured data corresponding to the template engine and task requirements, tasks corresponding to the task requirements are generated, specifically including: matching the corresponding directed acyclic graph DAG template based on the template engine, task name, task type, task dependencies and task scheduling strategy, and injecting the task parameters into the DAG template to obtain the task corresponding to the task requirements.
[0007] Preferably, after generating the task corresponding to the task requirement based on the template engine and the structured data corresponding to the task requirement, the task generation method also includes: generating an abstract syntax tree AST of the task, and judging whether the AST of the task conforms to the preset grammatical rules; constructing a dependency graph of the task, and judging whether there is a circular abnormal dependency in the dependency graph of the task; inputting the task requirement and the task into the large language model for consistency evaluation, and judging whether the consistency meets the first preset situation.
[0008] In the second aspect, the present invention provides a task deployment method, including: obtaining the scheduling characteristics of the target task and at least two queues at the current moment, wherein the target task is obtained by the task generation method described in the first aspect; based on the scheduling characteristics of at least two queues at the current moment, determining the target queue of the target task, wherein the scheduling characteristics include at least one of the following: the number of running tasks, CPU / memory utilization, the number and type distribution of tasks to be processed, the average completion time of tasks within a preset time, and the proportion of timed-out tasks; based on the grayscale release mechanism, publishing the target task to the target queue to deploy the target task.
[0009] Preferably, after determining the target queue of the target task based on the scheduling characteristics of at least two queues at the current moment, and before publishing the target task to the target queue based on the grayscale release mechanism, the task deployment method also includes: determining the scheduling parameters of the target queue at the current moment based on the reinforcement learning model and the scheduling characteristics of the target queue at the current moment, wherein the scheduling parameters include: the maximum number of parallel tasks, task routing rules and resource preemption rules.
[0010] Preferably, the reinforcement learning model includes a state space, an action space and a reward function. Based on the reinforcement learning model and the scheduling characteristics of the target queue at the current moment, the scheduling parameters of the target queue at the current moment are determined, specifically including: matching the state and action of the target queue corresponding to the scheduling characteristics from the state space and the action space respectively; calculating the reward for performing the action under the state of the target queue based on the reward function; and determining the action corresponding to the maximum value in the reward as the scheduling parameter of the target queue at the current moment.
[0011] Preferably, before matching the state and action of the target queue corresponding to the scheduling characteristics from the state space and action space, the task deployment method also includes: obtaining the scheduling characteristics of the target queue at the historical moment; determining N scheduling parameters of the target queue at the historical moment based on the scheduling characteristics of the target queue at the historical moment, where N represents a positive integer; determining the scheduling characteristics and N scheduling parameters of the target queue at the historical moment as the state and action of the target queue respectively; and summarizing the state and action of the target queue respectively to obtain the state space and action space.
[0012] Preferably, the target queue includes the target original queue and its corresponding grayscale queue. Based on the grayscale release mechanism, the target task is released to the target queue to deploy the target task, specifically including: releasing the target task to the grayscale queue corresponding to the target original queue, and monitoring the running data of the target task on the grayscale queue; judging whether the running data of the target task on the grayscale queue meets the second preset situation; in response to the running data meeting the second preset situation, releasing the target task to the target original queue to deploy the target task.
[0013] In the third aspect, the present invention also provides a task generation device, including a first acquisition module, an extraction module and a generation module. The first acquisition module is used to obtain the user's task requirements. The extraction module is connected to the first acquisition module and is used to input the task requirements into a large model to complete and extract the task requirements, and output structured data corresponding to the task requirements. The generation module is connected to the extraction module and is used to generate tasks corresponding to the task requirements based on the template engine and the structured data corresponding to the task requirements.
[0014] In the fourth aspect, the present invention also provides a task deployment device, including a second acquisition module, a first determination module and a publishing module, the second acquisition module is used to obtain the scheduling characteristics of the target task and at least two queues at the current moment, wherein the target task is generated by the task generation device described in the third aspect, the first determination module is connected to the second acquisition module, and is used to determine the target queue of the target task based on the scheduling characteristics of at least two queues at the current moment, wherein the scheduling characteristics include at least one of the following: the number of currently running tasks, CPU / memory utilization, the number and type distribution of tasks to be processed, the average completion time of tasks within a preset time, and the proportion of timed tasks; the publishing module is connected to the first determination module, and is used to publish the target task to the target queue based on the grayscale publishing mechanism to deploy the target task.
[0015] In a fifth aspect, the present invention also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to implement the task generation method provided in the first aspect or the task deployment method provided in the second aspect.
[0016] In a sixth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for generating tasks provided in the first aspect or the method for deploying tasks provided in the second aspect is implemented.
[0017] The present invention provides a task generation method, deployment method, device, electronic device, and computer-readable storage medium. By inputting task requirements into a large language model, guiding the large language model to output complete and accurate structured data, and rapidly integrating the structured data with the DAG template code corresponding to the task requirements in a template engine, the method enhances the intelligent level of task generation, reduces the errors and costs of manually generated tasks, and significantly improves the efficiency and accuracy of task generation. Therefore, the present invention enables intelligent task generation. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 This is a flowchart of a method for generating a task according to embodiment 1 of the present invention;
[0019] Figure 2 This is an example diagram of inputting task requirements into a large language model to complete and extract task requirements in Example 1 of the present invention;
[0020] Figure 3 This is an example diagram of the dependency graph of tasks in Example 1 of the present invention;
[0021] Figure 4 This is a flowchart of a method for deploying a task according to embodiment 2 of the present invention;
[0022] Figure 5 This is a flowchart of a method for deploying a task according to embodiment 3 of the present invention;
[0023] Figure 6 This is a schematic diagram of the structure of a task generation device according to Example 4 of the present invention;
[0024] Figure 7 This is a structural diagram of a task deployment device according to Example 5 of the present invention. DETAILED DESCRIPTION
[0025] In order to enable those skilled in the art to better understand the technical solutions of the present invention, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.
[0026] It should be understood that the specific embodiments and drawings described herein are only used to explain the present invention rather than to limit the present invention.
[0027] It is understood that, in the absence of conflict, the various embodiments of the present invention and the various features in the embodiments may be combined with each other.
[0028] It can be understood that, for the convenience of description, the drawings of the present invention only show parts related to the present invention, while parts unrelated to the present invention are not shown in the drawings.
[0029] It can be understood that each unit and module involved in the embodiments of the present invention may correspond to only one physical structure, or may be composed of multiple physical structures, or multiple units and modules may be integrated into one physical structure.
[0030] It will be understood that, without conflict, the functions and steps marked in the flowcharts and block diagrams of the present invention may occur in an order different from that marked in the drawings.
[0031] It is understood that the flowcharts and block diagrams of the present invention illustrate the possible architectures, functions, and operations of the systems, devices, equipment, and methods according to various embodiments of the present invention. Each box in the flowchart or block diagram may represent a unit, module, program segment, or code, which contains executable instructions for implementing the specified functions. Moreover, each box or combination of boxes in the block diagram and flowchart may be implemented using a hardware-based system that implements the specified functions, or may be implemented using a combination of hardware and computer instructions.
[0032] It can be understood that the units and modules involved in the embodiments of the present invention can be implemented by software or hardware. For example, the units and modules can be located in a processor.
[0033] Example 1:
[0034] like Figure 1 As shown, this embodiment provides a method for generating a task.
[0035] In this embodiment, a task refers to a specific execution unit defined by a workflow scheduling platform. This specific execution unit contains the code or commands required to complete a specific operation. Workflow scheduling platforms include, but are not limited to, Apache Airflow. Apache Airflow is a commonly used open source workflow scheduling platform used for programmatically defining, scheduling, and monitoring workflows. Therefore, this example uses an Airflow task as an example.
[0036] The task generation method includes:
[0037] S101, obtaining the user's task requirements.
[0038] In this embodiment, the user describes the task requirements in natural language, for example: "Execute the data backup task at 2:00 a.m. every day, the backup path is / data / backup, and send an email to notify the administrator after the backup is completed."
[0039] S102: Input the task requirements into the large language model to complete and extract the task requirements, and output structured data corresponding to the task requirements.
[0040] Specifically, the structured data includes task name, task type, task parameters, task dependencies and task scheduling strategy.
[0041] In this embodiment, Figure 2 As shown, the task requirements are input into the large language model for task requirement completion and extraction, specifically including: inputting the task requirements into the large language model; using the large language model to extract the task requirements of the task requirements input by the user, and extracting the structured data corresponding to the task requirements, such as: task name: data_backup, task type: data backup, task parameters: backup path / data / backup, task dependency: none, task scheduling strategy: execute at 2 am every day, and send an email notification after completion; monitoring whether the task requirements are fuzzy requirements (i.e., Figure 2 In the judgment of whether the task requirements meet the constraints, where the constraints include but are not limited to: whether the output result expectations are specific and clear); if the large language model detects that the task requirements are fuzzy, it will clarify them through dialogue (i.e. Figure 2 The agent prompts the user to re-enter the task requirements in the task request window to complete the task requirements and obtain the missing parameters of the structured data. For example, if it detects that retries (retry strategy) are not specified, the agent asks: "It is detected that the task does not define a failure retry strategy. Do you want to use the default strategy (exponential backoff)?"
[0042] It should be noted that the format of structured data is usually JSON format, for example:
[0043]
[0044] After inputting the task requirements into the large language model to complete and extract the task requirements, this embodiment further includes: collecting historical data of task requirement completion and extraction, and fine-tuning the large language model based on LoRa technology and historical data to ensure that the output results of the large language model are consistent with the historical data. The historical data includes but is not limited to: DAG (Directed Acyclic Graph) files of historically run Airflow tasks. This embodiment collects and organizes the DAG files of historically run Airflow tasks and uses them as knowledge base context for fine-tuning the large model.
[0045] S103: Generate a task corresponding to the task requirement based on the template engine and the structured data corresponding to the task requirement.
[0046] In this embodiment, template engines include, but are not limited to, the Django template engine and the Jinja2 template engine. The Jinja2 template engine's design philosophy is derived from the Django template engine, extending its syntax and offering a range of powerful features. Among these features, it adds sandbox execution and optional automatic escaping, significantly contributing to application security. Therefore, this embodiment uses the Jinja2 template engine as an example.
[0047] Specifically, S103: Based on the structured data corresponding to the template engine and the task requirements, generate tasks corresponding to the task requirements, including: matching the corresponding directed acyclic graph DAG template based on the template engine, task name, task type, task dependency and task scheduling strategy, and injecting task parameters into the DAG template to obtain the task corresponding to the task requirements.
[0048] In this embodiment, Apache Airflow defines tasks and their dependencies through DAGs and provides a rich user interface for managing and monitoring task execution. Therefore, this embodiment pre-configures DAG templates with different task names, task types, task dependencies, and task scheduling strategies in the jinja2 template engine. Subsequently, the corresponding DAG templates can be directly matched and task parameters can be injected into the DAG template to generate a DAG file that complies with the Airflow 2.0 specification (i.e., the tasks corresponding to the task requirements). For example:
[0049]
[0050]
[0051] backup_task>>email_task. This embodiment separates task parameters from DAG code. Dynamic task mapping and expansion can be achieved by simply updating the task parameters of structured data. This improves task maintainability and enables the generation and execution of multiple task instances by writing a DAG template once, avoiding code duplication. This is particularly suitable for parameterized task scenarios such as batch processing and multi-environment deployment.
[0052] It's important to note that the most important Airflow tasks are the various Airflow Operators. These allow you to generate various types of tasks. For example, you can use DataXOperator or SQLOperator to execute data exchange tasks, or use PySparkOperator or SqlOperator to execute data transformation logic and generate data conversion tasks. Therefore, different types of DAG templates correspond to different types of Airflow Operators.
[0053] Optionally, after S103: generating a task corresponding to the task requirement based on the template engine and the structured data corresponding to the task requirement, the task generation method further includes:
[0054] S104: Generate an abstract syntax tree (AST) of the task, and determine whether the AST of the task complies with preset syntax rules.
[0055] In this embodiment, the preset grammatical rules are as follows: ① a valid start_date parameter is included; ② the Bash command tar -czf ... contains no dangerous characters; ③ the EmailOperator correctly sets the to address.
[0056] S105: Build a dependency graph of the task and determine whether there is a circular abnormal dependency in the dependency graph of the task.
[0057] In this embodiment, the dependency graph of the task is as follows: Figure 3 As shown in the figure, the task is a simple linear dependency: backup_data→send_email, with no cyclic path. Therefore, there is no cyclic abnormal dependency in the task dependency graph.
[0058] S106: Input the task requirements and the task into the large language model for consistency evaluation, and determine whether the consistency meets a first preset condition.
[0059] In this embodiment, task requirements and task consistency assessments are shown in Table 1. This embodiment uses a three-tiered verification mechanism consisting of an AST (Abstract Syntax Tree), a dependency graph, and a large language model to perform syntax checks, logic checks, and parameter checks on tasks, ensuring that tasks conform to expected business logic and scheduling rules, ensuring task security and compliance, and further reducing risks.
[0060] Table 1 Task requirements and task consistency evaluation
[0061] Task Requirements Task consistency Execution time 0 2*** √ Backup Path / data / backup √ Email Notification EmailOperator √ Triggered after backup >>Dependency √ recipient admin@example.com √
[0062] It should be noted that after the three-tier verification mechanism completes the DAG file (i.e., task) check, it is necessary to confirm with the user again whether the DAG file (i.e., task) meets expectations. If it does not meet expectations, the user needs to modify it and then verify it again until the user confirms that there is no problem.
[0063] After generating a task corresponding to the task requirement based on the template engine and the structured data corresponding to the task requirement, this embodiment further includes: uploading the target task (i.e., the task corresponding to the generated task requirement) to the DAG directory of Airflow, and configuring the target task accordingly so that the target task can be correctly identified and deployed by Airflow, wherein the configuration includes but is not limited to: configuring a target queue for the target task and scheduling parameters of the target queue, and the scheduling parameters include but are not limited to: the maximum number of parallel tasks, task routing rules, and resource preemption rules.
[0064] The target task is configured accordingly, specifically including: ① obtaining the number of running tasks, CPU (Central Processing Unit, central processing unit) / memory usage, number and type distribution of pending tasks, average task completion time within the preset time, and proportion of timeout tasks of at least two queues (for example, three queues: high_priority, medium_priority, low_priority) at the current moment; ② matching the queues whose task types include the target task in the number and type distribution of pending tasks to obtain a first queue set, and sorting the first queue set based on the number of running tasks, CPU / memory usage, average task completion time within the preset time, and proportion of timeout tasks, to obtain the sorting results corresponding to the number of running tasks, CPU / memory usage, average task completion time within the preset time, and proportion of timeout tasks; ③ selecting the repeated queue from the first M queues of the sorting results corresponding to the number of running tasks, CPU / memory usage, average task completion time within the preset time, and proportion of timeout tasks as the target queue, where M represents a positive integer; ④ matching the number of running tasks, C respectively from the state space and action space of the reinforcement learning model. ⑤ The state and action of the target queue corresponding to the PU / memory usage rate, the number and type distribution of pending tasks, the average completion time of tasks within the preset time, and the proportion of timeout tasks; ⑤ Based on the reward function of the reinforcement learning model, for example: reward = (completed_tasks*0.5)-(timeout_tasks*2)-starvation_penalty*10), calculate the reward for performing an action under the state of the target queue, where reward represents the reward for performing an action under the state of the target queue, completed_tasks represents the task completion rate of the target queue, timeout_tasks represents the task timeout rate of the target queue, and starvation_penalty represents the process starvation penalty of the target queue; ⑥ The action corresponding to the maximum value of the reward is determined as the scheduling parameter of the target queue at the current moment.
[0065] Before matching the state and action of the target queue corresponding to the number of running tasks, CPU / memory usage, number and type distribution of pending tasks, average task completion time within a preset time, and proportion of timed-out tasks in the state space and action space of the reinforcement learning model, this embodiment further includes: using the number of running tasks, CPU / memory usage, number and type distribution of pending tasks, average task completion time within a preset time, and proportion of timed-out tasks of at least two queues at historical moments as the state of at least two queues, for example: state = {'high_priority': {'running': 4, 'pending': 2, 'cpu_usage': 80%}, 'medium_priority': {'running': 8, 'pending': 15, 'mem_usage': 65%}, 'low_priority': {'running': 12, 'pending': 30}, 'avg_completion_time': 142s, 'time out_rate': 5.2%}; based on the number of running tasks, CPU / memory usage, number and type distribution of pending tasks, average task completion time within a preset time, and proportion of timed-out tasks of at least two queues at historical moments, U / memory usage, number and type distribution of pending tasks, average task completion time within a preset time, and the proportion of timed-out tasks define the maximum number of N parallel tasks for at least two queues at a historical moment (e.g., high_priority: 5→6, medium_priority: 10→12, low_priority: 15→18), task routing rules (e.g., high_priority:'rout e_rules':{'realtime_task_threshold': 50ms}), and resource preemption rules (e.g., high_priority:'enable_preemption': True). This embodiment determines and adjusts the scheduling parameters of the target queue through a reinforcement learning model, so that the deployment of the target task adapts to the load changes of the target queue in real time, reducing queue resource waste and manual intervention costs.
[0066] After configuring the target task accordingly, this implementation also includes: based on a grayscale release mechanism, such as the Canary mechanism, publishing the target task to the target queue to deploy the target task. This embodiment uses the Canary mechanism to pre-deploy the task to the test cluster for operation, verify the task's running data on the test cluster, and synchronize the task to the production environment after verification. This allows for timely detection of potential problems, avoids the risk of failure caused by a full release, improves the security and compliance of task deployment, ensures the stability of the production environment through progressive deployment, and achieves efficient operation and maintenance by combining cloud native capabilities, reducing version rollback time from hours to minutes.
[0067] The task generation method provided in this embodiment improves the intelligence level of task generation, reduces the errors and costs of manually generated tasks, and significantly improves the efficiency and accuracy of task generation, thereby realizing intelligent task generation, by inputting task requirements into a large language model, guiding the large language model to output complete and accurate structured data, and quickly integrating the structured data with the DAG template code corresponding to the task requirements in the template engine.
[0068] Example 2:
[0069] like Figure 4 As shown, this embodiment provides a method for deploying a task. The method for deploying a task includes:
[0070] S201 , obtaining a target task and scheduling characteristics of at least two queues at the current moment, wherein the target task is obtained by the task generation method of Example 1.
[0071] In this embodiment, obtaining the target task specifically includes: the user describes the task requirements in natural language, for example: "Perform data backup task at 2 a.m. every day, the backup path is / data / backup, and send an email to notify the administrator after the backup is completed." The task requirements are input into the large language model; the large language model is used to extract the task requirements input by the user, and the structured data corresponding to the task requirements are extracted, for example: task name: data_backup, task type: data backup, task parameters: backup path / data / backup, task dependency: none, task scheduling strategy: execute at 2 a.m. every day, and send an email notification after completion; the agent framework is used to monitor whether the task requirements are fuzzy requirements (i.e. Figure 2 In the judgment of whether the task requirements meet the constraints, where the constraints include but are not limited to: whether the output result expectations are specific and clear); if the large language model detects that the task requirements are fuzzy, it will clarify them through dialogue (i.e. Figure 2 The system prompts the user to re-enter the task requirements) to complete the task requirements to obtain the missing parameters of the structured data. For example, if it is detected that retries (retry strategy) are not specified, the Agent asks: It is detected that the task has not defined a failure retry strategy. Do you want to use the default strategy (exponential backoff)? By pre-configuring DAG templates under various task names, task types, task dependencies and task scheduling strategies in the jinja2 template engine, the corresponding DAG templates can be directly matched later, and the task parameters can be injected into the DAG template to generate a DAG file that complies with the Airflow 2.0 specification (that is, the task corresponding to the task requirements). Through the three-layer verification mechanism of AST, dependency graph and large language model, the tasks are checked for syntax, logic and parameters to ensure that the tasks comply with the expected business logic and scheduling rules, ensure the security and compliance of the tasks, and further reduce risks.
[0072] S202, determining a target queue for a target task based on the scheduling characteristics of at least two queues at the current moment, wherein the scheduling characteristics include at least one of the following: the number of running tasks, CPU / memory usage, the number and type distribution of pending tasks, the average completion time of tasks within a preset time, and the proportion of timed-out tasks.
[0073] In this embodiment, queues containing the target task in the number and type distribution of tasks to be processed are matched to obtain a first queue set, and the first queue set is sorted based on the number of running tasks, CPU / memory usage, average task completion time within a preset time, and the proportion of timed tasks to obtain sorting results corresponding to the number of running tasks, CPU / memory usage, average task completion time within a preset time, and the proportion of timed tasks; the repeated queues in the first M queues of the sorting results corresponding to the number of running tasks, CPU / memory usage, average task completion time within a preset time, and the proportion of timed tasks are selected as the target queue, where M represents a positive integer.
[0074] S203: Based on the grayscale release mechanism, the target task is released to the target queue to deploy the target task.
[0075] In this embodiment, the grayscale release mechanism includes but is not limited to: Canary mechanism.
[0076] Optionally, after S202: determining a target queue for a target task based on the scheduling characteristics of at least two queues at the current moment, and before S203: publishing the target task to the target queue based on a grayscale release mechanism to deploy the target task, the task deployment method further includes:
[0077] S204 , determining scheduling parameters of the target queue at the current moment based on the reinforcement learning model and the scheduling characteristics of the target queue at the current moment, wherein the scheduling parameters include: a maximum number of parallel tasks, a task routing rule, and a resource preemption rule.
[0078] Specifically, the reinforcement learning model includes a state space, an action space, and a reward function.
[0079] Specifically, S204: determining the scheduling parameters of the target queue at the current moment based on the reinforcement learning model and the scheduling characteristics of the target queue at the current moment, including steps S2041 to S2043:
[0080] S2041 , matching the state and action of the target queue corresponding to the scheduling feature from the state space and action space respectively.
[0081] S2042: Calculate the reward for executing the action in the state of the target queue based on the reward function.
[0082] In this embodiment, a reward function based on a reinforcement learning model, for example, reward = (completed_tasks*0.5)-(timeout_tasks*2)-starvation_penalty*10), is used to calculate the reward for performing an action in the state of the target queue, where reward represents the reward for performing an action in the state of the target queue, completed_tasks represents the task completion rate of the target queue, timeout_tasks represents the task timeout rate of the target queue, and starvation_penalty represents the process starvation penalty of the target queue.
[0083] S2043: Determine the action corresponding to the maximum value of the reward as the scheduling parameter of the target queue at the current moment.
[0084] Optionally, before S2041: matching the state and action of the target queue corresponding to the scheduling feature from the state space and the action space, the task deployment method further includes steps S2044 to S2047:
[0085] S2044: Obtain the scheduling characteristics of the target queue at a historical moment.
[0086] S2045 : Determine N scheduling parameters of the target queue at the historical moment based on the scheduling characteristics of the target queue at the historical moment, where N represents a positive integer.
[0087] S2046 : Determine the scheduling characteristics and N scheduling parameters of the target queue at a historical moment as the state and action of the target queue respectively.
[0088] S2047 , respectively summarize the states and actions of the target queue to obtain a state space and an action space.
[0089] In this embodiment, the number of running tasks, CPU / memory usage, number and type distribution of pending tasks, average task completion time within a preset time, and timeout task ratio of at least two queues at historical moments are used as the status of at least two queues, for example: e={'high_priority':{'running':4,'pending':2,'cpu_usage':80%},'medium_priority':{'running':8,'pending':15,'mem_usage':65%},'low_priority':{'running':12,'pending':30},'avg_completion_time':142s,'timeout_rate':5.2%};Based on the number of running tasks, CPU / memory usage, number and type distribution of pending tasks, average task completion time within the preset time, and timeout task ratio of at least two queues at historical moments, define the maximum number of N parallel tasks of at least two queues at historical moments (for example: high_priority:5→6, medium_priority:10→12, low_priority priority:15→18), task routing rules (e.g., high_priority:'route_rules':{'realtime_task_threshold':50ms}), and resource preemption rules (e.g., high_priority:'enable_preemption':True). This embodiment uses a reinforcement learning model to determine and adjust the scheduling parameters of the target queue, so that the deployment of target tasks adapts to the load changes of the target queue in real time, reducing queue resource waste and manual intervention costs.
[0090] Specifically, S203: Based on the grayscale release mechanism, the target task is released to the target queue to deploy the target task, including steps S2031 to S2033:
[0091] S2031: Publish the target task to the grayscale queue corresponding to the target original queue, and monitor the running data of the target task on the grayscale queue.
[0092] S2032: Determine whether the running data of the target task in the grayscale queue meets a second preset condition.
[0093] S2033: In response to the running data satisfying the second preset condition, the target task is published to the target original queue to deploy the target task.
[0094] In this embodiment, taking the Canary mechanism as an example of a grayscale release mechanism, the task is pre-deployed on the test cluster through the Canary mechanism to verify the running data of the task on the test cluster. After the verification is passed, the task is synchronized to the production environment to timely discover potential problems and avoid the risk of failure caused by full release. The security and compliance of task deployment are improved, and the stability of the production environment is guaranteed by progressive deployment. Efficient operation and maintenance are achieved by combining cloud native capabilities, and the version rollback time is shortened from hours to minutes.
[0095] The task deployment method provided in this embodiment determines and adjusts the target queue of the target task through a reinforcement learning model, so that the deployment of the target task adapts to the load changes of the target queue in real time, reducing queue resource waste and manual intervention costs; through the grayscale release mechanism, the target task is deployed progressively, potential problems are discovered in time, and the risk of failure caused by full release is avoided, thereby improving the security and compliance of task deployment and realizing intelligent task deployment.
[0096] Example 3:
[0097] like Figure 5 As shown, this embodiment provides a method for deploying a task. The method for deploying a task includes:
[0098] S301, obtaining the user's task requirements.
[0099] In this embodiment, the user's task requirements are obtained. Figure 5 The task requirements input in .
[0100] S302: Input the task requirements into the large language model to complete and extract the task requirements, and output structured data corresponding to the task requirements.
[0101] In this embodiment, task requirement completion and extraction Figure 5 Requirements analysis in .
[0102] S303: Generate a task corresponding to the task requirement based on the template engine and the structured data corresponding to the task requirement.
[0103] S304, generate the abstract syntax tree AST of the task, and determine whether the AST of the task conforms to the preset grammatical rules; construct the dependency graph of the task, and determine whether there is a circular abnormal dependency in the dependency graph of the task; input the task requirements and the task into the large language model for consistency evaluation, and determine whether the consistency meets the first preset situation.
[0104] In this embodiment, it is determined whether the AST of the task conforms to the preset grammatical rules, whether there is a circular abnormal dependency in the dependency graph of the task, and whether the consistency meets the first preset condition, i.e. Figure 5 Verification of tasks in .
[0105] S305, obtain the scheduling characteristics of the task and at least two queues at the current moment; determine the target queue of the task based on the scheduling characteristics of at least two queues at the current moment, wherein the scheduling characteristics include at least one of the following: the number of running tasks, CPU / memory utilization, the number and type distribution of pending tasks, the average completion time of tasks within a preset time, and the proportion of timed-out tasks; based on the grayscale release mechanism, release the task to the target queue to deploy the task.
[0106] The task deployment method provided in this embodiment inputs task requirements into a large language model, guides the large language model to output complete and accurate structured data, and quickly integrates the structured data with the DAG template code corresponding to the task requirements in the template engine, thereby improving the intelligence level of task generation, reducing the errors and costs of manually generated tasks, and thus greatly improving the efficiency and accuracy of task generation; through the reinforcement learning model, the target queue of the target task is determined and adjusted, so that the deployment of the target task adapts to the load changes of the target queue in real time, reducing queue resource waste and manual intervention costs; through the grayscale release mechanism, the target task is deployed progressively, potential problems are discovered in time, and the risk of failure caused by full release is avoided, thereby improving the security and compliance of task deployment and realizing intelligent task generation and deployment.
[0107] Example 4:
[0108] like Figure 6 As shown, this embodiment provides a task generation device, including a first acquisition module 41, an extraction module 42 and a generation module 43. The first acquisition module 41 is used to obtain the user's task requirements. The extraction module 42 is connected to the first acquisition module 41 and is used to input the task requirements into a large model to complete and extract the task requirements, and output structured data corresponding to the task requirements. The generation module 43 is connected to the extraction module 42 and is used to generate tasks corresponding to the task requirements based on the template engine and the structured data corresponding to the task requirements.
[0109] Specifically, the generation module 43 includes: a generation unit 431, which is used to match the corresponding directed acyclic graph DAG template based on the template engine, task name, task type, task dependency and task scheduling strategy, and inject task parameters into the DAG template to obtain the task corresponding to the task requirement.
[0110] Optionally, the task generation device also includes: a first judgment module 44, a second judgment module 45 and a third judgment module 46. The first judgment module 44 is used to generate an abstract syntax tree AST of the task and judge whether the AST of the task conforms to the preset grammatical rules. The second judgment module 45 is used to construct a dependency graph of the task and judge whether there is a circular abnormal dependency in the dependency graph of the task. The third judgment module 46 is used to input the task requirements and tasks into the large language model for consistency evaluation and judge whether the consistency meets the first preset situation.
[0111] It can be understood that the task generation device provided above executes the task generation method corresponding to the embodiment 1 provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the solution corresponding to the task generation method of the embodiment 1 above, and will not be repeated here.
[0112] Example 5:
[0113] like Figure 7 As shown, this embodiment provides a task deployment device, including a second acquisition module 51, a first determination module 52 and a publishing module 53, the second acquisition module 51 is used to obtain the target task and the scheduling characteristics of at least two queues at the current moment, wherein the target task is obtained by the task generation device of Example 4, the first determination module 52 is connected to the second acquisition module 51, and is used to determine the target queue of the target task based on the scheduling characteristics of at least two queues at the current moment, wherein the scheduling characteristics include at least one of the following: the number of currently running tasks, CPU / memory utilization, the number and type distribution of tasks to be processed, the average completion time of tasks within a preset time, and the proportion of timed tasks, the publishing module 53 is connected to the first determination module 52, and is used to publish the target task to the target queue based on the grayscale publishing mechanism to deploy the target task.
[0114] Optionally, the task generation device also includes: a second determination module 54, which is used to determine the scheduling parameters of the target queue at the current moment based on the reinforcement learning model and the scheduling characteristics of the target queue at the current moment, wherein the scheduling parameters include: the maximum number of parallel tasks, task routing rules and resource preemption rules.
[0115] Specifically, the second determination module 54 includes: a matching unit 541, a calculation unit 542 and a first determination unit 543. The matching unit 541 is used to match the state and action of the target queue corresponding to the scheduling feature from the state space and the action space respectively. The calculation unit 542 is used to calculate the reward for performing the action under the state of the target queue based on the reward function. The first determination unit 543 is used to determine the action corresponding to the maximum value in the reward as the scheduling parameter of the target queue at the current moment.
[0116] Optionally, the second determination module 54 also includes: an acquisition unit 544, a second determination unit 545, a third determination unit 546 and a summary unit 547, the acquisition unit 544 is used to obtain the scheduling characteristics of the target queue at a historical moment, the second determination unit 545 is used to determine N scheduling parameters of the target queue at a historical moment based on the scheduling characteristics of the target queue at a historical moment, where N represents a positive integer, the third determination unit 546 is used to determine the scheduling characteristics and N scheduling parameters of the target queue at a historical moment as the state and action of the target queue, respectively, and the summary unit 547 is used to summarize the state and action of the target queue, respectively, to obtain a state space and an action space.
[0117] Specifically, the publishing module 53 includes: a first publishing unit 531, a judgment unit 532 and a second publishing unit 533. The first publishing unit 531 is used to publish the target task to the grayscale queue corresponding to the target original queue, and monitor the running data of the target task on the grayscale queue. The judgment unit 532 is used to judge whether the running data of the target task on the grayscale queue meets the second preset situation. The second publishing unit 533 is used to publish the target task to the target original queue in response to the running data meeting the second preset situation to deploy the target task.
[0118] It can be understood that the task deployment device provided above executes the task deployment method corresponding to the embodiment 2 provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the solution corresponding to the task deployment method of embodiment 2 above, and will not be repeated here.
[0119] Example 6:
[0120] This embodiment provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to implement the task generation method in the above-mentioned embodiment 1 or the task deployment method in embodiment 2 or embodiment 3.
[0121] Example 7:
[0122] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for generating tasks in the above-mentioned embodiment 1 or the method for deploying tasks in embodiment 2 or embodiment 3 is implemented.
[0123] It will be understood that the above embodiments are merely exemplary embodiments for illustrating the principles of the present invention, and the present invention is not limited thereto. Those skilled in the art will appreciate that various modifications and improvements can be made without departing from the spirit and substance of the present invention, and such modifications and improvements are also considered to be within the scope of protection of the present invention.
Claims
1. A method for generating a task, characterized in that: include: Obtain user task requirements; Input the task requirements into the large language model to complete and extract the task requirements, and output the structured data corresponding to the task requirements; Generate tasks corresponding to the task requirements based on the template engine and structured data corresponding to the task requirements.
2. The task generation method according to claim 1, characterized in that: The structured data includes task name, task type, task parameters, task dependencies and task scheduling strategy. The generating of tasks corresponding to the task requirements based on the template engine and the structured data corresponding to the task requirements specifically includes: Based on the template engine, task name, task type, task dependency and task scheduling strategy, the corresponding directed acyclic graph (DAG) template is matched, and the task parameters are injected into the DAG template to obtain the task corresponding to the task requirement.
3. The task generation method according to claim 1, characterized in that: After generating the task corresponding to the task requirement based on the template engine and the structured data corresponding to the task requirement, the method further includes: Generate the abstract syntax tree AST of the task and determine whether the AST of the task conforms to the preset grammatical rules; Build a dependency graph for the task and determine whether the task dependency graph contains any abnormal circular dependencies. The task requirements and tasks are input into the large language model for consistency evaluation, and it is determined whether the consistency meets the first preset condition.
4. A task deployment method, characterized in that: include: Obtaining a target task and scheduling characteristics of at least two queues at a current moment, wherein the target task is obtained by the task generation method according to any one of claims 1 to 3; Determine a target queue for the target task based on the current scheduling characteristics of at least two queues, where the scheduling characteristics include at least one of the following: number of running tasks, CPU / memory usage, number and type distribution of pending tasks, average task completion time within a preset time, and proportion of timed-out tasks; Based on the grayscale release mechanism, the target task is published to the target queue to deploy the target task.
5. The task deployment method according to claim 4, characterized in that: After determining the target queue of the target task based on the scheduling characteristics of the at least two queues at the current moment, and before publishing the target task to the target queue based on the grayscale publishing mechanism, the method further includes: Based on the reinforcement learning model and the scheduling characteristics of the target queue at the current moment, the scheduling parameters of the target queue at the current moment are determined, where the scheduling parameters include: a maximum number of parallel tasks, a task routing rule, and a resource preemption rule.
6. The task deployment method according to claim 5, characterized in that: The reinforcement learning model includes a state space, an action space, and a reward function. The step of determining the scheduling parameters of the target queue at the current moment based on the reinforcement learning model and the scheduling characteristics of the target queue at the current moment specifically includes: Match the state and action of the target queue corresponding to the scheduling feature from the state space and action space respectively; Based on the reward function, calculate the reward for executing the action in the state of the target queue; The action corresponding to the maximum value of the reward is determined as the scheduling parameter of the target queue at the current moment.
7. The task deployment method according to claim 6, characterized in that: Before matching the state and action of the target queue corresponding to the scheduling feature from the state space and action space, the method further includes: Get the scheduling characteristics of the target queue at a historical moment; Determine N scheduling parameters of the target queue at the historical moment based on the scheduling characteristics of the target queue at the historical moment, where N represents a positive integer; The scheduling characteristics and N scheduling parameters of the target queue at a historical moment are respectively determined as the state and action of the target queue; Summarize the states and actions of the target queue respectively to obtain the state space and action space.
8. The task deployment method according to claim 4, characterized in that: The target queue includes the target original queue and its corresponding grayscale queue. The target task is published to the target queue based on the grayscale release mechanism to deploy the target task, specifically including: Publish the target task to the grayscale queue corresponding to the target original queue, and monitor the running data of the target task on the grayscale queue; Determining whether the running data of the target task on the grayscale queue meets a second preset condition; In response to the running data satisfying a second preset condition, the target task is published to the target original queue to deploy the target task.
9. A task generating device, characterized in that: It includes a first acquisition module, an extraction module and a generation module, The first acquisition module is used to obtain the user's task requirements. The extraction module is connected to the first acquisition module and is used to input the task requirements into the large model to complete and extract the task requirements, and output the structured data corresponding to the task requirements. The generation module is connected with the extraction module and is used to generate tasks corresponding to the task requirements based on the template engine and the structured data corresponding to the task requirements.
10. A task deployment device, characterized in that: It includes a second acquisition module, a first determination module and a publishing module, The second acquisition module is configured to acquire the target task and the scheduling characteristics of at least two queues at the current moment, wherein the target task is generated by the task generation device according to claim 9. The first determination module is connected to the second acquisition module and is used to determine the target queue of the target task based on the scheduling characteristics of at least two queues at the current moment, wherein the scheduling characteristics include at least one of the following: the number of currently running tasks, CPU / memory usage, the number and type distribution of pending tasks, the average completion time of tasks within a preset time, and the proportion of timed tasks. The publishing module is connected to the first determining module and is used to publish the target task to the target queue based on the grayscale publishing mechanism to deploy the target task.
11. An electronic device, characterized in that: It includes a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to implement a task generation method as described in any one of claims 1 to 3 or a task deployment method as described in any one of claims 4 to 8.
12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the task generation method according to any one of claims 1 to 3 or the task deployment method according to any one of claims 4 to 8.