Task processing method, system and device and readable storage medium

By analyzing and optimizing the operation and maintenance scripts in the cloud operation and maintenance system, the problems of low execution efficiency and major security risks of traditional script automation are solved, and efficient and secure operation and maintenance script execution and resource management are achieved.

CN119025244APending Publication Date: 2024-11-26SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202411155656.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

Traditional operation and maintenance script automation has not been effectively solved in the problems of low execution efficiency and major security risks, and it is difficult to meet the needs of rapid response and efficient management.

Method used

By introducing script management module, task orchestration module and distributed scheduling engine in the cloud operation and maintenance system, the script parser is used to parse and optimize the execution script, identify and repair code defects, and compare and test the optimized scripts to ensure that the optimized scripts are optimized The script is effectively called between the task scheduler and the execution engine.

Benefits of technology

It improves script execution efficiency, reduces security risks, realizes dynamic scheduling and load balancing of resources, and improves operation and maintenance convenience and security.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025244A_ABST
    Figure CN119025244A_ABST
Patent Text Reader

Abstract

The invention discloses a task processing method, system and device and a readable storage medium, a cloud operation and maintenance system realizes the task processing method, a script management module in the cloud operation and maintenance system can analyze an execution script of a target task, find out defects and make up the defects through optimization, and the task processing efficiency is improved. And thus, the optimized target execution script corresponding to the target task is obtained. Therefore, the task arrangement module determines the engine list and the target execution script corresponding to the target task, and can send the engine list and the target execution script to the distributed scheduling engine; the executor executes the target execution footstep to process the target task. According to the method and the device, the execution script of the target task is detected and optimized, so that code defects in the execution script can be effectively removed, and when the target execution script is executed subsequently, the execution efficiency can be improved, and potential safety hazards can be reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer application technology, and in particular to a task processing method, system, device and readable storage medium. Background Art

[0002] As enterprise IT systems become increasingly complex, traditional manual operation and maintenance methods can no longer meet the needs of rapid response and efficient management. As an important means to improve operation and maintenance efficiency, operation and maintenance script automation still faces problems such as script defects in actual applications, resulting in low execution efficiency and many security risks.

[0003] In summary, how to effectively solve problems such as script optimization is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0004] The purpose of this application is to provide a task processing method, system, device and readable storage medium, which can effectively improve execution efficiency and reduce safety hazards by optimizing script defects.

[0005] In order to solve the above technical problems, this application provides the following technical solutions:

[0006] A task processing method is applied to a cloud operation and maintenance system, wherein the cloud operation and maintenance system includes a task scheduling center and an execution engine, wherein the task scheduling center includes a script management module, a task management module, a task scheduling module and a distributed scheduling engine, and the execution engine includes a register, an executor and a callback; the method includes:

[0007] The script management module obtains the execution script of the target task, uses a script parser to parse the execution script, obtains the constituent elements of the execution script, and uses the constituent elements to generate an abstract syntax tree; based on the abstract syntax tree, analyzes the execution logic in the execution script to determine code defects; optimizes the code defects, and performs a comparison test on the execution scripts before and after optimization, and determines the optimized execution script that passes the test as the target execution script;

[0008] The task scheduling module determines an engine list and a target execution script corresponding to the target task, and sends the engine list and the target execution script to the distributed scheduling engine;

[0009] The distributed scheduling engine calls the executor according to the engine list;

[0010] The executor executes the target service component task based on the target execution script, and feeds back the execution result to the callback;

[0011] The callback device feeds back the execution result to the distributed scheduling engine;

[0012] The distributed scheduling engine feeds back the execution result to the task management module;

[0013] The task management module determines the execution status of the service component task based on the execution result.

[0014] Preferably, the task scheduling module determines the engine list corresponding to the target task and the target execution script, including:

[0015] The task orchestration module publishes the target task as a service component task, and uses a workflow engine to determine a workflow based on the execution process and dependency relationships between the service component tasks; based on the workflow, a target service component task is determined from the service component tasks to be executed, and a list of engines that execute the target service component task and the target execution script are obtained by sequentially scheduling the script management module, the security control module, and the slicing engine.

[0016] Preferably, the distributed scheduling engine calls the executor according to the engine list, including:

[0017] The distributed scheduling engine monitors the resources in the system in real time through the built-in resource monitoring probe and stores the monitoring data in the central database; according to the actual requirements of the operation and maintenance script and the current status of the system resources, the executor is determined from the engine list and called.

[0018] Preferably, the task scheduling center further includes a registration center; and the method further includes:

[0019] The executor sends information data and monitoring data to the registration center, obtains configuration file information of the corresponding executor, and then initializes the execution engine;

[0020] In response to the viewing request, the registration center outputs an executor list.

[0021] Preferably, it also includes:

[0022] The registration center deploys multiple instances to provide registration services for the execution engine;

[0023] When the registration center manages the executor, it does so in the form of database row locks.

[0024] Preferably, the task scheduling center further includes a batch callback interface, and the method further includes:

[0025] After the task scheduling center is started, the callback task queue is initialized, and the completed tasks enter the callback task queue and wait for callback execution;

[0026] The asynchronous callback thread obtains tasks in batches from the callback task queue and calls the batch callback interface to perform callback operations.

[0027] Preferably, the task scheduling center further includes a visual operation and maintenance management platform, and the method further includes:

[0028] The visual operation and maintenance management platform outputs task progress and / or execution results.

[0029] A cloud operation and maintenance system, comprising: a task scheduling center and an execution engine, wherein the task scheduling center comprises a script management module, a task scheduling module, and a distributed scheduling engine, and the execution engine comprises a registrar, an executor, and a callback;

[0030] The steps of the task processing method described above are performed in the cloud operation and maintenance system.

[0031] An electronic device, comprising:

[0032] Memory for storing computer programs;

[0033] A processor is used to implement the steps of the above-mentioned task processing method when executing the computer program.

[0034] A readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the task processing method are implemented.

[0035] The method provided by the embodiment of the present application is applied to a cloud operation and maintenance system, which includes a task scheduling center and an execution engine. The task scheduling center includes a script management module, a task management module, a task scheduling module, and a distributed scheduling engine. The execution engine includes a registrar, an executor, and a callback. The method includes: the script management module obtains the execution script of the target task, uses a script parser to parse the execution script, obtains the constituent elements of the execution script, and uses the constituent elements to generate an abstract syntax tree; based on the abstract syntax tree, the execution logic in the execution script is analyzed to determine the code defects; the code defects are optimized, and the execution scripts before and after the optimization are compared and tested, and the optimized execution script that passes the test is determined as the target execution script; the task scheduling module determines the engine list and the target execution script corresponding to the target task, and sends the engine list and the target execution script to the distributed scheduling engine; the distributed scheduling engine calls the executor according to the engine list; the executor executes the target service component task based on the target execution script, and feeds back the execution result to the callback; the callback feeds back the execution result to the distributed scheduling engine; the distributed scheduling engine feeds back the execution result to the task management module; the task management module determines the execution status of the service component task based on the execution result.

[0036] In this application, the script management module obtains the execution script of the target task. First, based on the script parser, the execution script is parsed to obtain the constituent elements of the execution script. Then, an abstract syntax tree is obtained using the constituent elements; based on the abstract syntax tree, the execution logic of the execution script can be analyzed more deeply to determine the code defects of the execution script; the code defects are optimized, and the execution scripts before and after the optimization are compared and tested, and then the optimized execution script that passes the test is determined as the target execution script, so as to ensure the accuracy and effectiveness of the execution script optimization process; that is, the script management module can analyze the execution script of the target task, find out the defects, and make up for the defects through optimization, so as to obtain the optimized target execution script corresponding to the target task. In this way, the task scheduling module determines the engine list and target execution script corresponding to the target task, and can send the engine list and target execution script to the distributed scheduling engine; the distributed scheduling engine calls the executor according to the engine list; the executor executes the target service component task based on the target execution script, and feeds back the execution result to the callback; the distributed scheduling engine and the task management module obtain the execution results in turn; the task management module determines the execution status of the service component task based on the execution result.

[0037] In the present application, by detecting and optimizing the execution script of the target task, code defects in the execution script can be effectively removed, and the execution efficiency can be improved and security risks can be reduced when the target execution script is subsequently executed.

[0038] Correspondingly, the embodiments of the present application also provide a cloud operation and maintenance system, equipment and readable storage medium corresponding to the above-mentioned task processing method, which have the above-mentioned technical effects and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 This is a flowchart of an implementation method of a task processing method in an embodiment of the present application;

[0041] Figure 2 This is a structural diagram of a task processing system in an embodiment of the present application;

[0042] Figure 3 This is a schematic diagram of a specific implementation of a task processing method in an embodiment of the present application;

[0043] Figure 4 This is a schematic diagram of the structure of an electronic device in an embodiment of the present application;

[0044] Figure 5 This is a schematic diagram of the specific structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0046] Please refer to Figure 2 In the present application, the task scheduling center 100 and the execution engine 200, the task scheduling center includes a script management module 101, a task management module 104, a task scheduling module 102, and a distributed scheduling engine 103, and the execution engine includes a register 201, an executor 202, and a callback 203;

[0047] Execute the steps of the task processing method provided in the embodiment of the present application in the cloud operation and maintenance system.

[0048] That is, in this application, the cloud operation and maintenance system includes two services, namely, the task scheduling center service and the execution engine service. The task scheduling center includes the script management module, the task management module 104, the task scheduling module, the sharding engine 105, the visual operation and maintenance management platform 107, the distributed task scheduling engine, the security control module 106, the system docking interface, the registration center 108 and other core parts. The execution engine includes the core modules such as the registrar, the executor, and the callback.

[0049] Please refer to Figure 3 The task scheduling center service and the execution engine service work together to form an efficient, secure, and easy-to-use IT operation and maintenance script automation execution system. The following is a detailed description of the purpose and implementation of the main core modules:

[0050] The script management module can achieve the following purposes, including but not limited to:

[0051] Script management: easy-to-use script management function to manage the script name, description, version, category directory, source file storage and retrieval method, etc.

[0052] Intelligent script parsing and optimization parser: An efficient script parser that supports multiple script languages ​​(such as Bash, Python, PowerShell, etc.) and has powerful syntax analysis and semantic understanding capabilities. By deeply parsing the script, the parser can identify the various components in the script (such as variable definition, conditional judgment, loop control, etc.) and build an abstract syntax tree for the script.

[0053] Optimization strategy: Based on the abstract syntax tree, the parser can further analyze the execution logic in the script and identify possible performance bottlenecks and redundant code. For these problems, the parser will automatically apply a series of optimization strategies, such as code refactoring, loop unrolling, instruction reordering, etc., to optimize the execution path of the script and reduce unnecessary computing overhead.

[0054] Automated testing: In order to ensure that the optimized script can run normally and achieve the expected results, the present invention also integrates an automated testing framework. The framework can automatically generate test cases and conduct comprehensive functional and performance tests on the scripts before and after optimization to ensure the accuracy and effectiveness of the optimization process.

[0055] The task management module can achieve the following purposes, including but not limited to:

[0056] Task management function: It has convenient task management functions (used for creating, editing, deleting, operating, testing, and executing tasks), viewing task execution logs, and monitoring task execution status.

[0057] Task strategy function: This invention provides management of task execution strategy and provides data support for automated task execution. Configure the task execution cycle (such as every day, every hour, every minute, etc.) and trigger conditions in the scheduled task framework.

[0058] Execution monitoring (used to monitor the execution status and progress of operation and maintenance scripts in real time) and security auditing (used to view and analyze security logs and alarm information), etc.

[0059] The sharding engine can achieve the following purposes, including but not limited to:

[0060] Task execution sharding strategy design: Select a reasonable data sharding strategy, such as range sharding, hash sharding, or list sharding, based on business characteristics and data scale.

[0061] Sharding implementation: Implement data sharding at the database level and application level to ensure that data is distributed to different physical nodes as needed.

[0062] Shard management: Use custom management tools to monitor and manage shard status, including data migration, capacity expansion and reduction, etc.

[0063] The task scheduling module can achieve the following purposes, including but not limited to:

[0064] Task service componentization: Tasks are published as service components, each of which is responsible for specific business logic or data processing tasks.

[0065] Workflow definition: Use the workflow engine Flowable to define the execution process and dependencies between service components.

[0066] Service component scheduling: The workflow engine automatically schedules task execution according to the defined workflow, supporting multiple execution modes such as sequential execution, parallel execution, and conditional branching.

[0067] Service component task monitoring: monitors the task execution status in real time and provides a visual interface to display task progress, execution results and other information.

[0068] The task scheduling engine completes the current activity by listening to the task completion message queue.

[0069] The operation and maintenance management module can achieve the following purposes, including but not limited to:

[0070] Visual operation and maintenance management platform user interface: It provides a friendly user interface and an intuitive operation interface, allowing operation and maintenance personnel to easily perform operations such as script editing, task scheduling, execution monitoring and security auditing.

[0071] Data analysis and reporting: In order to further improve the intelligence level of operation and maintenance work, the platform also provides data analysis and reporting functions. Through in-depth mining and analysis of operation and maintenance data, operation and maintenance personnel can find potential problems and optimization points in the system and formulate more scientific and reasonable operation and maintenance strategies accordingly. At the same time, the platform can also generate various forms of reports and charts (such as bar charts, line charts, pie charts, etc.) so that operation and maintenance personnel can more intuitively understand the operation status and operation and maintenance effects of the system.

[0072] Distributed task scheduling engine can achieve the following purposes including but not limited to:

[0073] Dynamic resource scheduling and load balancing resource monitoring: Through the built-in resource monitoring probe, the usage of various resources in the system (such as CPU, memory, disk, network, etc.) is monitored in real time. These monitoring data will be collected and stored in the central database for subsequent resource scheduling and load balancing decisions.

[0074] Scheduling algorithm: Based on resource monitoring data, the intelligent scheduling algorithm can dynamically adjust the resource allocation strategy according to the actual needs of the operation and maintenance script and the current status of system resources. For example, when resources are tight, the algorithm will give priority to allocating resources to scripts with high priority or short execution time; when resources are sufficient, it will improve resource utilization through parallel execution and other methods.

[0075] Load balancing: In order to achieve balanced utilization of resources, a load balancing mechanism is introduced. This mechanism can automatically detect the load of each node and assign new tasks to nodes with lower loads according to the load conditions. At the same time, in order to avoid task interruption due to node failure, fault tolerance and backup mechanisms can be implemented to ensure the reliability and continuity of tasks.

[0076] Trigger immediately: Execute the calling script immediately.

[0077] Scheduled triggering: A self-developed scheduled task framework is used to define, manage, and trigger scheduled tasks to achieve scheduled processing of sharded data. It includes task timeout strategy, exception retry strategy, blocking strategy (discard, wait, overwrite, concurrency), etc. It supports defining the execution cycle through cron expressions.

[0078] Task completion callback: By default, the completed task is called back through the form of docking with the message queue. The specific method is to compress the result data and logs generated by the task and send them to the corresponding message queue. The client program executed by the callback obtains data by monitoring the queue and performs subsequent business processing.

[0079] In order to ensure that the execution process of the operation and maintenance script complies with security specifications and requirements, the security control module can achieve the following purposes, including but not limited to:

[0080] Security Audit and Protection Module Security Audit: The module can record and analyze the execution logs of operation and maintenance scripts and system operation logs to identify potential security risks and abnormal behaviors. At the same time, the module also supports custom audit rules and alarm strategies to timely discover and deal with security issues.

[0081] Script security detection and whitelist mechanism: In order to ensure the security of operation and maintenance scripts, script execution detection can be performed, the script can be matched by intercepting the data of the rule library, and a whitelist can be provided to release high-risk commands.

[0082] Protection mechanism: Provides a series of security protection mechanisms. These mechanisms include access control, permission management, data encryption, etc., to ensure that the execution process of the operation and maintenance script is not maliciously attacked or illegally accessed. At the same time, it also supports integration and linkage with existing security systems to improve the overall security protection capabilities.

[0083] The registration center can achieve the following purposes including but not limited to:

[0084] Dynamic service discovery, configuration management and service management: The executor calls the Rest interface to POST information data and monitoring data to the registration center service, obtains the configuration file information of the corresponding executor, and then initializes the execution engine service. The registration center can view the executor list, including basic information, status, busyness, etc.

[0085] Multi-instance deployment: the registration center can deploy multiple instances and provide registration services for the execution engine through nginx reverse proxy. When the registration center manages the executor, it avoids concurrency problems through database row locks. Even if one registration center goes down, it will not affect the normal execution scheduling of the execution engine.

[0086] The registration center provides an execution engine service query list: it provides support for the execution engine service instance data for the distributed task scheduling engine and sharding engine.

[0087] The execution engine service can be used for the following purposes, including but not limited to:

[0088] Efficient execution engine and concurrent control execution engine: Provides an efficient execution engine that can directly parse and execute optimized scripts. Compared with traditional interpreters or compilers, this execution engine has lower latency and higher throughput. It uses technical means such as multi-threading and asynchronous IO to fully utilize the advantages of modern multi-core CPUs and high-speed networks. In other words, the execution engine is used to actually execute tasks, and it uses hormone means such as multi-threading and asynchronous IO internally. The execution engine can be deployed in multiple instances, horizontally expanded, and executed in shards, thereby improving the execution efficiency of tasks.

[0089] Concurrency control: In order to handle a large number of concurrently executed operation and maintenance scripts, a fine-grained concurrency control mechanism can be implemented. This mechanism can ensure that multiple scripts will not interfere with each other or conflict during execution. Through locking mechanisms, transaction management, data isolation and other technical means, the present invention can effectively manage resource competition and data consistency issues in concurrent execution.

[0090] Horizontal expansion: The execution engine supports multi-instance deployment. The configuration parameter group identifies whether they are the same group. The same group can execute the same task and participate in scheduling strategies such as sharding execution and load balancing.

[0091] The registrar can be used for the following purposes, including but not limited to:

[0092] Node health data collection: Collect data such as CPU, memory, disk, connectivity, etc. of the executor node to monitor the health status of the executor node.

[0093] Heartbeat reporting: Data is reported every 30 seconds, and health data and device information are reported to the registration center service of the task scheduling center.

[0094] The asynchronous callback service (i.e., callback device) can be used for the following purposes, including but not limited to:

[0095] Callback task queue: After the service is started, a callback task queue will be initialized. The completed tasks will directly enter the queue and wait for callback execution.

[0096] Batch callback: asynchronous callback thread, batch obtain tasks from the queue, call the batch callback interface of the task scheduling center to perform callback operations.

[0097] The system docking interface can be used for purposes including but not limited to the following:

[0098] Data platform docking: Provides a variety of docking methods (message queues, local databases, remote libraries, CMDB, etc.). By default, the local library stores temporary data and the central library stores final data. It can be docked with any data platform through script plug-ins to facilitate the distribution and push of data results after execution.

[0099] Unified task management API: Provides a unified task API for easy integration and standardizes the format of input and output parameters.

[0100] The application of this cloud operation and maintenance system has the following technical effects:

[0101] 1. Improve script execution efficiency, as follows:

[0102] Intelligent script parsing and optimization: By automatically identifying and optimizing redundant code and unreasonable logic in the script, the script execution speed is greatly improved without changing the original function.

[0103] Efficient execution engine and concurrency control: Adopting technologies such as multi-threading and asynchronous IO, fully utilizing the advantages of multi-core CPU and high-speed network to improve execution efficiency and concurrent processing capabilities.

[0104] Sharded execution capability: Shard the tasks to be executed through the sharding strategy, greatly improving the task processing capability.

[0105] 2. Maximize resource utilization, as follows:

[0106] Dynamic resource scheduling and load balancing: Real-time monitoring of system resource status, dynamic adjustment of resource allocation strategies based on script requirements, avoiding performance bottlenecks caused by uneven resource allocation, and maximizing resource utilization.

[0107] 3. Improve execution security, as follows:

[0108] Security audit and protection module: can monitor the entire script execution process, promptly discover and handle potential security risks, and ensure stable system operation and data security.

[0109] Script security detection and whitelist mechanism: intercept high-risk commands and prevent malicious script execution through the interception rule library and whitelist mechanism.

[0110] Access control, permission management, data encryption: Multi-level security protection mechanism to prevent malicious attacks and illegal access.

[0111] 4. Improve the convenience of operation and maintenance, as follows:

[0112] Visual operation and maintenance management platform: provides a user-friendly interface and rich management functions to facilitate operations such as script editing, task scheduling, execution monitoring and security auditing.

[0113] Data analysis and reporting: Deeply mine operation and maintenance data, discover potential problems and optimization points, generate visual reports, and help formulate scientific and reasonable operation and maintenance strategies.

[0114] Please refer to Figure 2 , Figure 2 It is a flowchart of a task processing method in an embodiment of the present application, which is applied to a cloud operation and maintenance system. The cloud operation and maintenance system includes a task scheduling center and an execution engine. The task scheduling center includes a script management module, a task management module, a task scheduling module, and a distributed scheduling engine. The execution engine includes a registrar, an executor, and a callback.

[0115] The method comprises the following steps:

[0116] S101, the script management module obtains the execution script of the target task, uses the script parser to parse the execution script, obtains the constituent elements of the execution script, and uses the constituent elements to generate an abstract syntax tree; based on the abstract syntax tree, analyzes the execution logic in the execution script to determine the code defects; optimizes the code defects, and performs a comparison test on the execution scripts before and after the optimization, and determines the optimized execution script that passes the test as the target execution script;

[0117] That is, in the script management module, after obtaining the execution script of the target task, the script parser is first used to execute the script. Specifically, an efficient script parser can be used, which can support multiple script languages ​​(such as Bash, Python, PowerShell, etc.) and has powerful syntax analysis and semantic understanding capabilities. By deeply parsing the script, the parser can identify the various components in the script (i.e., components such as variable definition, conditional judgment, loop control, etc.) and construct an abstract syntax tree of the script.

[0118] Among them, the script parser: distinguishes the suffixes of the script files and presets different suffixes to correspond to different script languages.

[0119] Construct an abstract syntax tree: First, perform lexical analysis on the source program code, perform syntactic analysis on the word stream generated by the lexical analysis, and finally perform semantic analysis to form a syntax tree of the source code.

[0120] Abstract syntax tree: It is a tree representation of the syntax structure of the source code. It represents the syntax structure of the programming language in a tree form.

[0121] Optimization strategy: Based on the abstract syntax tree, the parser can further analyze the execution logic in the script and identify possible performance bottlenecks and redundant code. Specifically, it identifies the logic with performance bottlenecks, such as repeated IO operations. Redundant code logic: Identify the same code blocks and prompt that the redundancy can be extracted.

[0122] To address these issues, the parser will automatically apply a series of optimization strategies, such as code refactoring, loop unrolling, and instruction reordering, to optimize the script execution path and reduce unnecessary computing overhead. After refactoring the script code, multiple scripts will eventually be merged into one executable script, which only contains one script during execution, and does not need to reference multiple paths, reducing IO operations.

[0123] In order to ensure that the optimized script can run normally and achieve the expected results, test cases can be automatically generated based on the automated testing framework to conduct comprehensive functional and performance tests on the scripts before and after optimization to ensure the accuracy and effectiveness of the optimization process. Specifically, the script can be automatically run and tested through preset devices and default execution parameters. Specific structure: input parameter definition, device information, script execution, return results, statistical execution time observation summary.

[0124] S102, the task scheduling module determines the engine list and target execution script corresponding to the target task, and sends the engine list and target execution script to the distributed scheduling engine;

[0125] The task encoding module can determine the engine list and target execution script corresponding to the target task, and provide the engine list and target execution script to the distributed scheduling engine so that the distributed scheduling engine can schedule the corresponding executor.

[0126] In a specific implementation of the present application, the task scheduling module determines an engine list and a target execution script corresponding to a target task, including:

[0127] The task scheduling module publishes the target task as a service component task, and uses the workflow engine to determine the workflow based on the execution process and dependency relationships between the service component tasks; based on the workflow, the target service component task is determined from the service component tasks to be executed, and the engine list and target execution script for executing the target service component task are obtained by scheduling the script management module, security control module and sharding engine in sequence.

[0128] That is, in the task coding module, it is necessary to implement task service componentization, workflow definition, service component scheduling, and service component task monitoring.

[0129] Among them, task service componentization: publish the target task as a service component, each component is responsible for a specific business logic or data processing task. Specifically, the task consists of three parts: input, execution, and output. The task can be published as a service component through the rest interface or message queue consumer. After receiving the http or message queue message, the task generates a job and executes it.

[0130] Workflow definition: Use the workflow engine Flowable to define the execution process and dependencies between service components.

[0131] Service component scheduling: The workflow engine automatically schedules task execution according to the defined workflow, supporting multiple execution modes such as sequential execution, parallel execution, and conditional branching.

[0132] Service component task monitoring: monitors the task execution status in real time and provides a visual interface to display task progress, execution results and other information.

[0133] The task scheduling engine completes the current activity by listening to the task completion message queue.

[0134] S103, the distributed scheduling engine calls the executor according to the engine list;

[0135] The distributed scheduling engine can schedule executors based on the engine list.

[0136] In one implementation of the present application, the distributed scheduling engine calls the executor according to the engine list, including:

[0137] The distributed scheduling engine monitors the resources in the system in real time through the built-in resource monitoring probe and stores the monitoring data in the central database. According to the actual needs of the operation and maintenance script and the current status of the system resources, the executor is determined from the engine list and called.

[0138] That is to say, when the distributed scheduling engine schedules the executor, it can select the executor suitable for executing the service component task from the engine list according to the actual needs of the operation and maintenance steps and the current status of the system font, and then schedule the executor.

[0139] S104, the executor executes the target service component task based on the target execution script, and feeds back the execution result to the callback;

[0140] The executor can execute the target service component task based on the target execution steps, and after the execution is completed, it will feedback the execution result to the callback.

[0141] S105, the callback device feeds back the execution result to the distributed scheduling engine;

[0142] The callback will feed back the execution result to the distributed scheduling engine, and the distributed scheduling engine will determine the execution status.

[0143] S106, the distributed scheduling engine feeds back the execution result to the task management module;

[0144] The distributed scheduling engine will also feed back the learning results to the task management simulation module.

[0145] S107: The task management module determines the execution status of the service component task based on the execution result.

[0146] The task management module can determine the execution status of the service component based on the execution results.

[0147] The method provided by the embodiment of the present application is applied to a cloud operation and maintenance system, which includes a task scheduling center and an execution engine. The task scheduling center includes a script management module, a task management module, a task scheduling module, and a distributed scheduling engine. The execution engine includes a registrar, an executor, and a callback. The method includes: the script management module obtains the execution script of the target task, uses a script parser to parse the execution script, obtains the constituent elements of the execution script, and uses the constituent elements to generate an abstract syntax tree; based on the abstract syntax tree, the execution logic in the execution script is analyzed to determine the code defects; the code defects are optimized, and the execution scripts before and after the optimization are compared and tested, and the optimized execution script that passes the test is determined as the target execution script; the task scheduling module determines the engine list and the target execution script corresponding to the target task, and sends the engine list and the target execution script to the distributed scheduling engine; the distributed scheduling engine calls the executor according to the engine list; the executor executes the target service component task based on the target execution script, and feeds back the execution result to the callback; the callback feeds back the execution result to the distributed scheduling engine; the distributed scheduling engine feeds back the execution result to the task management module; the task management module determines the execution status of the service component task based on the execution result.

[0148] In this application, the script management module obtains the execution script of the target task. First, based on the script parser, the execution script is parsed to obtain the constituent elements of the execution script. Then, an abstract syntax tree is obtained using the constituent elements; based on the abstract syntax tree, the execution logic of the execution script can be analyzed more deeply to determine the code defects of the execution script; the code defects are optimized, and the execution scripts before and after the optimization are compared and tested, and then the optimized execution script that passes the test is determined as the target execution script, so as to ensure the accuracy and effectiveness of the execution script optimization process; that is, the script management module can analyze the execution script of the target task, find out the defects, and make up for the defects through optimization, so as to obtain the optimized target execution script corresponding to the target task. In this way, the task scheduling module determines the engine list and target execution script corresponding to the target task, and can send the engine list and target execution script to the distributed scheduling engine; the distributed scheduling engine calls the executor according to the engine list; the executor executes the target service component task based on the target execution script, and feeds back the execution result to the callback; the distributed scheduling engine and the task management module obtain the execution results in turn; the task management module determines the execution status of the service component task based on the execution result.

[0149] In the present application, by detecting and optimizing the execution script of the target task, code defects in the execution script can be effectively removed, and the execution efficiency can be improved and security risks can be reduced when the target execution script is subsequently executed.

[0150] It should be noted that, based on the above embodiments, the embodiments of the present application also provide corresponding improved solutions. In the preferred / improved embodiments, the same steps or corresponding steps as those in the above embodiments can be referenced to each other, and the corresponding beneficial effects can also be referenced to each other, which will not be described one by one in the preferred / improved embodiments of this article.

[0151] In a specific implementation of the present application, the task scheduling center also includes a registration center; the method further includes:

[0152] The executor sends information data and monitoring data to the registration center, obtains the configuration file information of the corresponding executor, and then initializes the execution engine;

[0153] In response to a query request, the registry outputs a list of executors.

[0154] That is, the executor can obtain the corresponding configuration file information and initialize the execution engine by sending information data and monitoring data to the registration center. When the registration center receives the break view request, it can output the executor list for calling.

[0155] That is, the executor calls the Rest interface to POST information data and monitoring data to the registration center service, obtains the configuration file information of the corresponding executor, and then initializes the execution engine service. The registration center can view the executor list, including basic information, status, busyness, etc.

[0156] In a specific implementation of the present application, it also includes:

[0157] The registration center deploys multiple instances to provide registration services for the execution engine;

[0158] When the registration center manages the executor, it does so by locking the database row.

[0159] Multi-instance deployment: The registration center can deploy multiple instances and provide registration services for the execution engine through nginx reverse proxy. When the registration center manages the executor, it avoids concurrency problems through database row locks. Even if one registration center goes down, it will not affect the normal execution scheduling of the execution engine. The registration center provides an execution engine service query list: it provides support for the execution engine service instance data for the distributed task scheduling engine and the sharding engine.

[0160] In a specific implementation of the present application, the task scheduling center also includes a batch callback interface, and the method also includes:

[0161] After the task scheduling center is started, the callback task queue is initialized, and the completed tasks enter the callback task queue and wait for callback execution;

[0162] The asynchronous callback thread obtains tasks from the callback task queue in batches and calls the batch callback interface to perform callback operations.

[0163] The callback task queue is a queue of callback tasks that will be initialized after the service is started. The completed tasks will directly enter the queue and wait for the callback to be executed.

[0164] Batch callback, corresponding to asynchronous callback thread, can obtain tasks from the callback task queue in batches and call the batch callback interface of the task scheduling center to perform callback operations.

[0165] In a specific implementation of the present application, the task scheduling center also includes a visual operation and maintenance management platform, and the method also includes:

[0166] The visual operation and maintenance management platform outputs task progress and / or execution results.

[0167] Among them, outputting the task progress and / or the execution result means outputting at least one of the task progress and the execution result.

[0168] Specifically, a user-friendly interface and an intuitive operation interface can be provided in the visual operation and maintenance management platform user interface, so that operation and maintenance personnel can easily perform operations such as script editing, task scheduling, execution monitoring and security auditing.

[0169] In order to further improve the intelligent level of operation and maintenance work, the visual operation and maintenance management platform can also provide data analysis and reporting functions. Through in-depth mining and analysis of operation and maintenance data, operation and maintenance personnel can find potential problems and optimization points in the system and formulate more scientific and reasonable operation and maintenance strategies accordingly. At the same time, the platform can also generate various forms of reports and charts (such as bar charts, line charts, pie charts, etc.) so that operation and maintenance personnel can more intuitively understand the operation status and operation and maintenance effects of the system.

[0170] Corresponding to the above method embodiment, the embodiment of the present application further provides an electronic device. The electronic device described below and the task processing method described above can refer to each other.

[0171] See also Figure 4 As shown, the electronic device includes:

[0172] A memory 332, for storing computer programs;

[0173] The processor 322 is used to implement the steps of the task processing method of the above method embodiment when executing a computer program.

[0174] For details, please refer to Figure 5 , Figure 5A schematic diagram of the specific structure of an electronic device provided for this embodiment, which may have relatively large differences due to different configurations or performances, may include one or more processors (central processing units, CPU) (for example, one or more processors) and a memory 332, and the memory 332 stores one or more computer programs 342 or data 344. Among them, the memory 332 can be a temporary storage or a permanent storage. The program stored in the memory 332 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the data processing device. Furthermore, the processor 322 can be configured to communicate with the memory 332 to execute a series of instruction operations in the memory 332 on the electronic device 301.

[0175] The electronic device 301 may further include one or more power supplies 326 , one or more wired or wireless network interfaces 350 , one or more input and output interfaces 358 , and / or one or more operating systems 341 .

[0176] The steps in the task processing method described above can be implemented by the structure of an electronic device.

[0177] Corresponding to the above method embodiment, the embodiment of the present application further provides a readable storage medium. The readable storage medium described below and the task processing method described above can refer to each other.

[0178] A readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the task processing method of the above method embodiment are implemented.

[0179] The readable storage medium may specifically be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or other readable storage medium that can store program codes.

[0180] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0181] Those skilled in the art may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0182] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0183] Finally, it should be noted that, in this article, relationships such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms include, include or any other variations are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0184] Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A task processing method, characterized in that: Applied to a cloud operation and maintenance system, the cloud operation and maintenance system includes a task scheduling center and an execution engine, the task scheduling center includes a script management module, a task management module, a task scheduling module and a distributed scheduling engine, and the execution engine includes a registrar, an executor and a callback; the method includes: The script management module obtains the execution script of the target task, uses a script parser to parse the execution script, obtains the constituent elements of the execution script, and uses the constituent elements to generate an abstract syntax tree; based on the abstract syntax tree, analyzes the execution logic in the execution script to determine code defects; optimizes the code defects, and performs a comparison test on the execution scripts before and after optimization, and determines the optimized execution script that passes the test as the target execution script; The task scheduling module determines an engine list and a target execution script corresponding to the target task, and sends the engine list and the target execution script to the distributed scheduling engine; The distributed scheduling engine calls the executor according to the engine list; The executor executes the target service component task based on the target execution script, and feeds back the execution result to the callback; The callback device feeds back the execution result to the distributed scheduling engine; The distributed scheduling engine feeds back the execution result to the task management module; The task management module determines the execution status of the service component task based on the execution result.

2. The method according to claim 1, characterized in that The task scheduling module determines the engine list corresponding to the target task and the target execution script, including: The task orchestration module publishes the target task as a service component task, and uses a workflow engine to determine a workflow based on the execution process and dependency relationships between the service component tasks; based on the workflow, a target service component task is determined from the service component tasks to be executed, and a list of engines that execute the target service component task and the target execution script are obtained by sequentially scheduling the script management module, the security control module, and the slicing engine.

3. The method according to claim 1, characterized in that The distributed scheduling engine calls the executor according to the engine list, including: The distributed scheduling engine monitors the resources in the system in real time through the built-in resource monitoring probe and stores the monitoring data in the central database; according to the actual requirements of the operation and maintenance script and the current status of the system resources, the executor is determined from the engine list and called.

4. The method according to claim 1, characterized in that: The task scheduling center also includes a registration center; the method also includes: The executor sends information data and monitoring data to the registration center, obtains configuration file information of the corresponding executor, and then initializes the execution engine; In response to the viewing request, the registration center outputs an executor list.

5. The method according to claim 4, characterized in that Also includes: The registration center deploys multiple instances to provide registration services for the execution engine; When the registration center manages the executor, it does so in the form of database row locks.

6. The method according to claim 1, characterized in that The task scheduling center also includes a batch callback interface, and the method further includes: After the task scheduling center is started, the callback task queue is initialized, and the completed tasks enter the callback task queue and wait for callback execution; The asynchronous callback thread obtains tasks in batches from the callback task queue and calls the batch callback interface to perform callback operations.

7. The method according to any one of claims 1 to 6, characterized in that: The task scheduling center also includes a visual operation and maintenance management platform, and the method further includes: The visual operation and maintenance management platform outputs task progress and / or execution results.

8. A cloud operation and maintenance system, characterized in that: include: A task scheduling center and an execution engine, wherein the task scheduling center includes a script management module, a task management module, a task scheduling module, and a distributed scheduling engine, and the execution engine includes a registrar, an executor, and a callback; The steps of the task processing method according to any one of claims 1 to 7 are performed in the cloud operation and maintenance system.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the task processing method according to any one of claims 1 to 7 when executing the computer program.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the task processing method according to any one of claims 1 to 7 are implemented.

Citation Information

Cited By

  • Multi-host automatic operation and maintenance control method and system based on Ansible framework

    CN120416045A

  • Hybrid cloud resource operation and maintenance scheduling method and device based on large model, equipment and storage medium

    CN120631593A

  • A large model-based hybrid cloud resource operation and maintenance scheduling method, device, equipment and storage medium

    CN120631593B