System for implementing unified job automation management and execution under cloudy environment

Abstract

The application relates to a system for realizing unified job automation management and execution in a multi-cloud environment, comprising a script management module for centrally managing, version controlling, classifying and label managing and online testing of multiple script languages; a template management module for defining standardized job process templates, supporting global variables, inter-step data transmission and execution strategy configuration; a scheme management module for providing execution strategy configuration; and an execution record module for recording detailed processes of each execution and performing multidimensional query and visual display. The system for realizing unified job automation management and execution in a multi-cloud environment adopts the application, establishes a complete execution tracing system, records detailed states of the whole life cycle of jobs, supports quick positioning and optimization of problems, designs a dynamic resource scheduling mechanism, automatically adjusts execution resources according to loads, improves resource utilization efficiency, constructs an open and extensible system architecture, and supports unified job management in a multi-cloud and mixed environment.

Description

Technical Field

[0001] This invention relates to the field of enterprise IT environment, and more particularly to the field of enterprise IT infrastructure, specifically to a system for achieving unified automated management and execution of operations in a multi-cloud environment. Background Technology

[0002] As digital transformation deepens, enterprise IT infrastructure is rapidly expanding in scale, leading to a sharp increase in operational complexity and the challenge of managing heterogeneous environments. Against this backdrop, automated job management has become a core capability for IT operations. Enterprises need a solution that can uniformly manage various automated tasks in complex, heterogeneous environments and support flexible business processes.

[0003] The mainstream automated job management solutions currently on the market can be mainly divided into the following categories: 1) Simple wrappers around open-source automation tools: such as configuration management tools like Ansible, SaltStack, and Puppet, and continuous integration tools like Jenkins. These tools typically provide basic script execution and task scheduling functions, but lack enterprise-level job workflow management, fine-grained access control, and complete execution traceability capabilities. Users need to develop additional modules to meet enterprise needs, increasing implementation complexity.

[0004] 2) Job modules in commercial ITSM systems: such as job automation modules in IT service management platforms like ServiceNow and BMC Remedy. These solutions are typically deeply integrated with ticketing systems, providing comprehensive workflow approval and auditing functions. However, they have a heavyweight architecture, poor customizability, high deployment costs, and difficulty adapting to the rapid changes in cloud environments.

[0005] 3) Cloud platform native automation services: such as AWS Systems Manager and Azure Automation. These services are deeply integrated with specific cloud platforms, providing convenient automation capabilities, but their cross-cloud management capabilities are weak. If an enterprise adopts a multi-cloud strategy, it needs to maintain multiple automation systems on different platforms, increasing management complexity.

[0006] 4) Dedicated job scheduling systems: such as Control-M, Tidal, and other traditional job scheduling tools. These systems have advantages in large batch processing scenarios, but they lack support for cloud environments and containerized applications, have complex interfaces, and have a steep learning curve.

[0007] Currently, the closest approach to this invention on the market is a hybrid solution combining open-source automation tools (such as Ansible) with a self-developed management interface. These solutions typically have the following characteristics: 1) Provides script execution capabilities based on open-source tools; 2) Provides basic script management and execution functions through a web interface; 3) Supports simple scheduled task configuration; 4) Provides basic execution log viewing functionality.

[0008] Current automated job management solutions generally suffer from the following key shortcomings, which urgently need to be improved: 1) Fragmented Management Dimensions: Existing solutions typically treat scripts, execution flows, scheduling strategies, and execution logs as independent components, managing them across different systems. For example, scripts are stored in a Git repository, execution flows are defined in Jenkins, scheduling is configured in Cron, and execution logs are stored in a logging system. This fragmentation leads to high management complexity, requiring switching between multiple systems; version consistency is difficult to guarantee, as script updates do not synchronize with the flow; it is difficult to trace the complete execution chain, making problem localization challenging; and team collaboration is inefficient, resulting in fragmented knowledge.

[0009] 2) Weak process definition capabilities: Existing solutions typically only support linear execution processes, lack support for complex business scenarios, cannot flexibly define manual review nodes, and must rely on external systems to achieve this; they lack conditional branching capabilities, making it difficult to dynamically adjust subsequent steps based on execution results; they do not support data flow between steps, making it impossible to realize complex data processing processes; and they cannot dynamically adjust execution parameters based on resource characteristics, resulting in poor adaptability.

[0010] 3) Insufficient execution traceability: Existing execution records typically only focus on the final success / failure status, lacking detail. Performance metrics such as execution time and resource consumption for each step are not fully recorded; snapshots of environmental variables and input parameters are missing, making it difficult to reproduce problems; intermediate states and data changes during execution cannot be traced; and there is a lack of complete audit records for manual intervention operations.

[0011] 4) Low resource efficiency: Existing execution engines typically use a fixed resource pool model. They cannot dynamically adjust the number of execution nodes based on job load; their job scheduling strategies are simplistic and cannot intelligently schedule tasks based on priority, resource requirements, or other factors; resource isolation is poor, and high-load jobs may affect the execution of other jobs; and they lack analysis of execution history, making it impossible to optimize resource allocation strategies.

[0012] 5) Limited scalability and integration capabilities: Existing solutions perform poorly in multi-cloud and hybrid environments. It is difficult to manage operations on different infrastructures in a unified manner; there is a lack of standardized interfaces, making integration with other systems difficult; it is not flexible enough to support new types of execution steps and resource types; custom development is costly and difficult to adapt to rapidly changing business needs. Summary of the Invention

[0013] The purpose of this invention is to overcome the shortcomings of the prior art and provide a system that meets the requirements of flexible, efficient, traceable, and widely applicable unified operation automation management and execution in a multi-cloud environment.

[0014] To achieve the above objectives, the present invention provides a system for unified automated management and execution of jobs in a multi-cloud environment as follows: This system for unified automated management and execution of jobs in a multi-cloud environment is characterized by the following: The script management module is used for centralized management, version control, category tag management, and online testing of various scripting languages; The template management module, connected to the script management module, is used to define standardized work process templates and supports global variables, data transfer between steps, and execution strategy configuration. The solution management module, connected to the template management module, is used to apply the standardized work process template to specific business scenarios and provide execution strategy configuration. The execution record module, connected to the solution management module, is used to record the detailed process of each execution, enabling multi-dimensional queries and visualization.

[0015] Preferably, the system further includes an execution engine, which employs a layered architecture and includes: The request parsing layer is used to receive user operation requests and perform parameter validation and permission checks. The process orchestration layer, connected to the request parsing layer, is used to parse the execution process based on the template and generate an execution plan; The resource scheduling layer, connected to the process orchestration layer, is used to dynamically allocate execution resources based on the execution plan and the current load. The task execution layer, connected to the resource scheduling layer, is used to execute tasks on allocated resources and monitor the execution status. The results processing layer, connected to the task execution layer, is used to collect execution results, update the status, and trigger subsequent operations.

[0016] Preferably, the execution engine enables dynamic scaling up and down. The execution engine collects CPU, memory, and queue length metrics in real time, dynamically adjusts the number of execution nodes according to preset thresholds, releases resources after the executed task is completed, and schedules execution resources in the nearest cloud environment.

[0017] Preferably, the script management module provides a script asset library, centrally manages multiple scripting languages, and provides script classification, tag management, and version rollback.

[0018] Preferably, the template management module implements step execution strategy configuration, specifically including timeout control, error handling strategy and permission requirements; the template management module adopts an inter-step data transfer mechanism to realize data flow in complex business processes.

[0019] Preferably, the scheme management module has multiple execution strategies, including single execution, periodic execution, and event triggering, and performs precondition checks.

[0020] Preferably, the execution record module also implements lifecycle management of execution records, realizes complete preservation and traceable query of data throughout the execution process, and supports archiving and cleanup strategies.

[0021] This invention employs a system for unified automated job management and execution in a multi-cloud environment. It constructs a unified job management architecture, organically integrating scripts, templates, schemes, and execution records to break down management silos. It provides a flexible process definition mechanism, supporting seamless integration of manual and automated operations to adapt to complex business scenarios. A complete execution traceability system is established, recording detailed status throughout the job lifecycle, supporting rapid problem localization and optimization. A dynamic resource scheduling mechanism is designed to automatically adjust execution resources based on load, improving resource utilization efficiency. An open and scalable system architecture is constructed to support unified job management in multi-cloud and hybrid environments. Attached Figure Description

[0022] Figure 1 This is an architecture diagram of the system for achieving unified automated management and execution of jobs in a multi-cloud environment, as described in this invention. Detailed Implementation

[0023] To more clearly describe the technical content of the present invention, the following description is provided in conjunction with specific embodiments.

[0024] The system for unified automated management and execution of jobs in a multi-cloud environment according to the present invention includes: The script management module is used for centralized management, version control, category tag management, and online testing of various scripting languages; The template management module, connected to the script management module, is used to define standardized work process templates and supports global variables, data transfer between steps, and execution strategy configuration. The solution management module, connected to the template management module, is used to apply the standardized work process template to specific business scenarios and provide execution strategy configuration. The execution record module, connected to the solution management module, is used to record the detailed process of each execution, enabling multi-dimensional queries and visualization.

[0025] In a preferred embodiment of the present invention, the system further includes an execution engine, the execution engine adopting a layered architecture, and the execution engine comprising: The request parsing layer is used to receive user operation requests and perform parameter validation and permission checks. The process orchestration layer, connected to the request parsing layer, is used to parse the execution process based on the template and generate an execution plan; The resource scheduling layer, connected to the process orchestration layer, is used to dynamically allocate execution resources based on the execution plan and the current load. The task execution layer, connected to the resource scheduling layer, is used to execute tasks on allocated resources and monitor the execution status. The results processing layer, connected to the task execution layer, is used to collect execution results, update the status, and trigger subsequent operations.

[0026] In a preferred embodiment of the present invention, the execution engine achieves dynamic scaling up and down. The execution engine collects CPU, memory, and queue length indicators in real time, dynamically adjusts the number of execution nodes according to preset thresholds, releases resources after the executed task is completed, and schedules execution resources in the nearest cloud environment.

[0027] In a preferred embodiment of the present invention, the script management module provides a script asset library, centrally manages multiple scripting languages, and provides script classification, tag management, and version rollback.

[0028] In a preferred embodiment of the present invention, the template management module implements step execution strategy configuration, specifically including timeout control, error handling strategy and permission requirements; the template management module adopts an inter-step data transfer mechanism to realize data flow in complex business processes.

[0029] As a preferred embodiment of the present invention, the scheme management module has multiple execution strategies, including single execution, periodic execution, and event triggering, and performs precondition checks.

[0030] In a preferred embodiment of the present invention, the execution record module also implements lifecycle management of execution records, realizes complete preservation and traceable query of data throughout the execution process, and supports archiving and cleanup strategies.

[0031] With the increasing complexity of modern enterprise IT infrastructure and the deepening of cloud transformation, automated job management has become a key element in ensuring stable and efficient business operations. However, with the expansion and heterogeneity of enterprise IT environments, traditional job management methods face multiple challenges. This invention aims to address the following core issues: 1) Lack of decentralized and standardized resource operations: Enterprise IT environments contain a large number of scattered manual scripts and automated tasks, maintained by different teams and individuals, lacking unified standards and centralized management. These scripts often suffer from problems such as redundant development, version chaos, and inconsistent quality, resulting in high operation and maintenance costs and difficulty in ensuring security and compliance.

[0032] 2) Fragmented Workflows and Difficulty in Collaboration: Enterprise business processes typically involve multiple systems and stages, requiring a combination of manual and automated operations. Existing solutions often can only handle single types of automated tasks and cannot organically integrate operations such as manual review, script execution, resource detection, and data processing to form an end-to-end automated process, resulting in fragmented processes and low collaboration efficiency.

[0033] 3) Insufficient transparency and traceability: Traditional automation tools often only record simple success or failure statuses after executing a task, lacking detailed records of the entire execution process, including the execution status of each step, changes in environmental variables, input and output data, and critical details such as exception information. When problems occur, it is difficult to quickly locate the cause, and it is also not conducive to continuous optimization and compliance auditing.

[0034] 4) Insufficient resource utilization efficiency and flexibility: As enterprises expand, the workload increases exponentially, and statically configured execution resources are unable to cope with dynamically changing workloads. Insufficient resources during peak periods lead to task backlogs, while idle resources during off-peak periods result in waste, and there is a lack of ability to automatically adjust resources based on real-time load.

[0035] 5) Weak adaptability to multiple environments: Enterprise IT environments typically include various infrastructures such as public cloud, private cloud, and traditional physical machines. Existing job management tools are often designed for specific environments, making it difficult to manage them uniformly across environments, resulting in high management complexity and high learning costs.

[0036] In a specific embodiment of the present invention, a unified job automation management and execution system in a cloud environment is proposed, the core architecture and working mechanism of which are as follows: System architecture diagram as follows Figure 1 As shown, the core functional modules are described in detail below: 1. Script Management: (1) Provides a unified script asset library that supports centralized management of multiple scripting languages ​​(shell, python, PowerShell, sql, etc.); (2) Implement script version control, record the historical version of each modification, and support version rollback and difference comparison; (3) Provide an online editor that supports features such as syntax highlighting and auto-completion to improve development efficiency; (4) Implement script classification and tag management to facilitate quick retrieval and reuse; (5) Provide a script testing environment to support the verification of script functions in an isolated environment and reduce the risk of production environment.

[0037] 2. Template Management: (1) Define standardized work process templates to encapsulate complex operations into reusable process units; (2) Supports multiple job step types, including: manual review, script execution, resource detection, resource operation, and data processing; (3) Provide a global variable mechanism to support dynamic injection of parameters at runtime; (4) Supports data transfer between steps, enabling data flow in complex business processes; (5) Configure the execution strategy for the steps, including timeout control, error handling strategy, permission requirements, etc.

[0038] 3. Solution Management: (1) Apply standardized templates to specific business scenarios and define the execution scope and execution strategy; (2) Supports multi-dimensional execution scope definition: Target resources can be selected from multiple dimensions such as image, cloud host, operating system, business system, host tag, physical machine, etc. (3) Provides flexible execution strategy configuration: single execution, periodic execution, event triggering; (4) Perform pre-execution condition checks to ensure that the execution environment meets the requirements; (5) Provide an execution plan preview function to avoid accidental operation.

[0039] 4. Execution Log: (1) Completely record the detailed process of each execution, including: execution start / end time, total time, execution status, time and resource consumption of each step, environment variable snapshot, input parameters and output results, exception information and error stack, and manual intervention record (auditor, audit time, audit comments). (2) Supports multi-dimensional query and analysis, which facilitates problem localization and performance optimization; (3) Provide visualization of execution results, intuitively displaying execution status and key indicators; (4) Implement lifecycle management of execution records and support archiving and cleanup strategies.

[0040] Execution engine working mechanism: The execution engine of this system adopts a layered architecture design, and its specific working mechanism is as follows: 1) Request parsing layer: Receives user operation requests, performs parameter validation and permission checks; 2) Process orchestration layer: Based on the template definition, parse the execution process and generate an execution plan; 3) Resource scheduling layer: Dynamically allocates execution resources based on the execution plan and current load; 4) Task Execution Layer: Executes specific tasks on allocated resources and monitors execution status; 5) Result processing layer: Collect execution results, update status, and trigger subsequent operations.

[0041] The execution engine supports dynamic scaling, primarily through the following mechanisms: 1) Load monitoring: Real-time collection of metrics such as CPU, memory, and queue length; 2) Flexible strategy: Dynamically adjust the number of execution nodes based on preset thresholds; 3) Graceful scaling down: Ensures resources are released only after the currently executing task is completed; 4) Cross-regional scheduling: Supports scheduling execution resources in a multi-cloud environment.

[0042] The execution status tracking mechanism adopts an event-driven architecture, using a publish-subscribe pattern to achieve real-time status updates. The system publishes status events at key nodes, and subscribers can be user interfaces, alarm systems, or analysis modules, ensuring the timeliness and consistency of status information.

[0043] The key points that this invention needs to protect are as follows: (1) Unified job management architecture: It organically integrates script management, template management, scheme management and execution record management to form a unified job lifecycle management system, breaking the traditional decentralized management model. This architecture achieves seamless collaboration between modules through a unified data model and API interface, providing users with a consistent operating experience.

[0044] (2) Multi-dimensional resource selection and execution mechanism: It innovatively supports defining the execution scope from multiple dimensions such as image, cloud host, operating system, business system, host tag, physical machine, etc. The system automatically maps the logical scope to the specific execution target. This mechanism abstracts through the middle layer, shields the differences of underlying resources, and realizes unified operation management across environments.

[0045] (3) Hybrid process definition framework: It organically combines manual review and automated operation, and supports complex process definitions such as conditional branching and data flow. In particular, the seamless connection between manual review nodes and automated processes solves the problem that traditional automated systems cannot handle scenarios that require manual decision-making.

[0046] (4) Full execution status tracking technology: It not only records the final execution result, but also completely saves key information such as environmental snapshots, inputs and outputs, and intermediate states during the execution process, forming a traceable chain of execution evidence. This technology controls storage costs while ensuring data integrity through low-overhead sampling and intelligent compression algorithms.

[0047] (5) Dynamic resource scheduling algorithm: Based on real-time load and historical data, a predictive scaling strategy is adopted to achieve optimal allocation of execution resources. This algorithm considers factors such as task priority, resource requirements, and execution history, and continuously optimizes scheduling decisions through machine learning, significantly improving resource utilization efficiency.

[0048] Compared with the prior art, the present invention has the following significant advantages: This invention improves management efficiency by reducing system switching and lowering learning costs through a unified management interface. Centralized management of scripts, templates, and solutions reduces redundant development and increases reusability. Version control and change auditing reduce the risk of configuration drift. Real-world testing shows that after adopting this system, enterprise operational management efficiency improves by over 40%, and the error rate decreases by 60%.

[0049] This invention enhances process flexibility, supporting the flexible definition of complex business processes and adapting to diverse scenarios. Seamless integration of manual review and automated steps balances security and efficiency. The data flow mechanism between steps supports complex data processing workflows.

[0050] This invention accelerates problem localization, provides a complete execution log, offers "time machine"-like problem backtracking capabilities, enables multi-dimensional query analysis, and quickly locates performance bottlenecks. Environment variable snapshots ensure problem reproducibility.

[0051] This invention optimizes resource utilization, dynamically schedules resources, automatically expands capacity during peak periods to ensure service level alignment (SLAs), and automatically scales down during off-peak periods to reduce resource costs. Intelligent task scheduling balances the load and avoids resource contention. Real-world testing shows that, with the same hardware resources, this system can handle 2-3 times the workload of traditional solutions.

[0052] This invention is adaptable and scalable, enabling unified management across cloud and hybrid environments, thus reducing management complexity. Its modular architecture facilitates functional expansion and customized development. Standardized API interfaces facilitate integration with existing systems.

[0053] This invention enhances security and compliance by providing fine-grained access control to ensure the principle of least privilege; complete audit trails to meet compliance requirements; encrypted storage of sensitive information to prevent data leakage; and manual review nodes to ensure that critical operations are supervised by humans, reducing the risk of misoperation.

[0054] For the specific implementation scheme of this embodiment, please refer to the relevant descriptions in the above embodiments, which will not be repeated here.

[0055] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0056] It should be noted that in the description of this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means at least two.

[0057] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0058] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution device. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0059] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The corresponding program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0060] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0061] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0062] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0063] This invention employs a system for unified automated job management and execution in a multi-cloud environment. It constructs a unified job management architecture, organically integrating scripts, templates, schemes, and execution records to break down management silos. It provides a flexible process definition mechanism, supporting seamless integration of manual and automated operations to adapt to complex business scenarios. A complete execution traceability system is established, recording detailed status throughout the job lifecycle, supporting rapid problem localization and optimization. A dynamic resource scheduling mechanism is designed to automatically adjust execution resources based on load, improving resource utilization efficiency. An open and scalable system architecture is constructed to support unified job management in multi-cloud and hybrid environments.

[0064] In this specification, the invention has been described with reference to specific embodiments thereof. However, it will be apparent that various modifications and variations can be made without departing from the spirit and scope of the invention. Therefore, the specification and drawings should be considered illustrative rather than restrictive.

Claims

1. A system for unified automated management and execution of tasks in a multi-cloud environment, characterized in that, The system includes: The script management module is used for centralized management, version control, category tag management, and online testing of various scripting languages; The template management module, connected to the script management module, is used to define standardized work process templates and supports global variables, data transfer between steps, and execution strategy configuration. The solution management module, connected to the template management module, is used to apply the standardized work process template to specific business scenarios and provide execution strategy configuration. The execution record module, connected to the solution management module, is used to record the detailed process of each execution, enabling multi-dimensional queries and visualization.

2. The system for unified automated management and execution of jobs in a multi-cloud environment according to claim 1, characterized in that, The system also includes an execution engine, which adopts a layered architecture and includes: The request parsing layer is used to receive user operation requests and perform parameter validation and permission checks. The process orchestration layer, connected to the request parsing layer, is used to parse the execution process based on the template and generate an execution plan; The resource scheduling layer, connected to the process orchestration layer, is used to dynamically allocate execution resources based on the execution plan and the current load. The task execution layer, connected to the resource scheduling layer, is used to execute tasks on allocated resources and monitor the execution status. The results processing layer, connected to the task execution layer, is used to collect execution results, update the status, and trigger subsequent operations.

3. The system for unified automated management and execution of jobs in a multi-cloud environment according to claim 2, characterized in that, The execution engine enables dynamic scaling up and down. It collects CPU, memory, and queue length metrics in real time, dynamically adjusts the number of execution nodes based on preset thresholds, releases resources after the executed task is completed, and schedules execution resources in the nearest cloud environment.

4. The system for unified automated management and execution of jobs in a multi-cloud environment according to claim 1, characterized in that, The script management module provides a script asset library, centrally manages multiple scripting languages, and offers script categorization, tag management, and version rollback.

5. The system for unified automated management and execution of jobs in a multi-cloud environment according to claim 1, characterized in that, The template management module implements step execution strategy configuration, specifically including timeout control, error handling strategy and permission requirements; the template management module adopts an inter-step data transfer mechanism to realize data flow in complex business processes.

6. The system for unified automated management and execution of jobs in a multi-cloud environment according to claim 1, characterized in that, The aforementioned scheme management module has multiple execution strategies, including single execution, periodic execution, and event-triggered execution, and performs precondition checks.

7. The system for unified automated management and execution of jobs in a multi-cloud environment according to claim 1, characterized in that, The execution record module also implements lifecycle management of execution records, enabling complete preservation and traceable query of data throughout the execution process, and supports archiving and cleanup strategies.

Images