An automated operation and maintenance system, method, device, medium and program product

Abstract

The application provides an automatic operation and maintenance system, method, device, medium and program product, the method comprising: an orchestration execution layer, configured to orchestrate an operation and maintenance job process according to an atomic capability corresponding to an operation and maintenance operation instruction configured in an atomic capability layer, wherein each step in the operation and maintenance job process is an asynchronous task, an interception process of a high-risk instruction is set for each step in the operation and maintenance job process, and the operation and maintenance job process is issued to a command pipeline layer; the command pipeline layer comprises: a total agent node and a sub-agent node configured at a terminal; the total agent node is configured to receive the operation and maintenance job process issued by the orchestration execution layer and issue the operation and maintenance job process to a corresponding sub-agent node; and the sub-agent node is configured to execute the operation and maintenance job process. In the application, by constructing an automatic operation and maintenance system architecture, asynchronous task design and high-risk instruction interception are performed on the orchestration execution layer and the command pipeline layer, thereby improving the automatic operation and maintenance efficiency and safety of the system.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of cloud computing, and particularly relate to an automatic operation and maintenance system, method, device, medium and program product. BACKGROUND

[0002] The existing automatic operation and maintenance technology is based on the Ansible tool for automatic operation and maintenance, and the operation is irreversible and cannot be rolled back, cannot intercept high-risk instructions, has high operation and maintenance risk coefficient for large-scale nodes, poor safety and reliability, and needs manual intervention. Moreover, the Ansible tool does not support rolling execution, and when facing high-risk operation and maintenance operations, it does not have a flexible rollback mechanism, and the adaptability cannot be guaranteed. In the processing of complex operation and maintenance scenarios, the serial task flow in the Ansible tool cannot be completely covered, that is, multiple Ansible-Playbooks arrangement tasks need to be pre-arranged, and multiple tasks are run. When the node scale expands to 100,000 levels, the repetitive work is seriously accumulated, and the operation and maintenance efficiency is extremely low. The cloud management platform integrated with the Ansible tool does not support the arrangement of parallel tasks, branch tasks, convergence tasks and loop tasks, and cannot realize automation for complex public cloud operation and maintenance scenarios, has low observability and measurability, lacks index monitoring and call chain tracking mechanism, and does not support active reporting and statistics of public cloud node operation and maintenance data. SUMMARY

[0003] Embodiments of the present application provide an automatic operation and maintenance system, method, device, medium and program product to solve the problems of low efficiency and poor safety of existing public cloud large-scale node automatic operation and maintenance.

[0004] To solve the above technical problems, the present application is implemented as follows:

[0005] In a first aspect, the embodiments of the present application provide an automatic operation and maintenance system, comprising:

[0006] an arrangement execution layer and a command pipeline layer;

[0007] The arrangement execution layer is configured to arrange operation and maintenance job processes according to atomic capabilities corresponding to operation and maintenance operation instructions configured in the atomic capability layer, wherein each step in the operation and maintenance job processes is an asynchronous task, a high-risk instruction interception process is set for each step in the operation and maintenance job processes, and the operation and maintenance job processes are issued to the command pipeline layer; wherein each operation and maintenance scenario corresponds to an operation and maintenance job process.

[0008] The command pipeline layer comprises: a general agent node and a sub-agent node arranged at a terminal; the general agent node is configured to receive an operation and maintenance job flow issued by the orchestration execution layer and issue the operation and maintenance job flow to at least one corresponding sub-agent node; the sub-agent node is configured to execute the operation and maintenance job flow and feed back an execution result to the corresponding general agent node; and a long link listening mode is set for the general agent node and the sub-agent node, and the listening content is centrally stored.

[0009] Optionally, the method further comprises:

[0010] The atomic capability layer is configured to solidify the preset at least one operation and maintenance instruction into corresponding atomic capabilities and audit and publish the atomic capabilities in the orchestration execution layer.

[0011] Optionally, the method further comprises:

[0012] The operation and maintenance scenario layer is configured to manage an operation and maintenance job flow corresponding to an operation and maintenance scenario arranged in the orchestration execution layer and provide an interface to an external system.

[0013] In the second aspect, an embodiment of the present application provides an automatic operation and maintenance method, which applies the automatic operation and maintenance system according to any one of the first aspect.

[0014] The operation and maintenance job flow is arranged according to the atomic capabilities corresponding to the operation and maintenance instructions arranged in the atomic capability layer in the orchestration execution layer, wherein each step in the operation and maintenance job flow is an asynchronous task, and a high-risk instruction interception process is set for each step; and each operation and maintenance scenario corresponds to an operation and maintenance job flow.

[0015] The operation and maintenance job flow is issued to the command pipeline layer; the command pipeline layer comprises: a general agent node and a sub-agent node arranged at a terminal; the general agent node is configured to receive an operation and maintenance job flow issued by the orchestration execution layer and issue the operation and maintenance job flow to at least one corresponding sub-agent node; the sub-agent node is configured to execute the operation and maintenance job flow and feed back an execution result to the corresponding general agent node; and a long link listening mode is set for the general agent node and the sub-agent node, and the listening content is centrally stored.

[0016] Optionally, before the operation and maintenance job flow is arranged according to the operation and maintenance instructions arranged in the atomic capability layer, the method further comprises:

[0017] The preset at least one operation and maintenance instruction is solidified into corresponding atomic capabilities, and the atomic capabilities corresponding to the operation and maintenance instructions are audited and published.

[0018] Optionally, the method further comprises:

[0019] The management is performed on the operation and maintenance job flow corresponding to the operation and maintenance scene arranged in the arrangement execution layer, and an interface is provided to an external system.

[0020] Optionally, after the operation and maintenance job flow is arranged in the arrangement execution layer according to the atomic ability corresponding to the operation and maintenance operation instruction configured in the atomic ability layer, the method further includes:

[0021] An online log, an index monitoring and a call chain tracking are provided for each step in the operation and maintenance job flow.

[0022] Optionally, after the operation and maintenance job flow is arranged in the arrangement execution layer according to the atomic ability corresponding to the operation and maintenance operation instruction configured in the atomic ability layer, the method further includes:

[0023] A risk level attribute is set for the operation and maintenance job flow, and the operation and maintenance job flow is processed according to the corresponding risk level attribute.

[0024] Optionally, after the operation and maintenance job flow is arranged in the arrangement execution layer according to the atomic ability corresponding to the operation and maintenance operation instruction configured in the atomic ability layer, the method further includes:

[0025] A rolling execution strategy is set for each step in the operation and maintenance job flow according to a fixed node number, a percentage or a specific expression.

[0026] In a third aspect, an electronic device is provided, which includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, and the program or instructions, when executed by the processor, implement the steps in the automated operation and maintenance method according to any one of the first aspect.

[0027] In a fourth aspect, a readable storage medium is provided, which stores a program or instructions, and the program or instructions, when executed by a processor, implement the steps in the automated operation and maintenance method according to any one of the first aspect.

[0028] In a fifth aspect, a computer program product is provided, which includes computer instructions, and the computer instructions, when executed by a processor, implement the steps in the automated operation and maintenance method according to any one of the first aspect.

[0029] In the present application, by constructing an automated operation and maintenance system architecture, in the orchestration execution layer, the operation and maintenance operation instruction corresponding to the atomic ability configured in the atomic ability layer is arranged to orchestrate the operation and maintenance job process, and each step in the operation and maintenance job process is set as an asynchronous task and the interception process of the high-risk instruction in each step in the operation and maintenance job process; the total agent node in the command pipeline layer and the sub-agent node configured in the terminal set the long link listening mode, and the listening content is centralized stored, the orchestration execution layer and the command pipeline layer are designed as asynchronous tasks and high-risk instructions are intercepted, the system automation operation and maintenance efficiency and safety are improved, and the problems of low efficiency and poor safety of the existing public cloud large-scale node automation operation and maintenance are solved. BRIEF DESCRIPTION OF DRAWINGS

[0030] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments with reference made to the accompanying drawings. The drawings are for purposes of illustration only and are not intended to be limiting in any respect. Moreover, in the drawings, like reference numerals designate like parts throughout the various figures, wherein:

[0031] Figure 1 is a structural schematic diagram of an automated operation and maintenance system provided by an embodiment of the present application;

[0032] Figure 2 is a structural schematic diagram of an orchestration execution layer of an automated operation and maintenance system provided by an embodiment of the present application;

[0033] Figure 3 is a schematic diagram of a visual canvas of the structure of an automated operation and maintenance system provided by an embodiment of the present application;

[0034] Figure 4 is an interaction schematic diagram of an orchestration execution layer and a command pipeline layer of an automated operation and maintenance system provided by an embodiment of the present application;

[0035] Figure 5 is a flowchart of an automated operation and maintenance method provided by an embodiment of the present application;

[0036] Figure 6 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0038] Please refer to Figure 1 The embodiment of the application provides an automatic operation and maintenance system, comprising:

[0039] The arrangement execution layer 2 is used for arranging operation and maintenance job processes according to atomic capabilities corresponding to operation and maintenance operation instructions configured in the atomic capability layer 3, wherein each step in the operation and maintenance job processes is an asynchronous task, an interception process of a high-risk instruction in each step in the operation and maintenance job processes is set, and the operation and maintenance job processes are issued to the command pipeline layer 1; wherein each operation and maintenance scene corresponds to an operation and maintenance job process.

[0040] In the embodiment of the application, please refer to The arrangement execution layer 2 takes a process engine as a core capability, is supplemented by visual canvas, asynchronous tasks, process control, process analysis, context management and execution engine and the like, and drives large-scale node automatic operation and maintenance; supports arrangement of parallel tasks, branch tasks, converging tasks and cyclic tasks, covers all operation and maintenance scenes of a public cloud, each operation and maintenance scene corresponds to an arrangement process, each step in the process supports interception of a high-risk instruction, each step internally calls an interface service (including HTTP, TCP private protocol, GRPC and the like) provided by the command pipeline layer 1 to issue instructions to corresponding nodes for execution; and the independent process supports combination, nesting and calling, for example, a fault discovery process and a fault disposal process are combined to complete a whole-process operation and maintenance scene of fault self-healing; wherein, logs are provided for each process step in the arrangement execution layer 2 based on an OpenTelemetry protocol, so as to meet calling link observability, report and statistics of internal operation data based on a private protocol, facilitate system service operation analysis, and improve system automatic operation and maintenance efficiency and safety.

[0041] Figure 2 Specifically, the visual canvas supports arrangement of a process by a user in a drag-and-drop manner; supports arrangement of parallel tasks, branch tasks, converging tasks and cyclic tasks; supports rapid copying of a step; supports direct nested reference of a process to a process; supports canvas beautification; supports rapid preview and the like; please refer to

[0042] Figure 3 ​The complete public cloud batch node problem automation processing flow arranged by the visual canvas includes: setting a flow start, performing problem bounding, triggering parallel checking of problems in four layers of calculation, storage, network and application through the capability of a parallel gateway; dynamically transferring the problem checking result in each layer through the capability of a branch gateway; after the processing results of the four layers of calculation, storage, network and application are all collected to a convergence gateway, the flow continues to flow down; one branch of the branch gateway is connected to a loop start node, and a loop task is implemented by using the branch gateway, if the problem is not recovered, the problem processing step is looped until all the four-layer problems are recovered, and the flow ends.

[0043] The flow control includes flow parsing and context management; global parameters are configured in the flow to be provided to each step; the out parameters and the in parameters of each step can also refer to each other; the parameters in the flow support encrypted storage; the parameters in the flow can be referred to as variables, and support reference through the mode of "${key}"; the branch task in the flow supports binary operators such as "==,!=, >, >=, <, <=, in, not in" and keywords such as "and, or, True / true, False / false", and is parsed in real time by the parser of the flow engine during task execution; the branch gateway task can be matched with a single selection drop-down box variable to ensure that only one branch hits, and the parallel gateway task can be matched with a multi-selection drop-down box variable to realize multiple branch hits.

[0044] The asynchronous task is set as an asynchronous task for each step execution in the flow engine, through the setting of the asynchronous task, resource occupation can be effectively released for a task with a long execution time; according to actual needs, an asynchronous task channel can be built with Celery as a core component, when a task fails or connection interruption occurs during execution, Celery will automatically try to re-execute the task, meet the high availability mechanism, and improve the system automation operation efficiency and safety.

[0045] The execution engine sends an execution signal when processing large-scale parallel tasks, so that the command pipeline layer 1 receives the execution signal and starts execution, improving the system automation operation efficiency and safety.

[0046] The command pipeline layer 1 includes: a total agent node and a sub-agent node configured in a terminal; the total agent node is used to receive an operation and maintenance job flow issued by the arrangement execution layer 2, and issue the operation and maintenance job flow to at least one corresponding sub-agent node; the sub-agent node is used to execute the operation and maintenance job flow and feed back the execution result to the corresponding total agent node; wherein the total agent node and the sub-agent node are set in a long link listening mode, and the listening content is centralized stored.

[0047] In the embodiment of the application, based on the architecture of the total agent node and the sub-agent node configured in the terminal, i.e. Proxy+Agent, first, the Agent is installed on all server, network device, security device or application system nodes, second, 5-10 nodes are selected as Proxy and the Task Server is deployed, the Task Server has remote job task execution capability (such as Bash, Perl, Bat, PowerShel, Python, SQL, etc.), the Proxy and the Agent establish long connection for communication, set long link listening mode, and centralized storage is performed on the listening content; the execution layer sends the job execution task instruction, the Proxy receives the instruction and issues it to the corresponding Agent node, and finally the actual node completes the execution action of a task.

[0048] Specifically, the specific interaction flowchart of the orchestration execution layer 2 and the command pipeline layer 1 is as shown in Figure 4 The orchestration execution layer arranges the operation and maintenance job process according to the atomic ability corresponding to the operation and maintenance operation instruction configured in the atomic ability layer; the orchestration execution layer internally integrates a safety mechanism of high-risk instruction interception and risk level judgment function: and a review mechanism before triggering the process execution; each step supports rolling execution to minimize the impact of failure; all steps provide online logs, index monitoring and call chain tracking based on the OpenTelemetry protocol; wherein the scripts and instructions in each step of the operation and maintenance job process are scanned in full text to detect whether they contain high-risk instructions; as long as a step involves a high-risk instruction, the expert review mechanism is triggered, the entire process is packaged and transferred to the operation and maintenance expert, and only the process reviewed by the operation and maintenance professional can interact with the command pipeline layer; in addition to the interception of high-risk instructions, the operation and maintenance job process can also be individually set with a risk level attribute, wherein the query type process is set as low-risk by default, the operation type process is set as medium-risk by default, and the change type process is set as high-risk by default; the processes with medium and high risk attributes are triggered by default before execution, the review mechanism is triggered, the entire process is packaged and transferred to the operation and maintenance expert, and only the process reviewed by the operation and maintenance professional can interact with the command pipeline layer; each step in the operation and maintenance job process can be set with a rolling execution strategy according to a fixed node number, a percentage and a specific expression; in a large-scale node operation and maintenance scenario, setting the rolling execution strategy can avoid the impact on services due to excessive concurrency, and can also gradually observe the effect after execution, thereby increasing the safety and reliability of automatic operation and maintenance; the orchestration execution layer from step arrangement, high-risk instruction scanning, risk level setting, expert review to rolling execution, all processes are connected with logs based on the OpenTelemetry protocol, which can facilitate and quickly track the call chain, facilitate system service operation analysis, and improve the efficiency and safety of system automatic operation and maintenance.

[0049] The execution engine in the orchestration execution layer generates an execution task request, and initiates a request to the command pipeline layer in an asynchronous task mode; after the command pipeline layer receives the request, an execution signal is placed in a message queue;

[0050] The total proxy node Proxy (Task Server) in the command pipeline layer synchronizes the request to other Proxy after consuming the execution signal in the message queue, and synchronously executes the task request.

[0051] The total proxy node Proxy (Task Server) issues an instruction to the corresponding sub-proxy node Agent.

[0052] The sub-proxy node Agent triggers an execution action, and returns the result of the instruction execution to the total proxy node Proxy after completing the running of the instruction on the node.

[0053] The total proxy node Proxy receives the result of the instruction execution, and stores the result in a DB.

[0054] After the total proxy node Proxy completes the result storage, the command pipeline layer main service is notified, and the command pipeline layer main service pushes the execution result to the execution engine in the orchestration execution layer; all steps in the command pipeline layer provide online logs, index monitoring and call chain tracking based on the OpenTelemetry protocol, so that the call chain tracking can be conveniently and quickly performed, system service operation analysis is facilitated, and system automatic operation and maintenance efficiency and safety are improved.

[0055] In the embodiment of the application, by constructing an automatic operation and maintenance system architecture, the orchestration execution layer arranges an operation and maintenance job process according to the atomic ability corresponding to the configured operation and maintenance instruction in the atomic ability layer, and sets each step in the operation and maintenance job process as an asynchronous task and sets an interception process for high-risk instructions in each step in the operation and maintenance job process; the total proxy node in the command pipeline layer and the sub-proxy node configured in the terminal set a long link listening mode, and the listening content is centralized stored, the orchestration execution layer and the command pipeline layer are designed as asynchronous tasks and high-risk instructions are intercepted, the system automatic operation and maintenance efficiency and safety are improved, and the problems of low efficiency and poor safety of the existing public cloud large-scale node automatic operation and maintenance are solved.

[0056] In the embodiment of the application, optionally, the method further comprises:

[0057] The atomic ability layer 3 is configured to solidify at least one preset operation and maintenance instruction as a corresponding atomic ability, and to audit and publish the atomic ability in the orchestration execution layer 2.

[0058] In the embodiment of the application, please refer to Figure 1The atomic capability layer 3 integrates OS layer, middleware layer, application environment layer, database layer, and service application layer, and all atoms are scripts, and all scripts are instructions. The operation and maintenance personnel solidify a series of operation and maintenance operation instructions into corresponding atomic capabilities, that is, OS management capability, middleware management capability, environment management capability, database management capability, and service management capability. All atomic capabilities are audited by operation and maintenance experts and published on the system. The risk of executing atomic capabilities on the node is evaluated in advance, and the operation and maintenance safety risk is reduced. After the atomic capability is published to the system, it is integrated into the process steps of the orchestration execution layer 2. The atomic capability is configured once and reused permanently, and the communication and coordination time cost of multiple roles is reduced.

[0059] In the embodiment of the application, the atomic capability layer 3 is configured to integrate OS layer, middleware layer, application environment layer, database layer, and service application layer.

[0060] The operation and maintenance scene layer 4 is configured to manage the operation and maintenance job flow corresponding to the operation and maintenance scene arranged in the orchestration execution layer 2, and provide an interface to an external system.

[0061] In the embodiment of the application, the operation and maintenance scene layer 4 manages the operation and maintenance job flow corresponding to the operation and maintenance scene arranged in the orchestration execution layer 2. Through the fusion of the atomic capability layer and the orchestration execution layer, various daily operation and maintenance operations are encapsulated, such as automatic inspection, acceptance deployment, automatic backup, deployment installation, daily inspection, file backup, monitoring management, fault diagnosis, fault self-healing, and operation analysis.

[0062] The operation and maintenance scene layer 4 provides a native interface, and an external system (such as a monitoring platform, an alarm platform, and an acceptance platform) can remotely call and execute corresponding operation and maintenance operations. All operation and maintenance operation instructions are black boxes for users, and the temporary intervention of users on the operation instructions is limited. Through the scene mode, automatic operation and maintenance is realized, the occurrence rate of human-caused faults is greatly reduced, and the efficiency and safety of automatic operation and maintenance are improved.

[0063] Please refer to Figure 5 The embodiment of the application provides an automatic operation and maintenance method, and the automatic operation and maintenance system as described in Figure 1 The embodiment of the application provides an automatic operation and maintenance method, and the automatic operation and maintenance system as described in

[0064] Step 51: In the orchestration execution layer, the operation and maintenance job flow is arranged according to the atomic capability corresponding to the operation and maintenance operation instruction configured in the atomic capability layer. Each step in the operation and maintenance job flow is an asynchronous task, and a high-risk instruction interception process is set for each step. Each operation and maintenance scene corresponds to an operation and maintenance job flow.

[0065] In the embodiment of the application, the orchestration execution layer 2 takes the process engine as the core capability, and is supplemented by functions such as a visual canvas, an asynchronous task, process control, process analysis, context management and an execution engine to drive large-scale node automatic operation and maintenance; the orchestration execution layer 2 supports the orchestration of parallel tasks, branch tasks, convergent tasks and cyclic tasks, covers all operation and maintenance scenarios of public clouds, each operation and maintenance scenario corresponds to an orchestration process, each step in the process supports the interception of high-risk instructions, and each step internally calls the interface services (including HTTP, TCP private protocol, GRPC and various protocol family communications) provided by the command pipeline layer 1 to issue instructions to the corresponding nodes for execution; and the independent process supports combination, nesting and calling, for example, the combination of a fault discovery process and a fault handling process to complete a full-process operation and maintenance scenario of fault self-healing; wherein, logs are provided for each process step in the orchestration execution layer 2 based on the OpenTelemetry protocol, thereby meeting the call link observability capability, reporting and statistics of internal operation data based on the private protocol, facilitating system service operation analysis, and improving the efficiency and safety of system automatic operation and maintenance.

[0066] Step 52: issuing the operation and maintenance job process to the command pipeline layer; the command pipeline layer includes a total agent node and a sub-agent node configured in a terminal; the total agent node is used to receive the operation and maintenance job process issued by the orchestration execution layer and issue it to at least one corresponding sub-agent node; the sub-agent node is used to execute the operation and maintenance job process and feed back the execution result to the corresponding total agent node; wherein, the total agent node and the sub-agent node are set in a long link listening mode, and the listening content is centralized stored.

[0067] In the embodiment of the application, the command pipeline layer takes the total agent node and the sub-agent node configured in the terminal, that is, the Proxy+Agent architecture as the basis, first installs the Agent on the nodes of all servers, network devices, security devices or application systems, and then selects 5-10 nodes as the Proxy and deploys the Task Server, the Task Server has remote job task execution capability (such as Bash, Perl, Bat, PowerShel, Python, SQL, etc.), the Proxy and the Agent establish a long connection for communication, set a long link listening mode, and centralized store the listening content; the orchestration execution layer sends a job execution task instruction, the Proxy receives the instruction and issues it to the corresponding Agent node, and finally completes the execution action of a task on the actual node.

[0068] Specifically, the specific interaction flowchart of the orchestration execution layer 2 and the command pipeline layer 1 is as follows Figure 4As shown, the orchestration execution layer arranges the operation and maintenance job process according to the atomic capability configured in the atomic capability layer; the orchestration execution layer internally integrates a security mechanism of high-risk instruction interception and risk level judgment function: and a review mechanism before triggering the process execution; each step supports rolling execution to minimize the impact of failure; all steps provide online logs, index monitoring and call chain tracking based on OpenTelemetry protocol; wherein the scripts and instructions in each step of the operation and maintenance job process are full-text scanned to detect whether they contain high-risk instructions; as long as a step involves high-risk instructions, the expert review mechanism is triggered, the entire process is packaged and transferred to the operation and maintenance experts, and only the process reviewed by the operation and maintenance experts can interact with the command pipeline layer; in addition to the interception of high-risk instructions, the operation and maintenance job process can also be individually set with a risk level attribute, wherein the query type process is by default low-risk, the operation type process is by default medium-risk, and the change type process is by default high-risk; the processes with medium and high-risk attributes are by default triggered by the review mechanism before execution, the entire process is packaged and transferred to the operation and maintenance experts, and only the process reviewed by the operation and maintenance experts can interact with the command pipeline layer; each step in the operation and maintenance job process can set a rolling execution strategy according to a fixed node number, a percentage and a specific expression; in a large-scale node operation and maintenance scenario, setting a rolling execution strategy can avoid the impact on services due to excessive concurrency, and can also gradually observe the effect after execution, thereby increasing the safety and reliability of automated operation and maintenance; the orchestration execution layer from step arrangement, high-risk instruction scanning, risk level setting, expert review to rolling execution, all processes are connected to logs based on OpenTelemetry protocol, which can facilitate fast call chain tracking, facilitate system service operation analysis, and improve system automated operation and maintenance efficiency and safety.

[0069] The execution engine in the orchestration execution layer generates an execution task request and initiates a request to the command pipeline layer in an asynchronous task mode; after receiving the request, the command pipeline layer places an execution signal in a message queue;

[0070] After the total proxy node Proxy (Task Server) in the command pipeline layer consumes the execution signal in the message queue, it synchronizes the request to other Proxy and synchronously executes the task request.

[0071] The total proxy node Proxy (Task Server) issues instructions to the corresponding sub-proxy node Agent;

[0072] The sub-proxy node Agent triggers an execution action, returns the result of the instruction execution to the total proxy node Proxy after completing the execution of the instruction on the node;

[0073] The total proxy node Proxy receives the result of the instruction execution and stores the result in the DB.

[0074] After the total agent node Proxy completes the result storage, the total agent node Proxy notifies the command pipeline layer master service, and the command pipeline layer master service pushes the execution result to the execution engine in the orchestration execution layer; Wherein, all steps in the command pipeline layer provide online logs, index monitoring and call chain tracking based on OpenTelemetry protocol, which can facilitate and quickly track the call chain, facilitate system service operation analysis, and improve the efficiency and safety of system automation operation.

[0075] In the embodiment of the application, by constructing an automated operation and maintenance system architecture, the orchestration execution layer is configured to orchestrate the operation and maintenance job process according to the atomic ability corresponding to the operation and maintenance operation instruction configured in the atomic ability layer, and each step in the operation and maintenance job process is set as an asynchronous task and an interception process of high-risk instructions in the operation and maintenance job process; The total agent node in the command pipeline layer and the sub-agent node configured in the terminal are set to a long link listening mode, and the listening content is centralized stored, the orchestration execution layer and the command pipeline layer are designed as asynchronous tasks and high-risk instructions are intercepted, the efficiency and safety of system automation operation are improved, and the problems of low efficiency and poor safety of existing public cloud large-scale node automation operation are solved.

[0076] In the embodiment of the application, before the operation and maintenance operation instruction configured in the atomic ability layer is orchestrated, the operation and maintenance operation instruction is audited and released.

[0077] The pre-set at least one operation and maintenance operation instruction is solidified into corresponding atomic ability, and the atomic ability corresponding to the operation and maintenance operation instruction is audited and released.

[0078] In the embodiment of the application, please refer to Figure 1 The atomic ability layer 3 integrates atomic abilities of multiple dimensions such as OS layer, middleware layer, application environment layer, database layer and business application layer, all atoms are scripts, and all scripts are instructions; That is, the operation and maintenance personnel solidify a series of operation and maintenance operation instructions accumulated into corresponding atomic abilities, that is, OS management ability, middleware management ability, environment management ability, database management ability and business management ability, all atomic abilities will be audited by operation and maintenance experts and released on the system, the risk of executing atomic abilities on the node is evaluated in advance, and the operation and maintenance safety risk is reduced; After the atomic is released to the system, it is integrated into the process steps of the orchestration execution layer 2, the atomic ability is configured once and reused permanently, and the communication and coordination time cost of multiple roles is reduced.

[0079] In the embodiment of the application, optionally, it further includes:

[0080] The operation and maintenance job process corresponding to the operation and maintenance scene orchestrated in the orchestration execution layer is managed, and an interface is provided to an external system.

[0081] In the embodiment of the present application, the operation and maintenance scene layer 4 manages the operation and maintenance job flow corresponding to the operation and maintenance scene arranged in the arrangement execution layer 2, and through the fusion of the atomic capability layer and the arrangement execution layer, encapsulates various daily operation and maintenance operations such as automatic inspection, acceptance deployment, automatic backup, deployment installation, daily inspection, file backup, monitoring management, fault diagnosis, fault self-healing and operation analysis.

[0082] The operation and maintenance scene layer 4 provides a native interface, and external systems such as a monitoring platform, an alarm platform and an acceptance platform can remotely call and execute corresponding operation and maintenance operations; and all operation and maintenance operation instructions are black boxes for users, limiting temporary intervention of the users on the operation instructions, realizing automatic operation and maintenance through a scene mode, greatly reducing the failure rate caused by human factors, and improving the efficiency and safety of automatic operation and maintenance.

[0083] In the embodiment of the present application, after the operation and maintenance job flow is arranged in the arrangement execution layer according to the atomic capability corresponding to the operation and maintenance operation instruction configured in the atomic capability layer, the operation and maintenance job flow further includes:

[0084] Online logs, index monitoring and call chain tracking are provided for each step in the operation and maintenance job flow.

[0085] In the embodiment of the present application, each step supports rolling execution, and the impact of a fault is minimized; all steps provide online logs, index monitoring and call chain tracking based on the OpenTelemetry protocol; system service operation analysis is facilitated, and the call chain is more conducive to being tracked, and the fault scene is analyzed.

[0086] In the embodiment of the present application, after the operation and maintenance job flow is arranged in the arrangement execution layer according to the atomic capability corresponding to the operation and maintenance operation instruction configured in the atomic capability layer, the operation and maintenance job flow further includes:

[0087] A risk level attribute is set for the operation and maintenance job flow, and the operation and maintenance job flow is processed according to the corresponding risk level attribute.

[0088] In the embodiment of the present application, scripts and instructions in each step in the operation and maintenance job flow are scanned in full text, and it is detected whether high-risk instructions are included; as long as a step involves a high-risk instruction, an expert review mechanism is triggered, the entire flow is packaged and transferred to an operation and maintenance expert, and only the flow that passes the operation and maintenance professional review can interact with the command pipeline layer; in addition to the interception of high-risk instructions, the operation and maintenance job flow can also be separately set with a risk level attribute, wherein a query type flow is by default low-risk, an operation type flow is by default medium-risk, and a change type flow is by default high-risk; the flow with a medium or high risk attribute triggers a review mechanism by default before execution, the entire flow is packaged and transferred to an operation and maintenance expert, and only the flow that passes the operation and maintenance professional review can interact with the command pipeline layer, thereby improving the safety capability of automatic operation and maintenance.

[0089] Optionally, after the atomic capability operation instruction corresponding to the operation and maintenance operation instruction configured in the atomic capability layer is arranged in the operation and maintenance operation process, the embodiment of the application further comprises:

[0090] The rolling execution strategy is set for each step in the operation and maintenance operation process according to a fixed node number, a percentage or a specific expression.

[0091] In the embodiment of the application, the rolling execution strategy can be set for each step in the operation and maintenance operation process according to a fixed node number, a percentage or a specific expression. In a large-scale node operation and maintenance scenario, setting the rolling execution strategy can avoid the impact on services due to excessive concurrency, and can also gradually observe the effect after execution, thereby increasing the safety and reliability of automatic operation and maintenance. From step arrangement, high-risk instruction scanning, risk level setting, expert review to rolling execution, the whole process of the arrangement execution layer accesses logs based on the OpenTelemetry protocol, which can facilitate fast call chain tracking, facilitate system service operation analysis, and improve system automatic operation and maintenance efficiency and safety.

[0092] The embodiment of the application provides an electronic device 60, as shown in Figure 6 The principle block diagram of the electronic device 60 is shown in Figure 6 The principle block diagram of the electronic device 60 is shown in

[0093] The embodiment of the application provides a readable storage medium, and the readable storage medium stores programs or instructions. The programs or instructions are executed by a processor to realize each process of the embodiment of the automatic operation and maintenance method, and the same technical effects can be achieved. To avoid repetition, details are not described here.

[0094] The embodiment of the application further provides a computer program product, which comprises computer instructions. The computer instructions are executed by a processor to realize each process of the method embodiment shown in Figure 5 The same technical effects can be achieved. To avoid repetition, details are not described here.

[0095] Computer-readable media includes permanent and non-permanent, movable and non-movable media, which can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0096] It should be noted that in the technical solutions of the present disclosure, the collection, collection, update, analysis, processing, use, transmission, storage and other aspects of user personal information are in line with relevant laws and regulations, are used for legal purposes, and do not violate public order and good customs. Necessary measures are taken to prevent illegal access to user personal information data, and user personal information security and network security are maintained.

[0097] It should be noted that in this paper, the term "includes", "contains" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, article or device. Without more limitations, the element defined by the statement "includes a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0098] The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0099] Those skilled in the art can clearly understand the above-mentioned embodiment method can be realized by means of software and necessary general hardware platform, of course, also can be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application essentially or say the part of the prior art contribution can be embodied in the form of software products, the computer software product is stored in a storage medium (such as ROM / RAM, magnetic disc, optical disc), including a number of instructions to make a service classification device (may be a mobile phone, computer, server, air conditioner, or network equipment, etc.) executes the method described in various embodiments of the present application.

[0100] The above-mentioned is only the preferred embodiment of the present application, it should be pointed out that, for those skilled in the art, without departing from the principle of the present application, can make a number of improvements and refinements, these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. An automated operations system, characterized by, Comprise: The orchestration execution layer and the command pipeline layer; The orchestration execution layer is used for orchestrating operation and maintenance job processes according to atomic capabilities corresponding to configured operation and maintenance operation instructions in the atomic capability layer, wherein each step in the operation and maintenance job processes is an asynchronous task, an interception process of a high-risk instruction is set for each step in the operation and maintenance job processes, and the operation and maintenance job processes are issued to the command pipeline layer; wherein each operation and maintenance scene corresponds to an operation and maintenance job process; a rolling execution strategy is set for each step in the operation and maintenance job processes according to a fixed node number, a percentage or a specific expression; The command pipeline layer comprises a total agent node and a sub-agent node configured in a terminal; the total agent node is used for receiving the operation and maintenance job processes issued by the orchestration execution layer and issuing the operation and maintenance job processes to at least one corresponding sub-agent node; the sub-agent node is used for executing the operation and maintenance job processes and feeding back execution results to the corresponding total agent node; wherein a long link monitoring mode is set for the total agent node and the sub-agent node, and the monitoring content is centralized stored. 2.The automated operations system of claim 1, wherein, Further comprise: The atomic capability layer is used for solidifying at least one preset operation and maintenance operation instruction into corresponding atomic capabilities, and auditing and publishing the atomic capabilities in the orchestration execution layer. 3.The automated operations system of claim 1, wherein, Further comprise: The operation and maintenance scene layer is used for managing operation and maintenance job processes corresponding to the operation and maintenance scenes in the orchestration execution layer, and providing interfaces to external systems.

4. An automated operation and maintenance method, characterized in that, Applied to the automatic operation and maintenance system in any one of claims 1-3, comprising: In the orchestration execution layer, operation and maintenance job processes are orchestrated according to atomic capabilities corresponding to configured operation and maintenance operation instructions in the atomic capability layer, wherein each step in the operation and maintenance job processes is an asynchronous task, and an interception process of a high-risk instruction is set for each step; wherein each operation and maintenance scene corresponds to an operation and maintenance job process; a rolling execution strategy is set for each step in the operation and maintenance job processes according to a fixed node number, a percentage or a specific expression; The operation and maintenance job processes are issued to the command pipeline layer; the command pipeline layer comprises a total agent node and a sub-agent node configured in a terminal; the total agent node is used for receiving the operation and maintenance job processes issued by the orchestration execution layer and issuing the operation and maintenance job processes to at least one corresponding sub-agent node; the sub-agent node is used for executing the operation and maintenance job processes and feeding back execution results to the corresponding total agent node; wherein a long link monitoring mode is set for the total agent node and the sub-agent node, and the monitoring content is centralized stored.

5. The method of claim 4, wherein, Before the operation and maintenance job processes are orchestrated according to the atomic capabilities corresponding to the configured operation and maintenance operation instructions in the atomic capability layer, further comprising: At least one preset operation and maintenance operation instruction is solidified into corresponding atomic capabilities, and the atomic capabilities corresponding to the operation and maintenance operation instructions are audited and published.

6. The automation operation and maintenance method according to claim 4, characterized in that, Further comprise: The operation and maintenance scene layer is used for managing operation and maintenance job processes corresponding to the operation and maintenance scenes in the orchestration execution layer, and providing interfaces to external systems.

7. The automation operation and maintenance method according to claim 4, characterized in that, After the operation and maintenance job processes are orchestrated according to the atomic capabilities corresponding to the configured operation and maintenance operation instructions in the atomic capability layer in the orchestration execution layer, further comprising: Online logs, index monitoring and call chain tracking are provided for each step in the operation and maintenance job flow. 8.The automation operation and maintenance method of claim 4, wherein, After the operation and maintenance job flow is arranged according to the atomic ability in the atomic ability layer, the method further comprises: A risk level attribute is set for the operation and maintenance job flow, and the operation and maintenance job flow is processed according to the corresponding risk level attribute.

9. An electronic device, comprising: The processor, the memory and the program or instructions stored on the memory and executable on the processor are included, and the program or instructions are executed by the processor to implement the steps in the automatic operation and maintenance method according to any one of claims 4 to 8.

10. A readable storage medium characterized by: The program or instructions are stored on the readable storage medium, and the program or instructions are executed by the processor to implement the steps in the automatic operation and maintenance method according to any one of claims 4 to 8.

11. A computer program product, characterised in that, The computer instructions are executed by the processor to implement the steps in the automatic operation and maintenance method according to any one of claims 4 to 8.

Images