A batch automation processing execution system for large cloud platforms

Abstract

The application relates to a batch automation processing execution system for a large cloud platform, comprising a job package management module, a batch job runtime module, a cloud server agent module and a service center module. The application can solve the technical problems that large-scale cloud resource operation and maintenance operations cannot be concurrently executed, operation and maintenance operations are not unified, concurrent operation and maintenance scripts are long to compile and are prone to errors.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of cloud server resource batch operation and maintenance, and particularly relates to a batch automatic processing execution system for large-scale cloud platforms. BACKGROUND

[0002] At present, cloud computing technology has been widely applied to production and life, and many enterprises have established their own private cloud platforms. Some large private cloud platforms have reached thousands of physical host nodes, and the distributed virtual cloud servers have reached tens of thousands. With the growth of business, the number of cloud servers will continue to grow.

[0003] On the one hand, the cloud platform operation and maintenance side needs to perform software installation, version upgrade, vulnerability repair, regular inspection and other operation and maintenance operations on the cloud server. If there is no good batch processing tool, it is unrealistic to maintain so many cloud servers. At present, although there are operation and maintenance tools such as Ansible and Puppet, these tools have problems such as complex deployment, high learning cost and low running efficiency. Therefore, it is necessary to develop a batch management tool for cloud platforms.

[0004] On the other hand, many cloud platform operation and maintenance personnel can only write simple shell or python scripts, and they do not have the ability to write batch job running time. Therefore, it is necessary to abstract and strip the common and running related contents, so that the operation and maintenance personnel only focus on the script writing of the operation and maintenance job, and can be integrated with the running time through simple integration specifications.

[0005] On the other hand, there are many interactive inputs in the operation and maintenance job. The traditional method needs to use an expect command for processing. The most difficult thing is the use of regular expressions. For complex regular expressions, it is difficult to write correctly at one time.

[0006] Finally, the cloud server carries business, and the operation of the business system or the business personnel will cause the SSH service or the infrastructure to be unavailable. Therefore, it is necessary to install the corresponding agent on the cloud server to monitor the state of the cloud server and ensure the availability of the cloud server.

[0007] Therefore, in order to solve the above problems, the application provides a batch automatic processing execution system for large-scale cloud platforms.

[0008] Through retrieval, no existing technical disclosure literature identical or similar to the application has been found. SUMMARY

[0009] The purpose of the present application is to overcome the deficiencies of the prior art, and provide a batch automation processing execution system for large-scale cloud platforms, which can solve the technical problems of large-scale cloud resource operation and maintenance operations that cannot be executed concurrently, and operation and maintenance operations that are not unified, and the long time and error-prone of preparing operation and maintenance concurrent scripts.

[0010] The present application solves its practical problems by adopting the following technical solutions:

[0011] A batch automation processing execution system for large-scale cloud platforms comprises a job package management module, a batch job runtime module, a cloud server agent module and a service center module.

[0012] The batch job runtime module is used to abstract and encapsulate general batch running capabilities, and provide high concurrency, retry mechanism, timeout mechanism and exception handling mechanism required by batch processing operations.

[0013] The job package management module is used to encapsulate daily operation and maintenance jobs, and is responsible for the creation / update / deletion / query management responsibilities of operation and maintenance job packages. The encapsulation of daily operation and maintenance job packages needs to follow the integration specifications of job package encapsulation. The output end of the job package management module is connected with the batch job runtime module, which is used to receive the custom interactive language set by the platform operation and maintenance personnel, and package the complex job command logic into a job package, so as to seamlessly integrate with the runtime during job execution. At the same time, the job package management module provides a job orchestration language to orchestrate multiple job packages together to achieve the purpose of executing multiple jobs simultaneously.

[0014] The output end of the batch job runtime module is connected with multiple cloud server agent modules respectively, which is used to detect and repair problems existing in the cloud server, and deploy multiple agents on the corresponding cloud server to provide protection for batch job execution.

[0015] The output end of the service center is connected with the batch job runtime module, which is used to collect the status of all cloud servers of the cloud platform and provide a list of available cloud servers for the batch job runtime module.

[0016] Moreover, the batch job runtime module is also used for:

[0017] Concurrent execution: multiple connections are established simultaneously using the coroutine mechanism, and even a dedicated connection is established for each cloud server to ensure that the job can be quickly executed and completed.

[0018] Middleware: custom behaviors are inserted at the key stage of job execution.

[0019] Exception handling: retry is performed on the cloud server that fails to process, and the execution is abandoned after multiple retries fail, and the error information is recorded.

[0020] Timeout detection: provide timeout settings for the entire job or single execution of a single cloud server, when the timeout is not completed, the job will exit and be marked as a timeout exception.

[0021] Moreover, the encapsulation of the daily operation and maintenance job package needs to follow the integrated specification of the job package encapsulation, including:

[0022] The job package must adopt a certain compression format;

[0023] The job package must contain an entry file with a specified name; when the cloud server executes the job, it will be executed from the specified entry file;

[0024] If the execution is successful, 0 will be returned, and if other values are returned, it will indicate an error;

[0025] The job package must contain a version, which cannot be updated once released. If it needs to be updated, the version number must be upgraded;

[0026] If the job contains complex interaction logic, it can be described by the interaction language described in the application.

[0027] Moreover, the custom interaction language set by the platform operation and maintenance personnel is:

[0028] A set of more readable syntax, which is then converted into a regular expression by an interpreter; when actually encapsulating the job package, the interaction requirements can be described using the interaction language; when the job package is executed, these interaction requirements will be converted into the final regular expression and executable interaction logic.

[0029] Moreover, the job is divided into simple jobs and complex jobs; simple jobs represent jobs that execute a single job package, and complex jobs are jobs composed of multiple job packages, commands, and simple scripts; complex jobs need to be described by a job package arrangement language, and when the complex job is executed, the system runtime will execute the parts in sequence according to the arrangement requirements.

[0030] Moreover, the cloud server agent module is used to monitor the health status of the cloud server, ensure that the batch job can normally connect to the cloud server where it is located and normally execute the job; when the cloud server appears abnormally, the cloud server agent module can automatically repair. When a cloud server that cannot be repaired appears, the cloud server agent module will report the details of the cloud server to the service center.

[0031] Advantages and beneficial effects of the application:

[0032] The application provides a batch automation processing execution system for a large cloud platform, and provides a complete operation and maintenance script and a command batch execution development system.According to the constraints of the system, an operation and maintenance personnel can only focus on writing operation and maintenance atomic operations, and a large-scale batch execution operation can be realized by using the system running instruction, so that the difficulty of completing batch jobs by the operation and maintenance personnel is greatly reduced, and the execution efficiency of the batch jobs is improved.Firstly, the system abstracts and encapsulates batch job runtime related capabilities, such as high concurrency, exception handling, timeout and retry mechanism, so that the operation and maintenance personnel can better focus on the commands, scripts and logic of the job itself.Secondly, the system provides a custom interactive language, which reduces the difficulty of writing interactive commands, especially regular expressions, thereby replacing the traditional expect tool.Thirdly, the system provides an integration mechanism, which can package complex job command logic into a job package, so as to be seamlessly integrated with the runtime during job execution.Fourthly, the system provides a job orchestration language, which can orchestrate multiple job packages together to achieve the purpose of simultaneously executing multiple jobs.Finally, the system provides a special agent, which is deployed on the target cloud server and can provide functions such as abnormal detection and self-recovery of the ssh service, to ensure the health of the cloud server state. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 A batch automation processing system of the application is shown in the figure;

[0034] Figure 2 A job runtime middleware implementation principle of the application is shown in the figure;

[0035] Figure 3 A syntax tree structure of the application is shown in the figure. DETAILED DESCRIPTION

[0036] The embodiments of the application are further described in detail below with reference to the accompanying drawings:

[0037] A batch automation processing execution system for a large cloud platform, as shown in Figure 1 The system includes a job package management module, a batch job runtime module, a cloud server agent module and a service center module.

[0038] The batch job runtime module is used for abstracting and encapsulating general batch running capabilities, and providing high concurrency, retry mechanism, timeout mechanism and exception handling mechanism required by batch processing operations.

[0039] The job package management module is used for packaging daily operation and maintenance jobs, is responsible for the management responsibilities such as creation / update / deletion / query of operation and maintenance job packages, and needs to follow the integration specification of job package packaging for packaging daily operation and maintenance job packages; an output end of the job package management module is connected with the batch job runtime module, is used for receiving a self-defined interactive language set by a platform operation and maintenance personnel, and packs complex job command logic into a job package, so as to be seamlessly integrated with the runtime when the job is executed; meanwhile, a job arrangement language is provided, multiple job packages are arranged together, and the purpose of simultaneously executing multiple jobs is achieved.

[0040] An output end of the batch job runtime module is connected with multiple cloud server agent modules respectively, is used for detecting and repairing problems existing in the cloud servers, deploys multiple agents on the corresponding cloud servers, and provides guarantee for batch job execution.

[0041] An output end of the service center is connected with the batch job runtime module, is used for collecting the states of all cloud servers of the cloud platform, and provides a list of available cloud servers for the batch job runtime module.

[0042] In the embodiment, the batch job runtime module is further used for:

[0043] Concurrent execution: a coroutine mechanism is adopted to simultaneously establish multiple connections, and even a special connection is established for each cloud server, so as to ensure that the job can be quickly executed and ended.

[0044] Middleware: custom behaviors are inserted in the key stage of job execution.

[0045] For example, before or after the whole job starts, before or after the job of each cloud server starts. The middleware can be flexibly customized according to needs.

[0046] Exception handling: the cloud server with a processing failure is retried, and when the multiple retries fail, the execution is abandoned and error information is recorded.

[0047] Timeout detection: timeout setting is provided for the whole job or single execution of a single cloud server, and when the timeout is not completed, the job will be exited and is marked as a timeout exception.

[0048] In the embodiment, the packaging of the daily operation and maintenance job package needs to follow the integration specification of job package packaging, including:

[0049] The job package must adopt a compression format, such as a zip format.

[0050] The job package must contain an entry file with a specified name, such as main.sh or main.py. When the cloud server executes the job, the specified entry file will be executed.

[0051] If the execution is successful, 0 will be returned, and if other values are returned, it will indicate an error.

[0052] The job package must contain a version, which cannot be updated once released. If it is to be updated, the version number must be upgraded.

[0053] If complex interaction logic is included in the job, it can be described by the interaction language described in the application.

[0054] In the embodiment, the custom interaction language set by the platform operation personnel is:

[0055] The target of the interaction language is to reduce the difficulty of writing regular expressions, and the core capability is to invent a set of more readable descriptive syntax, and then convert it into a regular expression through an interpreter; when the job package is actually encapsulated, the interaction requirements can be described using the interaction language; when the job package is executed, these interaction requirements will be converted into the final regular expression and executable interaction logic.

[0056] In the embodiment, the job is divided into a simple job and a complex job; the simple job represents a job of executing a single job package, and the complex job is a job composed of multiple job packages, commands, and simple scripts; the complex job needs to be described by a job package arrangement language, and when the complex job is executed, the system runtime will execute the parts in sequence according to the arrangement requirements.

[0057] In the embodiment, the cloud server agent module is used for monitoring the health status of the cloud server, ensuring that the batch job can be normally connected to the cloud server where it is located and normally executed when the batch job is running; when the cloud server is abnormal, the cloud server agent module can automatically repair. When the cloud server cannot be repaired, the cloud server agent module will report the details of the cloud server to the service center.

[0058] The working principle of the application is:

[0059] In a first aspect, the application proposes a batch processing system for a large cloud platform. The system can be divided into four parts: a batch job runtime, a job package management, a cloud server agent, and a service center.

[0060] The batch job runtime abstracts and encapsulates the general batch running capability, provides the high concurrency, retry mechanism, timeout mechanism, and exception handling mechanism required by the batch processing operation. The job package management is responsible for the management duties such as creation, update, deletion, and query of the job package. The encapsulated job package needs to follow the job package integration specification. The cloud server agent is responsible for detecting and repairing problems existing in the cloud server, and provides protection for the batch job execution. The service center is responsible for collecting the state of all cloud servers of the cloud platform, and providing a list of available cloud servers for the job runtime.

[0061] In a second aspect, the batch job runtime implements mechanisms or functions including high concurrency execution, middleware, exception handling, retry mechanism, timeout detection, etc.

[0062] The high concurrency execution mechanism uses coroutine technology to establish multiple connections at the same time, or even a dedicated connection for each cloud server, to ensure that the job can be quickly executed and completed. The coroutine technology has an internal scheduling mechanism, is not limited by the number of local threads, and has the advantages of easy control of concurrency. The middleware mechanism provides the ability to customize behaviors for job execution. Before or after the entire job starts, before or after each cloud server job starts. The middleware can be flexibly customized as needed. The exception handling mechanism retries the cloud server that fails to process, and gives up execution and records error information when multiple retries fail. The timeout detection mechanism provides a timeout setting for the entire job or a single execution of a single cloud server. When the timeout is not completed, the job will exit and be marked as a timeout exception.

[0063] In a third aspect, when packaging a job package, the integration specification must be followed, otherwise it will not be integrated with the job runtime.

[0064] The integration specification includes: the job package must use a certain compression format, such as zip format; the job package must contain an entry file with a specified name, such as main.sh or main.py. When the cloud server executes the job, it will execute from the specified entry file; if the execution is successful, it will return 0, and if it returns other values, it will indicate an error; the job package must contain a version, which cannot be updated once released. If you want to update, you must upgrade the version number; if the job contains interactive logic, it can be described by the interactive language described in the present application.

[0065] In a fourth aspect, when complex interactive logic is involved, support for a custom interactive language is provided.

[0066] The interactive language reduces the difficulty of writing regular expressions. During job package execution, these interactive languages are converted into final regular expressions and executable interactive logic. The core of the interactive language is to simplify the description of regular expressions.

[0067] The present application divides the basic patterns of regular expressions into eleven kinds, including maybe, one_or_more, etc., and their corresponding meanings are shown in the following table.

[0068] Pattern Regular expression notation Meaning zero_or_more('a') a* Zero or more a's one_or_more('a') a+ One or more a's repeat(m,n,'a') a{m,n} Repeat a m to n times numberic() 0-9 Numeric maybe('a') a？ Zero or one a's alpha() a-zA-Z Alphabetic char() . Any character one_char_of('a','b') [ab] One of a or b range('a','z') a-z From a to z line('a') ^a$ Line containing only a's group('ab') (ab) Group consisting of a and b

[0069] In writing a complex regular expression, for example, a regular expression for matching an email address is "[a-zA-Z0-9_-]+@[a-zA-Z0-9_-]+(\.[a-zA-Z0-9_-]+)+", which can be rewritten as "$range,'@ ', $range, one_or_more(char(), group($range))" using the interactive language. The variable range represents the string one_or_more(one_char_of(alpha(), numeric(), '_', '-')). It can be seen that the custom interactive language has better readability and description capability.

[0070] During the job package runtime, the job runtime converts the above-described language string into a regular expression. The following describes the specific conversion process. The entire conversion process is divided into two steps. First, the input string is tokenized and a syntax tree is constructed. Then, the conversion result is output by evaluating and traversing in the middle order.

[0071] Taking the above regular email statement as an example. Tokenization is to identify keywords, string constants, and variables from the above statement. With the tokenization result as input, a syntax tree as shown in Figure 3 It can be seen that the three sub-trees in the tree framed by the dashed line correspond to the $range clause of the interactive statement.

[0072] Next, the syntax tree is post-ordered, and then each node is evaluated in turn. The specific evaluation algorithm is as follows.

[0073]

[0074] T is a logical tree with syntax relationships obtained by syntax escaping a statement, S

[0075] is the set of all branches of the syntax tree, containing all tokens of the statement, s is a token in the syntax set, and rs is a regular expression string to be generated.

[0076] In a fifth aspect, a job can be divided into a simple job and a complex job.

[0077] The simple job represents a job for executing a single job package, and the complex job is a job composed of multiple job packages and commands, and simple scripts. The complex job needs to be described by the job package arrangement language. During execution of the complex job, the system runtime will execute the parts in order according to the arrangement requirements.

[0078] The following is an example of a complex job orchestration. As can be seen, this job contains two steps: check and vmtools. The check is used to execute a command to check whether the vm-agent service exists, and if it exists (success), the job ends. Otherwise, the vmtools step will be continued, which is responsible for installing the vmtools tool with version number 2.3, and the installation location can be specified through the location parameter.

[0079] version: '1'

[0080] jobs:

[0081] check:

[0082] kind: cmd

[0083] command: systemctl status vm-agent

[0084] success: break

[0085] vmtools:

[0086] kind: package

[0087] package: vmtools: 2.3

[0088] params:

[0089] -location: / opt / vmtools

[0090] In a sixth aspect, the cloud server agent is used to monitor the health status of the cloud server, to ensure that the batch job can be normally connected to the cloud server where it is located and normally executed when running. When the cloud server appears abnormal, the agent can automatically repair. When the cloud server that cannot be repaired appears, the agent will report the details of the cloud server to the service center.

[0091] Embodiment 1

[0092] In actual embodiments, the job can be deployed as a service, in which case, a client can be configured for the service, such as a command line, a web interface, etc. At the same time, in addition to job package management, common scenarios can be encapsulated as auxiliary functions. For the daily work of cloud platform operation and maintenance, these auxiliary functions can include batch command execution, simple script batch execution, batch file copying, etc. Finally, in the formal implementation, log management, user authentication, operation audit, related batch execution record management, etc. functions are generally also needed to be provided.

[0093] In actual embodiments, the job runtime is generally implemented in Go language, and batch execution is an excellent scenario for applying Go language coroutine technology. During the running, the cloud servers to be processed can be divided into several groups, each group containing one or more cloud servers. Each coroutine is responsible for the job execution of a group of cloud servers. After the execution, the execution result is reported through the channel message.

[0094] Regarding the login mode of the job runtime, the SSH login based on password is generally used, and the root is generally not allowed to log in directly. According to the actual needs of operation and maintenance, multiple login modes should be supported during login. Common modes include su, sudo, direct, and sudo-i, wherein the su and sudo-i modes represent root user switching by using the su command and the sudo-i command, and the root user is maintained during the execution process, the direct mode represents executing the job directly as the login user, and the sudo mode represents using the sudo prefix to ensure that the root permission can be obtained during the job execution process.

[0095] The middleware mechanism of the job runtime is an important mechanism for ensuring scalability. The capabilities of the job runtime, such as the running time of the job, the history record, and the like, can be extended through the middleware mechanism. Next, a feasible design method for defining the middleware mechanism is described. The middleware function type can be defined as type HandlerFunc handle(*Context), wherein the Context type has a Next() method. It is assumed that two custom middlewares, Logger and Timer, are registered before the job execution, which are respectively used to record the logs before and after the job running and the execution time. Finally, the method Execute function for executing the job is also wrapped as a middleware Executor, and its execution process is as shown in Figure 2 .

[0096] From Figure 2As can be seen, the three registered middlewares are saved into ctx, ctx will first execute the first middleware logger, the handle function of logger will call ctx.Next function during execution, thus triggering timer middleware, similarly, the handle function of timer will execute Executor middleware during execution, which is the last middleware and the middleware for executing job logic. When Executor middleware is executing, ctx.Next method will not be called, thus returning to timer middleware to continue executing the unfinished logic, after the execution of timer middleware, logger middleware will be returned to execute the unfinished logic, until the final return call. As can be seen, this is a chain of calling process, during which, as long as one of the middlewares does not call ctx.Next function, the job will not be executed. On the other hand, parameters can be passed between middlewares through ctx object, thus forming a dependency, at this time, the order of middleware registration is necessary.

[0097] Regarding the job package management, in actual embodiments, the job package can be compressed into an rpm package (or other packaging format supported by the operating system, such as a deb package), and the rpm package can be published to a public rpm repository, and the yum source can be configured in all cloud servers, then when the job is executed, it can be automatically downloaded from the rpm repository without transmission by the job runtime. In this case, only necessary parameters need to be passed when installing the rpm package command. On the other hand, a special syntax can be designed to define environment variable parameters and set them before job execution, thus achieving global reference at the job level.

[0098] In some embodiments, regarding the acquisition of cloud server list information, it can be customized according to the general situation of the cloud platform. For example, the cloud platform generally runs several resource sets (business systems), each resource set has several cloud servers, and the resource set also belongs to different deployment environments, and the cloud servers have different tags, etc. The cloud server list that meets the scope of this job can be obtained according to the attributes of these cloud servers. Similarly, when the job is finished, the corresponding attributes can be transferred into the output result as needed.

[0099] It should be emphasized that the embodiments described in the present application are illustrative rather than restrictive, and thus the present application includes but is not limited to the embodiments described in the specific embodiments, and any other embodiments derived by those skilled in the art according to the technical solutions of the present application also belong to the scope of protection of the present application.

Claims

1. A batch automation processing execution system for large cloud platforms, characterized by: The application relates to a cloud platform operation and maintenance system, which comprises a job package management module, a batch job runtime module, a cloud server agent module and a service center module. The batch job runtime module is used for abstracting and encapsulating general batch running capability, providing high concurrency, retry mechanism, timeout mechanism and exception handling mechanism required by batch processing operation. The job package management module is used for encapsulating daily operation and maintenance jobs, and is responsible for the management of operation and maintenance job packages. The output end of the job package management module is connected with the batch job runtime module, is used for receiving a self-defined interactive language set by a platform operation and maintenance personnel, and packs complex job command logic into a job package, so that the job package can be seamlessly integrated with the runtime during job execution. Meanwhile, the job package management module provides a job arrangement language, arranges multiple job packages together, and realizes the purpose of simultaneously executing multiple jobs. The output end of the batch job runtime module is connected with multiple cloud server agent modules, is used for detecting and repairing problems existing in cloud servers, and deploys multiple agents on corresponding cloud servers to provide guarantee for batch job execution. The output end of the service center is connected with the batch job runtime module, is used for collecting the states of all cloud servers of a cloud platform, and provides a usable cloud server list for the batch job runtime module. The job is divided into a simple job and a complex job. The simple job represents a job of executing a single job package, and the complex job is a job combined by multiple job packages, commands and simple scripts. 2.The batch automation processing execution system for large cloud platform of claim 1, wherein: The complex job needs to be described by a job package arrangement language, and the system runtime executes parts of the complex job in sequence according to the arrangement requirement during execution of the complex job. The self-defined interactive language set by the platform operation and maintenance personnel is a set of more readable descriptive syntax, which is converted into a regular expression through an interpreter. The batch job runtime module is also used for: Concurrent execution: a coroutine mechanism is adopted to simultaneously establish multiple connections, and a special connection is established for each cloud server to ensure that the job can be quickly executed and ended; Middleware: custom behaviors are inserted in a key stage of job execution; 3.The batch automation processing execution system for large cloud platform of claim 1, wherein: Exception handling: a failed cloud server is retried, and the execution is abandoned and error information is recorded after multiple retries fail; Timeout detection: timeout setting is provided for the whole job or single execution of a single cloud server, and the job will be exited and marked as a timeout exception when the timeout is not completed. The encapsulated daily operation and maintenance job package needs to follow the integration specification of job package encapsulation, and the integration specification comprises that: The job package must adopt a certain compression format; The job package must contain an entry file with a specified name, and the specified entry file will be executed when the cloud server executes the job; If the execution is successful, 0 will be returned, and if other values are returned, errors will be indicated; The job package must contain a version, and the version cannot be updated once published; if the version needs to be updated, the version number must be upgraded; The job package must contain a version, and the version cannot be updated once published; if the version needs to be updated, the version number must be upgraded; If the job contains complex interaction logic, the complex interaction logic is described by the custom interaction language.

4. The batch automation processing execution system for large cloud platform of claim 1 or 2, wherein: The cloud server agent module is used for monitoring the health state of the cloud server, ensuring that the batch job is normally connected to the cloud server where the batch job is located and the batch job is normally executed; when the cloud server is abnormal, the cloud server agent module can be automatically repaired; when the cloud server cannot be repaired, the cloud server agent module reports the details of the cloud server to the service center.

Images