Batch automatic processing execution system for large-scale cloud platform

Abstract

The invention relates to a batch automatic processing execution system for a large cloud platform. The batch automatic processing execution system comprises a job package management module, a batch job runtime module, a cloud server agent module and a service center module. The technical problems that large-scale cloud resource operation and maintenance operations cannot be executed concurrently, the operation and maintenance operations are not unified, the time for compiling operation and maintenance concurrent scripts is long, and errors are prone to occurring can be solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of batch operation and maintenance of cloud server resources, and in particular to a batch automated processing execution system for large cloud platforms. Background Art

[0002] At present, cloud computing technology has been widely used in production and life. Many companies have established their own private cloud platforms. Some large private cloud platforms have a scale of thousands of physical host nodes and tens of thousands of allocated virtual cloud servers. As the business grows, the number of cloud servers will continue to grow.

[0003] On the one hand, the cloud platform operation and maintenance side needs to perform software installation, version upgrades, vulnerability repairs, regular inspections and other operation and maintenance operations on cloud servers. Without a good batch processing tool, it is unrealistic to maintain so many cloud servers. Although there are operation and maintenance tools such as Ansible and Puppet, these tools have problems such as complex deployment, high learning cost, and low operating efficiency. Therefore, it is necessary to develop a batch management tool for cloud platforms.

[0004] On the other hand, many cloud platform operation and maintenance personnel can only write simple shell or Python scripts. They do not have the ability to write batch job runtimes. Therefore, they must abstract and strip away common operation-related content so that operation and maintenance personnel can only focus on script writing of the operation and maintenance jobs themselves and integrate them with the runtime through simple integration specifications.

[0005] On the other hand, many operation and maintenance tasks require interactive input. Traditional methods require the use of commands such as expect to process them. The most difficult part is the use of regular expressions. It is difficult to write complex regular expressions correctly in one go.

[0006] Finally, the cloud server carries the business, and the operations of the business system or business personnel may cause the SSH service or infrastructure to be unavailable. Therefore, it is necessary to install the corresponding agent on the cloud server to monitor the status of the cloud server and ensure the availability of the cloud server.

[0007] Therefore, without solving the above problems, the present invention proposes a batch automated processing execution system for a large cloud platform.

[0008] After searching, no public documents of the prior art identical or similar to the present invention were found. Summary of the invention

[0009] The purpose of the present invention is to overcome the shortcomings of the prior art and propose a batch automated processing execution system for large cloud platforms, which can solve the technical problems that large-scale cloud resource operation and maintenance operations cannot be executed concurrently, the operation and maintenance operations are not unified, and the preparation of operation and maintenance concurrent scripts is time-consuming and error-prone.

[0010] The present invention solves the practical problem by adopting the following technical solutions:

[0011] A batch automated processing execution system for a large cloud platform, comprising: a job package management module, a batch job runtime module, a cloud server proxy module and a service center module;

[0012] The batch job runtime module is used to abstract and encapsulate general batch operation capabilities, and provide high concurrency, retry mechanism, timeout mechanism, and exception handling mechanism required for batch processing operations;

[0013] The job package management module is used to encapsulate daily operation and maintenance operations, and is responsible for the management duties of creating / updating / deleting / querying operation and maintenance job packages. The encapsulation of daily operation and maintenance job packages must comply with the integration specifications of job package encapsulation; the output end of the job package management module is connected to the batch job runtime module, which is used to receive the custom interactive language set by the platform operation and maintenance personnel, and package the complex job command logic into a job package so that it can be seamlessly integrated with the runtime during job execution; at the same time, a job scheduling language is provided to schedule multiple job packages together to achieve the purpose of executing multiple jobs at the same time;

[0014] The output end of the batch job runtime module is respectively connected to multiple cloud server agent modules to detect and repair problems existing in the cloud servers, deploy multiple agents on corresponding cloud servers, and provide guarantee for batch job execution.

[0015] The output end of the service center is connected to the batch job runtime module, and is used to collect the status of all cloud servers on the cloud platform and provide the batch job runtime module with a list of available cloud servers.

[0016] Furthermore, the batch job runtime module is further used for:

[0017] Concurrent execution: Use the coroutine mechanism to establish multiple connections at the same time, and even establish a dedicated connection for each cloud server to ensure that the job can be executed and completed quickly.

[0018] Middleware: Insert custom behaviors at key stages of job execution.

[0019] Exception handling: Retry the failed cloud server. If multiple retries fail, give up the execution and record the error information.

[0020] Timeout detection: Provides a timeout setting for the entire job or a single execution of a single cloud server. If the timeout is not completed, the job will exit and be marked as a timeout exception.

[0021] Furthermore, the encapsulated daily operation and maintenance work packages must comply with the integration specifications for work package encapsulation, including:

[0022] The job package must be in some compressed format;

[0023] The job package must contain an entry file with a specified name; when the cloud server executes the job, it will be executed from the specified entry file;

[0024] If the execution is successful, it will return 0, if it returns any other value, it will indicate an error;

[0025] The job package must contain a version. Once released, it cannot be updated. If you want to update, you must upgrade the version number;

[0026] If the job contains complex interactive logic, it can be described using the interactive language described in the present invention.

[0027] Moreover, the custom interaction language set by the platform operation and maintenance personnel is:

[0028] A set of more readable and descriptive grammars is then converted into regular expressions through an interpreter; when actually encapsulating the job package, the interactive language can be used to describe the interactive requirements; when the job package is executed, these interactive requirements will be converted into the final regular expressions and executable interactive logic.

[0029] Moreover, the jobs are divided into simple jobs and complex jobs; a simple job refers to a job that executes a single job package, and a compound job is a job composed of multiple job packages, commands, and simple scripts; a compound job needs to be described using a job package orchestration language, and when a compound job is executed, the system runtime will execute its parts in sequence according to the orchestration requirements.

[0030] Furthermore, the cloud server proxy module is used to monitor the health status of the cloud server, ensuring that the batch job can be normally connected to the cloud server and execute the job normally when it is running; when an abnormality occurs in the cloud server, the cloud server proxy module can automatically repair it. When a cloud server cannot be repaired, the cloud server proxy module will report the details of the cloud server to the service center.

[0031] Advantages and beneficial effects of the present invention:

[0032] The present invention proposes a batch automatic processing execution system for large cloud platforms, and provides a complete set of operation and maintenance scripts and commands that can be executed in batches. According to the constraints of this system, the operation and maintenance personnel can only focus on writing operation and maintenance atomic operations, and use this system to run instructions to realize large-scale batch execution operations, which can greatly reduce the difficulty of the operation and maintenance personnel to complete batch jobs and improve the execution efficiency of batch jobs. First, the system abstracts and encapsulates the relevant capabilities of batch job runtime, such as high concurrency, exception handling, timeout and retry mechanism, etc., so that the operation and maintenance personnel can better focus on the commands, scripts and logic of the job itself; secondly, the system provides a custom interactive language, which reduces the difficulty of writing interactive commands, especially regular expressions, thereby replacing the traditional expect tool; thirdly, the system provides an integration mechanism, which can package complex job command logic into job packages, so as to seamlessly integrate with the runtime when the job is executed; fourthly, the system provides a job scheduling language, which can schedule multiple job packages together to achieve the purpose of executing multiple jobs at the same time; finally, the system provides a special agent, which is deployed on the target cloud server and can provide functions such as anomaly detection and self-healing of ssh services to ensure the health of the cloud server. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic diagram of a batch automated processing system of the present invention;

[0034] Figure 2 This is a schematic diagram of the implementation principle of the middleware during job runtime of the present invention;

[0035] Figure 3 It is a schematic diagram of the syntax tree structure of the present invention. DETAILED DESCRIPTION

[0036] The embodiments of the present invention are further described in detail below with reference to the accompanying drawings:

[0037] A batch automation processing execution system for large cloud platforms, such as Figure 1 As shown, it includes: a job package management module, a batch job runtime module, a cloud server agent module and a service center module;

[0038] The batch job runtime module is used to abstract and encapsulate general batch operation capabilities, provide high concurrency, retry mechanism, timeout mechanism, exception handling mechanism, etc. required for batch processing operations;

[0039] The job package management module is used to encapsulate daily operation and maintenance operations, and is responsible for the management duties of creating / updating / deleting / querying operation and maintenance job packages. The encapsulation of daily operation and maintenance job packages must comply with the integration specifications of job package encapsulation; the output end of the job package management module is connected to the batch job runtime module, which is used to receive the custom interactive language set by the platform operation and maintenance personnel, and package the complex job command logic into a job package so that it can be seamlessly integrated with the runtime during job execution; at the same time, a job scheduling language is provided to schedule multiple job packages together to achieve the purpose of executing multiple jobs at the same time;

[0040] The output end of the batch job runtime module is connected to multiple cloud server agent modules to detect and repair problems in the cloud servers. Multiple agents are deployed on the corresponding cloud servers to provide guarantee for batch job execution.

[0041] The output end of the service center is connected to the batch job runtime module to collect the status of all cloud servers on the cloud platform and provide the batch job runtime module with a list of available cloud servers.

[0042] In this embodiment, the batch job runtime module is further used for:

[0043] Concurrent execution: Use the coroutine mechanism to establish multiple connections at the same time, and even establish a dedicated connection for each cloud server to ensure that the job can be executed and completed quickly.

[0044] Middleware: Insert custom behaviors at key stages of job execution.

[0045] For example, before or after the entire job starts or ends, before or after each cloud server job starts or ends. The middleware can be flexibly customized as needed.

[0046] Exception handling: Retry the failed cloud server. If multiple retries fail, give up the execution and record the error information.

[0047] Timeout detection: Provides a timeout setting for the entire job or a single execution of a single cloud server. If the timeout is not completed, the job will exit and be marked as a timeout exception.

[0048] In this embodiment, the packaging of daily operation and maintenance job packages must comply with the integration specifications of job package packaging, including:

[0049] The job package must be in some compressed format, such as zip format.

[0050] The job package must contain an entry file with a specified name, such as main.sh or main.py. When the cloud server executes the job, it will be executed from the specified entry file.

[0051] If the execution is successful, it will return 0, if it returns any other value, it will indicate an error.

[0052] The job package must contain a version. Once released, it cannot be updated. If you want to update it, you must upgrade the version number.

[0053] If the job contains complex interactive logic, it can be described using the interactive language described in the present invention.

[0054] In this embodiment, the custom interaction language set by the platform operation and maintenance personnel is:

[0055] The goal of the interactive language is to reduce the difficulty of writing regular expressions. The core capability is to invent a set of more readable and descriptive grammars, and then convert them into regular expressions through an interpreter. When actually encapsulating the job package, the interactive language can be used to describe the interactive requirements. When the job package is executed, these interactive requirements will be converted into the final regular expressions and executable interactive logic.

[0056] In this embodiment, the jobs are divided into simple jobs and complex jobs; a simple job refers to a job that executes a single job package, and a compound job is a job composed of multiple job packages, commands, and simple scripts; a compound job needs to be described using a job package orchestration language, and when a compound job is executed, the system runtime will execute parts of it in sequence according to the orchestration requirements.

[0057] In this embodiment, the cloud server proxy module is used to monitor the health status of the cloud server to ensure that the batch job can be normally connected to the cloud server where it is located and execute the job normally when it is running; when an abnormality occurs in the cloud server, the cloud server proxy module can automatically repair it. When a cloud server that cannot be repaired appears, the cloud server proxy module will report the details of the cloud server to the service center.

[0058] The working principle of the present invention is:

[0059] In the first aspect, the present invention proposes a batch processing system for a large cloud platform. The system can be divided into four parts: batch job runtime, job package management, cloud server agent, and service center.

[0060] The batch job runtime abstracts and encapsulates the general batch operation capabilities, providing high concurrency, retry mechanism, timeout mechanism, exception handling mechanism, etc. required for batch processing operations. The job package management is responsible for the creation / update / deletion / query of job packages. The encapsulated job package must follow the job package integration specification. The cloud server agent is responsible for detecting and repairing problems with the cloud server to provide guarantees for batch job execution. The service center is responsible for collecting the status of all cloud servers on the cloud platform and providing a list of available cloud servers for job runtime.

[0061] Secondly, when batch jobs are running, mechanisms or functions such as high-concurrency execution, middleware, exception handling, retry mechanism, and timeout detection are implemented.

[0062] The high-concurrency execution mechanism uses coroutine technology to establish multiple connections at the same time, and even establish a dedicated connection for each cloud server to ensure that the job can be executed and completed quickly. Coroutine technology has an internal scheduling mechanism, is not limited by the number of local threads, and is easy to control concurrency. The middleware mechanism provides the ability to customize the behavior of job execution. Before or after the start or end of the entire job, before or after the start or end of each cloud server job. The middleware can be flexibly customized as needed. The exception handling mechanism retries the cloud server that failed to process, and abandons the execution and records the error information after multiple retries fail. The timeout detection mechanism provides a timeout setting for the single execution of the entire job or a single cloud server. When the timeout is not completed, the job will exit and be marked as a timeout exception.

[0063] Third, when packaging the job package, you must comply with the integration specifications, otherwise it will not be integrated with the job runtime.

[0064] The integration specifications include: the job package must be in a certain compression format, such as zip format; the job package must contain an entry file with a specified name, such as main.sh or main.py. When the cloud server executes the job, it will be executed from the specified entry file; if the execution is successful, it will return 0, and if other values ​​are returned, it will indicate an error; the job package must contain a version, and once released, it cannot be updated. If you want to update, you must upgrade the version number; if the job contains interactive logic, it can be described by the interactive language described in this invention.

[0065] Fourthly, when complex interaction logic is involved, support for custom interaction languages ​​is provided.

[0066] Interactive languages ​​reduce the difficulty of writing regular expressions. When the job package is executed, these interactive languages ​​will be converted into the final regular expressions and executable interactive logic. The core of the interactive language is to simplify the description of regular expressions.

[0067] The present invention divides the basic patterns of regular expressions into eleven types, such as maybe, one_or_more, etc., and their corresponding meanings are shown in the following table.

[0068] model Regular Expression Symbols meaning zero_or_more('a') a* Zero or more a one_or_more('a') a+ One or more a repeat(m,n,'a') a{m,n} Repeat a from m to n times numberic() 0-9 number maybe('a') a？ Not appearing or appearing oncea alpha() a-zA-Z letter char() . Any character one_char_of('a','b') [ab] Select One range('a','z') az From A to Z line('a') ^a$ Lines containing only the letter a group('ab') (ab) ab group

[0069] When writing complex regular expressions, for example, the regular expression for matching email addresses is "[a-zA-Z0-9_-]+@[a-zA-Z0-9_-]+(\.[a-zA-Z0-9_-]+)+", the interactive language can be rewritten as "$range,'@',$range,one_or_more(char(),group($range))". The variable range represents the string one_or_more(one_char_of(alpha(),numberic(),'_','-'). It can be seen that the custom interactive language has better readability and description capabilities.

[0070] When the job package is running, the above description language string will be converted into a regular expression. The following describes the specific conversion process. The entire conversion process is divided into two steps. First, the input string is segmented and a syntax tree is constructed. Then, the in-order traversal is performed to evaluate and output the conversion result.

[0071] Take the regular sentence of the email as an example. Word segmentation is to identify keywords, string constants, and variables from the above sentence. Using the result of word segmentation as input, we can get the following: Figure 3 In the syntax tree shown, we can see that the three subtrees framed by dotted lines in this tree correspond to the $range clauses of the aforementioned interactive statements.

[0072] Next, perform a post-order traversal of the syntax tree and evaluate each node in turn. The specific evaluation algorithm is as follows.

[0073]

[0074] T is a logical tree with grammatical relations obtained by grammatically escaping a statement, S

[0075] It is the set of all branches of this syntax tree, including all the word segments of the sentence. s is a word segment in the syntax set, and rs is a regular expression string to be generated.

[0076] Fifthly, homework can be divided into simple homework and complex homework.

[0077] A simple job is a job that executes a single job package, while a compound job is a job that is composed of multiple job packages, commands, and simple scripts. A compound job needs to be described using a job package orchestration language. When a compound job is executed, the system will execute its parts in order according to the orchestration requirements.

[0078] The following is an example of a complex job arrangement. It can be seen that this job contains two steps: check and vmtools. The check command is used to execute a command to check whether the vm-agent service exists. If it exists (success), the job ends. Otherwise, the following vmtools step will continue. This step is responsible for installing the vmtools tool with version number 2.3, and the installation location can be specified through the location parameter.

[0079] version:'1'

[0080] jobs:

[0081] check:

[0082] kind:cmd

[0083] command:systemctl status vm-agent

[0084] success:break

[0085] vmtools:

[0086] kind:package

[0087] package:vmtools:2.3

[0088] params:

[0089] -location: / opt / vmtools

[0090] Sixth, the cloud server agent is used to monitor the health status of the cloud server, ensuring that the batch job can be connected to the cloud server normally when it is running and execute the job normally. When the cloud server is abnormal, the agent can automatically repair it. When a cloud server cannot be repaired, the agent will report the details of the cloud server to the service center.

[0091] Example 1

[0092] In actual implementations, jobs can be deployed as services when running, in which case clients can be configured for the service, such as command lines, web interfaces, etc. At the same time, in addition to job package management, commonly used scenarios can be encapsulated as auxiliary functions. For the daily work of cloud platform operation and maintenance, these auxiliary functions can include batch command execution, batch execution of simple scripts, batch file copying, etc. Finally, in the formal implementation, log management, user authentication, operation auditing, and related batch execution record management functions are generally required.

[0093] In actual implementations, the Go language is generally used to implement the job runtime, and batch execution is an excellent scenario for applying the Go language coroutine technology. During operation, the cloud servers to be processed can be divided into several groups, each group containing one or more cloud servers. Each coroutine is responsible for the job execution of a group of cloud servers. After the execution is completed, the execution results are reported by channel messages.

[0094] Regarding the login method when the job is running, password-based SSH login is generally used, and root direct login is generally not allowed. According to the actual needs of operation and maintenance, multiple login modes should be supported during login. Common modes include su, sudo, direct, and sudo-i. The su and sudo-i modes indicate that the root user is switched using the su and sudo-i commands, and the root user is maintained during the execution process. The direct mode indicates that the job is executed directly as the logged-in user. The sudo mode indicates that the sudo prefix is ​​used to ensure that the root privilege can be obtained during the job execution process.

[0095] The middleware mechanism during job runtime is an important mechanism to ensure scalability. The middleware mechanism can be used to extend the capabilities of the job runtime, such as the job's running time, historical records, etc. Next, a feasible design method for defining the middleware mechanism is described. The middleware function type can be defined as type HandlerFunc handle(*Context), where the Context type has a Next() method. Suppose that two custom middleware Logger and Timer are registered before the job is executed, which are used to record the logs before and after the job runs and the execution time respectively. Finally, the Execute function, which is the method for executing the job, is also wrapped as a middleware Executor, and its execution process is as follows: Figure 2 shown.

[0096] from Figure 2It can be seen that the three registered middlewares are saved in ctx. Ctx will first execute the first middleware logger. The handle function of logger will call the ctx.Next function during the execution process, thereby triggering the timer middleware. Similarly, the timer.handle function executes the Executor middleware during the execution process. This is the last middleware and the middleware used to execute the job logic. When the Executor middleware is executing, it will not call the ctx.Next method, so it will return to the timer middleware to continue to execute the unfinished logic. After the timer middleware is executed, it will return to the logger middleware to execute the unfinished logic until the call is finally returned. It can be seen that this is a chain calling process. In this process, as long as one middleware does not call the ctx.Next function, the job will fail to execute. On the other hand, parameters can be passed between middlewares through the ctx object, thus forming a dependency. At this time, the order of middleware registration is very necessary.

[0097] Regarding job package management, in actual embodiments, the job package can be compressed into an rpm package (or other packaging formats supported by the operating system, such as a deb package), and the rpm package can be published in a public rpm warehouse, and the yum source can be configured in all cloud servers. Then, when executing the job, it can be automatically downloaded from the rpm warehouse without being transmitted when the job is running. In this case, it is only necessary to pass the necessary parameters when installing the rpm package command. On the other hand, a special syntax can also be designed to achieve the definition of environment variable parameters, and set them before the job is executed, thereby playing a global reference at the job level.

[0098] In some embodiments, the acquisition of cloud server list information can be customized according to the general situation of the cloud platform. For example, the cloud platform generally runs several resource sets (business systems), each resource set has several cloud servers, and the resource sets also belong to different deployment environments. The cloud servers have different tags, etc., and the cloud servers can be filtered according to their attributes to obtain a cloud server list that meets the scope of this operation. Similarly, when the operation is completed, the corresponding attributes can be transferred to the output results as needed.

[0099] It should be emphasized that the embodiments described in the present invention are illustrative rather than restrictive. Therefore, the present invention includes but is not limited to the embodiments described in the specific implementation manner. Any other implementation manners derived by those skilled in the art based on the technical solution of the present invention also fall within the scope of protection of the present invention.

Claims

1. A batch automated processing execution system for large cloud platforms, characterized by: include: Job package management module, batch job runtime module, cloud server proxy module and service center module; The batch job runtime module is used to abstract and encapsulate general batch operation capabilities, and provide high concurrency, retry mechanism, timeout mechanism, and exception handling mechanism required for batch processing operations; The job package management module is used to encapsulate daily operation and maintenance operations, and is responsible for the management duties of creating / updating / deleting / querying operation and maintenance job packages. The encapsulation of daily operation and maintenance job packages must comply with the integration specifications of job package encapsulation; the output end of the job package management module is connected to the batch job runtime module, which is used to receive the custom interactive language set by the platform operation and maintenance personnel, and package the complex job command logic into a job package so that it can be seamlessly integrated with the runtime during job execution; at the same time, a job scheduling language is provided to schedule multiple job packages together to achieve the purpose of executing multiple jobs at the same time; The output end of the batch job runtime module is respectively connected to multiple cloud server agent modules to detect and repair problems existing in the cloud servers, and multiple agents are deployed on the corresponding cloud servers to provide guarantee for batch job execution; The output end of the service center is connected to the batch job runtime module, and is used to collect the status of all cloud servers on the cloud platform and provide the batch job runtime module with a list of available cloud servers.

2. The batch automated processing execution system for large cloud platforms according to claim 1, characterized in that: The batch job runtime module is also used for: Concurrent execution: Use the coroutine mechanism to establish multiple connections at the same time, and even establish a dedicated connection for each cloud server to ensure that the job can be executed and completed quickly; Middleware: inserts custom behaviors at key stages of job execution; Exception handling: Retry the failed cloud server, and give up the execution and record the error information after multiple retries fail; Timeout detection: Provides a timeout setting for the entire job or a single execution of a single cloud server. If the timeout is not completed, the job will exit and be marked as a timeout exception.

3. The batch automated processing execution system for large cloud platforms according to claim 1, characterized in that: The encapsulated daily operation and maintenance work packages must comply with the integration specifications of the work package encapsulation, including: The job package must be in some compressed format; The job package must contain an entry file with a specified name; when the cloud server executes the job, it will be executed from the specified entry file; If the execution is successful, it will return 0, if it returns any other value, it will indicate an error; The job package must contain a version. Once released, it cannot be updated. If you want to update it, you must upgrade the version number. If the job contains complex interactive logic, it can be described using the interactive language described in the present invention.

4. The batch automated processing execution system for large cloud platforms according to claim 1, characterized in that: The custom interactive language set by the platform operation and maintenance personnel is: A set of more readable and descriptive grammars is then converted into regular expressions through an interpreter; when actually encapsulating the job package, the interactive language can be used to describe the interactive requirements; when the job package is executed, these interactive requirements will be converted into the final regular expressions and executable interactive logic.

5. The batch automated processing execution system for large cloud platforms according to claim 1, characterized in that: The jobs are divided into simple jobs and complex jobs; a simple job refers to a job that executes a single job package, and a compound job is a job composed of multiple job packages, commands, and simple scripts; a compound job needs to be described using a job package orchestration language, and when a compound job is executed, the system runtime will execute its parts in sequence according to the orchestration requirements.

6. A batch automated processing execution system for a large cloud platform according to claim 1 or 2, characterized in that: The cloud server proxy module is used to monitor the health status of the cloud server to ensure that the batch job can be normally connected to the cloud server where it is located and execute the job normally when it is running; when an abnormality occurs in the cloud server, the cloud server proxy module can automatically repair it; when an unrepairable cloud server occurs, the cloud server proxy module will report the details of the cloud server to the service center.

Images