Distributed inspection system for operation and maintenance side of large-scale cloud platform

Abstract

The invention relates to a distributed checking system for a large-scale cloud platform operation and maintenance side. The distributed checking system comprises a checking client and a server cluster, the checking client runs on a terminal of a checking user and provides an operable interface for the checking user; the server-side cluster comprises one or more control nodes and working nodes; the control node is used for receiving a checking request of a checking client, analyzing a checking operation into a series of tasks, scheduling the tasks to proper working nodes for execution, monitoring task execution of the working nodes and providing checking task mirror image management; the working node is used for executing tasks and reporting task states and log information to the control node at the same time. According to the method, hundreds of thousands of cloud resource objects can be accurately checked within the minute level, and the operation and maintenance efficiency is greatly improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of cloud computing resource inspection, and relates to a distributed inspection system, in particular to a distributed inspection system for the operation and maintenance side of a large cloud platform. Background Art

[0002] At present, cloud computing technology has been widely used in production and life. Many enterprises have established their own private cloud platforms. Some large private cloud platforms have thousands of physical host nodes, and the number of business systems running on the cloud is as high as hundreds, including some important business systems related to the national economy and people's livelihood. On the other hand, in order to better manage the various resources of the cloud platform, the cloud platform operation and maintenance management side usually formulates relevant specifications and requirements, and puts forward specific requirements for the use of cloud resources on the platform and business side. For example, the resource utilization rate of the cloud server must reach a specified threshold. In order to ensure that these specifications are implemented, the cloud platform management and operation side often carries out some standardized governance activities to check whether there are any violations of the specifications for various cloud resources. This kind of inspection of cloud resources is collectively referred to as inspection. For example, the online inspection of the business system is an important task of the daily operation and maintenance of the cloud platform. Only after the online inspection can the business system be officially put into operation.

[0003] At present, the inspection work of many cloud platforms is still in the traditional manual inspection stage. Manual operation is inefficient, time-consuming, and prone to errors, which greatly occupies the manpower and time costs of the operation and maintenance team. Taking the above-mentioned online inspection as an example, a complete inspection involves up to 30 indicators, and the number of cloud resource objects to be inspected for some indicators is huge. For example, in order to check whether the workloads in the business system are configured with health checks, all workloads need to be manually checked one by one, and some business systems have dozens or even hundreds of workloads.

[0004] In summary, manual operation is basically unable to complete the inspection work efficiently and correctly, which reduces the quality of inspection services. On the other hand, the inspection process is lengthened, which also slows down the progress of the business system online, leading to dissatisfaction with the business system.

[0005] In order to solve the above problems, the present invention proposes a distributed inspection system for the operation and maintenance side of a large cloud platform.

[0006] After searching, no prior art documents identical or similar to the present invention were found. Summary of the invention

[0007] The purpose of the present invention is to overcome the shortcomings of the prior art and propose a distributed inspection system for the operation and maintenance side of a large cloud platform, which can ensure accurate inspection of hundreds of thousands of cloud resource objects within minutes, greatly improving the operation and maintenance efficiency.

[0008] The present invention solves the practical problem by adopting the following technical solutions:

[0009] A distributed inspection system for the operation and maintenance side of a large cloud platform, including an inspection client and a server cluster;

[0010] The inspection client runs on the terminal of the inspection user and provides an operable interface for the inspection user;

[0011] The server-side cluster includes one or more control nodes and working nodes;

[0012] The control node is used to receive the inspection request from the inspection client, parse the inspection job into a series of tasks and dispatch them to the appropriate working node for execution, monitor the execution of the working node tasks, and provide inspection task image management;

[0013] The working node is used to execute tasks and report task status and log information to the control node.

[0014] Moreover, the output end of the verification client is connected to the server-side cluster, and is used to send a task mirroring request to the server-side cluster; send a request to create a verification to the server-side cluster; send a verification request to the server-side cluster and display the verification result.

[0015] Moreover, the control node includes: an API service module, a scheduler, a task manager, and an image warehouse; the output end of the inspection client is connected to the API service module, and the API service module is connected to the scheduler, the task manager, and the image warehouse respectively;

[0016] The API service module is used to receive requests from the client and requests from the working node related to the execution of tasks; the scheduler is responsible for parsing the inspection requests and scheduling the inspection tasks to the working nodes for execution according to the scheduling algorithm; the task manager is responsible for monitoring the execution of the inspection tasks and handling various possible situations; the image repository is responsible for the management of task images.

[0017] Moreover, the working node includes agent tasks and multiple inspection tasks; the task agent is connected to the API service module and is a background service run by the working node, which is used to manage various tasks running locally in real time, and report the task status, logs, exceptions and other operation-related information to the control node through the API; the task agent accepts commands from the control node; the task agent also regularly sends heartbeats, resource utilization and other data to the control node to provide a basis for the control node scheduling strategy.

[0018] Moreover, the multiple inspection tasks are divided into collectors and inspectors; the collectors and inspectors are connected to the proxy tasks respectively, the collectors are used to collect resources required for inspection, and the inspectors are used to inspect resource data.

[0019] Advantages and beneficial effects of the present invention:

[0020] 1. The present invention proposes a distributed inspection system for the operation and maintenance side of a large cloud platform, which has the advantage of automation. The system checks the interaction between the control nodes and the working nodes in the client and server cluster, defines the inspection rules, and automates the inspection process, truly realizing the customization and rapid execution of inspections, ensuring accurate inspection of hundreds of thousands of cloud resource objects within minutes, and greatly improving the operation and maintenance efficiency.

[0021] 2. The inspection system of the present invention has the advantage of being distributed and has parallel processing capabilities. The distribution enables this framework to have the ability to globally inspect large cloud platforms. With the appropriate scheduling algorithm, this framework can complete the collection and inspection of hundreds of thousands of cloud platform resources within minutes, providing technical support for real-time tracking of cloud platform resource status. Parallelization allows the inspection job to be decomposed into multiple inspection tasks, which can be scheduled to different nodes for execution. Depending on the specific circumstances of the scheduling algorithm, the same inspection task can even be scheduled to multiple nodes for simultaneous execution, achieving the effect of data parallelism. Regardless of how the inspection tasks are distributed, the task manager has the ability to collect the results of each inspection task, thereby summarizing them as the results of the entire inspection job.

[0022] 3. The inspection system of the present invention has the advantage of scalability, which is mainly reflected in two aspects. The first is the scalability of the scheduling algorithm. If the default scheduling algorithm cannot meet the actual performance requirements, it can be written and replaced with a more efficient scheduling algorithm. The second is the scalability of the inspection task. In order to adapt to the changing inspection requirements of the cloud platform, new inspection tasks or new versions of existing inspection tasks can be added by mirroring. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the structure of the inspection system of the present invention. DETAILED DESCRIPTION

[0024] The following further describes the embodiments of the present invention:

[0025] A distributed inspection system for the operation and maintenance side of large cloud platforms, such as Figure 1 As shown, it includes checking the client and server clusters;

[0026] The inspection client runs on the terminal of the inspection user and provides an operable interface for the inspection user;

[0027] The server-side cluster includes one or more control nodes and working nodes;

[0028] The control node is used to receive the inspection request from the inspection client, parse the inspection job into a series of tasks and dispatch them to the appropriate working node for execution, monitor the execution of the working node tasks, and provide inspection task image management;

[0029] The working node is used to execute tasks and report task status and log information to the control node.

[0030] In this embodiment, the output end of the verification client is connected to the server-side cluster, which is used to send a task mirroring request to the server-side cluster; send a request to create a verification to the server-side cluster; send a verification request to the server-side cluster and display the verification result.

[0031] In this embodiment, the control node includes: an API service module, a scheduler, a task manager, and an image warehouse; the output end of the inspection client is connected to the API service module, and the API service module is connected to the scheduler, the task manager, and the image warehouse respectively;

[0032] The API service module is used to receive requests from the client and requests from the working node related to the execution of tasks; the scheduler is responsible for parsing the inspection requests and scheduling the inspection tasks to the working nodes for execution according to the scheduling algorithm; the task manager is responsible for monitoring the execution of the inspection tasks and handling various possible situations; the image repository is responsible for the management of task images.

[0033] In this embodiment, the working node includes agent tasks and multiple inspection tasks; the task agent is connected to the API service module and is a background service run by the working node, which is used to manage various tasks running locally in real time, and report the task status, logs, exceptions and other operation-related information to the control node through the API; the task agent accepts commands from the control node; the task agent also regularly sends heartbeats, resource utilization and other data to the control node to provide a basis for the control node scheduling strategy.

[0034] In this embodiment, the various inspection tasks are divided into a collector and a checker; the collector and the checker are respectively connected to the proxy tasks, the collector is used to collect resources required for inspection, and the checker is used to check the resource data.

[0035] The working principle of the present invention is:

[0036] In a first aspect, the present invention proposes a distributed verification system for a large cloud platform, which can be divided into a client and a server cluster.

[0037] The client runs on the terminal of the verification user and provides an operable interface for the verification user. The server-side cluster consists of control nodes and working nodes. The control node is responsible for receiving the verification request from the verification client, parsing the verification job into a series of tasks and dispatching them to the appropriate working node for execution, monitoring the execution of the working node tasks, and providing verification task image management. The working node is mainly responsible for the execution of the task, and reports the task status and log information to the control node.

[0038] Secondly, the inspection client mainly includes three core functions: sending task image-related requests to the inspection server cluster; sending creation inspection requests to the inspection server cluster; sending inspection requests to the inspection server cluster and displaying the inspection results.

[0039] The task image encapsulates all the static resources required to run the inspection task and is provided in the form of a compressed package, generally compressed by zip. Task images can be divided into two types of images: inspection tasks and collection tasks. The image file mainly includes the executable file, version, configuration information, etc. of the inspection task. The configuration information mainly includes basic image information, image default configuration, and dependent image list. The basic information of the image includes name, type, version number, etc. The image default configuration provides the configuration information required to run the task and can be overwritten when defining the inspection. The dependent image list indicates the current image's dependence on other images (including version numbers). Specifically, the inspector image and the collector image both depend on the collector image. When running the inspector task or the collector task, it is necessary to calculate the image it depends on and schedule it for execution first. To ensure integrity, the image repository should use a hash algorithm to calculate the fingerprint of the image.

[0040] Users can create new checks based on actual needs. Each check consists of one or more check items. Each check item corresponds to one or more check tasks at runtime. Each check task has corresponding image, version, configuration and other information. This information needs to be specified when defining a new check. The newly created check will be saved to the server, and the created check can be edited as needed.

[0041] The main task of the client is to send a verification request to the server. The verification request information mainly includes the verification name, verification scope and output results. The verification name is the name of the created verification, the output result specifies the save location of the verification result, and the verification scope specifies the scope of the verification project. The verification system proposed in the present invention supports single-project verification, multi-project verification, and cloud platform verification. Single-project verification only verifies one business system, and a complete and unique project name must be provided when applying. Multi-project verification can verify multiple business systems, and a list of the names of all business systems must be provided when applying, or the business system names can be described using wildcards and other methods. Cloud platform verification verifies all business systems, and the business system name does not need to be provided when applying. After the verification is initiated, the client can view the progress and real-time log of the verification task.

[0042] Thirdly, the server-side control node is mainly used to receive client requests, parse the inspection requests, and use the scheduling algorithm to schedule the parsed inspection tasks to the working nodes for execution, and monitor the task execution status during the task execution process.

[0043] The control node consists of four major components: API service, scheduler, task manager, and image repository. The API service is mainly used to receive requests from clients and requests from working nodes related to task execution. The scheduler is mainly responsible for parsing inspection requests and scheduling inspection tasks to working nodes for execution according to the scheduling algorithm. The task manager is responsible for monitoring the execution of inspection tasks and handling various possible situations. The image repository is responsible for managing task images.

[0044] The API service mainly receives three types of requests from the client. When receiving a request to create a new inspection definition, the API service will save the inspection definition for subsequent requests. When receiving a request for a task image from the client, the API service will forward the request to the image warehouse component for processing. When receiving a inspection request from the client, the API service will forward the request to the scheduler for processing.

[0045] The scheduler is responsible for parsing the inspection request forwarded by the API service, converting the inspection job into a series of inspection tasks, and then scheduling each task to the appropriate working node for execution according to the actual situation of the cluster. When none of the working nodes meet the scheduling conditions, the scheduler also acts as a queuing service. After receiving the inspection request from the client, the scheduler will parse the inspection request. The main function of the inspection parsing is to verify the legitimacy of the inspection orchestration information. For each inspection item in the orchestration, the scheduler will check the legitimacy of the image. The image legitimacy check mainly includes two parts. First, check whether the image file exists in the image warehouse, and secondly check whether the collector image on which the image file depends is legal, especially to check whether there is a circular dependency between images. If the dependency list of image A directly or indirectly contains image B, and the dependency list of image B also directly or indirectly contains image B, then it is said that image A and image B have a circular dependency.

[0046] The scheduler performs scheduling according to a specific scheduling algorithm. The scheduling algorithm should be extensible and can be managed in a plug-in manner according to the requirements of the algorithm. The default scheduling algorithm of the scheduler supports parallel inspection capabilities, and simultaneously schedules the various inspection and collection tasks involved in the inspection job to the appropriate working nodes for execution. The scheduling mainly refers to the node status, resource utilization, and running task information of the working nodes.

[0047] Scheduling is divided into collector scheduling and inspector scheduling. The inspector needs to request resource data from the task manager during runtime, and these resource data are obtained by the collector. Therefore, before the inspector task is scheduled, it must be ensured that the collector task it depends on is fully scheduled and executed. In order to improve scheduling efficiency and job execution speed, it is necessary to adopt a caching algorithm to accelerate the resource collection process. When a certain resource already exists in the cache of the task manager, the resource collector scheduling can be canceled. Of course, a certain validity period should be set for the cached resources to avoid generating dirty data, which will affect the inspection results. The same caching mechanism also applies to the inspector task.

[0048] There are thousands of business systems and resource sets in large cloud platforms. When the inspection involves multiple business systems or even the entire cloud platform, it takes a lot of time to collect all the resources of these business systems or resource sets, which will greatly reduce the efficiency of the entire inspection operation. To this end, a fine-grained parallel mechanism is needed. On the one hand, once the resources that a certain inspector depends on are collected, the inspector can be scheduled for execution immediately, without having to wait until all the resources required by all inspectors are collected before scheduling. On the other hand, once the resources required by a business system are met, the dispatch of the inspector for that business system can be initiated immediately, without having to wait until the resources of all business systems are collected before scheduling. After the resources of other business systems are collected, the dispatch of the inspector can be initiated again.

[0049] The task manager is responsible for monitoring and checking the execution of tasks. When a task has an abnormal situation such as timeout, it will re-apply for scheduling and execution. When a task is not completed, the task manager is responsible for collecting progress information; when a task is completed, the task manager is also responsible for receiving the inspection results of the task, as well as merging and other operations.

[0050] The image repository is responsible for the management of task images. The images package the programs and configurations required for task execution. The corresponding images must be obtained before the task is executed.

[0051] Fourthly, the working node consists of proxy tasks and various inspection tasks. The task agent is the background service of the working node, which manages the tasks running locally in real time and reports the task status, logs, exceptions and other operation-related information to the control node through the API. The task agent accepts commands from the control node, mainly task-related commands. The task agent also regularly sends heartbeat, resource utilization and other data to the control node to provide a basis for the control node scheduling strategy.

[0052] The working node is responsible for executing various inspection tasks. Inspection tasks are part of the inspection job, which are divided into two types: collectors and inspectors. Collectors are used to collect resources required for inspection, and inspectors are used to inspect resource data.

[0053] During the execution of the collector and the checker, a large number of interfaces of cloud platform related services will be called. When the scale of the check operation is large, it is very likely to trigger the traffic limit of the cloud platform API gateway, resulting in interface call failure. In order to ease the traffic pressure, it is necessary to introduce a queuing and retry mechanism for the interface calls of all tasks.

[0054] In this embodiment, the inspection client can include multiple forms, which can be a command line or a front-end web interface. In addition to the aforementioned core functions, the client can also include some auxiliary functions. These auxiliary functions include inspection progress, inspection history and related logs, cluster related information display, etc. Some configuration information and operations related to the framework can also be implemented on the client, such as replacing a new scheduling algorithm, managing new nodes, removing existing cluster nodes, checking node health status, etc.

[0055] In this embodiment, both the control node and the working node can have one or more nodes, and all components of the control node and the working node can also be deployed on one node in a single-point mode. To avoid single-point failure, multiple control nodes can be configured, and the nodes are in a master-slave relationship. When the master node fails, the standby node can quickly take over the cluster management. When deploying this inspection framework in a production environment, a multi-control node mode can generally be used. When deploying in a test environment, or when performing conceptual testing, the client and the cluster can be deployed on one node for testing convenience.

[0056] During the operation of the framework service, in order to adapt to load changes, the number of working nodes can be adjusted dynamically, manually or automatically. At this time, the control node API service should provide the corresponding interface for node registration and removal operations. When incorporating a new node, the corresponding information needs to be provided so that the new working node can successfully find the API service to complete the registration and automatically complete the installation of the proxy service through the API service. When removing an existing node, you only need to send the relevant node deregistration request to the API service. If there are inspection tasks running on the working node, you need to wait until these services are completely migrated to other nodes or until these services are executed before you can perform the removal operation. In fact, the working node will actively send some heartbeat information to the control node. Once the heartbeat information of a node is not received for a long time, this node can be regarded as an abnormal node, which is actually removed.

[0057] In this embodiment, the dependency relationship between images can be abstracted as a directed graph G without weights. The vertex set V of the graph is all the inspector or collector images involved in the inspection operation, and the edge set E of the graph represents the dependency relationship between images, from the dependent image to the dependent image. The cyclic dependency check between images can be abstracted as a loop-free check of the graph. In graph theory, the depth-first forest G generated by the deep search algorithm running on the graph G is π , if a vertex u in an edge (u,v)∈V in the graph can be connected to its vertex in G π If the ancestor node v in a certain query tree is (u,v), then (u,v) is called a backward edge. A directed graph G = (V,E) is acyclic if and only if a priority search on it does not produce backward edges. A specific feasible algorithm is shown in the following algorithm 1-3.

[0058]

[0059]

[0060]

[0061]

[0062] In the actual implementation, in order to improve the execution speed of the inspection job, the task execution time should be reduced. A feasible method is to introduce a parallel mechanism to decompose the inspection job into sub-jobs that can be executed independently in parallel. The basis for decomposition is to find a directed subgraph composed of completely independent inspection items. The decomposition algorithm is shown in Algorithm 4.

[0063]

[0064]

[0065] It should be emphasized that the embodiments of the present invention are illustrative rather than restrictive. Therefore, the present invention includes but is not limited to the embodiments described in the specific implementation modes. Any other implementation modes derived by those skilled in the art based on the technical solutions of the present invention also fall within the scope of protection of the present invention.

Claims

1. A distributed inspection system for the operation and maintenance side of a large cloud platform, characterized by: Including checking the client and server clusters; The inspection client runs on the terminal of the inspection user and provides an operable interface for the inspection user; The server-side cluster includes one or more control nodes and working nodes; The control node is used to receive the inspection request from the inspection client, parse the inspection job into a series of tasks and dispatch them to the appropriate working node for execution, monitor the execution of the working node tasks, and provide inspection task image management; The working node is used to execute tasks and report task status and log information to the control node.

2. A distributed inspection system for the operation and maintenance side of a large cloud platform according to claim 1, characterized in that: The output end of the inspection client is connected to the server cluster and is used to send a task image request to the server cluster; Send a request to create a check to the server cluster; Send a query request to the server cluster and display the query result.

3. A distributed inspection system for the operation and maintenance side of a large cloud platform according to claim 1, characterized in that: The control node includes: an API service module, a scheduler, a task manager, and an image warehouse; the output end of the inspection client is connected to the API service module, and the API service module is connected to the scheduler, the task manager, and the image warehouse respectively; The API service module is used to receive requests from the client and requests from the working node related to the execution of tasks; the scheduler is responsible for parsing the inspection requests and scheduling the inspection tasks to the working nodes for execution according to the scheduling algorithm; the task manager is responsible for monitoring the execution of the inspection tasks and handling various possible situations; the image repository is responsible for the management of task images.

4. A distributed inspection system for the operation and maintenance side of a large cloud platform according to claim 1, characterized in that: The working node includes proxy tasks and multiple inspection tasks; the task agent is connected to the API service module and is a background service run by the working node. It is used to manage various tasks running locally in real time and report task status, logs, exceptions and other operation-related information to the control node through the API; the task agent accepts commands from the control node; the task agent also regularly sends heartbeats, resource utilization and other data to the control node to provide a basis for the control node scheduling strategy.

5. A distributed inspection system for the operation and maintenance side of a large cloud platform according to claim 4, characterized in that: The multiple inspection tasks are divided into collectors and inspectors; the collectors and inspectors are connected to the proxy tasks respectively, the collectors are used to collect resources required for inspection, and the inspectors are used to inspect resource data.

Images