Resource integrated monitoring and scheduling device based on Kubernetes intelligent board cluster

Abstract

The present invention discloses a resource integration monitoring and scheduling device based on a Kubernetes smart board cluster. The device includes: a custom scheduler, a smart board integration component, a smart board monitoring component, and a node priority queue; the present invention develops a custom scheduler based on the Kubernetes scheduling framework to solve the problem of cluster resource fragmentation and optimize cluster resource utilization; the domestic smart board resources are integrated into the Kubernetes cluster through the smart board integration component framework, so that the Kubernetes cluster can discover heterogeneous hardware devices and enable containers to apply for the resources; the smart board monitoring component reports the runtime parameters of the smart board to the distributed monitoring component Prometheus; the node priority cache queue solves the problem of low efficiency of using the default scheduling strategy in a large-scale cluster from the performance point of view.

Description

Technical Field

[0001] The embodiment of the present invention relates to the technical field of Kubernetes smart board cluster, and in particular to a resource integrated monitoring and scheduling device based on Kubernetes smart board cluster. Background Art

[0002] Orchestration and management of large-scale containers has become an inevitable trend to ensure the convenience of cloud service deployment and operational stability. The Kubernetes container orchestration engine adds a new abstract object Pod on the basis of Docker containers to better utilize containers to divide application functional modules. Once it was announced, it received support from many IaaS cloud service providers, and with its active community and super scalability, it quickly occupied the main market of container cloud and gradually became the de facto standard of container cloud platform.

[0003] However, in a world with big data and diverse workloads, the demand for computing is endless. With the development of artificial intelligence, more and more instances in a cluster will handle AI-related business, and complex neural network operations have gradually proved that the current demand for computing power is endless. At the same time, as the complexity of the business continues to increase, domestic special chips used in a specific field have continued to emerge in the domestic market, such as the MLU100 and MLU200 series of high-performance deep learning processors launched by Cambrian Technology. The domestic smart board cluster environment of this application is also built based on Cambrian MLU100 and MLU270, but such devices are often heterogeneous with conventional CPUs and other devices in the cluster. As a general container orchestration engine, Kubernetes often encounters the following problems when it comes to unified virtualization and management of domestic smart board clusters:

[0004] 1. The inability to integrate special domestic smart boards by default causes the container to be unable to apply for resources

[0005] The huge problem this brings is that when Kubernetes executes container scheduling, it can only use CPU and memory utilization as the basis for scheduling. This relatively single scheduling indicator cannot fully take into account the resource situation of some heterogeneous domestic smart boards, which often leads to the inability to fully utilize the resources of these devices when the container is scheduled to a certain node and meets the CPU and memory requirements, resulting in task failure.

[0006] 2. Distributed monitoring components cannot obtain information about domestic smart boards

[0007] In today's Kubernetes cluster management, Prometheus distributed monitoring components are usually used to monitor the cluster's software services and hardware. However, due to the existence of heterogeneous devices, Prometheus can only monitor conventional hardware devices, such as CPU, memory, network card, etc. Therefore, it is impossible to monitor the runtime information of heterogeneous hardware devices, which brings difficulties to the troubleshooting of device anomalies and the deployment of services.

[0008] 3. The default scheduling strategy causes cluster resources to be fragmented

[0009] When the Kubernetes Api Server receives a container scheduling request from a developer or operator, it executes the specific scheduling process through the scheduler component. The default scheduling algorithm tends to schedule containers in a load-balanced manner. This default scheduling strategy often leads to the following problems: When the cluster load is high, the balanced scheduling strategy results in a small amount of remaining resources on each node. When a container with high resource requirements arrives, the total amount of remaining resources in the cluster is sufficient but no node can meet its scheduling requirements, forcing the container to be suspended (Pending). The remaining resources in the cluster are also difficult to fully utilize in the subsequent container scheduling process, and are in a fragmented distribution state.

[0010] 4. The default scheduling strategy is inefficient in large-scale clusters

[0011] The default scheduling strategy of Kubernetes uses multiple scheduling extension points to work together. Based on the process of the Kubernetes Scheduling Framework, in the final scoring strategy of each Pod scheduling process, a full list operation will be performed for each Pod to score and bind (that is, all nodes selected in the previous link will be scored, and finally the node with the highest score will be selected for scheduling and binding). In the case of a large cluster, each full list operation will take a lot of time, resulting in low efficiency in container scheduling. Summary of the invention

[0012] In order to solve the above-mentioned problems arising in the process of virtualization management of domestic smart boards, the present invention provides a resource integration monitoring and scheduling device based on Kubernetes smart board clusters to realize resource integration monitoring and container scheduling of domestic smart board clusters.

[0013] An embodiment of the present invention provides a resource integrated monitoring and scheduling device based on a Kubernetes smart card cluster, comprising:

[0014] Custom scheduler, smart board integration component, smart board monitoring component, and node priority queue;

[0015] The custom scheduler is used to schedule the container according to the preset scheduling strategy, the operation results of the node priority queue and the operation results of the domestic smart board integration component;

[0016] The domestic smart board integration component is used to integrate the target smart board into the Kubernetes cluster, and connect the integrated target smart board to the Kubernetes cluster as a third-party resource;

[0017] The domestic smart board monitoring component is used to regularly report the operating parameter information of the target smart board to the distributed monitoring component, and collect the operating parameter information of the target smart board in a multi-threaded manner according to the interface;

[0018] The node priority queue is used to dynamically adjust the node queue cached internally, and provide the operation result to the custom scheduler, so that the custom scheduler can score the nodes in the adjusted node priority queue by traversing.

[0019] Optionally, the step of integrating the target smart board into the Kubernetes cluster by the domestic smart board integration component includes:

[0020] Integrate the target smart card object into Kubernetes by extending resources;

[0021] Scan the device files in the target directory and update the proxy component;

[0022] The target smart card information is reported to the Kubernetes application program interface server through the proxy component in a control loop manner, and the binding relationship between the corresponding node object and the smart card object is changed when the smart card object information changes.

[0023] Optionally, the domestically produced smart board monitoring component is also used to report user-defined monitoring data to a distributed monitoring component server through a Prometheus Exporter mechanism.

[0024] Optionally, the dynamic adjustment process of the node priority queue includes:

[0025] The node priority queue controller observes the relevant events of the node object through a control loop;

[0026] Add, delete, modify and check the node elements in this queue according to the event type of the relevant event, including updating the remaining amount of resources of the smart board object bound to the node;

[0027] The positions of elements in the queue are dynamically adjusted based on the principle that the less remaining resources, the higher the priority.

[0028] Optionally, the custom scheduler includes extension points for the pre-selection phase and extension points for the optimization phase;

[0029] Among them, the extension point in the pre-selection stage is used to preliminarily screen the runnable nodes according to the pre-selection strategy;

[0030] The extension point of the optimization phase is used to select the best node in the pre-selected result list for binding scheduling according to the optimization algorithm.

[0031] Optionally, the custom scheduler includes a filtering module for filtering nodes that cannot run the currently scheduled Pod during the scheduling process.

[0032] Optionally, the custom scheduler includes a scoring module for scoring and ranking the nodes that have passed the filtering stage.

[0033] Optionally, the custom scheduler includes a normalization module for modifying node scores before the scheduler calculates the final ranking of the nodes.

[0034] Beneficial effects of the present invention:

[0035] 1. The present invention integrates domestic smart card resources into the Kubernetes cluster through the smart card integration component (Device Plugin) framework, so that the Kubernetes cluster can discover heterogeneous hardware devices and enable containers to apply for the resources;

[0036] 2. The present invention reports the runtime parameters of the domestic smart board to the distributed monitoring component Prometheus through the Prometheus Exporter mechanism;

[0037] 3. The present invention develops a custom scheduler based on the Kubernetes scheduling framework to solve the problem of cluster resource fragmentation and optimize cluster resource utilization;

[0038] 4. The present invention designs a node priority cache queue, which solves the problem of low efficiency of using the default scheduling strategy in large-scale clusters from the performance perspective. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a general architecture diagram of a resource integration monitoring and scheduling device based on a Kubernetes smart card cluster in the present invention;

[0040] Figure 2It is a schematic diagram of the interaction process of each component in the integrated monitoring and scheduling process of the present invention;

[0041] Figure 3 It is the overall architecture diagram of the domestically produced smart board integrated component in the present invention;

[0042] Figure 4 It is a schematic diagram of the working process of the domestic smart board integrated component in the present invention;

[0043] Figure 5 It is a schematic diagram of the working process of the domestic intelligent board monitoring component in the present invention;

[0044] Figure 6 It is the timing diagram of the domestic intelligent board monitoring component in the present invention;

[0045] Figure 7 This is a schematic diagram of the Kubernetes scheduling framework in the present invention;

[0046] Figure 8 It is the class diagram of the custom scheduler design in the present invention;

[0047] Fig. 9 It is a timing diagram of the custom scheduler in the present invention;

[0048] Fig.10 It is the cluster node management information operation diagram in the present invention;

[0049] Fig.11 It is a node detailed information display diagram in the present invention;

[0050] Fig.12 It is a command line test result display diagram in the present invention;

[0051] Fig.13 It is a display diagram of the monitoring-related component Pod in the present invention;

[0052] Fig.14 It is a display diagram of the temperature information of the board card during operation in the present invention;

[0053] Fig.15 It is a comparison diagram of cluster resource utilization in the present invention;

[0054] Fig.16 It is a comparison chart of the optimization ratio of task running time in the present invention. DETAILED DESCRIPTION

[0055] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0056] The present invention provides a resource integrated monitoring and scheduling device (Kube-Rubber) based on Kubernetes smart card cluster. Typically, the smart card cluster environment in this embodiment is constructed based on domestic MLU100 and MLU270. Figure 1 Kube-Rubber is located in the Kubernetes extension layer as a whole and is developed based on the API provided by Kubernetes. The device includes: a custom scheduler, an intelligent board integration component (MLU Device Plugin), an intelligent board monitoring component (MLU Exporter), and a node priority queue (Node Queue).

[0057] The custom scheduler is used to schedule the container according to a preset scheduling strategy, the operation results of the node priority queue and the operation results of the domestic smart board integration component.

[0058] The domestically produced smart board integration component is used to integrate the target smart board into the Kubernetes cluster, and connect the integrated target smart board to the Kubernetes cluster as a third-party resource, so that users and other components can apply for and operate it; the above-mentioned target smart boards are Cambrian MLU270 and MLU100 boards.

[0059] The domestic smart board monitoring component is used to regularly report the operating parameter information of the target smart board to the distributed monitoring component Prometheus, and collect the operating parameter information of the target smart board through multi-threading according to the Exporter interface, such as board utilization, running memory, temperature, power consumption, and the number of single-node boards.

[0060] The node priority queue is used to dynamically adjust the node queue cached internally, and provide the operation results to the custom scheduler, so that the custom scheduler can score the nodes by traversing the adjusted node priority queue. When scoring nodes, the custom scheduler does not need to use the Kubernetes native full pull (list) API, but finds the optimal node by traversing the final adjusted node priority queue. This method can complete the optimal node selection process with a smaller time complexity coefficient.

[0061] See also Figure 2 , Figure 2 The diagram shows the interaction process of various components in the integrated monitoring and scheduling process.

[0062] Furthermore, the above four components are described in detail below.

[0063] 1. Domestic smart card integrated components (MLU Device Plugin)

[0064] The overall architecture designed for MLU Device Plugin is as follows Figure 3 As shown, its workflow is as follows Figure 4 The Device Plugin designed for Cambrian MLU domestic boards integrates domestic board information into Kubernetes through the following three key steps:

[0065] Step 1: Integrate the MLU API object into Kubernetes through Extender Resource.

[0066] Step 2: Scan the device files in the / dev directory in Watch mode and update the proxy component Kubelet;

[0067] Step 3: Report MLU information to the Kubernetes API Server through Kubelet in a control loop, and change the corresponding Node API object binding relationship when changes occur;

[0068] At the same time, MLU Device Plugin not only includes the function of reporting third-party device information to the Kubernetes API Server, but also includes the function of allocating and mounting devices for containers during node startup. When the container starts, Kubelet calls the Allocate interface to MLU Device Plugin to allocate a card for it, and finally performs the specific mounting action through the Docker container runtime interface (CRI).

[0069] Device Plugin is a third-party hardware integration framework provided by Kubernetes. Third-party developers and operation and maintenance personnel develop hardware integration components through the Device Plugin framework to integrate heterogeneous hardware into the cluster. Through the integration method of Device Plugin, the Kubernetes cluster can discover heterogeneous hardware devices and enable containers to apply for the resources.

[0070] 2. Domestic intelligent board monitoring components (MLU Exporter)

[0071] The main function of MLU Exporter is to report the runtime parameters of MLU boards to the distributed monitoring component Prometheus. It reports user-defined monitoring data to the Prometheus central server through the Prometheus Exporter mechanism. Prometheus Exporter is the officially recommended way to integrate user-defined monitoring data by Prometheus. It reports the runtime information of boards to the distributed monitoring component Prometheus regularly for external observation. Specifically, the workflow of MLU Exporter is as follows: Figure 5 shown.

[0072] MLU Exporter connects to the listening port of Prometheus Server through HTTP. The process of collecting and submitting data is deployed on the client through DaemonSet daemon process container, so that a process of collecting and submitting data runs on each node. In addition, each process uses multi-threading to report data to the listening port, such as Figure 6 In addition, MLU Exporter also needs to consider the form of data. Prometheus only supports four data types. Because the temperature, power consumption, quantity, utilization, and memory information of the board are values ​​that can be increased or decreased, the Gauge format is used to encapsulate the data before sending it to inform the Prometheus Server of the specific type of custom data.

[0073] 3. Node Priority Queue

[0074] like Figure 2 As shown, the node priority queue plays a role in optimizing scheduling efficiency in the device designed in this application. The dynamic adjustment process of the node priority queue (Node Queue) mainly includes the following key steps: Step 1, the node priority queue controller observes the relevant events of the node object through a control loop;

[0075] Step 2: Add, delete, modify and check the node elements in this queue according to the event type, including updating the remaining amount of resources of the domestic smart card object (MLU) bound to the node;

[0076] Step 3: According to the principle that the less remaining resources, the higher the priority, the position of the elements in the queue is dynamically adjusted. The higher the priority, the closer to the head of the queue.

[0077] 4. Custom Scheduler

[0078] like Figure 7As shown in the figure, Kubernetes Scheduling Framework is the latest pluggable framework supported by Kubernetes Scheduler. It adds a set of plugin APIs to the existing scheduler, and finally runs the user-defined algorithm into a separate Scheduler Pod through dynamic registration. Users match the scheduler by specifying the Scheduler in the yaml file.

[0079] The Kubernetes Scheduling Framework defines a series of extension points, and developers can achieve different scheduling effects by implementing different extension points. These extension points are mainly divided into two categories, namely the pre-selection stage and the optimization stage. The purpose of the extension points in the pre-selection stage is to preliminarily screen the runnable nodes according to the pre-selection strategy, while the purpose of the extension points in the optimization stage is to select the best node in the pre-selection result list according to the optimization algorithm for binding scheduling.

[0080] Furthermore, the custom scheduler includes a filter module for filtering nodes that cannot run the currently scheduled Pod during the scheduling process. For each node, the scheduler will call this type of plug-in in the configured order. If any filter plug-in marks a node as infeasible, the remaining plug-ins will not be called for the node. Nodes can be evaluated at the same time, and the Filter plug-in can be called multiple times in the same scheduling cycle. During the calling process, the scheduler actually starts multiple goroutines to call filters concurrently on multiple nodes to improve filtering efficiency. In this extension point, the filter will pull the contents of the node cache queue and keep a copy in the cache of its own extension point for scoring in the next stage.

[0081] The custom scheduler includes a scoring module, which is used to score and rank the nodes that have passed the filtering stage. This extension point implements the scoring strategy in the scheduling process, and the scheduler will call each scoring plug-in for each Score node. The scheduling algorithm of the Score extension point implemented by this device is designed based on the cache queue in the filter. First, traverse from the head of the cache queue to determine whether the remaining amount of domestic smart board resources has met the needs of the current container. If it does, set it to the highest score. If it does not meet the requirements, continue to traverse backwards until a node that meets the requirements is found or traverse to the end of the queue.

[0082] This algorithm not only solves the problem of resource fragmentation of domestic boards in the cluster, but also improves scheduling efficiency. Since the traversal search is based on the priority cache queue, the less the remaining resources of all domestic boards, the closer they are to the head of the queue. Therefore, the first node that meets the demand is the best matching node. After the actual resource allocation, compared with the balanced scheduling method, the nodes with smaller remaining resources can be allocated first, reducing the probability of fragmentation. At the same time, based on the O(n) time complexity of traversal, the difference from the default full list of Kubernetes is that when Score finds the first node that meets its own needs during the traversal process, it exits the Score link and enters the next link, while the traditional full list method requires scoring each node one by one, and finally selecting the node with the highest score. Therefore, in terms of the traversal coefficient, the time complexity coefficient of this application is smaller.

[0083] The custom scheduler includes a Normalize Score module, which is used to modify the node scores before the scheduler calculates the final ranking of the nodes. The "Normalize" extension point is used to modify the scores before the scheduler calculates the final ranking of the nodes. It is necessary to ensure that the output of the Score plugin must be an integer in the range of [MinNodeScore, MaxNodeScore] ([0-100]).

[0084] The design of the custom scheduler is relatively complex and core, and plays the role of interacting with other components at the same time. Therefore, in order to understand its internal logic more clearly, this embodiment will provide the design class diagram and timing diagram of the custom scheduler, such as Figure 8 and Fig. 9 shown.

[0085] Experimental verification

[0086] 1. Test environment

[0087] The specific parameters of the test server are shown in Table 1:

[0088] Table 1 Specific parameters of the test server

[0089]

[0090] For the integration and monitoring test of domestic smart boards in the cluster, it is required that the test cluster of this device can be expanded to 10 smart nodes. Therefore, this achievement is proved from two aspects. On the one hand, the screenshot from the cluster control program (command line) proves that the cluster manages 10 smart nodes; on the other hand, through the front-end page of the grafana component, the monitoring data is observed by integrating the Prometheus data source, and the screenshot proves that Prometheus has successfully obtained the board runtime information reported by the MLU Exporter.

[0091] For resource defragmentation and scheduling efficiency, the solution adopted by this application is to manage resource fragments and accelerate scheduling through a priority-based dynamic adjustment strategy. In order to verify that the cluster has the defragmentation function and has good scheduling efficiency, it is measured from two dimensions: the system layer and the task layer.

[0092] On the one hand, from the perspective of the system layer, better fragmentation should provide a better resource allocation solution. That is, the resource utilization rate can be increased as much as possible under the premise of the same task scale. Therefore, cluster resource utilization is selected as an indicator. On the other hand, from the perspective of the task layer, under the input of the same scale of tasks, this device reduces the waiting time caused by insufficient resources as much as possible, and is accompanied by the efficient search principle of the node cache queue, thereby shortening the overall execution time of the task. Therefore, the overall running time of the task is selected as another test indicator, and the task running time optimization ratio is compared.

[0093] Among them, resource utilization:

[0094]

[0095] Task runtime optimization ratio:

[0096]

[0097] Based on the above analysis, we divide the test content into cluster domestic intelligent board integration and monitoring test and scheduler resource defragmentation and performance test.

[0098] 2. Integration and monitoring testing

[0099] 1. Integration testing

[0100] Run the following command in the cluster console: kubectl get node;

[0101] get Fig.10 As a result, 10 nodes are successfully included in the cluster management:

[0102] Next, execute the following command in the cluster console: kubectl describe node;

[0103] Fig.11 To output some screenshots, it can be found that 4 MLU devices are integrated on each node (for unified management, the domestic smart boards MLU270 and MLU100 of the same category are labeled with one label):

[0104] Finally, execute the following command in the cluster console:

[0105] kubectl describe nodes|tr-d'\000'|sed-n-e' / ^Name / , / Roles / p'-e' / ^Capacity / , / Allocatable / p'-e' / ^Allocated resources / , / Events / p'|grep-e Name-emlu270|perl-pe's / \n / / '|perl-pe's / Name: / \n / g'|sed's / ustc.com\ / mlu100:\? / / g'|sed'1s / ^ / Node Available(MLUs) / '|sed's / $ / 0 0 0 / '|awk'{print$2,$3}'|column–t;

[0106] Then get Fig.12 There are a total of 10 nodes equipped with domestically produced smart boards, with each node having 4 boards.

[0107] The above tests prove that the domestic smart boards MLU100 and MLU270 have been successfully integrated into the Kubernetes cluster.

[0108] 2. Monitoring test

[0109] Execute the following command in the cluster control: kubectl get pod-n monitoring;

[0110] get Fig.13 The output of the Kubernetes monitoring namespace includes the running Pods of the Prometheus monitoring component, the grafana front-end component, and the MLU Exporter component:

[0111] Enter https: / / 192.168.30.248:2999 in the URL to enter the grafana console, select the Prometehus data source in the new Pannel, and you can find the runtime parameters of the corresponding domestic smart board. This proves that the MLU Exporter works normally and successfully reports the board runtime information to Prometheus.

[0112] 3. Defragmentation and performance testing

[0113] 1. Dataset preparation

[0114] This test selected the large-scale production cluster data of Alibaba's artificial intelligence platform as the test data task set, and used the default scheduler and the scheduler with resource defragmentation function to schedule the task set based on the resources of domestic smart boards in the verification cluster. This test data set tracks and records the data in a large production cluster with 6,500 GPUs (distributed on 1,800 machines) in Alibaba PAI (artificial intelligence platform) in July and August 2020. Its tasks include training and reasoning tasks of Alibaba's machine learning platform. In addition, the data set contains relevant information about the running of task instances and relevant resource indicators of the machine.

[0115] For this test set, we first perform reasonable data cleaning to meet the server configuration of this experimental environment. This mainly includes the following aspects:

[0116] (1) Filter incomplete data and filter columns with missing information.

[0117] (2) Filter tasks with more than 40 instances and limit the number of generated instances according to the conditions of the test cluster.

[0118] (3) Normalize the maximum and minimum values ​​of the CPU and memory for the test cluster. The number of CPU cores is normalized to 0 to 32 cores. The memory size is normalized to 0 to 64000 Mi.

[0119] (4) Perform quantity simulation on the gpu column data for domestic intelligent computing boards. First, convert the number of gpus to the number of MLUs in a 1:1 ratio. Then perform normalization to normalize the number of MLUs to 1 to 4.

[0120] (5) Convert the start time and completion time into task running time.

[0121] 2. Test data and conclusions

[0122] The task sets for this test are all sampled from the source data of the Alibaba Cloud platform. Among them, high-performance tasks are task sequences with 3 or 4 board requirements in the data set, and low-performance tasks are task sequences with 1 or 2 board requirements. Intensive tasks are tasks with more than 5 instances and a runtime greater than 5 units, and sparse tasks are the opposite. Mixed tasks are random sampling methods that select continuous tasks in the data set for simulation testing.

[0123] The verification cluster uses the platform configuration of Haiguang CPU + domestic smart card and Intel + domestic smart card, and has a complete test plan. The following results are collected through automated testing (the test objects are the default scheduler and the custom scheduler):

[0124] Table 2 Resource utilization data of resource fragmentation test

[0125]

[0126] According to the above data, after the cluster is managed by the custom scheduler, the resource utilization rate can be increased by an average of 8.85%; the optimization rate of task running time can reach 6.88%. And the scheduler will perform better when the task is intensive. After testing, the custom scheduler component in this device can effectively reduce the generation of resource fragmentation and has a high scheduling efficiency.

[0127] Example

[0128] Example 1 provides a custom scheduler based on the Kubernetes Scheduling Framework. In this example, there is a method for developing a custom scheduler based on the Kubernetes Extender method. This method is applicable to versions before Kubernetes 1.16 and is another method for implementing the custom scheduler function. This method integrates the pre-selection and optimization strategies in the scheduling process, and redeploys the scheduler to the cluster by recompiling the source code. It has poor scalability and is not conducive to secondary development by third-party developers based on Kubernetes.

[0129] Embodiment 2 provides a method for integrating domestically produced smart devices based on Kubernetes Device Plugin. The detailed process of this embodiment has been fully introduced in the above content. The process of Device Plugin can be replaced by command line when reporting, but this method is not convenient for unified management. Therefore, the MLU Device Plugin designed by this application can be used to easily integrate third-party domestically produced smart board resources through the deployment method of DaemonSet.

[0130] Example 3 provides a domestic intelligent device monitoring method based on Prometheus Exporter. Prometheus is a distributed monitoring component that is widely used today. Similarly, integrating custom monitoring information using a push gateway is another solution to achieve this function. However, prometheus does not recommend using this method. The main reason is that pushgateway cannot make intelligent judgments on the data sent. If there are errors in the data collected by the program itself, pushgateway will still report all collected data to the prometheus server, and this method can only be integrated into the prometheus UI, and it is more complicated to integrate into the more convenient grafana front-end access platform.

[0131] Example 4 provides a container scheduling algorithm based on node priority queues. The purpose of this embodiment is to optimize the efficiency of container scheduling while reducing the generation of resource fragments. Similarly, the Kubernetes rescheduler reschedules the Pod in this way when the Pod Pending is suspended by migration and redeployment. However, this solution is essentially based on a passive approach and has a large stop and start overhead.

[0132] Embodiment 5 provides a solution for cluster domestic smart card integration and monitoring test and scheduler resource defragmentation and performance test. The specific solution has been described in detail in the previous part.

[0133] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A resource integrated monitoring and scheduling device based on Kubernetes smart card cluster, characterized in that: include: Custom scheduler, smart board integration component, smart board monitoring component, and node priority queue; The custom scheduler is used to schedule the container according to the preset scheduling strategy, the operation results of the node priority queue and the operation results of the domestic smart board integration component; The domestic smart board integration component is used to integrate the target smart board into the Kubernetes cluster, and connect the integrated target smart board to the Kubernetes cluster as a third-party resource; The domestic smart board monitoring component is used to regularly report the operating parameter information of the target smart board to the distributed monitoring component, and collect the operating parameter information of the target smart board in a multi-threaded manner according to the interface; The node priority queue is used to dynamically adjust the node queue cached internally, and provide the operation result to the custom scheduler, so that the custom scheduler can score the nodes in the adjusted node priority queue by traversing; The smart board monitoring component is also used to report user-defined monitoring data to the distributed monitoring component server through the Prometheus Exporter mechanism.

2. The device according to claim 1, characterized in that The steps of integrating the target smart card into the Kubernetes cluster by the domestic smart card integration component include: Integrate the target smart card object into Kubernetes by extending resources; Scan the device files in the target directory and update the proxy component; The target smart card information is reported to the Kubernetes application program interface server through the proxy component in a control loop manner, and the binding relationship between the corresponding node object and the smart card object is changed when the smart card object information changes.

3. The device according to claim 1, characterized in that The dynamic adjustment process of the node priority queue includes: The node priority queue controller observes the relevant events of the node object through a control loop; Add, delete, modify and check the node elements in this queue according to the event type of the relevant event, including updating the remaining amount of resources of the smart board object bound to the node; The positions of elements in the queue are dynamically adjusted based on the principle that the less remaining resources, the higher the priority.

4. The device according to claim 1, characterized in that The custom scheduler includes extension points in the pre-selection phase and extension points in the optimization phase; Among them, the extension point in the pre-selection stage is used to preliminarily screen the runnable nodes according to the pre-selection strategy; The extension point of the optimization phase is used to select the best node in the pre-selected result list for binding scheduling according to the optimization algorithm.

5. The device according to claim 4, characterized in that The custom scheduler includes a filtering module, which is used to filter nodes that cannot run the currently scheduled Pod during the scheduling process.

6. The device according to claim 4, characterized in that The custom scheduler includes a scoring module for scoring and ranking the nodes that have passed the filtering stage.

7. The device according to claim 4, characterized in that The custom scheduler includes a normalization module for modifying node scores before the scheduler calculates the final ranking of the nodes.

Images