Container fault processing method, container scheduling method, related device and medium

Abstract

The present disclosure provides a container fault processing method, a container scheduling method, related devices and media. The container fault processing method comprises: collecting resource usage data of a heterogeneous instruction set architecture node, the heterogeneous instruction set architecture node being provided with a container, the resource usage data comprising host level and container level resource usage data, and performance indicators related to the heterogeneous instruction set architecture; using a machine learning algorithm to detect whether the heterogeneous instruction set architecture node has a fault based on the resource usage data; in the case that the heterogeneous instruction set architecture node has a fault, diagnosing the cause of the fault according to a pre-established fault knowledge graph, the fault knowledge graph comprising related container fault modes under the heterogeneous instruction set architecture; and according to the diagnosis result, recovering the fault related to the container of the heterogeneous instruction set architecture node to accurately locate and solve the container fault in the heterogeneous instruction set architecture environment.

Description

TECHNICAL FIELD

[0001] The present disclosure belongs to the technical field of virtualization, and particularly relates to a container fault processing method, a container scheduling method, related devices and media. BACKGROUND

[0002] With the rapid development of cloud computing and edge computing technologies, the instruction set architectures of nodes in data centers are gradually developing towards heterogeneity. Heterogeneous instruction set architectures can provide higher performance and lower energy consumption, meeting the needs of data centers for diversified workloads. Among them, the RISC-V instruction set architecture, as an open-source instruction set architecture, has been widely used in servers and edge devices due to its low power consumption and high performance. The X86 instruction set architecture dominates in data centers due to its mature ecosystem and extensive software support. In the environment of heterogeneous instruction set architectures, container technology is widely used in data centers due to its lightweight and efficient characteristics. Container technology provides an isolated environment to run applications, achieving fast deployment, expansion and management of applications. However, when running containers on heterogeneous instruction set architecture nodes, due to the differences in instruction sets between heterogeneous instruction set architecture nodes, some architecture-specific container faults may occur. The current container fault diagnosis and recovery scheme is mainly designed for a single instruction set architecture and cannot effectively handle container faults in a heterogeneous instruction set architecture environment. Therefore, it is urgent to provide a container fault processing method to accurately locate and solve container faults in a heterogeneous instruction set architecture environment. SUMMARY

[0003] In view of the above problems, the present disclosure provides a container fault processing method, a container scheduling method, related devices and media, aiming to accurately locate and solve container faults in a heterogeneous instruction set architecture environment.

[0004] According to a first aspect of the present disclosure, a container fault processing method is provided, comprising:

[0005] Collecting resource usage data of a heterogeneous instruction set architecture node, the heterogeneous instruction set architecture node being provided with a container, the resource usage data including host-level and container-level resource usage data, and performance indicators related to the heterogeneous instruction set architecture;

[0006] Based on the resource usage data, using a machine learning algorithm to detect whether the heterogeneous instruction set architecture node has a fault;

[0007] In the case that the heterogeneous instruction set architecture node has a fault, diagnosing the fault cause according to a pre-established fault knowledge graph, the fault knowledge graph including related container fault patterns under the heterogeneous instruction set architecture;

[0008] According to the diagnosis result, a container-related fault of the heterogeneous instruction set architecture node is recovered.

[0009] Optionally, the detecting whether the heterogeneous instruction set architecture node has a fault based on the resource usage data includes:

[0010] The resource usage data of the heterogeneous instruction set architecture node is detected in real time using a rule engine, and it is determined that the heterogeneous instruction set architecture node has a fault when the resource usage data does not meet a preset numerical range.

[0011] Optionally, the detecting whether the heterogeneous instruction set architecture node has a fault based on the resource usage data includes:

[0012] The resource usage data of the heterogeneous instruction set architecture node is detected in real time using a normal behavior model established based on a long short-term memory network algorithm, and it is determined that the heterogeneous instruction set architecture node has a fault when the resource usage data deviates from a normal behavior benchmark.

[0013] Optionally, the detecting whether the heterogeneous instruction set architecture node has a fault based on the resource usage data includes:

[0014] The resource usage data of the heterogeneous instruction set architecture node is detected in real time using an anomaly detection model established for a feature of the heterogeneous instruction set architecture, and it is determined that the heterogeneous instruction set architecture node has a fault when the resource usage data has an anomaly.

[0015] Optionally, the diagnosing a fault cause according to a pre-established fault knowledge graph when the heterogeneous instruction set architecture node has a fault includes:

[0016] A fault cause is analyzed based on a container fault mode related to the heterogeneous instruction set architecture.

[0017] A root cause of the fault is located by performing causal analysis on the analyzed fault cause using a causal reasoning engine.

[0018] Optionally, the heterogeneous instruction set architecture node includes a first node supporting a first instruction set architecture and a second node supporting a second instruction set architecture, and the recovering a container-related fault of the heterogeneous instruction set architecture node according to the diagnosis result includes:

[0019] In a case where the fault root cause is that a specific application is incompatible with a first instruction set architecture supported by the first node, container migration is performed on a container corresponding to the specific application between the first node and the second node.

[0020] Optionally, the recovering, according to the diagnosis result, of the container-related fault of the heterogeneous instruction set architecture node further includes:

[0021] The recovering, according to the diagnosis result, of the container-related fault of the heterogeneous instruction set architecture node further includes:

[0022] According to a second aspect of the present disclosure, a container scheduling method is provided, including:

[0023] The container scheduling is performed for the heterogeneous instruction set architecture node using a scheduling strategy, the heterogeneous instruction set architecture node including a first node supporting a first instruction set architecture and a second node supporting a second instruction set architecture;

[0024] In a case where the fault root cause determined according to the container fault processing method is that a specific application is incompatible with a first instruction set architecture supported by the first node, the scheduling strategy is updated so as to deploy the specific application to the second node.

[0025] According to a third aspect of the present disclosure, a container fault processing apparatus is provided, including:

[0026] A data collection module is configured to collect resource usage data of a heterogeneous instruction set architecture node, the heterogeneous instruction set architecture node being provided with a container, the resource usage data including host-level and container-level resource usage data and a performance index related to the heterogeneous instruction set architecture;

[0027] A fault detection module is configured to detect, based on the resource usage data, whether the heterogeneous instruction set architecture node has a fault using a machine learning algorithm;

[0028] A fault cause diagnosis module is configured to, in a case where the heterogeneous instruction set architecture node has a fault, perform fault cause diagnosis according to a pre-established fault knowledge graph, the fault knowledge graph including a container fault mode related to the heterogeneous instruction set architecture;

[0029] A fault recovery module is configured to recover, according to a diagnosis result, a container-related fault of the heterogeneous instruction set architecture node.

[0030] Optionally, the fault detection module is further configured to:

[0031] The rule engine is used to detect resource usage data of the heterogeneous instruction set architecture node in real time, and the heterogeneous instruction set architecture node is determined to have a fault when the resource usage data does not meet a preset numerical range.

[0032] Optionally, the fault detection module is further configured to:

[0033] The normal behavior model established based on the long short-term memory network algorithm is used to detect the resource usage data of the heterogeneous instruction set architecture node in real time, and the heterogeneous instruction set architecture node is determined to have a fault when the resource usage data deviates from a normal behavior benchmark.

[0034] Optionally, the fault detection module is further configured to:

[0035] The anomaly detection model established based on features of the heterogeneous instruction set architecture is used to detect the resource usage data of the heterogeneous instruction set architecture node in real time, and the heterogeneous instruction set architecture node is determined to have a fault when the resource usage data is abnormal.

[0036] Optionally, the fault cause diagnosis module is further configured to:

[0037] The fault cause is analyzed based on a related container fault mode under the heterogeneous instruction set architecture.

[0038] The fault cause diagnosis module is further configured to:

[0039] The root cause of the fault is located by using a causal reasoning engine to perform causal analysis on the analyzed fault cause.

[0040] Optionally, the heterogeneous instruction set architecture node includes a first node supporting a first instruction set architecture and a second node supporting a second instruction set architecture, and the fault recovery module is further configured to:

[0041] In a case where the root cause of the fault is that a specific application program is incompatible with the first instruction set architecture supported by the first node, the container corresponding to the specific application program is migrated between the first node and the second node.

[0042] Optionally, the fault recovery module is further configured to:

[0043] According to the diagnosis result, a recovery operation is automatically performed on the container of the heterogeneous instruction set architecture node, and the recovery operation includes container restart and container expansion.

[0044] a container scheduling module, configured to perform container scheduling for the heterogeneous instruction set architecture nodes using a scheduling strategy, the heterogeneous instruction set architecture nodes including a first node supporting a first instruction set architecture and a second node supporting a second instruction set architecture;

[0045] a container scheduling optimization module, configured to update the scheduling strategy so as to deploy a specific application to the second node in a case where the root cause of the fault determined according to the container fault processing method is that the specific application is incompatible with the first instruction set architecture supported by the first node.

[0046] According to a fifth aspect of the present disclosure, an electronic device is provided, including a memory, a processor, and a program stored in the memory and executable on the processor, the program being executable by the processor to implement the method described above.

[0047] According to a sixth aspect of the present disclosure, a storage medium is provided, the storage medium storing a computer program or instructions, the computer program or instructions being executable by a processor to implement the steps of the method according to any one of the above.

[0048] The present disclosure brings the following beneficial effects:

[0049] The container fault processing method provided by the present disclosure detects the container-related fault of the heterogeneous instruction set architecture node according to the resource usage data of the host level and the container level, and the resource usage data such as the performance indicators related to the heterogeneous instruction set architecture, diagnoses the root cause of the fault according to the fault knowledge graph including the container fault mode related to the heterogeneous instruction set architecture, locates the root cause of the container-related fault of the heterogeneous instruction set architecture node, and recovers the container-related fault of the heterogeneous instruction set architecture node according to the diagnosis result. In this way, the performance indicators related to the heterogeneous instruction set architecture and the container fault mode related to the heterogeneous instruction set architecture both consider the instruction set difference between different instruction set architecture nodes, can deeply understand the characteristics of different instruction set architectures, can realize the diagnosis of the fault across the instruction set architectures, and can accurately locate and solve the container fault in the heterogeneous instruction set architecture environment.

[0050] Other features and advantages of the present disclosure will be described in the following description, and some will become apparent from the description, or will be learned from the practice of the present disclosure. The purposes and other advantages of the present disclosure will be realized and achieved by the structures particularly pointed out in the specification as well as the appended drawings.

[0051] In order to make the above objectives, features and advantages of the present disclosure more apparent and easy to understand, the following preferred embodiments are specifically described with reference to the attached drawings. BRIEF DESCRIPTION OF DRAWINGS

[0052] The above and other objects, features and advantages of the present disclosure will become more apparent from the following description when taken in conjunction with the accompanying drawings, in which:

[0053] Figure 1 is a structural diagram of a data center to which one embodiment of the present disclosure is applied;

[0054] Figure 2 is a structural diagram of a worker node according to one embodiment of the present disclosure;

[0055] Figure 3 shows a flowchart of a container failure processing method according to one embodiment of the present disclosure;

[0056] Figure 4 shows a flowchart of a container scheduling method according to one embodiment of the present disclosure;

[0057] Figure 5 shows a structural diagram of a container failure processing apparatus according to one embodiment of the present disclosure;

[0058] Figure 6 shows a structural diagram of a scheduler according to one embodiment of the present disclosure;

[0059] Figure 7 shows a structural diagram of an electronic device according to one embodiment of the present disclosure. DETAILED DESCRIPTION

[0060] Various embodiments of the present disclosure will be described hereinafter with reference to the accompanying drawings. In the drawings, like reference numerals are used to indicate the same elements throughout the several views. Each part of the drawings is not drawn to scale for the sake of clarity.

[0061] The following terms are used herein:

[0062] Container: As a lightweight virtualization technology, it is a set of processes that are limited in resources and isolated from each other. Container technology creates an independent running environment for different applications, realizes resource isolation, configuration and security protection, and can meet the resource demand of application on-demand allocation and guarantee the isolation and availability of the application. The application carried in the container is called container instance, or also called containerized application. In order to meet the needs of large-scale applications, many containers are often deployed in computer clusters for unified management and external service in practice, so container orchestration tools are needed. Container orchestration tools use container services and orchestrate them to decide how to interact between containers, extend the life cycle management capability to complex, multi-container workloads deployed on a large number of computer clusters, and provide an abstraction layer for developers and infrastructure teams to handle large-scale containerized deployment. The container orchestration tool is, for example, the K8s (full name is Kubernetes, a system that runs and coordinates containerized application processes) system, which is an open source system for automatically deploying, expanding and managing containerized applications. Docker is an open source application container engine, which is a tool for starting and stopping containers, allowing developers to package their applications and dependent packages into a portable container, and then publishing them to any popular machine. It can also achieve virtualization. Containers are completely using the sandbox mechanism and will not have any interface with each other.

[0063] Pod: It is the basic operation unit of the container orchestration tool (for example, K8s) and is the smallest deployment unit that can be created, debugged and managed. All containers in the same pod share the same IP address, IPC, host name and other resources. The pod abstracts the network and storage from the underlying containers, so it can be more easily moved in the cluster. One or more containers for carrying applications can be encapsulated in each pod. The containers of the pod are scheduled as a whole to a worker node for running.

[0064] Figure 1 is the structure diagram of a data center to which an embodiment of the present disclosure is applied. As shown in Figure 1 , for convenience of description, Figure 1 only the control node 110 and the worker node 120 are shown. In implementation, the control node 110 or the worker node 120 can be a server, and also can be a virtual machine on the server. It should be understood that although Figure 1Only a limited number of worker nodes 120 are shown in the figure, but the present disclosure should not be limited thereto. In some embodiments, the worker nodes 120 are heterogeneous instruction set architecture nodes, including worker node A 220 and worker node B 220, where the worker node A 220 supports the RISC-V instruction set architecture and the worker node B 220 supports the X86 instruction set architecture. It can be understood that the worker nodes 120 can also support other instruction set architectures such as the ARM instruction set architecture, and for the sake of convenience of description, Figure 1 Not shown in the figure.

[0065] In some embodiments, the data center can be applied to various application scenarios such as content delivery network (CDN), e-commerce, game, audio and video, Internet of Things, logistics, industrial brain, city brain, etc., and provide computing services for terminal users in various scenarios. Specifically, for each application scenario, an application program that can provide computing services in the application scenario can be deployed in the worker nodes 120 of the data center. Considering that a large number of applications may need to be deployed in the data center, a container technology can be used to build a container group in the worker nodes 120, carry the application program through the container, and then deploy the application program in units of containers. By running these container instances deployed on the worker nodes 120, the corresponding computing services can be provided for terminal users. It should be noted that for the worker node A 220, the containers deployed thereon support the RISC-V instruction set architecture. For the worker node B 220, the containers deployed thereon support the X86 instruction set architecture.

[0066] In some embodiments, a container orchestration tool (e.g., K8s) can be running on the control node 110 to implement orchestration management of the container instances in the data center to which the control node 110 belongs, where the orchestration management of the container instances includes at least one of the following: creation, elastic scaling, rolling update, reconstruction, migration, shutdown, etc. of the container instances. In some embodiments, in addition to the container orchestration management function, the control node 110 can also be responsible for other management operations in the data center to which the control node 110 belongs, such as monitoring and management of operation and maintenance, logs, network status, etc. Commands can be sent to each worker node 120 through the control node 110. In simple terms, the control node 110 is the manager and the worker node 120 is the managed. The background services running on the control node 110 can generally include an application programming interface server (API server) 111 and a scheduler 130. The application programming interface server 111 is the front-end interface of the container orchestration tool (e.g., K8s), and various client tools and other components of the container orchestration tool (e.g., K8s) can manage various resources of the cluster of the data center through the application programming interface server 111. The scheduler 130 can decide where to place the container group on the worker node to run and decide the resources allocated to the container group.

[0067] In some embodiments, since the worker node 120 is the real processing device of the data center, Figure 2 An internal structure diagram of the worker node of one embodiment of the present disclosure is shown. In some embodiments, the worker node 120 can include a plurality of processors 22 and a memory 29. The memory 29 in the worker node 120 can be a main memory (referred to as main memory or memory for short) for storing instruction information and / or data information represented by data signals, such as storing data provided by the processor 22 (e.g., as an operation result), and can also be used to implement data exchange between the processor 22 and the external storage device 26 (or referred to as auxiliary storage or external memory).

[0068] In some cases, the processor 22 can need to access the memory 29 to fetch data stored in the memory 29 or to modify data stored in the memory 29. Since the memory 29 has a slower access speed, to alleviate the speed gap between the processor 22 and the memory 29, the worker node 120 further includes a cache memory 28 coupled with the bus 21, which is configured to cache some data in the memory 29 that can be repeatedly invoked, such as program data or packet data. The cache memory 28 is implemented by a type of storage device such as static random access memory (SRAM). The cache memory 28 can be a multi-level structure, such as a three-level cache structure having a level one cache (L1 Cache), a level two cache (L2 Cache), and a level three cache (L3 Cache), or a cache structure with more than three levels or other types of cache structures. In some embodiments, a part of the cache memory 28 (e.g., the level one cache, or the level one cache and the level two cache) can be integrated inside the processor 22 or in the same system on chip as the processor 22.

[0069] Based on this, the processor 22 can include an instruction execution unit 221, a memory management unit 222, and the like. The instruction execution unit 221 initiates a write access request when executing some instructions that need to modify the memory, the write access request specifying write data to be written into the memory and a corresponding physical address; and the memory management unit 222 is configured to translate a virtual address specified by the instructions into a physical address mapped by the virtual address, and the physical address specified by the write access request can be consistent with the physical address specified by the corresponding instructions.

[0070] The information exchange between the memory 29 and the cache memory 28 is usually organized in blocks. In some embodiments, the cache memory 28 and the memory 29 can be divided into data blocks according to the same spatial size, and the data blocks can be the smallest unit (including one or more data of a preset length) of data exchange between the cache memory 28 and the memory 29. For the sake of brevity and clarity of description, each data block in the cache memory 28 is referred to as a cache block (which can be referred to as a cacheline or a cache line) below, and different cache blocks have different cache block addresses; and each data block in the memory 29 is referred to as a memory block, and different memory blocks have different memory block addresses. The cache block address includes, for example, a physical address tag for locating the data block.

[0071] Due to space and resource limitations, the cache memory 28 cannot cache all the contents in the memory 29, i.e., the storage capacity of the cache memory 28 is usually smaller than that of the memory 29, and the cache memory 28 cannot provide all the memory block addresses provided by the memory 29. When the processor 22 needs to access the memory, it first accesses the cache memory 28 via the bus 21 to determine whether the content to be accessed has been stored in the cache memory 28. If yes, the cache memory 28 hits, and the processor 22 directly calls the content to be accessed from the cache memory 28. If the content to be accessed by the processor 22 is not in the cache memory 28, the processor 22 needs to access the memory 29 via the bus 21 to find the corresponding information in the memory 29. Because the access rate of the cache memory 28 is very fast, the efficiency of the processor 22 can be significantly improved when the cache memory 28 hits, and thus the performance and efficiency of the entire worker node 120 are improved.

[0072] As shown in the figure, the processor 22, the cache memory 28, and the memory 29 are packaged in a system on chip (SoC) 201. Designers can configure the SoC architecture so that the communication between the various elements in the worker node 120 is secure.

[0073] In this example, the worker node 120 can also include various software. In some embodiments, as shown in Figure 2 the operating system 202, container support 203, and container group 204 are arranged on top of the underlying hardware (i.e., the system on chip 201). The operating system 202 is, for example, a UNIX operating system, a Linux operating system, or other operating system that can be used on a server. The worker node 120 supports the RISC-V instruction set architecture or other instruction set architectures such as the X86 instruction set architecture or the ARM instruction set architecture. The container support 203 is various underlying implementations required to support the containers running thereon. For example, for docker, the container support 203 needs to implement two technologies of cgroup (control groups) and namespace, the cgroup implements resource limits, and the namespace implements resource isolation. Docker allows developers to package their applications and dependent runtime environments into a portable container and then publish them on the worker node 120. As Figure 2As shown, based on the container support 203, a plurality of container groups 204 can be run, a running environment and a running application program are implemented in each container of the container group 204, and isolation and non-interference between the respective application programs are achieved through container technology. The application programs can include, but are not limited to, programs for controlling or responding to external devices (for example, biometric sensors, printers, microphones, speakers, flow valves, or other I / O components, sensors, actuators, or devices), programs for various I / O tasks, security programs, authentication programs, various computing modules, communication programs, communication support protocols, or other programs, or combinations thereof.

[0074] In some embodiments, as Figure 2 As shown, on the underlying hardware (i.e., the system on a chip 201), there is also provided

[0075] The container orchestration tool (for example, K8s) is a service agent (Kubelet) 205 on the worker node 120. In implementation, the service agent (Kubelet) 205 can be a program module of software or hardware, for example, implemented based on FPGA or CPLD, etc. In some embodiments, the service agent 205 can receive and execute instructions sent by the control node 110, manage the container groups and the containers in the container groups. The service agent 205 can register information of the worker node 120 on the application program interface server 111 of the control node 110, monitor resource usage of the worker node 120, and periodically report resource usage data of the worker node 120 to the control node 110. In some embodiments, on the underlying hardware (i.e., the system on a chip 201), a garbage collection (GC) module (not shown in the figure) can also be provided. In implementation, the garbage collection (GC) module can be a program module of software or hardware, for example, implemented based on FPGA or CPLD, etc. In some embodiments, the garbage collection (GC) module can establish a garbage collection process for each container, and clear configuration information of the container group in the case that the container group is destroyed.

[0076] In addition, the worker node 120 can also include display devices 23, audio devices 24, input / output devices 25, storage devices 26, communication devices 27, and other hardware devices. Of course, different computer systems can have different structures according to different mainboards, operating systems, and instruction set architectures.

[0077] Again referring to Figure 1 As Figure 1As shown, the background service running on the control node 110 can also include a container fault handling apparatus 140. The container fault handling apparatus 140 can collect resource usage data of each heterogeneous instruction set architecture node (including the worker node A 220 and the worker node B 220), implement diagnosis of cross-instruction set architecture faults, accurately locate and solve container faults in a heterogeneous instruction set architecture environment in a data center. Specifically, the container fault handling apparatus 140 includes a data collection module 141, a fault detection module 142, a fault cause diagnosis module 143, and a fault recovery module 144. In some embodiments, the data collection module 141 collects resource usage data of each heterogeneous instruction set architecture node. Specifically, the resource usage data can be collected by the service agent (Kubelet) 205 arranged on each heterogeneous instruction set architecture node. In some embodiments, the resource usage data includes host-level and container-level resource usage data, and heterogeneous instruction set architecture (for example, RISC-V instruction set architecture or X86 instruction set architecture) related performance indicators. It is easy to understand that by collecting resource usage data of each heterogeneous instruction set architecture node in the data center, comprehensive monitoring of the entire data center can be ensured, providing data support for subsequent fault diagnosis and scheduling optimization.

[0078] In some embodiments, cAdvisor (Container Advisor) is used to collect container-level resource usage data. Specifically, cAdvisor is deployed in the Kubernetes cluster. A DaemonSet can be created using the following YAML file:

[0079] apiVersion: apps / v1

[0080] kind: DaemonSet

[0081] metadata:

[0082] name: cadvisor

[0083] namespace: monitoring

[0084] labels:

[0085] app: cadvisor

[0086] spec:

[0087] selector:

[0088] matchLabels:

[0089] app: cadvisor

[0090] template:

[0091] metadata:

[0092] labels:

[0093] app: cadvisor

[0094] spec:

[0095] containers:

[0096] - name: cadvisor

[0097] image: gcr.io / cadvisor / cadvisor:v0.39.0

[0098] resources:

[0099] limits:

[0100] memory: 200Mi

[0101] requests:

[0102] cpu: 150m

[0103] memory: 200Mi

[0104] volumeMounts:

[0105] - name: rootfs

[0106] mountPath: / rootfs

[0107] readOnly: true

[0108] - name: var-run

[0109] mountPath: / var / run

[0110] readOnly: true

[0111] - name: sys

[0112] mountPath: / sys

[0113] readOnly: true

[0114] - name: docker

[0115] mountPath: / var / lib / docker

[0116] readOnly: true

[0117] ports:

[0118] - name: http

[0119] containerPort: 8080

[0120] protocol: TCP

[0121] volumes:

[0122] - name: rootfs

[0123] hostPath:

[0124] path: /

[0125] - name: var-run

[0126] hostPath:

[0127] path: / var / run

[0128] - name: sys

[0129] hostPath:

[0130] path: / sys

[0131] - name: docker

[0132] hostPath:

[0133] path: / var / lib / docker

[0134] In some embodiments, host-level resource usage data is collected using Node Exporter. Specifically, Node Exporter is deployed in the Kubernetes cluster. A DaemonSet can be created using the following YAML file:

[0135] apiVersion: apps / v1

[0136] kind: DaemonSet

[0137] metadata:

[0138] name: node-exporter

[0139] namespace: monitoring

[0140] labels:

[0141] app: node-exporter

[0142] spec:

[0143] selector:

[0144] matchLabels:

[0145] app: node-exporter

[0146] template:

[0147] metadata:

[0148] labels:

[0149] app: node-exporter

[0150] spec:

[0151] containers:

[0152] - name: node-exporter

[0153] image: prom / node-exporter:v1.1.2

[0154] args:

[0155] - --path.procfs= / host / proc

[0156] - --path.sysfs= / host / sys

[0157] ports:

[0158] - name: metrics

[0159] containerPort: 9100

[0160] volumeMounts:

[0161] - name: proc

[0162] mountPath: / host / proc

[0163] readOnly: true

[0164] - name: sys

[0165] mountPath: / host / sys

[0166] readOnly: true

[0167] volumes:

[0168] - name: proc

[0169] hostPath:

[0170] path: / proc

[0171] - name: sys

[0172] hostPath:

[0173] path: / sys

[0174] In some embodiments, resource collection plugins specific to a heterogeneous instruction set architecture (e.g., RISC-V instruction set architecture or X86 instruction set architecture) are built. A Python script can be created to collect RISC-V instruction set architecture and X86 instruction set architecture related performance metrics:

[0175] import psutil

[0176] import platform

[0177] from prometheus_client import start_http_server, Gauge

[0178] # Define metrics

[0179] cpu_freq = Gauge('cpu_frequency', 'CPU Frequency in MHz')

[0180] cpu_temp = Gauge('cpu_temperature', 'CPU Temperature in Celsius')

[0181] riscv_custom_metric = Gauge('riscv_custom_metric', 'Custom metric for RISC-V')

[0182] def collect_metrics():

[0183] arch = platform.machine()

[0184] if arch == "x86_64":

[0185] # Collect performance metrics related to X86 instruction set architecture

[0186] cpu_freq.set(psutil.cpu_freq().current)

[0187] if hasattr(psutil, "sensors_temperatures"):

[0188] temps = psutil.sensors_temperatures()

[0189] if "coretemp" in temps:

[0190] cpu_temp.set(temps["coretemp"][0].current)

[0191] elif arch == "riscv64":

[0192] # Collect performance metrics related to RISC-V instruction set architecture

[0193] # Add appropriate metrics based on RISC-V instruction set architecture characteristics

[0194] riscv_custom_metric.set(get_custom_riscv_metric())

[0195] def get_custom_riscv_metric():

[0196] # Implement logic for collecting metrics specific to RISC-V instruction set architecture

[0197] # This might require reading specific system files or using specific tools

[0198] return 0# Replace with actual metric value

[0199] if __name__ == '__main__':

[0200] start_http_server(8000)

[0201] while True:

[0202] collect_metrics()

[0203] time.sleep(15) # collect every 15 seconds

[0204] In some embodiments, the collected resource usage data is stored into Prometheus. Configure Prometheus to scrape these metrics. Add the following to Prometheus' configuration file:

[0205] scrape_configs:

[0206] - job_name: 'cadvisor'

[0207] kubernetes_sd_configs:

[0208] - role: node

[0209] relabel_configs:

[0210] - source_labels: [__meta_kubernetes_node_label_kubernetes_io_hostname]

[0211] target_label: node

[0212] - job_name: 'node-exporter'

[0213] kubernetes_sd_configs:

[0214] - role: node

[0215] relabel_configs:

[0216] - source_labels: [__meta_kubernetes_node_label_kubernetes_io_hostname]

[0217] target_label: node

[0218] - job_name: 'arch-specific-exporter'

[0219] static_configs:

[0220] - targets: ['localhost:8000']

[0221] In some embodiments, the fault detection module 142 detects whether each heterogeneous instruction set architecture node has a fault based on the resource usage data using a machine learning algorithm. In some embodiments, the resource usage data of each heterogeneous instruction set architecture node is detected in real time using a rule engine, and if the resource usage data does not satisfy a preset numerical range, it is determined that the corresponding worker node has a fault. It can be understood that using a rule engine can provide a fast and low-latency fault detection mechanism, which can detect some common and obvious fault patterns, such as high CPU usage, insufficient memory, etc., so as to quickly respond to obvious faults before the machine learning model detects an anomaly. In some embodiments, the implementation code of the fast fault detection logic based on the rule engine is as follows:

[0222] class RuleBasedDetector:

[0223] def __init__(self):

[0224] self.rules = {

[0225] 'high_cpu_usage': lambda m: m['cpu_usage']>90,

[0226] 'low_memory': lambda m: m['memory_available']<100 * 1024 * 1024,#100MB

[0227] 'disk_full': lambda m: m['disk_usage_percent']>95,

[0228] 'high_network_io': lambda m: m['network_io']>1000000000,# 1GB / s

[0229] }

[0230] def detect_issues(self, metrics):

[0231] detected_issues = []

[0232] for rule_name, rule_func in self.rules.items():

[0233] if rule_func(metrics):

[0234] detected_issues.append(rule_name)

[0235] return detected_issues

[0236] def add_rule(self, name, rule_func):

[0237] self.rules[name] = rule_func

[0238] # Usage Example

[0239] detector = RuleBasedDetector()

[0240] In some embodiments, a normal behavior model based on a Long Short-Term Memory (LSTM) network algorithm is used to monitor the resource usage data of each heterogeneous instruction set architecture node in real time. If the resource usage data deviates from the normal behavior baseline, a fault is identified in the heterogeneous instruction set architecture node. It is understood that the normal behavior model based on the LSTM algorithm can learn the behavior patterns of nodes under normal operating conditions, thereby identifying abnormal behavior based on the real-time resource usage data of the heterogeneous instruction set architecture node. When the resource usage data deviates from the normal behavior baseline, the normal behavior model based on the LSTM algorithm can determine that a fault exists in the heterogeneous instruction set architecture node. In some embodiments, the implementation code for building the normal behavior model using the LSTM algorithm is as follows:

[0241] import numpy as np

[0242] from tensorflow.keras.models import Sequential

[0243] from tensorflow.keras.layers import LSTM, Dense

[0244] from sklearn.preprocessing import MinMaxScaler

[0245] class LSTMDetector:

[0246] def __init__(self, lookback=60):

[0247] self.lookback = lookback

[0248] self.model = self._build_model()

[0249] self.scaler = MinMaxScaler(feature_range=(0, 1))

[0250] def _build_model(self):

[0251] model = Sequential()

[0252] model.add(LSTM(50, activation='relu', input_shape=(self.lookback,1)))

[0253] model.add(Dense(1))

[0254] model.compile(optimizer='adam', loss='mse')

[0255] return model

[0256] def train(self, data):

[0257] scaled_data = self.scaler.fit_transform(data.reshape(-1, 1))

[0258] X, y = self._create_dataset(scaled_data)

[0259] self.model.fit(X, y, epochs=100, batch_size=32, verbose=0)

[0260] def detect_anomalies(self, data, threshold=0.1):

[0261] scaled_data = self.scaler.transform(data.reshape(-1, 1))

[0262] X, _ = self._create_dataset(scaled_data)

[0263] predictions = self.model.predict(X)

[0264] mse = np.mean(np.power(X[:, -1] - predictions, 2), axis=1)

[0265] return mse>threshold

[0266] def _create_dataset(self, data):

[0267] X, y = [], []

[0268] for i in range(len(data) - self.lookback):

[0269] X.append(data[i:(i + self.lookback), 0])

[0270] y.append(data[i + self.lookback, 0])

[0271] return np.array(X), np.array(y)

[0272] # Usage example

[0273] detector = LSTMDetector()

[0274] normal_data = np.random.randn(1000) # Assume this is normal data

[0275] detector.train(normal_data)

[0276] new_data = np.random.randn(100) # New data

[0277] anomalies = detector.detect_anomalies(new_data)

[0278] print("Anomalies detected:", anomalies)

[0279] In some embodiments, an anomaly detection model built using features specific to heterogeneous instruction set architectures is used to detect resource usage data for each heterogeneous instruction set architecture node in real time. If anomalies are found in the resource usage data, a fault is identified in the heterogeneous instruction set architecture node. It is understood that different instruction set architectures have different performance characteristics and anomaly patterns. Through targeted training, the model can better understand and identify anomalies associated with a specific instruction set architecture. For example, the RISC-V instruction set architecture may exhibit specific anomaly patterns in branch prediction or CSR access, while the x86 instruction set architecture may have unique anomalies in SIMD instructions or virtualization performance. Training a specific anomaly detection model for each instruction set architecture can improve the accuracy and sensitivity of detection. In some embodiments, the implementation code for the anomaly detection model built using features specific to heterogeneous instruction set architectures is as follows:

[0280] # Add architecture-specific rules

[0281] detector.add_rule('riscv_specific_issue', lambda m: m['riscv_custom_metric']>100)

[0282] # Detection Issues

[0283] metrics = {

[0284] 'cpu_usage': 95,

[0285] 'memory_available': 50 * 1024 * 1024,

[0286] 'disk_usage_percent': 98,

[0287] 'network_io': 500000000,

[0288] 'riscv_custom_metric': 150

[0289] }

[0290] issues = detector.detect_issues(metrics)

[0291] print("Detected issues:", issues)

[0292] In some embodiments, the fault detection module 142 is designed with a scalable plugin mechanism that supports the addition of new fault detection algorithms, thus providing a flexible framework that allows new detection algorithms to be added as needed to address new fault patterns and requirements.

[0293] In some embodiments, the fault cause diagnosis module 143 performs fault cause diagnosis based on a pre-established fault knowledge graph in the presence of faults in the heterogeneous instruction set architecture nodes. The fault knowledge graph includes related container fault patterns under the heterogeneous instruction set architecture (including RISC-V instruction set architecture and X86 instruction set architecture). In some embodiments, based on the related container fault patterns under the heterogeneous instruction set architecture, the fault cause analysis is performed using a causal reasoning engine to perform causal analysis on the analyzed fault causes and locate the root cause of the fault. It can be understood that the fault knowledge graph provides the basic knowledge for fault diagnosis, including related container fault patterns under the heterogeneous instruction set architecture (including RISC-V instruction set architecture and X86 instruction set architecture), supporting accumulation and updating of knowledge. The causal reasoning engine can analyze the fault causes based on the causal reasoning system, can handle complex fault scenarios in the heterogeneous instruction set architecture environment, and accurately locate the root cause of the container-related fault of the heterogeneous instruction set architecture node. In some embodiments, the implementation code for building the fault knowledge graph is as follows:

[0294] from py2neo import Graph, Node, Relationship

[0295] class FaultKnowledgeGraph:

[0296] def __init__(self, uri, user, password):

[0297] self.graph = Graph(uri, auth=(user, password))

[0298] def build_graph(self):

[0299] # Create architecture nodes

[0300] x86 = Node("Architecture", name="x86_64")

[0301] riscv = Node("Architecture", name="riscv64")

[0302] # Create fault type nodes

[0303] cpu_fault = Node("FaultType", name="CPU Fault")

[0304] memory_fault = Node("FaultType", name="Memory Fault")

[0305] disk_fault = Node("FaultType", name="Disk Fault")

[0306] # Create symptom nodes

[0307] high_cpu_usage = Node("Symptom", name="High CPU Usage")

[0308] low_memory = Node("Symptom", name="Low Memory")

[0309] disk_full = Node("Symptom", name="Disk Full")

[0310] # Create relationships

[0311] self.graph.create(Relationship(x86, "HAS_FAULT", cpu_fault))

[0312] self.graph.create(Relationship(x86, "HAS_FAULT", memory_fault))

[0313] self.graph.create(Relationship(riscv, "HAS_FAULT", cpu_fault))

[0314] self.graph.create(Relationship(riscv, "HAS_FAULT", disk_fault))

[0315] self.graph.create(Relationship(cpu_fault, "HAS_SYMPTOM", high_cpu_usage))

[0316] self.graph.create(Relationship(memory_fault, "HAS_SYMPTOM", low_memory))

[0317] self.graph.create(Relationship(disk_fault, "HAS_SYMPTOM", disk_full))

[0318] def get_possible_causes(self, symptom):

[0319] query = """

[0320] MATCH (s:Symptom {name: $symptom})<-[:HAS_SYMPTOM]-(f:FaultType)

[0321] RETURN f.name AS fault_type

[0322] """

[0323] return [record["fault_type"] for record in self.graph.run(query,symptom=symptom)]

[0324] # Usage example

[0325] kg = FaultKnowledgeGraph("bolt: / / localhost:7687", "neo4j", "password")

[0326] kg.build_graph()

[0327] causes = kg.get_possible_causes("High CPU Usage")

[0328] print("Possible causes:", causes)

[0329] In some embodiments, the implementation code for using a causal reasoning engine to perform causal analysis on the analyzed fault causes and locate the root cause of the fault is as follows:

[0330] import networkx as nx

[0331] class CausalInferenceEngine:

[0332] def __init__(self, knowledge_graph):

[0333] self.kg = knowledge_graph

[0334] def diagnose(self, symptoms):

[0335] G = nx.DiGraph()

[0336] # Constructing a cause-effect graph

[0337] for symptom in symptoms:

[0338] G.add_edge('Root Cause', symptom)

[0339] for possible_cause in self.kg.get_possible_causes(symptom):

[0340] G.add_edge(possible_cause, symptom)

[0341] # Use the minimum cut set algorithm to find the most likely root cause.

[0342] min_cut = nx.minimum_cut(G, 'Root Cause', 'Symptoms')

[0343] return list(min_cut[1])[1] # Return the most likely root cause

[0344] # Usage Example

[0345] kg = FaultKnowledgeGraph("bolt: / / localhost:7687", "neo4j", "password")

[0346] engine = CausalInferenceEngine(kg)

[0347] symptoms = ["High CPU Usage", "Low Memory"]

[0348] root_cause = engine.diagnose(symptoms)

[0349] print("Most likely root cause:", root_cause)

[0350] In some embodiments, the fault cause diagnosis module 143 can provide an interactive fault diagnosis interface to support human-machine collaborative diagnosis of fault causes, and can also score the diagnosis results to optimize the fault cause diagnosis algorithm based on the scores and improve the accuracy of the fault cause diagnosis module 143 in diagnosing fault causes.

[0351] In some embodiments, the fault recovery module 144 recovers the container-related faults of the heterogeneous instruction set architecture nodes according to the diagnosis results. In some embodiments, for simple faults, the fault recovery module 144 automatically performs recovery operations on the containers of the heterogeneous instruction set architecture nodes according to the diagnosis results, and the recovery operations include container restart and container expansion. In some embodiments, for complex faults, for example, in the case where the root cause of the fault is that a specific application is incompatible with the RISC-V instruction set architecture (or the X86 instruction set architecture) supported by the worker node A 220 (or the worker node B 220), the fault recovery module 144 performs container migration on the container corresponding to the specific application between the worker node A 220 and the worker node B 220. In some embodiments, for special faults, human intervention can be prompted. In some embodiments, the fault recovery module 144 can use a policy-based automatic recovery engine to recover the container-related faults of the heterogeneous instruction set architecture nodes according to the diagnosis results, can use a gradual recovery mechanism to recover the container-related faults of the heterogeneous instruction set architecture nodes, and can integrate Kubernetes in the fault recovery module 144 to use the self-healing capability of Kubernetes to recover the container-related faults of the heterogeneous instruction set architecture nodes, so as to expand the fault handling support of the fault recovery module 144 for the container-related faults of the heterogeneous instruction set architecture environment.

[0352] In some embodiments, the implementation code for migrating containers corresponding to a specific application between heterogeneous instruction set architecture nodes is as follows:

[0353] from kubernetes import client, config

[0354] class ContainerMigrator:

[0355] def __init__(self):

[0356] config.load_kube_config()

[0357] self.v1 = client.CoreV1Api()

[0358] def migrate_container(self, pod_name, namespace, target_node):

[0359] # Get the original Pod configuration

[0360] pod = self.v1.read_namespaced_pod(name=pod_name, namespace=namespace)

[0361] # Create a new Pod configuration and change the target node

[0362] new_pod = client.V1Pod(

[0363] metadata=client.V1ObjectMeta(name=f"{pod_name}-migrated"),

[0364] spec=client.V1PodSpec(

[0365] containers=pod.spec.containers,

[0366] node_name=target_node ) )

[0369] # Create a new Pod

[0370] self.v1.create_namespaced_pod(namespace=namespace, body=new_pod)

[0371] # Delete the original Pod

[0372] self.v1.delete_namespaced_pod(name=pod_name, namespace=namespace)

[0373] def get_available_nodes(self, arch):

[0374] nodes = self.v1.list_node()

[0375] return [node.metadata.name for node in nodes.items ifnode.metadata.labels.get('kubernetes.io / arch') == arch]

[0376] # Usage example

[0377] migrator = ContainerMigrator()

[0378] available_nodes = migrator.get_available_nodes('riscv64')

[0379] if available_nodes:

[0380] migrator.migrate_container('my-pod', 'default', available_nodes[0])

[0381] In some embodiments, the implementation code for recovering from a container-related failure of a heterogeneous instruction set architecture node using a policy-based automatic recovery engine is:

[0382] class RecoveryEngine:

[0383] def __init__(self):

[0384] self.strategies = {

[0385] 'restart': self.restart_container,

[0386] 'migrate': self.migrate_container,

[0387] 'scale_up': self.scale_up_deployment

[0388] }

[0389] self.migrator = ContainerMigrator()

[0390] config.load_kube_config()

[0391] self.v1 = client.CoreV1Api()

[0392] self.apps_v1 = client.AppsV1Api()

[0393] def execute_recovery(self, fault, context):

[0394] strategy = self.select_strategy(fault, context)

[0395] return self.strategies[strategy](context)

[0396] def select_strategy(self, fault, context):

[0397] if fault == 'CPU Fault':

[0398] return'migrate'

[0399] elif fault == 'Memory Fault':

[0400] return'restart'

[0401] elif fault == 'Disk Fault':

[0402] return'scale_up'

[0403] else:

[0404] return'restart'# Default policy

[0405] def restart_container(self, context):

[0406] pod_name = context['pod_name']

[0407] namespace = context['namespace']

[0408] self.v1.delete_namespaced_pod(name=pod_name, namespace=namespace)

[0409] print(f"Restarted pod {pod_name} in namespace {namespace}")

[0410] def migrate_container(self, context):

[0411] pod_name = context['pod_name']

[0412] namespace = context['namespace']

[0413] target_arch = context['target_arch']

[0414] available_nodes = self.migrator.get_available_nodes(target_arch)

[0415] if available_nodes:

[0416] self.migrator.migrate_container(pod_name, namespace, available_nodes[0])

[0417] print(f"Migrated pod {pod_name} to node {available_nodes[0]}")

[0418] else:

[0419] print(f"No available nodes with architecture {target_arch}")

[0420] def scale_up_deployment(self, context):

[0421] deployment_name = context['deployment_name']

[0422] namespace = context['namespace']

[0423] deployment = self.apps_v1.read_namespaced_deployment(deployment_name,namespace)

[0424] deployment.spec.replicas += 1

[0425] self.apps_v1.patch_namespaced_deployment(

[0426] name=deployment_name,

[0427] namespace=namespace,

[0428] body=deployment )

[0430] [[ID=​​​​​​​​​​​​​​​

[0436] 'target_arch': 'riscv64',

[0437] 'deployment_name': 'my-deployment'

[0438] }

[0439] recovery_engine.execute_recovery('CPU Fault', context)

[0440] Refer again Figure 1 ,like Figure 1 As shown, in some embodiments, the scheduler 130 includes a container scheduling module 131 and a container scheduling optimization module 132. In some embodiments, the container scheduling module 131 uses a scheduling policy to schedule containers for heterogeneous instruction set architecture nodes. When the container scheduling optimization module 132 determines, according to the container fault handling device 140 as described above, that the root cause of a fault is incompatibility between a specific application and the RISC-V instruction set architecture (or x86 instruction set architecture) supported by worker node A 220 (or worker node B 220), it updates the scheduling policy to facilitate the deployment of the specific application to worker node B 220 (or worker node A 220) to avoid similar container faults. It is understood that by updating the scheduling policy according to the root cause of the fault determined by the container fault handling device 140, the container scheduling optimization module 132 can reduce the probability of similar container faults and improve the reliability of container operation.

[0441] In some embodiments, the container scheduling module 131 can use an architecture affinity scheduling algorithm to schedule containers for heterogeneous instruction set architecture nodes, or an active scheduling algorithm based on fault prediction to schedule containers for heterogeneous instruction set architecture nodes, or a dynamic load balancing strategy to schedule containers for heterogeneous instruction set architecture nodes, so as to balance the load of worker node A 220 and worker node B 220.

[0442] In some embodiments, the implementation code for scheduling containers for heterogeneous instruction set architecture nodes using an architecture affinity scheduling algorithm is as follows:

[0443] from kubernetes import client, config

[0444] class ArchitectureAwareScheduler:

[0445] def __init__(self):

[0446] config.load_kube_config()

[0447] self.v1 = client.CoreV1Api()

[0448] def create_pod_with_affinity(self, name, image, preferred_arch):

[0449] affinity = client.V1Affinity(

[0450] node_affinity=client.V1NodeAffinity(

[0451] preferred_during_scheduling_ignored_during_execution=[

[0452] client.V1PreferredSchedulingTerm(

[0453] weight=1,

[0454] preference=client.V1NodeSelectorTerm(

[0455] match_expressions=[

[0456] client.V1NodeSelectorRequirement(

[0457] key="kubernetes.io / arch",

[0458] operator="In",

[0459] values=[preferred_arch] ) ] ) ) ] ) )

[0467] pod = client.V1Pod(

[0468] metadata=client.V1ObjectMeta(name=name),

[0469] spec=client.V1PodSpec(

[0470] containers=[

[0471] client.V1Container(

[0472] name="main",

[0473] image=image )

[0475] ],

[0476] affinity=affinity ) )

[0479] self.v1.create_namespaced_pod(namespace="default", body=pod)

[0480] print(f"Created pod {name} with preferred architecture {preferred_arch}")

[0481] # Usage Example

[0482] scheduler = ArchitectureAwareScheduler()

[0483] scheduler.create_pod_with_affinity("my-riscv-pod", "my-image:latest","riscv64")

[0484] In some embodiments, the implementation code for container scheduling on heterogeneous instruction set architecture nodes using the fault prediction-based proactive scheduling algorithm is as follows:

[0485] import numpy as np

[0486] from sklearn.ensemble import RandomForestClassifier

[0487] from kubernetes import client, config

[0488] class PredictiveScheduler:

[0489] def __init__(self):

[0490] self.model = RandomForestClassifier()

[0491] config.load_kube_config()

[0492] self.v1 = client.CoreV1Api()

[0493] def train(self, historical_data):

[0494] X = historical_data[['cpu_usage', 'memory_usage', 'disk_usage']]

[0495] y = historical_data['fault_occurred']

[0496] self.model.fit(X, y)

[0497] def predict_fault_probability(self, node_metrics):

[0498] return self.model.predict_proba(node_metrics.reshape(1, -1))[0][1]

[0499] def get_node_metrics(self, node_name):

[0500] # This section should implement the logic for retrieving node metrics from the monitoring system.

[0501] # For example, we return some random values

[0502] return np.random.rand(3)

[0503] def schedule_pod(self, pod_name, image):

[0504] nodes = self.v1.list_node().items

[0505] node_probabilities = []

[0506] for node in nodes:

[0507] metrics = self.get_node_metrics(node.metadata.name)

[0508] prob = self.predict_fault_probability(metrics)

[0509] node_probabilities.append((node.metadata.name, prob))

[0510] # Select the node with the lowest fault probability

[0511] best_node = min(node_probabilities, key=lambda x: x[1])

[0512] pod = client.V1Pod(

[0513] metadata=client.V1ObjectMeta(name=pod_name),

[0514] spec=client.V1PodSpec(

[0515] containers=

[0516] client.V1Container(

[0517] name="main",[[ID=3​​​​​​​​​​​​​​​​​​​print(f"Scheduled pod {pod_name} on node {best_node[0]} with faultprobability {best_node[1]}")

[0526] # Using the example

[0527] scheduler = PredictiveScheduler()

[0528] # Let's assume we have some historical data

[0529] historical_data = np.random.rand(1000, 4)

[0530] historical_data = pd.DataFrame(historical_data, columns=['cpu_usage','memory_usage', 'disk_usage', 'fault_occurred'])

[0531] scheduler.train(historical_data)

[0532] scheduler.schedule_pod("my-predictive-pod", "my-image:latest")

[0533] In some embodiments, the implementation code for container scheduling for heterogeneous instruction set architecture nodes based on dynamic load balancing strategies is:

[0534] from kubernetes import client, config

[0535] import numpy as np

[0536] class DynamicLoadBalancer:

[0537] def __init__(self):

[0538] config.load_kube_config()

[0539] self.v1 = client.CoreV1Api()

[0540] self.apps_v1 = client.AppsV1Api()

[0541] def get_node_load(self, node_name):

[0542] # Implement logic to get node load from monitoring system here

[0543] # For the sake of example, we return a random value

[0544] return np.random.rand()

[0545] def balance_load(self):

[0546] nodes = self.v1.list_node().items

[0547] node_loads = [(node.metadata.name, self.get_node_load(node.metadata.name)) for node in nodes]

[0548] # Calculate average load

[0549] avg_load = np.mean([load for _, load in node_loads])

[0550] # Find out nodes with too high and too low load

[0551] high_load_nodes = [name for name, load in node_loads if load>avg_load* 1.2]

[0552] low_load_nodes = [name for name, load in node_loads if load<avg_load* 0.8]

[0553] # Migrate Pods from high load nodes to low load nodes

[0554] for high_node in high_load_nodes:

[0555] if not low_load_nodes:

[0556] break

[0557] pods = self.v1.list_namespaced_pod(namespace="default", field_selector=f"spec.nodeName={high_node}").items

[0558] for pod in pods:

[0559] if pod.metadata.owner_references[0].kind != "DaemonSet":# Do not migrate Pods of DaemonSet

[0560] target_node = low_load_nodes.pop(0)

[0561] self.migrate_pod(pod.metadata.name, pod.metadata.namespace, target_node)

[0562] if not low_load_nodes:

[0563] break

[0564] def migrate_pod(self, pod_name, namespace, target_node):

[0565] pod = self.v1.read_namespaced_pod(name=pod_name, namespace=namespace)

[0566] new_pod = client.V1Pod(

[0567] metadata=client.V1ObjectMeta(name=f"{pod_name}-migrated", namespace=namespace),

[0568] spec=client.V1PodSpec(

[0569] containers=pod.spec.containers,

[0570] node_name=target_node ) )

[0573] self.v1.create_namespaced_pod(namespace=namespace, body=new_pod)

[0574] self.v1.delete_namespaced_pod(name=pod_name, namespace=namespace)

[0575] print(f"Migrated pod {pod_name} from {pod.spec.node_name} to {target_node}")

[0576] # Usage example

[0577] balancer = DynamicLoadBalancer()

[0578] balancer.balance_load()

[0579] Figure 3 A flowchart of a container fault processing method according to one embodiment of the present disclosure is shown. As shown in the figure, the container fault processing method of the present embodiment can include: Figure 3

[0580] In step S310, resource usage data of a heterogeneous instruction set architecture node is collected, the heterogeneous instruction set architecture node being provided with a container, the resource usage data including host level and container level resource usage data, and performance indicators related to the heterogeneous instruction set architecture.

[0581] In step S320, based on the resource usage data, a machine learning algorithm is used to detect whether the heterogeneous instruction set architecture node has a fault.

[0582] In step S330, in the case where the heterogeneous instruction set architecture node has a fault, fault cause diagnosis is performed according to a pre-established fault knowledge graph, the fault knowledge graph including related container fault patterns under the heterogeneous instruction set architecture.

[0583] In step S340, according to the diagnosis result, a container-related fault of the heterogeneous instruction set architecture node is recovered.

[0584] ​Since the process of diagnosing and recovering the container failure in the heterogeneous instruction set architecture environment using the container failure processing method of the embodiments of the present disclosure has been described in detail in the device embodiments above, no further description is given here.

[0585] Figure 4 A flowchart of a container scheduling method according to one embodiment of the present disclosure is shown. As shown in Figure 4 the container scheduling method of the embodiments of the present disclosure can include:

[0586] In step S410, a container scheduling strategy is used to schedule containers for the heterogeneous instruction set architecture nodes, which include the first nodes supporting the first instruction set architecture and the second nodes supporting the second instruction set architecture.

[0587] In step S420, in the case where the fault root cause determined according to the container failure processing method as claimed in any one of claims 1 to 7 is that a specific application is incompatible with the first instruction set architecture supported by the first nodes, the scheduling strategy is updated so as to deploy the specific application to the second nodes.

[0588] Since the process of scheduling containers for the heterogeneous instruction set architecture environment using the container scheduling method of the embodiments of the present disclosure has been described in detail in the device embodiments above, no further description is given here.

[0589] Figure 5 A schematic diagram of a container failure processing device according to one embodiment of the present disclosure is shown. As shown in Figure 5 the container failure processing device 500 includes a data acquisition module 510, a fault detection module 520, a fault cause diagnosis module 530, and a fault recovery module 540.

[0590] The data acquisition module 510 is configured to acquire resource usage data of the heterogeneous instruction set architecture nodes on which containers are arranged, the resource usage data including host-level and container-level resource usage data, and performance indicators related to the heterogeneous instruction set architecture.

[0591] The fault detection module 520 is configured to detect whether the heterogeneous instruction set architecture nodes have a fault based on the resource usage data using a machine learning algorithm.

[0592] The fault cause diagnosis module 530 is configured to, in the case where the heterogeneous instruction set architecture nodes have a fault, diagnose the fault cause according to a pre-established fault knowledge graph, the fault knowledge graph including related container failure modes under the heterogeneous instruction set architecture.

[0593] The fault recovery module 540 is configured to recover the container-related fault of the heterogeneous instruction set architecture node according to the diagnosis result.

[0594] Since the process of diagnosing and recovering the container fault in the heterogeneous instruction set architecture environment using the container fault processing method of the embodiments of the present disclosure has been described in detail in the device embodiments above, no further description is given here.

[0595] Figure 6 A schematic diagram of a scheduler according to an embodiment of the present disclosure is shown. As shown in the figure, the scheduler 600 includes a container scheduling module 610 and a container scheduling optimization module 620. Figure 6

[0596] The container scheduling module 610 is configured to perform container scheduling for a heterogeneous instruction set architecture node using a scheduling strategy, wherein the heterogeneous instruction set architecture node includes a first node supporting a first instruction set architecture and a second node supporting a second instruction set architecture.

[0597] The container scheduling optimization module 620 is configured to update the scheduling strategy so as to deploy a specific application to the second node in a case where the fault root cause determined according to the container fault processing method of any one of claims 1 to 7 is that the specific application is incompatible with the first instruction set architecture supported by the first node.

[0598] Since the process of performing container scheduling for a container in a heterogeneous instruction set architecture environment using the container scheduling method of the embodiments of the present disclosure has been described in detail in the device embodiments above, no further description is given here.

[0599] The embodiments of the present disclosure further provide an electronic device, as shown in the figure, which includes a memory 720, a processor 710, and a program stored in the memory 720 and executable on the processor 710. Figure 7 The program, when executed by the processor 710, can implement the processes of each of the embodiments of the container fault processing method and / or the container scheduling method described above and achieve the same technical effects. To avoid repetition, no further description is given here.

[0600] Those skilled in the art can understand that all or part of the steps of the various methods of the above embodiments can be completed by instructions or by controlling relevant hardware through instructions, which can be stored in a computer-readable storage medium and loaded and executed by a processor. For this purpose, the embodiments of the present disclosure further provide a storage medium having a computer program or instructions stored thereon, which, when executed by a processor, can implement the processes of each of the embodiments of the container fault processing method and / or the container scheduling method described above.

[0601] ​Since the instructions stored in the storage medium can execute the steps in the container failure processing method and / or the container scheduling method provided by the embodiments of the present disclosure, the beneficial effects that can be achieved by the container failure processing method and / or the container scheduling method provided by the embodiments of the present disclosure can be achieved. Details are described in the foregoing embodiments, which will not be repeated here. The specific implementation of each operation can refer to the foregoing embodiments, which will not be repeated here.

[0602] In summary, the container failure processing method provided by the present disclosure detects the container-related failure of the heterogeneous instruction set architecture node according to the resource usage data of the host level and the container level, and the performance indicators related to the heterogeneous instruction set architecture, and performs failure cause diagnosis according to the failure knowledge graph including the container failure mode related to the heterogeneous instruction set architecture, locates the root cause of the container-related failure of the heterogeneous instruction set architecture node, and recovers the container-related failure of the heterogeneous instruction set architecture node according to the diagnosis result. In this way, the performance indicators related to the heterogeneous instruction set architecture and the container failure mode related to the heterogeneous instruction set architecture both consider the instruction set differences between different instruction set architecture nodes, can deeply understand the characteristics of different instruction set architectures, and can realize the diagnosis of cross-instruction set architecture failures, accurate positioning and solution of container failures in a heterogeneous instruction set architecture environment.

[0603] Finally, it should be noted that: obviously, the above embodiments are only examples for clearly illustrating the present disclosure, and are not limitations on the embodiments. Based on the above description, those skilled in the art can make other different forms of changes or modifications. Here, it is not necessary and impossible to enumerate all the embodiments. The obvious changes or modifications derived therefrom are still within the protection scope of the present disclosure.

Claims

1. A container failure handling method, comprising: Collect resource usage data of heterogeneous instruction set architecture nodes, on which containers are set up. The resource usage data includes host-level and container-level resource usage data, as well as performance indicators related to the heterogeneous instruction set architecture. Based on the resource usage data, machine learning algorithms are used to detect in real time whether there are faults in the heterogeneous instruction set architecture nodes. The machine learning algorithms include a rule engine, a long short-term memory network algorithm, and an anomaly detection model. In the event of a fault in the heterogeneous instruction set architecture node, the cause of the fault is diagnosed based on a pre-established fault knowledge graph, which includes relevant container fault modes under the heterogeneous instruction set architecture. Based on the diagnostic results, the container-related faults of the heterogeneous instruction set architecture node are recovered. The heterogeneous instruction set architecture node includes a first node supporting a first instruction set architecture and a second node supporting a second instruction set architecture. When the root cause of the fault is that a specific application is incompatible with the first instruction set architecture supported by the first node, the container corresponding to the specific application is migrated in real time between the first node and the second node.

2. The container failure handling method according to claim 1, wherein, The step of using machine learning algorithms to detect whether there are faults in the heterogeneous instruction set architecture nodes based on the resource usage data includes: The rule engine is used to monitor the resource usage data of the heterogeneous instruction set architecture node in real time. If the resource usage data does not meet the preset value range, it is determined that the heterogeneous instruction set architecture node is faulty.

3. The container failure handling method according to claim 1, wherein, The step of using machine learning algorithms to detect whether there are faults in the heterogeneous instruction set architecture nodes based on the resource usage data includes: A normal behavior model based on a long short-term memory network algorithm is used to detect the resource usage data of the heterogeneous instruction set architecture node in real time. If the resource usage data deviates from the normal behavior benchmark, it is determined that the heterogeneous instruction set architecture node is faulty.

4. The container failure handling method according to claim 1, wherein, The step of using machine learning algorithms to detect whether there are faults in the heterogeneous instruction set architecture nodes based on the resource usage data includes: An anomaly detection model is established using the characteristics of heterogeneous instruction set architecture to detect the resource usage data of the heterogeneous instruction set architecture nodes in real time. If anomalies are found in the resource usage data, it is determined that the heterogeneous instruction set architecture node is faulty.

5. The container failure handling method according to claim 1, wherein, In the event of a fault in the heterogeneous instruction set architecture node, the fault cause diagnosis is performed based on a pre-established fault knowledge graph, including: Fault cause analysis is performed based on the relevant container fault modes under the aforementioned heterogeneous instruction set architecture. Using a causal reasoning engine, we perform causal analysis on the identified fault causes to pinpoint the root cause of the fault.

6. The container failure handling method according to claim 5, wherein, The step of recovering from container-related faults on the heterogeneous instruction set architecture node based on the diagnostic results also includes: Based on the diagnostic results, the containers of the heterogeneous instruction set architecture node are automatically restored. The restoration operation includes container restart and container expansion.

7. A container scheduling method, comprising: A scheduling strategy is used to schedule containers for nodes with heterogeneous instruction set architectures, wherein the nodes with heterogeneous instruction set architectures include a first node supporting a first instruction set architecture and a second node supporting a second instruction set architecture. If the root cause of the failure determined by the container failure handling method according to any one of claims 1 to 6 is that a specific application is incompatible with the first instruction set architecture supported by the first node, the scheduling policy is updated to facilitate the deployment of the specific application to the second node.

8. A container failure handling device, comprising: The data acquisition module is used to collect resource usage data of heterogeneous instruction set architecture nodes. The heterogeneous instruction set architecture nodes are equipped with containers. The resource usage data includes host-level and container-level resource usage data, as well as performance indicators related to the heterogeneous instruction set architecture. The fault detection module is used to detect whether there are faults in the heterogeneous instruction set architecture nodes in real time based on the resource usage data and using machine learning algorithms. The machine learning algorithms include a rule engine, a long short-term memory network algorithm, and an anomaly detection model. The fault cause diagnosis module is used to diagnose the fault cause based on a pre-established fault knowledge graph when a fault exists in the heterogeneous instruction set architecture node. The fault knowledge graph includes relevant container fault modes under the heterogeneous instruction set architecture. The fault recovery module is used to recover container-related faults of the heterogeneous instruction set architecture node based on the diagnostic results. The heterogeneous instruction set architecture node includes a first node supporting a first instruction set architecture and a second node supporting a second instruction set architecture. When the root cause of the fault is that a specific application is incompatible with the first instruction set architecture supported by the first node, the module performs real-time container migration of the container corresponding to the specific application between the first node and the second node.

9. The container failure handling device according to claim 8, wherein, The fault detection module is also used for: The rule engine is used to monitor the resource usage data of the heterogeneous instruction set architecture node in real time. If the resource usage data does not meet the preset value range, it is determined that the heterogeneous instruction set architecture node is faulty.

10. The container failure handling apparatus according to claim 8, wherein, The fault detection module is also used for: A normal behavior model based on a long short-term memory network algorithm is used to detect the resource usage data of the heterogeneous instruction set architecture node in real time. If the resource usage data deviates from the normal behavior benchmark, it is determined that the heterogeneous instruction set architecture node is faulty.

11. The container failure handling apparatus according to claim 8, wherein, The fault detection module is also used for: An anomaly detection model is established using the characteristics of heterogeneous instruction set architecture to detect the resource usage data of the heterogeneous instruction set architecture nodes in real time. If anomalies are found in the resource usage data, it is determined that the heterogeneous instruction set architecture node is faulty.

12. The container failure handling apparatus according to claim 8, wherein, The fault cause diagnosis module is also used for: Fault cause analysis is performed based on the relevant container fault modes under the aforementioned heterogeneous instruction set architecture. Using a causal reasoning engine, we perform causal analysis on the identified fault causes to pinpoint the root cause of the fault.

13. The container failure handling apparatus according to claim 12, wherein, The fault recovery module is also used for: Based on the diagnostic results, the containers of the heterogeneous instruction set architecture node are automatically restored. The restoration operation includes container restart and container expansion.

14. A scheduler, comprising: The container scheduling module is used to schedule containers for heterogeneous instruction set architecture nodes using a scheduling strategy. The heterogeneous instruction set architecture nodes include a first node supporting a first instruction set architecture and a second node supporting a second instruction set architecture. The container scheduling optimization module is used to update the scheduling policy so as to deploy the specific application to the second node when the root cause of the failure determined by the container failure handling method according to any one of claims 1 to 6 is that the specific application is incompatible with the first instruction set architecture supported by the first node.

15. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, performs the method as described in any one of claims 1 to 7.

16. A storage medium storing a computer program or instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1 to 7.

Images