Container fault processing method, container scheduling method, related device and medium

Abstract

The invention provides a container fault processing method, a container scheduling method, a related device and a medium. The container fault processing method comprises the steps that resource use condition data of a heterogeneous instruction set architecture node is collected, a container is arranged on the heterogeneous instruction set architecture node, and the resource use condition data comprises host-level and container-level resource use condition data and performance indexes related to a heterogeneous instruction set architecture; based on the resource use condition data, using a machine learning algorithm to detect whether the heterogeneous instruction set architecture node has a fault; under the condition that the nodes of the heterogeneous instruction set architecture have faults, fault reason diagnosis is carried out according to a pre-established fault knowledge graph, and the fault knowledge graph comprises related container fault modes under the heterogeneous instruction set architecture; and according to the diagnosis result, recovering the faults related to the container of the heterogeneous instruction set architecture node so as to accurately position and solve the container faults in the heterogeneous instruction set architecture environment.

Description

Technical Field

[0001] The present disclosure belongs to the field of virtualization technology, and specifically relates to a container failure handling method, a container scheduling method, and related devices and media. Background Art

[0002] With the rapid development of cloud computing and edge computing technologies, the instruction set architecture of nodes in data centers is gradually developing towards heterogeneity. Heterogeneous instruction set architectures can provide higher performance and lower energy consumption to meet the needs of data centers for diversified workloads. Among them, the RISC-V instruction set architecture, as an open source instruction set architecture, has been widely used in servers and edge devices due to its low power consumption and high performance. The X86 instruction set architecture occupies a dominant position in data centers due to its mature ecosystem and extensive software support. In a heterogeneous instruction set architecture environment, container technology is widely used in data centers due to its lightweight and high efficiency. Container technology provides an isolated environment to run applications, enabling rapid deployment, expansion and management of applications. However, when running containers on heterogeneous instruction set architecture nodes, some architecture-specific container failures may occur due to the instruction set differences between heterogeneous instruction set architecture nodes. The current container fault diagnosis and recovery solutions are mainly designed for a single instruction set architecture and cannot effectively handle container failures in a heterogeneous instruction set architecture environment. Therefore, it is urgent to provide a container fault handling method to accurately locate and solve container failures in a heterogeneous instruction set architecture environment. Summary of the invention

[0003] In view of the above problems, the present disclosure provides a container failure handling method, a container scheduling method, related devices and media, aiming to accurately locate and solve container failures in a heterogeneous instruction set architecture environment.

[0004] According to a first aspect of the present disclosure, a method for handling a container failure is provided, comprising:

[0005] Collecting resource usage data of a heterogeneous instruction set architecture node, where a container is set on the heterogeneous instruction set architecture node, the resource usage data including resource usage data at the host level and the container level, and performance indicators related to the heterogeneous instruction set architecture;

[0006] Based on the resource usage data, using a machine learning algorithm to detect whether the heterogeneous instruction set architecture node has a fault;

[0007] In the case where the heterogeneous instruction set architecture node has a fault, diagnosing the cause of the fault according to a pre-established fault knowledge graph, wherein the fault knowledge graph includes relevant container fault modes under the heterogeneous instruction set architecture;

[0008] Based on the diagnosis result, the container-related faults of the heterogeneous instruction set architecture node are recovered.

[0009] Optionally, the detecting whether the heterogeneous instruction set architecture node has a fault by using a machine learning algorithm based on the resource usage data includes:

[0010] A rule engine is used to detect the resource usage data of the heterogeneous instruction set architecture node in real time. When the resource usage data does not meet a preset value range, it is determined that the heterogeneous instruction set architecture node has a fault.

[0011] Optionally, the detecting whether the heterogeneous instruction set architecture node has a fault by using a machine learning algorithm based on the resource usage data includes:

[0012] Using a normal behavior model established based on a long short-term memory network algorithm, the resource usage data of the heterogeneous instruction set architecture node is detected in real time. When the resource usage data deviates from a normal behavior benchmark, it is determined that the heterogeneous instruction set architecture node has a fault.

[0013] Optionally, the detecting whether the heterogeneous instruction set architecture node has a fault by using a machine learning algorithm based on the resource usage data includes:

[0014] Using an anomaly detection model established according to the characteristics of a heterogeneous instruction set architecture, the resource usage data of the heterogeneous instruction set architecture node is detected in real time. If an anomaly is found in the resource usage data, it is determined that the heterogeneous instruction set architecture node has a fault.

[0015] Optionally, when a fault occurs in the heterogeneous instruction set architecture node, diagnosing the cause of the fault according to a pre-established fault knowledge graph includes:

[0016] Performing failure cause analysis based on relevant container failure modes under the heterogeneous instruction set architecture;

[0017] Use the causal reasoning engine to perform causal analysis on the analyzed fault causes and locate the root cause of the fault.

[0018] Optionally, the heterogeneous instruction set architecture node includes a first node supporting a first instruction set architecture and a second node supporting a second instruction set architecture, and the recovering the container-related fault of the heterogeneous instruction set architecture node according to the diagnosis result includes:

[0019] When the root cause of the failure is that the specific application is incompatible with the first instruction set architecture supported by the first node, a container corresponding to the specific application is migrated between the first node and the second node.

[0020] Optionally, the recovering the container-related fault of the heterogeneous instruction set architecture node according to the diagnosis result further includes:

[0021] According to the diagnosis result, a recovery operation is automatically performed on the container of the heterogeneous instruction set architecture node, and the recovery operation includes container restart and container expansion.

[0022] According to a second aspect of the present disclosure, a container scheduling method is provided, including:

[0023] Performing container scheduling for heterogeneous instruction set architecture nodes using a scheduling strategy, wherein the heterogeneous instruction set architecture nodes include a first node supporting a first instruction set architecture and a second node supporting a second instruction set architecture;

[0024] When the root cause of the fault determined according to the container fault handling method as described in any of the above items is that a specific application is incompatible with a first instruction set architecture supported by the first node, the scheduling policy is updated to facilitate deployment of the specific application to the second node.

[0025] According to a third aspect of the present disclosure, a container fault processing device is provided, comprising:

[0026] A data collection module is used to collect resource usage data of a heterogeneous instruction set architecture node, where a container is provided on the heterogeneous instruction set architecture node, and the resource usage data includes resource usage data at the host level and the container level, and performance indicators related to the heterogeneous instruction set architecture;

[0027] A fault detection module, configured to detect whether the heterogeneous instruction set architecture node has a fault using a machine learning algorithm based on the resource usage data;

[0028] A fault cause diagnosis module, used to diagnose the fault cause according to a pre-established fault knowledge graph when a fault occurs in the heterogeneous instruction set architecture node, wherein the fault knowledge graph includes relevant container failure modes under the heterogeneous instruction set architecture;

[0029] A fault recovery module is used to recover container-related faults of the heterogeneous instruction set architecture node based on the diagnosis result.

[0030] Optionally, the fault detection module is further used to:

[0031] A rule engine is used to detect the resource usage data of the heterogeneous instruction set architecture node in real time. When the resource usage data does not meet a preset value range, it is determined that the heterogeneous instruction set architecture node has a fault.

[0032] Optionally, the fault detection module is further used to:

[0033] Using a normal behavior model established based on a long short-term memory network algorithm, the resource usage data of the heterogeneous instruction set architecture node is detected in real time. When the resource usage data deviates from a normal behavior benchmark, it is determined that the heterogeneous instruction set architecture node has a fault.

[0034] Optionally, the fault detection module is further used to:

[0035] Using an anomaly detection model established according to the characteristics of a heterogeneous instruction set architecture, the resource usage data of the heterogeneous instruction set architecture node is detected in real time. If an anomaly is found in the resource usage data, it is determined that the heterogeneous instruction set architecture node has a fault.

[0036] Optionally, the fault cause diagnosis module is further used for:

[0037] Performing failure cause analysis based on relevant container failure modes under the heterogeneous instruction set architecture;

[0038] Use the causal reasoning engine to perform causal analysis on the analyzed fault causes and locate the root cause of the fault.

[0039] Optionally, the heterogeneous instruction set architecture node includes a first node supporting a first instruction set architecture and a second node supporting a second instruction set architecture, and the fault recovery module is further used to:

[0040] When the root cause of the failure is that the specific application is incompatible with the first instruction set architecture supported by the first node, a container corresponding to the specific application is migrated between the first node and the second node.

[0041] Optionally, the fault recovery module is further used to:

[0042] According to the diagnosis result, a recovery operation is automatically performed on the container of the heterogeneous instruction set architecture node, and the recovery operation includes container restart and container expansion.

[0043] According to a fourth aspect of the present disclosure, a scheduler is provided, including:

[0044] A container scheduling module, configured to perform container scheduling for a heterogeneous instruction set architecture node using a scheduling strategy, wherein the heterogeneous instruction set architecture node includes a first node supporting a first instruction set architecture and a second node supporting a second instruction set architecture;

[0045] A container scheduling optimization module is used to update the scheduling policy so as to deploy the specific application to the second node when the root cause of the fault determined according to the container fault handling method as described in any of the above items is that the specific application is incompatible with the first instruction set architecture supported by the first node.

[0046] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein the program can implement the method described above when executed by the processor.

[0047] According to a sixth aspect of the present disclosure, a storage medium is provided, on which a computer program or instruction is stored, and when the computer program or instruction is executed by a processor, the steps of any of the methods described above are implemented.

[0048] The present disclosure brings the following beneficial effects:

[0049] The container fault handling method provided by the present invention detects container-related faults of heterogeneous instruction set architecture nodes based on resource usage data such as host-level and container-level resource usage data and performance indicators related to heterogeneous instruction set architectures, and diagnoses the cause of the fault based on a fault knowledge graph including relevant container fault modes under the heterogeneous instruction set architecture, locates the root cause of the container-related faults of the heterogeneous instruction set architecture nodes, and recovers the container-related faults of the heterogeneous instruction set architecture nodes based on the diagnosis results. In this way, the performance indicators related to the heterogeneous instruction set architecture and the relevant container fault modes under the heterogeneous instruction set architecture take into account the instruction set differences between nodes of different instruction set architectures, can deeply understand the characteristics of different instruction set architectures, can realize the diagnosis of faults across instruction set architectures, and can accurately locate and resolve container faults in a heterogeneous instruction set architecture environment.

[0050] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or understood by practicing the present disclosure. The purpose and other advantages of the present disclosure are realized and obtained by the structures particularly pointed out in the description and the drawings.

[0051] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0053] Figure 1 is a structural diagram of a data center to which an embodiment of the present disclosure is applied;

[0054] Figure 2 is an internal structure diagram of a working node according to an embodiment of the present disclosure;

[0055] Figure 3 A schematic diagram showing a flow chart of a method for handling a container failure according to an embodiment of the present disclosure is shown;

[0056] Figure 4 A schematic diagram showing a flow chart of a container scheduling method provided according to an embodiment of the present disclosure is shown;

[0057] Figure 5 A schematic diagram showing the structure of a container fault processing device provided according to an embodiment of the present disclosure is shown;

[0058] Figure 6 A schematic diagram showing the structure of a scheduler provided according to an embodiment of the present disclosure is shown;

[0059] Figure 7 A schematic structural diagram of an electronic device provided according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0060] Various embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. In each of the accompanying drawings, the same elements are represented by the same or similar reference numerals. For the sake of clarity, the various parts in the accompanying drawings are not drawn to scale.

[0061] The following terms are used in this article:

[0062] Container: As a lightweight virtualization technology, it is a group of processes that are resource-constrained and isolated from each other. Container technology creates independent operating environments for different applications, implements resource isolation, configuration and security, and can meet the resource requirements of applications on demand and ensure the isolation and availability of applications. The application hosted in the container is called a container instance, or it can also be called a containerized application. In order to meet the needs of large-scale applications, in practice, it is often necessary to deploy many containers in a computer cluster for unified management and provide services to the outside world, so a container orchestration tool is needed. Container orchestration tools use container services and orchestrate them to determine how containers interact with each other, extending lifecycle management capabilities to complex, multi-container workloads deployed on a large number of computer clusters, and providing an abstraction layer for developers and infrastructure teams to handle large-scale containerized deployments. Container orchestration tools, such as K8s (full name is Kubernetes, a system for running and coordinating containerized application processes) system, are open source systems for automatically deploying, scaling and managing containerized applications. Docker is an open source application container engine and a tool for starting and stopping containers. It allows developers to package their applications and dependency packages into a portable container and then publish them to any popular machine. It can also achieve virtualization. The containers completely use the sandbox mechanism and there are no interfaces between them.

[0063] Pod: It is the basic operation unit of container orchestration tools (for example, K8s) and the smallest deployment unit that can be created, debugged, and managed. All containers in the same container group share the same IP address, IPC, host name, and other resources. The container group abstracts the network and storage from the underlying container, making it easier to move containers around the cluster. Each container group can encapsulate one or more containers for hosting applications. The containers in the container group are scheduled as a whole to run on a working node.

[0064] Figure 1 FIG. 1 is a structural diagram of a data center used in an embodiment of the present disclosure. Figure 1 For the convenience of description, Figure 1 Only the control node 110 and the working node 120 are shown in FIG. In implementation, the control node 110 or the working node 120 may be a server or a virtual machine on the server. Figure 1Only a limited number of worker nodes 120 are shown, but the present disclosure should not be limited thereto. In some embodiments, the worker nodes 120 are heterogeneous instruction set architecture nodes, including worker nodes A 220 and worker nodes B 220, wherein worker nodes A 220 support RISC-V instruction set architecture, and worker nodes B 220 support X86 instruction set architecture. It is understandable that the worker nodes 120 may also support other instruction set architectures such as ARM instruction set architecture. For the convenience of description, Figure 1 Not shown.

[0065] In some embodiments, the data center can be applied to various application scenarios such as content delivery network (CDN), e-commerce, games, audio and video, Internet of Things, logistics, industrial brain, city brain, etc., and provide computing services to end users in various scenarios. Specifically, for each application scenario, an application that can provide computing services in the application scenario can be deployed in the work node 120 of the data center. Considering that a large number of applications may need to be deployed in the data center, container technology can be used to build a container group on the work node 120, carry the application program through the container, and then deploy the application program in units of containers. By running these container instances deployed on the work node 120, corresponding computing services can be provided to end users. It should be noted that for the work node A 220, the container deployed thereon supports the RISC-V instruction set architecture. For the work node B 220, the container deployed thereon supports the X86 instruction set architecture.

[0066] In some embodiments, a container orchestration tool (e.g., K8s) may be run on the control node 110 to implement orchestration management of container instances in the data center to which it belongs, wherein the orchestration management of container instances includes at least one of the following: creation, elastic scaling, rolling update, reconstruction, migration, shutdown, etc. of container instances. In some embodiments, in addition to the container orchestration management function, the control node 110 may also be responsible for other management operations in the data center to which it belongs, such as monitoring and management of operation and maintenance, logs, network status, etc. A command may be sent to each working node 120 through the control node 110. In simple terms, the control node 110 is a manager and the working node 120 is a managed. The background services running on the control node 110 may generally include an application programming interface server (API server) 111 and a scheduler 130. The application programming interface server 111 is the front-end interface of the container orchestration tool (e.g., K8s), and various client tools and other components of the container orchestration tool (e.g., K8s) can manage various resources of the cluster of the data center through the application programming interface server 111. The scheduler 130 may determine on which worker node to place the container group to run and determine the resources allocated to the container group.

[0067] In some embodiments, since the working node 120 is the real processing device of the data center, Figure 2 The internal structure diagram of a working node of an embodiment of the present disclosure is shown. In some embodiments, the working node 120 may include multiple processors 22 and a memory 29. The memory 29 in the working node 120 may be a main memory (abbreviated as main memory or internal memory), which is used to store instruction information and / or data information represented by data signals, such as storing data provided by the processor 22 (such as calculation results), and may also be used to implement data exchange between the processor 22 and an external storage device 26 (or referred to as auxiliary memory or external memory).

[0068] In some cases, the processor 22 may need to access the memory 29 to obtain data in the memory 29 or modify the data in the memory 29. Since the access speed of the memory 29 is slow, in order to alleviate the speed gap between the processor 22 and the memory 29, the working node 120 also includes a cache memory 28 coupled to the bus 21, and the cache memory 28 is used to cache some program data or message data in the memory 29 that may be repeatedly called. The cache memory 28 is implemented by a storage device of the type of static random access memory (SRAM). The cache memory 28 can be a multi-level structure, such as a three-level cache structure with a first-level cache (L1Cache), a second-level cache (L2Cache) and a third-level cache (L3Cache), or a cache structure of more than three levels or other types of cache structures. In some embodiments, a portion of the cache memory 28 (such as a first-level cache, or a first-level cache and a second-level cache) can be integrated inside the processor 22 or integrated with the processor 22 in the same system on a chip.

[0069] Based on this, the processor 22 may include an instruction execution unit 221, a memory management unit 222, and other parts. The instruction execution unit 221 initiates a write access request when executing some instructions that need to modify the memory. The write access request specifies the write data and the corresponding physical address that need to be written into the memory; the memory management unit 222 is used to translate the virtual addresses specified by these instructions into the physical addresses mapped by the virtual addresses. The physical address specified by the write access request may be consistent with the physical address specified by the corresponding instruction.

[0070] The information exchange between the memory 29 and the cache memory 28 is usually organized in blocks. In some embodiments, the cache memory 28 and the memory 29 can be divided into data blocks according to the same spatial size, and the data block can be used as the minimum unit of data exchange between the cache memory 28 and the memory 29 (including one or more data of a preset length). For the sake of simplicity and clarity, the data blocks in the cache memory 28 are referred to as cache blocks (which can be called cache lines or cache lines), and different cache blocks have different cache block addresses; the data blocks in the memory 29 are referred to as memory blocks, and different memory blocks have different memory block addresses. The cache block address includes, for example, a physical address tag for locating the data block.

[0071] Due to space and resource limitations, the cache memory 28 cannot cache all the contents in the memory 29, that is, the storage capacity of the cache memory 28 is usually smaller than that of the memory 29, and the cache block addresses provided by the cache memory 28 cannot correspond to all the memory block addresses provided by the memory 29. When the processor 22 needs to access the memory, it first accesses the cache memory 28 via the bus 21 to determine whether the content to be accessed has been stored in the cache memory 28. If so, the cache memory 28 hits, and the processor 22 directly calls the content to be accessed from the cache memory 28; if the content that the processor 22 needs to access is not in the cache memory 28, the processor 22 needs to access the memory 29 via the bus 21 to find the corresponding information in the memory 29. Because the access rate of the cache memory 28 is very fast, when the cache memory 28 hits, the efficiency of the processor 22 can be significantly improved, thereby improving the performance and efficiency of the entire working node 120.

[0072] As shown in the figure, the processor 22, the cache memory 28 and the memory 29 are packaged in a system on chip (SoC) 201. Designers can configure the SoC architecture so that the communication between the various components in the working node 120 is secure.

[0073] In this example, the working node 120 may also include various software. Figure 2 As shown, an operating system 202, container support 203 and container group 204 are arranged on the underlying hardware (i.e., system on chip 201). The operating system 202 is, for example, an operating system that can be used on a server, such as a UNIX operating system, a Linux operating system, etc. The working node 120 supports other instruction set architectures such as the RISC-V instruction set architecture or the X86 instruction set architecture or the ARM instruction set architecture. The container support 203 is the various underlying implementations required to support the containers running thereon. For example, for docker, the container support 203 needs to implement two technologies, cgroup (abbreviation of controlgroups, control group) and namespace (namespace). Cgroup implements resource limits, and namespace implements resource isolation. Docker allows developers to package their applications and the dependent operating environment into a portable container, and then publish it to the working node 120. As Figure 2As shown, based on the container support 203, multiple container groups 204 can be run, and the operating environment and running applications are implemented in each container of the container group 204, and the isolation and non-interference between the various applications are achieved through the container technology. The application may include but is not limited to programs for controlling or responding to external devices (e.g., biometric sensors, printers, microphones, speakers, flow valves, or other I / O components, sensors, actuators, or devices), programs for various I / O tasks, security programs, authentication programs, various computing modules, communication programs, communication support protocols, or other programs, or a combination thereof.

[0074] In some embodiments, Figure 2 As shown, on the underlying hardware (i.e., the system on chip 201) there is also a

[0075] The service agent (Kubelet) 205 of the container orchestration tool (e.g., K8s) on the working node 120. In implementation, the service agent (Kubelet) 205 can be a software program module or a hardware one, for example, based on FPGA or CPLD. In some embodiments, the service agent 205 can receive and execute instructions sent by the control node 110 to manage the container group and the containers in the container group. The service agent 205 can register the information of the working node 120 on the application interface server 111 of the control node 110, monitor the resource usage of the working node 120, and regularly report the resource usage data of the working node 120 to the control node 110. In some embodiments, a garbage collection (GC) module (not shown in the figure) can also be set on the underlying hardware (i.e., the system on chip 201). In implementation, the garbage collection (GC) module can be a software program module or a hardware one, for example, based on FPGA or CPLD. In some embodiments, the garbage collection (GC) module can establish a garbage collection process for each container, and clear the configuration information of the container group when the container group is destroyed.

[0076] In addition, the working node 120 may also include hardware devices such as a display device 23, an audio device 24, an input / output device 25, a storage device 26, and a communication device 27. Of course, the structure of different computer systems may vary depending on the motherboard, operating system, and instruction set architecture.

[0077] Reference again Figure 1 ,like Figure 1As shown, the background service running on the control node 110 may also include a container fault processing device 140. The container fault processing device 140 may collect resource usage data of each heterogeneous instruction set architecture node (including working node A 220 and working node B 220), realize the diagnosis of cross-instruction set architecture faults, and accurately locate and solve container faults in the heterogeneous instruction set architecture environment in the data center. Specifically, the container fault processing device 140 includes a data collection module 141, a fault detection module 142, a fault cause diagnosis module 143, and a fault recovery module 144. In some embodiments, the data collection module 141 collects resource usage data of each heterogeneous instruction set architecture node. Specifically, the resource usage data may be collected through a service agent (Kubelet) 205 arranged on each heterogeneous instruction set architecture node. In some embodiments, the resource usage data includes resource usage data at the host level and the container level, and performance indicators related to the heterogeneous instruction set architecture (for example, RISC-V instruction set architecture or X86 instruction set architecture). It is easy to understand that by collecting resource usage data of each heterogeneous instruction set architecture node in the data center, comprehensive monitoring of the entire data center can be ensured, providing data support for subsequent fault diagnosis and scheduling optimization.

[0078] In some embodiments, cAdvisor (Container Advisor) is used to collect container-level resource usage data. Specifically, cAdvisor is deployed in a Kubernetes cluster. A DaemonSet can be created using the following YAML file:

[0079] apiVersion: apps / v1

[0080] kind: DaemonSet

[0081] metadata:

[0082] name: cadvisor

[0083] namespace: monitoring

[0084] labels:

[0085] app: cadvisor

[0086] spec:

[0087] selector:

[0088] matchLabels:

[0089] app: cadvisor

[0090] template:

[0091] metadata:

[0092] labels:

[0093] app: cadvisor

[0094] spec:

[0095] containers:

[0096] - name: cadvisor

[0097] image: gcr.io / cadvisor / cadvisor:v0.39.0

[0098] resources:

[0099] limits:

[0100] memory: 200Mi

[0101] requests:

[0102] cpu: 150m

[0103] memory: 200Mi

[0104] volumeMounts:

[0105] - name: rootfs

[0106] mountPath: / rootfs

[0107] readOnly: true

[0108] - name: var-run

[0109] mountPath: / var / run

[0110] readOnly: true

[0111] - name: sys

[0112] mountPath: / sys

[0113] readOnly: true

[0114] - name: docker

[0115] mountPath: / var / lib / docker

[0116] readOnly: true

[0117] ports:

[0118] - name: http

[0119] containerPort: 8080

[0120] protocol: TCP

[0121] volumes:

[0122] - name: rootfs

[0123] hostPath:

[0124] path: /

[0125] - name: var-run

[0126] hostPath:

[0127] path: / var / run

[0128] - name: sys

[0129] hostPath:

[0130] path: / sys

[0131] - name: docker

[0132] hostPath:

[0133] path: / var / lib / docker

[0134] In some embodiments, Node Exporter is used to collect host-level resource usage data. Specifically, Node Exporter is deployed in a Kubernetes cluster. A DaemonSet can be created using the following YAML file:

[0135] apiVersion: apps / v1

[0136] kind: DaemonSet

[0137] metadata:

[0138] name: node-exporter

[0139] namespace: monitoring

[0140] labels:

[0141] app: node-exporter

[0142] spec:

[0143] selector:

[0144] matchLabels:

[0145] app: node-exporter

[0146] template:

[0147] metadata:

[0148] labels:

[0149] app: node-exporter

[0150] spec:

[0151] containers:

[0152] - name: node-exporter

[0153] image: prom / node-exporter:v1.1.2

[0154] args:

[0155] - --path.procfs= / host / proc

[0156] - --path.sysfs= / host / sys

[0157] ports:

[0158] - name: metrics

[0159] containerPort: 9100

[0160] volumeMounts:

[0161] - name: proc

[0162] mountPath: / host / proc

[0163] readOnly: true

[0164] - name: sys

[0165] mountPath: / host / sys

[0166] readOnly: true

[0167] volumes:

[0168] - name: proc

[0169] hostPath:

[0170] path: / proc

[0171] - name: sys

[0172] hostPath:

[0173] path: / sys

[0174] In some embodiments, a resource collection plugin specific to a heterogeneous instruction set architecture (e.g., RISC-V instruction set architecture or X86 instruction set architecture) is built. A Python script can be created to collect performance metrics related to the RISC-V instruction set architecture and the X86 instruction set architecture:

[0175] import psutil

[0176] import platform

[0177] from prometheus_client import start_http_server, Gauge

[0178] # Define indicators

[0179] cpu_freq = Gauge('cpu_frequency', 'CPU Frequency in MHz')

[0180] cpu_temp = Gauge('cpu_temperature', 'CPU Temperature in Celsius')

[0181] riscv_custom_metric = Gauge('riscv_custom_metric', 'Custom metric forRISC-V')

[0182] def collect_metrics():

[0183] arch = platform.machine()

[0184] if arch == "x86_64":

[0185] # Collect performance indicators related to the X86 instruction set architecture

[0186] cpu_freq.set(psutil.cpu_freq().current)

[0187] if hasattr(psutil, "sensors_temperatures"):

[0188] temps = psutil.sensors_temperatures()

[0189] if "coretemp" in temps:

[0190] cpu_temp.set(temps["coretemp"][0].current)

[0191] elif arch == "riscv64":

[0192] # Collect performance indicators related to the RISC-V instruction set architecture

[0193] # Here we need to add corresponding indicators according to the characteristics of the RISC-V instruction set architecture

[0194] riscv_custom_metric.set(get_custom_riscv_metric())

[0195] def get_custom_riscv_metric():

[0196] # Implement the RISC-V instruction set architecture-specific indicator collection logic

[0197] # This may require reading specific system files or using specific tools

[0198] return 0# replaced with the actual indicator value

[0199] if __name__ == '__main__':

[0200] start_http_server(8000)

[0201] while True:

[0202] collect_metrics()

[0203] time.sleep(15)# Collect every 15 seconds

[0204] In some embodiments, the collected resource usage data is stored in Prometheus. Configure Prometheus to capture these metrics. Add the following to the Prometheus configuration file:

[0205] scrape_configs:

[0206] - job_name: 'cadvisor'

[0207] kubernetes_sd_configs:

[0208] - role: node

[0209] relabel_configs:

[0210] - source_labels: [__meta_kubernetes_node_label_kubernetes_io_hostname]

[0211] target_label: node

[0212] - job_name: 'node-exporter'

[0213] kubernetes_sd_configs:

[0214] - role: node

[0215] relabel_configs:

[0216] - source_labels: [__meta_kubernetes_node_label_kubernetes_io_hostname]

[0217] target_label: node

[0218] - job_name: 'arch-specific-exporter'

[0219] static_configs:

[0220] - targets: ['localhost:8000']

[0221] In some embodiments, the fault detection module 142 uses a machine learning algorithm to detect whether each heterogeneous instruction set architecture node has a fault based on the resource usage data. In some embodiments, a rule engine is used to detect the resource usage data of each heterogeneous instruction set architecture node in real time, and when the resource usage data does not meet the preset numerical range, it is determined that the corresponding working node has a fault. It can be understood that the use of a rule engine can provide a fast, low-latency fault detection mechanism that can detect some common and obvious fault modes, such as excessive CPU usage, insufficient memory, etc., so as to quickly respond to obvious faults before the machine learning model detects an anomaly. In some embodiments, the implementation code for implementing the fast fault detection logic based on the rule engine is:

[0222] class RuleBasedDetector:

[0223] def __init__(self):

[0224] self.rules = {

[0225] 'high_cpu_usage': lambda m: m['cpu_usage']>90,

[0226] 'low_memory': lambda m: m['memory_available']<100 * 1024 * 1024,#100MB

[0227] 'disk_full': lambda m: m['disk_usage_percent']>95,

[0228] 'high_network_io': lambda m: m['network_io']>1000000000,# 1GB / s

[0229] }

[0230] def detect_issues(self, metrics):

[0231] detected_issues = []

[0232] for rule_name, rule_func in self.rules.items():

[0233] if rule_func(metrics):

[0234] detected_issues.append(rule_name)

[0235] return detected_issues

[0236] def add_rule(self, name, rule_func):

[0237] self.rules[name] = rule_func

[0238] # Example

[0239] detector = RuleBasedDetector()

[0240] In some embodiments, a normal behavior model established based on a long short-term memory network (LSTM) algorithm is used to detect the resource usage data of each heterogeneous instruction set architecture node in real time. When the resource usage data deviates from the normal behavior benchmark, it is determined that the heterogeneous instruction set architecture node has a fault. It can be understood that the normal behavior model established based on the LSTM algorithm can learn the behavior pattern of the node under normal operation, so that abnormal behavior can be identified based on the real-time resource usage data of the heterogeneous instruction set architecture node. When the resource usage data deviates from the normal behavior benchmark, the normal behavior model established based on the LSTM algorithm can determine that the heterogeneous instruction set architecture node has a fault. In some embodiments, the implementation code for establishing a normal behavior model using the LSTM algorithm is:

[0241] import numpy as np

[0242] from tensorflow.keras.models import Sequential

[0243] from tensorflow.keras.layers import LSTM, Dense

[0244] from sklearn.preprocessing import MinMaxScaler

[0245] class LSTMDetector:

[0246] def __init__(self, lookback=60):

[0247] self.lookback = lookback

[0248] self.model = self._build_model()

[0249] self.scaler = MinMaxScaler(feature_range=(0, 1))

[0250] def _build_model(self):

[0251] model = Sequential()

[0252] model.add(LSTM(50, activation='relu', input_shape=(self.lookback,1)))

[0253] model.add(Dense(1))

[0254] model.compile(optimizer='adam', loss='mse')

[0255] return model

[0256] def train(self, data):

[0257] scaled_data = self.scaler.fit_transform(data.reshape(-1, 1))

[0258] X, y = self._create_dataset(scaled_data)

[0259] self.model.fit(X, y, epochs=100, batch_size=32, verbose=0)

[0260] def detect_anomalies(self, data, threshold=0.1):

[0261] scaled_data = self.scaler.transform(data.reshape(-1, 1))

[0262] X, _ = self._create_dataset(scaled_data)

[0263] predictions = self.model.predict(X)

[0264] mse = np.mean(np.power(X[:, -1] - predictions, 2), axis=1)

[0265] return mse>threshold

[0266] def _create_dataset(self, data):

[0267] X, y = [], []

[0268] for i in range(len(data) - self.lookback):

[0269] X.append(data[i:(i + self.lookback), 0])

[0270] y.append(data[i + self.lookback, 0])

[0271] return np.array(X), np.array(y)

[0272] # Usage example

[0273] detector = LSTMDetector()

[0274] normal_data = np.random.randn(1000) # Assume this is normal data

[0275] detector.train(normal_data)

[0276] new_data = np.random.randn(100) # New data

[0277] anomalies = detector.detect_anomalies(new_data)

[0278] print("Anomalies detected:", anomalies)

[0279] In some embodiments, an anomaly detection model established for the characteristics of a heterogeneous instruction set architecture is used to detect the resource usage data of each heterogeneous instruction set architecture node in real time. When there is an anomaly in the resource usage data, it is determined that the heterogeneous instruction set architecture node has a fault. It is understandable that different instruction set architectures have different performance characteristics and anomaly patterns. Through targeted training, the model can better understand and identify anomalies associated with a specific instruction set architecture. For example, the RISC-V instruction set architecture may exhibit specific anomaly patterns in branch prediction or CSR access, while the X86 instruction set architecture may have unique anomalies in SIMD instructions or virtualization performance. Training a specific anomaly detection model for each instruction set architecture can improve the accuracy and sensitivity of detection. In some embodiments, the implementation code of the anomaly detection model established using the characteristics of a heterogeneous instruction set architecture is:

[0280] # Adding architecture specific rules

[0281] detector.add_rule('riscv_specific_issue', lambda m: m['riscv_custom_metric']>100)

[0282] # Detection Problem

[0283] metrics = {

[0284] 'cpu_usage': 95,

[0285] 'memory_available': 50 * 1024 * 1024,

[0286] 'disk_usage_percent': 98,

[0287] 'network_io': 500000000,

[0288] 'riscv_custom_metric': 150

[0289] }

[0290] issues = detector.detect_issues(metrics)

[0291] print("Detected issues:", issues)

[0292] In some embodiments, the fault detection module 142 is designed with an extensible plug-in mechanism to support adding new fault detection algorithms, which can provide a flexible framework that allows adding new detection algorithms as needed to cope with new fault modes and requirements.

[0293] In some embodiments, when a fault occurs in a heterogeneous instruction set architecture node, the fault cause diagnosis module 143 performs fault cause diagnosis based on a pre-established fault knowledge graph. The fault knowledge graph includes relevant container failure modes under heterogeneous instruction set architectures (including RISC-V instruction set architectures and X86 instruction set architectures). In some embodiments, the fault cause analysis is performed based on the relevant container failure modes under the heterogeneous instruction set architecture, and a causal reasoning engine is used to perform causal analysis on the analyzed fault cause to locate the root cause of the fault. It can be understood that the fault knowledge graph provides basic knowledge for fault diagnosis, including relevant container failure modes under heterogeneous instruction set architectures (including RISC-V instruction set architectures and X86 instruction set architectures), and supports the accumulation and updating of knowledge. The causal reasoning engine can systematically analyze the cause of the fault based on causal reasoning, can handle complex fault scenarios under heterogeneous instruction set architecture environments, and accurately locate the root cause of the fault of the container-related faults of heterogeneous instruction set architecture nodes. In some embodiments, the implementation code for constructing the fault knowledge graph is:

[0294] from py2neo import Graph, Node, Relationship

[0295] class FaultKnowledgeGraph:

[0296] def __init__(self, uri, user, password):

[0297] self.graph = Graph(uri, auth=(user, password))

[0298] def build_graph(self):

[0299] # Create a schema node

[0300] x86 = Node("Architecture", name="x86_64")

[0301] riscv = Node("Architecture", name="riscv64")

[0302] # Create fault type nodes

[0303] cpu_fault = Node("FaultType", name="CPU Fault")

[0304] memory_fault = Node("FaultType", name="Memory Fault")

[0305] disk_fault = Node("FaultType", name="Disk Fault")

[0306] # Create symptom nodes

[0307] high_cpu_usage = Node("Symptom", name="High CPU Usage")

[0308] low_memory = Node("Symptom", name="Low Memory")

[0309] disk_full = Node("Symptom", name="Disk Full")

[0310] # Create relationships

[0311] self.graph.create(Relationship(x86, "HAS_FAULT", cpu_fault))

[0312] self.graph.create(Relationship(x86, "HAS_FAULT", memory_fault))

[0313] self.graph.create(Relationship(riscv, "HAS_FAULT", cpu_fault))

[0314] self.graph.create(Relationship(riscv, "HAS_FAULT", disk_fault))

[0315] self.graph.create(Relationship(cpu_fault, "HAS_SYMPTOM", high_cpu_usage))

[0316] self.graph.create(Relationship(memory_fault, "HAS_SYMPTOM", low_memory))

[0317] self.graph.create(Relationship(disk_fault, "HAS_SYMPTOM", disk_full))

[0318] def get_possible_causes(self, symptom):

[0319] query = """

[0320] MATCH (s:Symptom {name: $symptom})<-[:HAS_SYMPTOM]-(f:FaultType)

[0321] RETURN f.name AS fault_type

[0322] """

[0323] return [record["fault_type"] for record in self.graph.run(query,symptom=symptom)]

[0324] # Usage example

[0325] kg = FaultKnowledgeGraph("bolt: / / localhost:7687", "neo4j", "password")

[0326] kg.build_graph()

[0327] causes = kg.get_possible_causes("High CPU Usage")

[0328] print("Possible causes:", causes)

[0329] In some embodiments, a causal reasoning engine is used to perform causal analysis on the analyzed fault cause, and the implementation code for locating the root cause of the fault is:

[0330] import networkx as nx

[0331] class CausalInferenceEngine:

[0332] def __init__(self, knowledge_graph):

[0333] self.kg = knowledge_graph

[0334] def diagnose(self, symptoms):

[0335] G = nx.DiGraph()

[0336] # Build a causal graph

[0337] for symptom in symptoms:

[0338] G.add_edge('Root Cause', symptom)

[0339] for possible_cause in self.kg.get_possible_causes(symptom):

[0340] G.add_edge(possible_cause, symptom)

[0341] # Use the minimum cut set algorithm to find the most likely root cause

[0342] min_cut = nx.minimum_cut(G, 'Root Cause', 'Symptoms')

[0343] return list(min_cut[1])[1]# Returns the most likely root cause

[0344] # Example

[0345] kg = FaultKnowledgeGraph("bolt: / / localhost:7687", "neo4j", "password")

[0346] engine = CausalInferenceEngine(kg)

[0347] symptoms = ["High CPU Usage", "Low Memory"]

[0348] root_cause = engine.diagnose(symptoms)

[0349] print("Most likely root cause:", root_cause)

[0350] In some embodiments, the fault cause diagnosis module 143 can provide an interactive fault diagnosis interface, support human-computer collaborative diagnosis of fault causes, and can also score the diagnosis results, optimize the fault cause diagnosis algorithm based on the score, and improve the accuracy of the fault cause diagnosis module 143 in diagnosing the fault cause.

[0351] In some embodiments, the fault recovery module 144 recovers the container-related faults of the heterogeneous instruction set architecture node according to the diagnosis results. In some embodiments, for simple faults, according to the diagnosis results, the container of the heterogeneous instruction set architecture node is automatically restored, and the recovery operation includes container restart and container expansion. In some embodiments, for complex faults, for example, when the root cause of the fault is that a specific application is incompatible with the RISC-V instruction set architecture (or X86 instruction set architecture) supported by the working node A 220 (or the working node B 220), the container corresponding to the specific application is migrated between the working node A 220 and the working node B 220. In some embodiments, for special faults, manual intervention can be prompted. In some embodiments, the fault recovery module 144 can use a policy-based automatic recovery engine to recover container-related faults of heterogeneous instruction set architecture nodes based on the diagnosis results, and can also use a progressive recovery mechanism to recover container-related faults of heterogeneous instruction set architecture nodes. Kubernetes can also be integrated into the fault recovery module 144 to use Kubernetes's self-healing capabilities to recover container-related faults of heterogeneous instruction set architecture nodes, so as to expand the fault recovery module 144's fault handling support for container-related faults in heterogeneous instruction set architecture environments.

[0352] In some embodiments, the implementation code for migrating containers corresponding to specific applications between heterogeneous instruction set architecture nodes is:

[0353] from kubernetes import client, config

[0354] class ContainerMigrator:

[0355] def __init__(self):

[0356] config.load_kube_config()

[0357] self.v1 = client.CoreV1Api()

[0358] def migrate_container(self, pod_name, namespace, target_node):

[0359] # Get the original Pod configuration

[0360] pod = self.v1.read_namespaced_pod(name=pod_name, namespace=namespace)

[0361] # Create a new Pod configuration and change the target node

[0362] new_pod = client.V1Pod(

[0363] metadata=client.V1ObjectMeta(name=f"{pod_name}-migrated"),

[0364] spec = client.V1PodSpec(

[0365] containers=pod.spec.containers,

[0366] node_name=target_node ) )

[0369] # Create a new Pod

[0370] self.v1.create_namespaced_pod(namespace=namespace, body=new_pod)

[0371] # Delete the original Pod

[0372] self.v1.delete_namespaced_pod(name=pod_name, namespace=namespace)

[0373] def get_available_nodes(self, arch):

[0374] nodes = self.v1.list_node()

[0375] return [node.metadata.name for node in nodes.items ifnode.metadata.labels.get('kubernetes.io / arch') == arch]

[0376] # Example

[0377] migrator = ContainerMigrator()

[0378] available_nodes = migrator.get_available_nodes('riscv64')

[0379] if available_nodes:

[0380] migrator.migrate_container('my-pod', 'default', available_nodes[0])

[0381] In some embodiments, the implementation code for recovering container-related failures of heterogeneous instruction set architecture nodes using a policy-based automatic recovery engine is:

[0382] class RecoveryEngine:

[0383] def __init__(self):

[0384] self.strategies = {

[0385] 'restart': self.restart_container,

[0386] 'migrate': self.migrate_container,

[0387] 'scale_up': self.scale_up_deployment

[0388] }

[0389] self.migrator = ContainerMigrator()

[0390] config.load_kube_config()

[0391] self.v1 = client.CoreV1Api()

[0392] self.apps_v1 = client.AppsV1Api()

[0393] def execute_recovery(self, fault, context):

[0394] strategy = self.select_strategy(fault, context)

[0395] return self.strategies[strategy](context)

[0396] def select_strategy(self, fault, context):

[0397] if fault == 'CPU Fault':

[0398] return 'migrate'

[0399] elif fault == 'Memory Fault':

[0400] return 'restart'

[0401] elif fault == 'Disk Fault':

[0402] return 'scale_up'

[0403] else:

[0404] return 'restart' # Default strategy

[0405] def restart_container(self, context):

[0406] pod_name = context['pod_name']

[0407] namespace = context['namespace']

[0408] self.v1.delete_namespaced_pod(name=pod_name, namespace=namespace)

[0409] print(f"Restarted pod {pod_name} in namespace {namespace}")

[0410] def migrate_container(self, context):

[0411] pod_name = context['pod_name']

[0412] namespace = context['namespace']

[0413] target_arch = context['target_arch']

[0414] available_nodes = self.migrator.get_available_nodes(target_arch)

[0415] if available_nodes:

[0416] self.migrator.migrate_container(pod_name, namespace, available_nodes[0])

[0417] print(f"Migrated pod {pod_name} to node {available_nodes[0]}")

[0418] else:

[0419] print(f"No available nodes with architecture {target_arch}")

[0420] def scale_up_deployment(self, context):

[0421] deployment_name = context['deployment_name']

[0422] namespace = context['namespace']

[0423] deployment = self.apps_v1.read_namespaced_deployment(deployment_name,namespace)

[0424] deployment.spec.replicas += 1

[0425] self.apps_v1.patch_namespaced_deployment(

[0426] name=deployment_name,

[0427] namespace=namespace,

[0428] body=deployment )

[0430] print(f"Scaled up deployment {deployment_name} to{deployment.spec.replicas} replicas")

[0431] # Usage example

[0432] recovery_engine = RecoveryEngine()

[0433] context = {

[0434] 'pod_name':'my-pod',

[0435] 'namespace': 'default',

[0436] 'target_arch': 'riscv64',

[0437] 'deployment_name': 'my-deployment'

[0438] }

[0439] recovery_engine.execute_recovery('CPU Fault', context)

[0440] Reference again Figure 1 ,like Figure 1 As shown, in some embodiments, the scheduler 130 includes a container scheduling module 131 and a container scheduling optimization module 132. In some embodiments, the container scheduling module 131 uses a scheduling strategy to schedule containers for heterogeneous instruction set architecture nodes. The container scheduling optimization module 132 updates the scheduling strategy when the root cause of the failure determined by the container fault processing device 140 as described above is that the specific application is incompatible with the RISC-V instruction set architecture (or X86 instruction set architecture) supported by the working node A 220 (or the working node B 220), so as to deploy the specific application to the working node B 220 (or the working node A 220) to avoid similar container failures. It can be understood that the container scheduling optimization module 132 updates the scheduling strategy according to the root cause of the failure determined by the container fault processing device 140, which can reduce the probability of similar container failures and improve the reliability of container operations.

[0441] In some embodiments, the container scheduling module 131 can use an architecture-affinity scheduling algorithm to perform container scheduling for heterogeneous instruction set architecture nodes, can also use an active scheduling algorithm based on fault prediction to schedule containers for heterogeneous instruction set architecture nodes, and can also use a dynamic load balancing strategy to schedule containers for heterogeneous instruction set architecture nodes to balance the load of working node A 220 and working node B 220.

[0442] In some embodiments, the implementation code for performing container scheduling for heterogeneous instruction set architecture nodes using an architecture-affinity scheduling algorithm is:

[0443] from kubernetes import client, config

[0444] class ArchitectureAwareScheduler:

[0445] def __init__(self):

[0446] config.load_kube_config()

[0447] self.v1 = client.CoreV1Api()

[0448] def create_pod_with_affinity(self, name, image, preferred_arch):

[0449] affinity = client.V1Affinity(

[0450] node_affinity=client.V1NodeAffinity(

[0451] preferred_during_scheduling_ignored_during_execution=[

[0452] client.V1PreferredSchedulingTerm(

[0453] weight=1,

[0454] preference=client.V1NodeSelectorTerm(

[0455] match_expressions=[

[0456] client.V1NodeSelectorRequirement(

[0457] key="kubernetes.io / arch",

[0458] operator="In",

[0459] values=[preferred_arch] ) ] ) ) ] ) )

[0467] pod = client.V1Pod(

[0468] metadata=client.V1ObjectMeta(name=name),

[0469] spec = client.V1PodSpec(

[0470] containers=[

[0471] client.V1Container(

[0472] name="main",

[0473] image=image )

[0475] ],

[0476] affinity=affinity ) )

[0479] self.v1.create_namespaced_pod(namespace="default", body=pod)

[0480] print(f"Created pod {name} with preferred architecture {preferred_arch}")

[0481] # Example

[0482] scheduler = ArchitectureAwareScheduler()

[0483] scheduler.create_pod_with_affinity("my-riscv-pod", "my-image:latest","riscv64")

[0484] In some embodiments, the implementation code of the active scheduling algorithm based on fault prediction for container scheduling for heterogeneous instruction set architecture nodes is:

[0485] import numpy as np

[0486] from sklearn.ensemble import RandomForestClassifier

[0487] from kubernetes import client, config

[0488] class PredictiveScheduler:

[0489] def __init__(self):

[0490] self.model = RandomForestClassifier()

[0491] config.load_kube_config()

[0492] self.v1 = client.CoreV1Api()

[0493] def train(self, historical_data):

[0494] X = historical_data[['cpu_usage', 'memory_usage', 'disk_usage']]

[0495] y = historical_data['fault_occurred']

[0496] self.model.fit(X, y)

[0497] def predict_fault_probability(self, node_metrics):

[0498] return self.model.predict_proba(node_metrics.reshape(1, -1))[0][1]

[0499] def get_node_metrics(self, node_name):

[0500] # The logic of obtaining node indicators from the monitoring system should be implemented here

[0501] # For the sake of example, we return some random values

[0502] return np.random.rand(3)

[0503] def schedule_pod(self, pod_name, image):

[0504] nodes = self.v1.list_node().items

[0505] node_probabilities = []

[0506] for node in nodes:

[0507] metrics = self.get_node_metrics(node.metadata.name)

[0508] prob = self.predict_fault_probability(metrics)

[0509] node_probabilities.append((node.metadata.name, prob))

[0510] # Select the node with the lowest fault probability

[0511] best_node = min(node_probabilities, key=lambda x: x[1])

[0512] pod = client.V1Pod(

[0513] metadata=client.V1ObjectMeta(name=pod_name),

[0514] spec=client.V1PodSpec(

[0515] containers=

[0516] client.V1Container(

[0517] name="main",

[0518] image=image )

[0520] ,

[0521] node_name=best_node[0] ) )

[0524] self.v1.create_namespaced_pod(namespace="default", body=pod)

[0525] print(f"Scheduled pod {pod_name} on node {best_node[0]} with faultprobability {best_node[1]}")

[0526] # Example

[0527] scheduler = PredictiveScheduler()

[0528] # Assume we have some historical data

[0529] historical_data = np.random.rand(1000, 4)

[0530] historical_data = pd.DataFrame(historical_data, columns=['cpu_usage','memory_usage', 'disk_usage', 'fault_occurred'])

[0531] scheduler.train(historical_data)

[0532] scheduler.schedule_pod("my-predictive-pod", "my-image:latest")

[0533] In some embodiments, the implementation code for performing container scheduling for heterogeneous instruction set architecture nodes based on a dynamic load balancing strategy is:

[0534] from kubernetes import client, config

[0535] import numpy as np

[0536] class DynamicLoadBalancer:

[0537] def __init__(self):

[0538] config.load_kube_config()

[0539] self.v1 = client.CoreV1Api()

[0540] self.apps_v1 = client.AppsV1Api()

[0541] def get_node_load(self, node_name):

[0542] # The logic of obtaining node load from the monitoring system should be implemented here

[0543] # For the sake of example, we return a random value

[0544] return np.random.rand()

[0545] def balance_load(self):

[0546] nodes = self.v1.list_node().items

[0547] node_loads = [(node.metadata.name, self.get_node_load(node.metadata.name)) for node in nodes]

[0548] # Calculate the load average

[0549] avg_load = np.mean([load for _, load in node_loads])

[0550] # Find nodes with too high or too low load

[0551] high_load_nodes = [name for name, load in node_loads if load>avg_load* 1.2]

[0552] low_load_nodes = [name for name, load in node_loads if load <avg_load* 0.8]

[0553] # Migrate Pods from high-load nodes to low-load nodes

[0554] for high_node in high_load_nodes:

[0555] if not low_load_nodes:

[0556] break

[0557] pods = self.v1.list_namespaced_pod(namespace="default", field_selector=f"spec.nodeName={high_node}").items

[0558] for pod in pods:

[0559] if pod.metadata.owner_references[0].kind != "DaemonSet":# Do not migrate Pods of DaemonSet

[0560] target_node = low_load_nodes.pop(0)

[0561] self.migrate_pod(pod.metadata.name, pod.metadata.namespace, target_node)

[0562] if not low_load_nodes:

[0563] break

[0564] def migrate_pod(self, pod_name, namespace, target_node):

[0565] pod = self.v1.read_namespaced_pod(name=pod_name, namespace=namespace)

[0566] new_pod = client.V1Pod(

[0567] metadata=client.V1ObjectMeta(name=f"{pod_name}-migrated", namespace=namespace),

[0568] spec=client.V1PodSpec(

[0569] containers=pod.spec.containers,

[0570] node_name=target_node ) )

[0573] self.v1.create_namespaced_pod(namespace=namespace, body=new_pod)

[0574] self.v1.delete_namespaced_pod(name=pod_name, namespace=namespace)

[0575] print(f"Migrated pod {pod_name} from {pod.spec.node_name} to {target_node}")

[0576] # Example

[0577] balancer = DynamicLoadBalancer()

[0578] balancer.balance_load()

[0579] Figure 3 FIG. 1 is a flow chart showing a method for handling a container failure according to an embodiment of the present disclosure. Figure 3 As shown, the container fault handling method of the embodiment of the present disclosure may include:

[0580] In step S310, resource usage data of a heterogeneous instruction set architecture node is collected, where a container is set on the heterogeneous instruction set architecture node, and the resource usage data includes resource usage data at the host level and the container level, and performance indicators related to the heterogeneous instruction set architecture.

[0581] In step S320, based on the resource usage data, a machine learning algorithm is used to detect whether the heterogeneous instruction set architecture node has a fault.

[0582] In step S330, when a fault occurs in the heterogeneous instruction set architecture node, a fault cause diagnosis is performed according to a pre-established fault knowledge graph, where the fault knowledge graph includes relevant container failure modes under the heterogeneous instruction set architecture.

[0583] In step S340, based on the diagnosis result, the container-related faults of the heterogeneous instruction set architecture node are recovered.

[0584] Since the process of diagnosing and recovering container failures in a heterogeneous instruction set architecture environment using the container failure handling method of the embodiment of the present disclosure has been described in detail in the device embodiment above, it will not be repeated here.

[0585] Figure 4 A schematic diagram of a process flow of a container scheduling method provided according to an embodiment of the present disclosure is shown. Figure 4 As shown, the container scheduling method of the embodiment of the present disclosure may include:

[0586] In step S410, a scheduling strategy is used to perform container scheduling for heterogeneous instruction set architecture nodes, where the heterogeneous instruction set architecture nodes include a first node supporting a first instruction set architecture and a second node supporting a second instruction set architecture.

[0587] In step S420, when the root cause of the fault determined by the container fault handling method as described in any one of claims 1 to 7 is that a specific application is incompatible with a first instruction set architecture supported by the first node, the scheduling policy is updated to facilitate deployment of the specific application to the second node.

[0588] Since the process of using the container scheduling method of the embodiment of the present disclosure to schedule containers in a heterogeneous instruction set architecture environment has been described in detail in the device embodiment above, it will not be repeated here.

[0589] Figure 5 A schematic diagram of a container fault handling device provided according to an embodiment of the present disclosure is shown. Figure 5 As shown, the container fault processing device 500 includes a data acquisition module 510 , a fault detection module 520 , a fault cause diagnosis module 530 and a fault recovery module 540 .

[0590] The data collection module 510 is used to collect resource usage data of a heterogeneous instruction set architecture node, on which a container is set. The resource usage data includes resource usage data at the host level and the container level, and performance indicators related to the heterogeneous instruction set architecture.

[0591] The fault detection module 520 is used to detect whether there is a fault in the heterogeneous instruction set architecture node based on the resource usage data using a machine learning algorithm.

[0592] The fault cause diagnosis module 530 is used to diagnose the fault cause according to a pre-established fault knowledge graph when a fault occurs in the heterogeneous instruction set architecture node, wherein the fault knowledge graph includes relevant container failure modes under the heterogeneous instruction set architecture.

[0593] The fault recovery module 540 is used to recover the container-related faults of the heterogeneous instruction set architecture node according to the diagnosis results.

[0594] Since the process of diagnosing and recovering container failures in a heterogeneous instruction set architecture environment using the container failure handling method of the embodiment of the present disclosure has been described in detail in the device embodiment above, it will not be repeated here.

[0595] Figure 6 FIG. 1 is a schematic diagram of a scheduler provided according to an embodiment of the present disclosure. Figure 6 As shown, the scheduler 600 includes a container scheduling module 610 and a container scheduling optimization module 620 .

[0596] The container scheduling module 610 is used to use a scheduling strategy to perform container scheduling for heterogeneous instruction set architecture nodes, wherein the heterogeneous instruction set architecture nodes include a first node supporting a first instruction set architecture and a second node supporting a second instruction set architecture.

[0597] The container scheduling optimization module 620 is used to update the scheduling policy so as to deploy the specific application to the second node when the root cause of the fault determined by the container fault handling method as described in any one of claims 1 to 7 is that the specific application is incompatible with the first instruction set architecture supported by the first node.

[0598] Since the process of using the container scheduling method of the embodiment of the present disclosure to schedule containers in a heterogeneous instruction set architecture environment has been described in detail in the device embodiment above, it will not be repeated here.

[0599] The present disclosure also provides an electronic device, such as Figure 7 As shown, it includes a memory 720, a processor 710, and a program stored in the memory 720 and executable on the processor 710. When the program is executed by the processor 710, each process of each embodiment of the above-mentioned container fault handling method and / or container scheduling method can be implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0600] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling related hardware through instructions, and the instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, the embodiments of the present disclosure also provide a storage medium, on which a computer program or instruction is stored, and when the computer program or instruction is executed by the processor, each process of each embodiment of the above container fault handling method and / or container scheduling method can be implemented.

[0601] Since the instructions stored in the storage medium can execute the steps in the container fault handling method and / or container scheduling method provided in the embodiment of the present disclosure, the beneficial effects that can be achieved by the container fault handling method and / or container scheduling method provided in the embodiment of the present disclosure can be achieved. For details, please refer to the previous embodiments, which will not be repeated here. The specific implementation of each of the above operations can be referred to the previous embodiments, which will not be repeated here.

[0602] In summary, the container fault handling method provided by the present invention detects container-related faults of heterogeneous instruction set architecture nodes based on resource usage data such as host-level and container-level resource usage data and performance indicators related to heterogeneous instruction set architectures, and diagnoses the cause of the fault based on a fault knowledge graph including relevant container failure modes under the heterogeneous instruction set architecture, locates the root cause of the container-related faults of the heterogeneous instruction set architecture node, and recovers the container-related faults of the heterogeneous instruction set architecture node based on the diagnosis result. In this way, the performance indicators related to the heterogeneous instruction set architecture and the relevant container failure modes under the heterogeneous instruction set architecture take into account the instruction set differences between nodes of different instruction set architectures, can deeply understand the characteristics of different instruction set architectures, can realize the diagnosis of cross-instruction set architecture faults, and accurately locate and solve container faults in a heterogeneous instruction set architecture environment.

[0603] Finally, it should be noted that: Obviously, the above embodiments are only examples for clearly illustrating the present disclosure, and are not intended to limit the implementation methods. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from this are still within the scope of protection of the present disclosure.

Claims

1. A method for handling container failures, comprising: Collecting resource usage data of a heterogeneous instruction set architecture node, where a container is set on the heterogeneous instruction set architecture node, the resource usage data including resource usage data at the host level and the container level, and performance indicators related to the heterogeneous instruction set architecture; Based on the resource usage data, using a machine learning algorithm to detect whether the heterogeneous instruction set architecture node has a fault; In the case where the heterogeneous instruction set architecture node has a fault, diagnosing the cause of the fault according to a pre-established fault knowledge graph, wherein the fault knowledge graph includes relevant container fault modes under the heterogeneous instruction set architecture; Based on the diagnosis result, the container-related faults of the heterogeneous instruction set architecture node are recovered.

2. The container failure handling method according to claim 1, wherein: The detecting whether the heterogeneous instruction set architecture node has a fault by using a machine learning algorithm based on the resource usage data includes: A rule engine is used to detect the resource usage data of the heterogeneous instruction set architecture node in real time. When the resource usage data does not meet a preset value range, it is determined that the heterogeneous instruction set architecture node has a fault.

3. The method for handling container failure according to claim 1, wherein: The detecting whether the heterogeneous instruction set architecture node has a fault by using a machine learning algorithm based on the resource usage data includes: Using a normal behavior model established based on a long short-term memory network algorithm, the resource usage data of the heterogeneous instruction set architecture node is detected in real time. When the resource usage data deviates from a normal behavior benchmark, it is determined that the heterogeneous instruction set architecture node has a fault.

4. The container failure processing method according to claim 1, wherein: The detecting whether the heterogeneous instruction set architecture node has a fault by using a machine learning algorithm based on the resource usage data includes: Using an anomaly detection model established according to the characteristics of a heterogeneous instruction set architecture, the resource usage data of the heterogeneous instruction set architecture node is detected in real time. If an anomaly is found in the resource usage data, it is determined that the heterogeneous instruction set architecture node has a fault.

5. The method for handling container failure according to claim 1, wherein: When a fault occurs in the heterogeneous instruction set architecture node, diagnosing the cause of the fault according to a pre-established fault knowledge graph includes: Performing failure cause analysis based on relevant container failure modes under the heterogeneous instruction set architecture; Use the causal reasoning engine to perform causal analysis on the analyzed fault causes and locate the root cause of the fault.

6. The method for handling container failure according to claim 5, wherein: The heterogeneous instruction set architecture node includes a first node supporting a first instruction set architecture and a second node supporting a second instruction set architecture, and recovering a container-related fault of the heterogeneous instruction set architecture node according to a diagnosis result includes: When the root cause of the failure is that the specific application is incompatible with the first instruction set architecture supported by the first node, a container corresponding to the specific application is migrated between the first node and the second node.

7. The method for handling container failure according to claim 6, wherein: The recovering the container-related fault of the heterogeneous instruction set architecture node according to the diagnosis result also includes: According to the diagnosis result, a recovery operation is automatically performed on the container of the heterogeneous instruction set architecture node, and the recovery operation includes container restart and container expansion.

8. A container scheduling method, comprising: Performing container scheduling for heterogeneous instruction set architecture nodes using a scheduling strategy, wherein the heterogeneous instruction set architecture nodes include a first node supporting a first instruction set architecture and a second node supporting a second instruction set architecture; When the root cause of the fault determined by the container fault handling method as described in any one of claims 1 to 7 is that a specific application is incompatible with a first instruction set architecture supported by the first node, the scheduling policy is updated to facilitate deployment of the specific application to the second node.

9. A container fault processing device, comprising: A data collection module is used to collect resource usage data of a heterogeneous instruction set architecture node, where a container is provided on the heterogeneous instruction set architecture node, and the resource usage data includes resource usage data at the host level and the container level, and performance indicators related to the heterogeneous instruction set architecture; A fault detection module, configured to detect whether the heterogeneous instruction set architecture node has a fault based on the resource usage data using a machine learning algorithm; A fault cause diagnosis module, used to diagnose the fault cause according to a pre-established fault knowledge graph when a fault occurs in the heterogeneous instruction set architecture node, wherein the fault knowledge graph includes relevant container failure modes under the heterogeneous instruction set architecture; A fault recovery module is used to recover container-related faults of the heterogeneous instruction set architecture node based on the diagnosis result.

10. The container fault processing device according to claim 9, wherein: The fault detection module is also used for: A rule engine is used to detect the resource usage data of the heterogeneous instruction set architecture node in real time. When the resource usage data does not meet a preset value range, it is determined that the heterogeneous instruction set architecture node has a fault.

11. The container fault processing device according to claim 9, wherein: The fault detection module is also used for: Using a normal behavior model established based on a long short-term memory network algorithm, the resource usage data of the heterogeneous instruction set architecture node is detected in real time. When the resource usage data deviates from a normal behavior benchmark, it is determined that the heterogeneous instruction set architecture node has a fault.

12. The container fault processing device according to claim 9, wherein: The fault detection module is also used for: Using an anomaly detection model established according to the characteristics of a heterogeneous instruction set architecture, the resource usage data of the heterogeneous instruction set architecture node is detected in real time. If an anomaly is found in the resource usage data, it is determined that the heterogeneous instruction set architecture node has a fault.

13. The container fault processing device according to claim 9, wherein: The fault cause diagnosis module is also used for: Performing failure cause analysis based on relevant container failure modes under the heterogeneous instruction set architecture; Use the causal reasoning engine to perform causal analysis on the analyzed fault causes and locate the root cause of the fault.

14. The container fault processing device according to claim 13, wherein: The heterogeneous instruction set architecture node includes a first node supporting a first instruction set architecture and a second node supporting a second instruction set architecture, and the fault recovery module is further used for: When the root cause of the failure is that the specific application is incompatible with the first instruction set architecture supported by the first node, a container corresponding to the specific application is migrated between the first node and the second node.

15. The container fault processing device according to claim 14, wherein: The fault recovery module is also used for: According to the diagnosis result, a recovery operation is automatically performed on the container of the heterogeneous instruction set architecture node, and the recovery operation includes container restart and container expansion.

16. A scheduler, comprising: A container scheduling module, configured to perform container scheduling for a heterogeneous instruction set architecture node using a scheduling strategy, wherein the heterogeneous instruction set architecture node includes a first node supporting a first instruction set architecture and a second node supporting a second instruction set architecture; A container scheduling optimization module, for updating a scheduling policy to facilitate deployment of the specific application to the second node when the root cause of the fault determined according to the container fault handling method as described in any one of claims 1 to 7 is that the specific application is incompatible with the first instruction set architecture supported by the first node.

17. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein the program can implement the method according to any one of claims 1 to 8 when executed by the processor.

18. A storage medium having a computer program or instruction stored thereon, wherein the computer program or instruction, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8.

Images