Observable container cluster fault operation and maintenance method, device, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides an observable container cluster fault operation and maintenance method, device, equipment and medium, on one hand, a micro-service decoupling architecture is adopted to construct an observable container cluster fault operation and maintenance system, decoupling among functional layers can be realized, the functional layers cooperatively execute tasks, and the operation efficiency is improved; the expansibility and the task execution efficiency of the system are improved; on one hand, feature construction is carried out according to a correlation analysis result between observability index data, the internal state of the system can be predicted through external visible information, the constructed features have obvious correlation, and the accuracy of subsequent fault prediction can be improved; and on the other hand, fault prediction is carried out according to the dynamic threshold value and the fault prediction model with the time sequence, the self-healing strategy is automatically executed, more accurate fault prediction can be carried out in combination with an artificial intelligence means and a dynamic threshold value mechanism, and meanwhile, the automatic self-healing strategy is provided, so that the system stability is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an observability container cluster fault operation and maintenance method, device, equipment and medium. BACKGROUND

[0002] With the rapid development of cloud native technology, container deployment has become the mainstream way of modern application publishing. According to statistics, more than 75% of enterprises in the world use container technology in production environment, and the size of a single cluster ranges from hundreds to tens of thousands of pods.

[0003] The container cluster mainly faces the following problems in operation:

[0004] 1. Fragmentation of monitoring data: monitoring indicators are scattered in multiple systems, and there is a lack of correlation analysis of data, and the data is invisible.

[0005] 2. Lack of fault prediction capability: the traditional way usually relies on fixed threshold to trigger alarm, and lacks effective fault prevention.

[0006] 3. Self-healing strategy is not intelligent: the traditional way usually relies on manual intervention, and the repair strategy is too simple and fixed. SUMMARY

[0007] In view of the above, it is necessary to provide an observability container cluster fault operation and maintenance method, device, equipment and medium, which aims to solve the problem of lagging response of container cluster fault operation and maintenance and lack of self-healing capability.

[0008] An observability container cluster fault operation and maintenance method, the observability container cluster fault operation and maintenance method comprising:

[0009] Applied to an observability container cluster fault operation and maintenance system constructed based on a microservice decoupling architecture, the observability container cluster fault operation and maintenance system comprising a data collection layer, a data processing layer, a data analysis layer and a decision execution layer; the observability container cluster fault operation and maintenance method comprising:

[0010] In response to a fault operation and maintenance instruction for a target container cluster, collecting container indicator data, node indicator data and application log data of the target container cluster as observability indicator data based on a unified data collection framework in the data collection layer;

[0011] Performing correlation analysis on the observability indicator data in the data processing layer to obtain an analysis result;

[0012] Performing feature construction according to the analysis result and the observability indicator data in the data processing layer to obtain a to-be-processed feature;

[0013] inputting the to-be-processed features into a fault prediction model trained based on time-series attributes at the data analysis layer;

[0014] obtaining a current dynamic threshold at the data analysis layer, and generating a target fault prediction result according to output data of the fault prediction model and the current dynamic threshold;

[0015] calling a decision engine to obtain a target self-recovery strategy matched with the target fault prediction result from a self-recovery knowledge base at the decision execution layer, and executing the target self-recovery strategy.

[0016] According to a preferred embodiment of the present application, the correlation analysis of the observability index data at the data processing layer to obtain the analysis result comprises:

[0017] calculating the correlation between the container index data and the node index data to obtain an inter-index correlation analysis result;

[0018] extracting events from the application log data;

[0019] calling a correlation analysis algorithm to determine the correlation between the extracted events according to the container index data and the node index data to obtain an event correlation analysis result;

[0020] integrating the inter-index correlation analysis result and the event correlation analysis result to obtain the analysis result.

[0021] According to a preferred embodiment of the present application, the feature construction according to the analysis result and the observability index data at the data processing layer to obtain the to-be-processed features comprises:

[0022] data fusion of the observability index data according to the analysis result to obtain the to-be-processed features.

[0023] According to a preferred embodiment of the present application, before the inputting of the to-be-processed features into the fault prediction model trained based on time-series attributes at the data analysis layer, the method further comprises:

[0024] constructing a stacked multi-layer LSTM network; wherein each layer of the LSTM network comprises a forward LSTM layer and a backward LSTM layer;

[0025] constructing a training set according to the dimensions of the observability index data;

[0026] training the multi-layer LSTM network using the training set to obtain the fault prediction model.

[0027] According to a preferred embodiment of the present application, before the obtaining of the current dynamic threshold at the data analysis layer, the method further comprises:

[0028] acquire current environment data to which the target container cluster belongs;

[0029] perform threshold adjustment according to a prediction error of the failure prediction model under the current environment data, to obtain the current dynamic threshold.

[0030] According to a preferred embodiment of the present application, the observable container cluster failure operation and maintenance system further comprises a feedback optimization layer; after the target self-healing strategy is executed, the method further comprises:

[0031] acquiring an execution result of the target self-healing strategy at the feedback optimization layer;

[0032] feeding back the execution result to the failure prediction model, and updating the failure prediction model according to the execution result.

[0033] According to a preferred embodiment of the present application, a distributed cooperative processing mechanism and a plug-in extension mechanism are used between layers of the observable container cluster failure operation and maintenance system.

[0034] An observable container cluster failure operation and maintenance device runs in an observable container cluster failure operation and maintenance system constructed based on a micro-service decoupling architecture, and the observable container cluster failure operation and maintenance system comprises a data acquisition layer, a data processing layer, a data analysis layer and a decision execution layer; the observable container cluster failure operation and maintenance device comprises:

[0035] An acquisition unit is configured to, in response to a failure operation and maintenance instruction for a target container cluster, acquire container index data, node index data and application log data of the target container cluster as observable index data based on a unified data acquisition framework at the data acquisition layer.

[0036] An analysis unit is configured to perform correlation analysis on the observable index data at the data processing layer to obtain an analysis result.

[0037] A construction unit is configured to perform feature construction according to the analysis result and the observable index data at the data processing layer to obtain a to-be-processed feature.

[0038] An input unit is configured to input the to-be-processed feature to a failure prediction model trained based on time-series attributes at the data analysis layer.

[0039] A generation unit is configured to acquire a current dynamic threshold at the data analysis layer, and generate a target failure prediction result according to output data of the failure prediction model and the current dynamic threshold.

[0040] The execution unit is configured to call a decision engine to obtain a target self-healing strategy matched with the target fault prediction result from a self-healing knowledge base, and execute the target self-healing strategy.

[0041] A computer device comprises:

[0042] A memory configured to store at least one instruction; and

[0043] A processor configured to execute the instruction stored in the memory to implement the observability container cluster fault operation and maintenance method.

[0044] A computer readable storage medium having at least one instruction stored therein, the at least one instruction being executed by a processor in a computer device to implement the observability container cluster fault operation and maintenance method.

[0045] From the above technical solutions, on the one hand, the micro-service decoupling architecture is used to build the observability container cluster fault operation and maintenance system, which can realize the decoupling between each functional layer, and each functional layer cooperates to execute tasks, thereby improving the expansibility and task execution efficiency of the system; on the other hand, the feature construction is performed according to the correlation analysis result between the observability index data, which can not only predict the internal state of the system through external visible information, but also the constructed features have obvious correlation, which can improve the accuracy of subsequent fault prediction; on the other hand, the fault prediction is performed according to the dynamic threshold and the fault prediction model with time sequence, and the self-healing strategy is automatically executed, which can combine artificial intelligence means and dynamic threshold mechanism to more accurately predict faults, and at the same time, provide an automatic self-healing strategy, thereby improving the stability of the system. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 is a flowchart of a preferred embodiment of the observability container cluster fault operation and maintenance method of the present application.

[0047] Figure 2 is a functional module diagram of a preferred embodiment of the observability container cluster fault operation and maintenance device of the present application.

[0048] Figure 3 is a structural schematic diagram of a computer device of a preferred embodiment of the present application for implementing the observability container cluster fault operation and maintenance method. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be described in detail below with reference to the drawings and specific embodiments.

[0050] As Figure 1The flowchart shown is a flowchart of a preferred embodiment of the method for operating and maintaining the observability container cluster fault. The order of the steps in the flowchart can be changed according to different needs, and some steps can be omitted.

[0051] The method for operating and maintaining the observability container cluster fault is applied in one or more computer devices, which is a device capable of automatically performing numerical calculation and / or information processing according to pre-set or stored instructions. The hardware of the computer device includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0052] The computer device can be any electronic product that can interact with the user, such as a personal computer, a tablet computer, a smartphone, a personal digital assistant (PDA), a game console, an interactive Internet Protocol Television (IPTV), a smart wearable device, etc.

[0053] The computer device can also include network devices and / or user devices. The network devices include, but are not limited to, a single network server, a server group composed of multiple network servers, or a cloud composed of a large number of hosts or network servers based on cloud computing.

[0054] The server can be a standalone server, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, etc. basic cloud computing services.

[0055] Artificial intelligence (AI) is the use of digital computers or computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0056] The artificial intelligence basic technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The artificial intelligence software technology mainly includes computer vision technology, robot technology, biometric identification technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0057] The network in which the computer device is located includes, but is not limited to, the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN), etc.

[0058] The embodiment is applied to an observability container cluster fault operation and maintenance system constructed based on a micro-service decoupling architecture, the observability container cluster fault operation and maintenance system comprising a data collection layer, a data processing layer, a data analysis layer, and a decision execution layer; and the observability container cluster fault operation and maintenance method comprises:

[0059] S10, in response to a fault operation and maintenance instruction for a target container cluster, collecting container index data, node index data, and application log data of the target container cluster as observability index data based on a unified data collection framework in the data collection layer.

[0060] In the embodiment, the fault operation and maintenance instruction can be automatically triggered when the target container cluster starts serving, so as to realize protection of the whole life cycle of the target container cluster.

[0061] In the embodiment, the target container cluster comprises a plurality of container hosts, and resource allocation, task scheduling, and fault recovery are coordinated through a unified management platform, so as to realize the goals of high availability, scalability, and elastic computing.

[0062] For example, the target container cluster can be a log analysis system of a small micro-service architecture, a multi-user platform, a multi-application platform, etc.

[0063] In the embodiment, the container index data is microscopic data reflecting the running state of a single container, and the container index data is used to monitor the resource usage, performance, and health status of an application in the container.

[0064] For example, the container index data can include, but is not limited to, utilization rates of resources such as a CPU (Central Processing Unit), memory, disk, network, etc., and running state data such as the number of restarts.

[0065] In the embodiment, the node index data is macroscopic running state reflecting a container host (physical machine / virtual machine), and is used to evaluate the overall health and resource allocation rationality of the node.

[0066] For example, the node indicator data can include, but is not limited to, a kernel version, a system load, a Docker / Kubelet service status, a container runtime version, allocatable resources, a host CPU oversell rate, a virtualization layer latency, a hardware health status, and the like.

[0067] In this embodiment, the application log data is a text record actively output by an application in a container, which is used to track business logic, locate code-level problems, and audit operations.

[0068] For example, the application log data can include, but is not limited to, container runtime logs, operating system logs, request logs, error logs, audit logs, framework / middleware logs, and the like.

[0069] The container indicator data, the node indicator data, and the application log data all belong to externally observable data types, thereby facilitating fault prediction.

[0070] S11, performing correlation analysis on the observability indicator data at the data processing layer to obtain an analysis result.

[0071] In this embodiment, the correlation analysis on the observability indicator data at the data processing layer to obtain an analysis result includes:

[0072] calculating the correlation between the container indicator data and the node indicator data to obtain an inter-indicator correlation analysis result;

[0073] extracting events from the application log data;

[0074] calling a correlation analysis algorithm to determine the correlation between the extracted events according to the container indicator data and the node indicator data to obtain an event correlation analysis result;

[0075] integrating the inter-indicator correlation analysis result and the event correlation analysis result to obtain the analysis result.

[0076] For example, when performing inter-indicator correlation analysis, the correlation between container resource indicators (such as container CPU usage and memory usage) and node resource indicators (such as node CPU usage and memory usage) can be calculated. If the correlation is high, it means that the container resource usage is greatly affected by the node resource status. The relationship between container performance indicators (such as container application response time) and node resource indicators can also be analyzed to determine whether node resource shortage will lead to a decrease in container performance.

[0077] For example, when performing event correlation analysis, key events (such as container crashes and node failures) can be extracted from application logs. Combined with container and node status metrics, association rule mining algorithms can be used to identify the correlation rules between events. For instance, when node network bandwidth utilization is too high, the number of network request failure logs within the container increases. Furthermore, event causal relationship models can be established. By analyzing the sequence and correlation of events, it can be determined which node or container metric changes are the cause of the erroneous events in the application logs.

[0078] S12, in the data processing layer, features are constructed based on the analysis results and the observability index data to obtain the features to be processed.

[0079] In this embodiment, the feature construction performed at the data processing layer based on the analysis results and the observability index data to obtain the features to be processed includes:

[0080] Based on the analysis results, the observable index data are fused to obtain the feature to be processed.

[0081] For example, if correlation analysis reveals a high correlation between container CPU utilization and memory utilization, a new feature can be constructed, such as resource utilization rate = (CPU utilization + memory utilization) / 2, to comprehensively reflect container resource consumption. If analysis shows that excessive node disk I / O leads to increased application read / write latency within the container, a "node disk I / O - container application read / write latency correlation feature" can be constructed, for example, by using the ratio of peak node disk I / O to the average container application read / write latency to assess the impact of this causal relationship on application performance. If correlation analysis reveals a strong correlation between specific application log event combinations (such as frequent container restarts accompanied by insufficient memory error logs) and container failures, these event combinations can be constructed as "failure mode features" to serve as key indicators for failure prediction.

[0082] Through the above embodiments, feature construction can be carried out based on the correlation analysis between indicators. Since the constructed features can reflect the correlation between indicators, it helps to improve the accuracy of subsequent model fault prediction.

[0083] S13, the features to be processed are input into the fault prediction model trained based on time-series attributes in the data analysis layer.

[0084] In this embodiment, before the data analysis layer inputs the features to be processed into the fault prediction model trained based on time-series attributes, the method further includes:

[0085] Construct a superimposed multi-layer LSTM (Long Short-Term Memory) network; where each LSTM network includes a forward LSTM layer and a backward LSTM layer;

[0086] Construct a training set according to the dimensions of the observability index data;

[0087] The multi-layer LSTM network is trained using the training set to obtain the fault prediction model.

[0088] Specifically, forward and backward LSTM layers can simultaneously capture past and future temporal dependencies; stacking multiple LSTM networks can enhance the ability to abstract complex temporal patterns. First, stacked LSTMs are used to extract multi-level features from the data, and then bidirectional LSTMs are used on each layer to further capture bidirectional information of the sequence. This approach fully leverages the advantages of both methods to adapt to complex temporal data or scenarios with high model performance requirements.

[0089] S14, obtain the current dynamic threshold in the data analysis layer, and generate the target fault prediction result based on the output data of the fault prediction model and the current dynamic threshold.

[0090] In this embodiment, before obtaining the current dynamic threshold at the data analysis layer, the method further includes:

[0091] Obtain the current environment data of the target container cluster;

[0092] The threshold is adjusted based on the prediction error of the fault prediction model under the current environmental data to obtain the current dynamic threshold.

[0093] Through the above embodiments, the threshold can be dynamically adjusted based on the environmental perception mechanism, thereby avoiding the prediction error caused by using a single model.

[0094] In this embodiment, generating the target fault prediction result based on the output data of the fault prediction model and the current dynamic threshold includes:

[0095] Compare the output data with the current dynamic threshold;

[0096] When the output data is greater than the current dynamic threshold, it is determined that the target container cluster has a risk of failure.

[0097] When the output data is less than or equal to the current dynamic threshold, it is determined that the target container cluster has no risk of failure.

[0098] Through the above embodiments, it is possible to more accurately predict whether the target container cluster has a risk of failure by combining comparison with dynamic thresholds.

[0099] S15, the decision execution layer calls the decision engine to obtain the target self-healing strategy that matches the target fault prediction result from the self-healing knowledge base, and executes the target self-healing strategy.

[0100] In this embodiment, the self-healing knowledge base can be constructed based on historical fault recovery strategies.

[0101] The target self-healing strategy enables the execution of corresponding self-healing tasks based on the actual fault type, thereby allowing the target container cluster to quickly return to normal service status.

[0102] For example, when the fault type is excessive load or uneven load utilization, the resources of the target container cluster can be dynamically scheduled based on the resource profile model so that the load of the target container cluster can be quickly restored to normal.

[0103] In this embodiment, the observable container cluster fault operation and maintenance system further includes a feedback optimization layer; after executing the target self-healing strategy, the method further includes:

[0104] The execution result of the target self-healing strategy is obtained in the feedback optimization layer;

[0105] The execution result is fed back to the fault prediction model, and the fault prediction model is updated based on the execution result.

[0106] Through the above embodiments, the model can be fed back based on the execution of the self-healing strategy, making the model's prediction results more accurate.

[0107] In this embodiment, the observable container cluster fault operation and maintenance system adopts a distributed collaborative processing mechanism and a plug-in extension mechanism between its various layers.

[0108] The distributed collaborative processing mechanism addresses the performance, reliability, and elasticity issues in container cluster operation and maintenance, making it suitable for handling high-concurrency, large-scale data and computing tasks.

[0109] The plug-in extension mechanism addresses the issues of technology adaptability and functional scalability, enabling the system to quickly adapt to changes in the cluster technology stack and iterative business requirements.

[0110] This embodiment combines the distributed collaborative processing mechanism and the plug-in extension mechanism to build a highly available, easily scalable, and highly compatible fault operation and maintenance system, which is especially suitable for the observability and fault management of container clusters in complex cloud-native environments.

[0111] As can be seen from the above technical solutions, on the one hand, adopting a microservice decoupling architecture to build an observable container cluster fault operation and maintenance system can achieve decoupling between functional layers, enabling each functional layer to collaboratively execute tasks, thereby improving the system's scalability and task execution efficiency. On the other hand, feature construction based on the correlation analysis results between observable indicator data can not only predict the internal state of the system through externally visible information, but also the constructed features have obvious correlations, which can improve the accuracy of subsequent fault prediction. Furthermore, fault prediction based on dynamic thresholds and time-series fault prediction models, and automatic execution of self-healing strategies, can combine artificial intelligence methods and dynamic threshold mechanisms to achieve more accurate fault prediction, while providing automated self-healing strategies, thereby improving system stability.

[0112] like Figure 2 The diagram shown is a functional block diagram of a preferred embodiment of the observable container cluster fault operation and maintenance device of the present invention. The observable container cluster fault operation and maintenance device 11 runs on an observable container cluster fault operation and maintenance system built on a microservice decoupling architecture. The observable container cluster fault operation and maintenance system includes a data acquisition layer, a data processing layer, a data analysis layer, and a decision execution layer. The observable container cluster fault operation and maintenance device 11 includes an acquisition unit 110, an analysis unit 111, a construction unit 112, an input unit 113, a generation unit 114, and an execution unit 115. The module / unit referred to in this invention refers to a series of computer program segments that can be executed by a processor and perform a fixed function, stored in memory. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0113] The acquisition unit 110 is used to collect container indicator data, node indicator data and application log data of the target container cluster as observability indicator data in response to the fault operation and maintenance command of the target container cluster.

[0114] The analysis unit 111 is used to perform correlation analysis on the observable index data in the data processing layer to obtain analysis results;

[0115] The construction unit 112 is used to construct features in the data processing layer based on the analysis results and the observability index data to obtain the features to be processed.

[0116] The input unit 113 is used to input the features to be processed into the fault prediction model trained based on time-series attributes in the data analysis layer.

[0117] The generation unit 114 is used to obtain the current dynamic threshold in the data analysis layer and generate a target fault prediction result based on the output data of the fault prediction model and the current dynamic threshold.

[0118] The execution unit 115 is used to call the decision engine at the decision execution layer to obtain a target self-healing strategy that matches the target fault prediction result from the self-healing knowledge base, and execute the target self-healing strategy.

[0119] As can be seen from the above technical solutions, on the one hand, adopting a microservice decoupling architecture to build an observable container cluster fault operation and maintenance system can achieve decoupling between functional layers, enabling each functional layer to collaboratively execute tasks, thereby improving the system's scalability and task execution efficiency. On the other hand, feature construction based on the correlation analysis results between observable indicator data can not only predict the internal state of the system through externally visible information, but also the constructed features have obvious correlations, which can improve the accuracy of subsequent fault prediction. Furthermore, fault prediction based on dynamic thresholds and time-series fault prediction models, and automatic execution of self-healing strategies, can combine artificial intelligence methods and dynamic threshold mechanisms to achieve more accurate fault prediction, while providing automated self-healing strategies, thereby improving system stability.

[0120] like Figure 3 The diagram shown is a schematic representation of the computer device used in a preferred embodiment of the present invention for implementing an observable container cluster fault operation and maintenance method.

[0121] The computer device 1 may include a memory 12, a processor 13, and a bus (the arrow in the figure represents the bus), and may also include a computer program stored in the memory 12 and executable on the processor 13, such as an observable container cluster fault management program.

[0122] Those skilled in the art will understand that the schematic diagram is merely an example of computer device 1 and does not constitute a limitation on computer device 1. Computer device 1 can be either a bus topology or a star topology. Computer device 1 may also include more or fewer other hardware or software than shown in the diagram, or different component arrangements. For example, computer device 1 may also include input / output devices, network access devices, etc.

[0123] It should be noted that the computer device 1 described is merely an example. Other existing or future electronic products that are adaptable to this invention should also be included within the scope of protection of this invention and are incorporated herein by reference.

[0124] The memory 12 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the computer device 1, such as a portable hard drive of the computer device 1. In other embodiments, the memory 12 can be an external storage device of the computer device 1, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the computer device 1. Furthermore, the memory 12 can include both internal storage units and external storage devices of the computer device 1. The memory 12 can be used not only to store application software and various types of data installed on the computer device 1, such as the code of observable container cluster fault management programs, but also to temporarily store data that has been output or will be output.

[0125] In some embodiments, processor 13 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. Processor 13 is the control unit of the computer device 1, connecting various components of the computer device 1 via various interfaces and lines. It executes programs or modules stored in the memory 12 (e.g., executing observable container cluster fault management programs) and calls data stored in the memory 12 to perform various functions of the computer device 1 and process data.

[0126] The processor 13 executes the operating system of the computer device 1 and various installed applications. The processor 13 executes these applications to implement the steps in the various embodiments of the observable container cluster fault operation and maintenance method described above, for example... Figure 1 The steps are shown.

[0127] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of performing specific functions, which describe the execution process of the computer program in the computer device 1. For example, the computer program may be divided into a data acquisition unit 110, an analysis unit 111, a construction unit 112, an input unit 113, a generation unit 114, and an execution unit 115.

[0128] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, computer equipment, or network device, etc.) or processor to execute portions of the observable container cluster fault operation and maintenance method described in the various embodiments of this invention.

[0129] If the modules / units integrated in the computer device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware devices. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above.

[0130] The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, etc.

[0131] Furthermore, the computer-readable storage medium may primarily include a stored program area and a stored data area, wherein the stored program area may store the operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of blockchain nodes, etc.

[0132] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0133] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, in... Figure 3 The bus is represented by only one straight line, but this does not mean that there is only one bus or one type of bus. The bus is configured to enable communication between the memory 12 and at least one processor 13, etc.

[0134] Although not shown, the computer device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 13 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The computer device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0135] Furthermore, the computer device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish a communication connection between the computer device 1 and other computer devices.

[0136] Optionally, the computer device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the computer device 1 and to display a visual user interface.

[0137] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0138] It will be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the computer device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0139] Combination Figure 1 The memory 12 in the computer device 1 stores multiple instructions to implement an observable container cluster fault operation and maintenance method, and the processor 13 can execute the multiple instructions to achieve:

[0140] In response to the fault operation and maintenance instructions for the target container cluster, the data acquisition layer collects container indicator data, node indicator data and application log data of the target container cluster as observability indicator data based on a unified data acquisition framework.

[0141] The observable index data are subjected to correlation analysis in the data processing layer to obtain the analysis results;

[0142] In the data processing layer, features are constructed based on the analysis results and the observability index data to obtain the features to be processed;

[0143] The features to be processed are input into the fault prediction model trained based on time-series attributes in the data analysis layer;

[0144] The current dynamic threshold is obtained in the data analysis layer, and a target fault prediction result is generated based on the output data of the fault prediction model and the current dynamic threshold.

[0145] The decision execution layer invokes the decision engine to retrieve the target self-healing strategy from the self-healing knowledge base that matches the target fault prediction result, and executes the target self-healing strategy.

[0146] Specifically, the processor 13's implementation method for the above instructions can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0147] It should be noted that all data involved in this case was legally obtained. Software tools or components not belonging to this company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

[0148] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0149] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0150] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0151] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0152] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0153] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0154] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in this invention can also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for observable container cluster fault operation and maintenance, characterized in that, This is applied to an observable container cluster fault operation and maintenance system built on a microservice decoupled architecture. The observable container cluster fault operation and maintenance system includes a data acquisition layer, a data processing layer, a data analysis layer, and a decision execution layer. The observable container cluster fault operation and maintenance method includes: In response to the fault operation and maintenance instructions for the target container cluster, the data acquisition layer collects container indicator data, node indicator data and application log data of the target container cluster as observability indicator data based on a unified data acquisition framework. The observable index data are subjected to correlation analysis in the data processing layer to obtain the analysis results; In the data processing layer, features are constructed based on the analysis results and the observability index data to obtain the features to be processed; The features to be processed are input into the fault prediction model trained based on time-series attributes in the data analysis layer; The current dynamic threshold is obtained in the data analysis layer, and a target fault prediction result is generated based on the output data of the fault prediction model and the current dynamic threshold. The decision execution layer invokes the decision engine to retrieve the target self-healing strategy from the self-healing knowledge base that matches the target fault prediction result, and executes the target self-healing strategy.

2. The observable container cluster fault operation and maintenance method as described in claim 1, characterized in that, The correlation analysis performed on the observability index data in the data processing layer yields the following analysis results: Calculate the correlation between the container indicator data and the node indicator data to obtain the correlation analysis results between indicators; Extract events from the application log data; The correlation analysis algorithm is invoked to determine the correlation patterns between the extracted events based on the container index data and the node index data, and the event correlation analysis results are obtained. The analysis results are obtained by integrating the correlation analysis results between the indicators and the correlation analysis results between the events.

3. The observable container cluster fault operation and maintenance method as described in claim 1, characterized in that, The feature construction performed in the data processing layer based on the analysis results and the observability index data yields the following features to be processed: Based on the analysis results, the observable index data are fused to obtain the feature to be processed.

4. The observable container cluster fault operation and maintenance method as described in claim 1, characterized in that, Before inputting the features to be processed into the fault prediction model trained based on time-series attributes at the data analysis layer, the method further includes: Construct a superimposed multi-layer LSTM network; where each LSTM network includes a forward LSTM layer and a backward LSTM layer; Construct a training set according to the dimensions of the observability index data; The multi-layer LSTM network is trained using the training set to obtain the fault prediction model.

5. The observable container cluster fault operation and maintenance method as described in claim 1, characterized in that, Before obtaining the current dynamic threshold at the data analysis layer, the method further includes: Obtain the current environment data of the target container cluster; The threshold is adjusted based on the prediction error of the fault prediction model under the current environmental data to obtain the current dynamic threshold.

6. The observable container cluster fault operation and maintenance method as described in claim 1, characterized in that, The observable container cluster fault operation and maintenance system further includes a feedback optimization layer; after executing the target self-healing strategy, the method further includes: The execution result of the target self-healing strategy is obtained in the feedback optimization layer; The execution result is fed back to the fault prediction model, and the fault prediction model is updated based on the execution result.

7. The observable container cluster fault operation and maintenance method as described in claim 1, characterized in that, The observable container cluster fault operation and maintenance system adopts a distributed collaborative processing mechanism and a plug-in extension mechanism between its various layers.

8. An observable container cluster fault operation and maintenance device, characterized in that, An observable container cluster fault operation and maintenance system is built on a microservice decoupled architecture. The observable container cluster fault operation and maintenance system includes a data acquisition layer, a data processing layer, a data analysis layer, and a decision execution layer. The observable container cluster fault operation and maintenance device includes: The data acquisition unit is used to respond to the fault operation and maintenance instructions of the target container cluster and collect container indicator data, node indicator data and application log data of the target container cluster as observable indicator data based on a unified data acquisition framework at the data acquisition layer. An analysis unit is used to perform correlation analysis on the observable index data in the data processing layer to obtain analysis results; A construction unit is used to construct features in the data processing layer based on the analysis results and the observable index data to obtain the features to be processed; An input unit is used to input the features to be processed into a fault prediction model trained based on time-series attributes in the data analysis layer. The generation unit is used to obtain the current dynamic threshold in the data analysis layer and generate the target fault prediction result based on the output data of the fault prediction model and the current dynamic threshold. The execution unit is used to call the decision engine at the decision execution layer to obtain the target self-healing strategy that matches the target fault prediction result from the self-healing knowledge base, and to execute the target self-healing strategy.

9. A computer device, characterized in that, The computer device includes: Memory, storing at least one instruction; and The processor executes instructions stored in the memory to implement the observable container cluster fault operation and maintenance method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, which is executed by a processor in a computer device to implement the observable container cluster fault operation and maintenance method as described in any one of claims 1 to 7.

Images