Container environment-based fault diagnosis method and device, storage medium and program product

Abstract

This application relates to the field of computer technology, and further to a fault diagnosis method and apparatus, a computer-readable storage medium, and a computer program product based on a container environment. The method includes: monitoring preset monitoring indicators and triggering alarm information based on the status of the monitoring indicators; determining the real-time status, number of restarts, and termination reason of a target container group based on the alarm information; executing a first working mode based on the real-time status and number of restarts, or executing a second working mode based on the number of restarts and termination reason; the first working mode is used to collect process-level performance data and cache it in memory when the target container group is alive; the second working mode is used to read process-level performance data from memory and perform replay analysis when the target container group restarts or crashes. This method effectively reconstructs the code-level root cause at the moment of the crash and also improves the success rate of fault diagnosis in container environments.

Description

Technical Field

[0001] This application relates to the field of computer technology, and further to a fault diagnosis method and apparatus based on a container environment, a computer-readable storage medium, and a computer program product. Background Technology

[0002] With the widespread adoption of microservice architectures, Kubernetes has become the de facto standard for container orchestration. However, the dynamic scheduling mechanism of Kubernetes makes the lifecycle of container groups (Pods, or application instances) highly short-lived and unpredictable.

[0003] This characteristic poses a significant challenge to traditional monitoring systems: traditional monitoring methods based on fixed IP addresses or hostnames are gradually becoming ineffective. Especially in scenarios where container groups are forcibly terminated and restarted due to performance anomalies triggering system protection mechanisms, existing monitoring methods can only capture limited log records and cannot effectively reconstruct the code-level root cause at the moment of the crash.

[0004] Therefore, there is an urgent need for a fault diagnosis method based on container environments, which can restore the code-level root cause of the crash at the moment of the incident when an anomaly occurs in the container group, thereby improving the success rate of fault diagnosis. Summary of the Invention

[0005] To address the aforementioned technical issues, this application provides a fault diagnosis method and apparatus, a computer-readable storage medium, and a computer program product based on a container environment, which effectively restores the code-level root cause at the moment of the crash and also improves the success rate of fault diagnosis in container environments.

[0006] In a first aspect, this application provides a fault diagnosis method based on a container environment, comprising: monitoring preset monitoring indicators and triggering alarm information according to the status of the monitoring indicators; determining the real-time status, number of restarts, and termination reason of the target container group according to the alarm information; executing a first working mode according to the real-time status and number of restarts, or executing a second working mode according to the number of restarts and termination reason; the first working mode is used to collect process-level performance data and cache it in memory when the target container group is alive; the second working mode is used to read process-level performance data from memory and perform replay analysis when the target container group restarts or crashes; wherein, the monitoring indicators include CPU utilization and / or memory usage.

[0007] The above-described fault diagnosis method for container environments dynamically selects either a first or second working mode after triggering alarms through real-time monitoring metrics and intelligently judging real-time status, restart count, and termination reasons. The first working mode collects cached process-level performance data when the container is alive, while the second working mode reads and analyzes cached data when the container restarts or crashes, thus achieving rapid response to different scenarios. This method ensures real-time collection of performance data from running container groups, guarantees that critical data is not lost at the moment of failure, effectively reconstructs the code-level root cause at the moment of crash, and improves the success rate of fault diagnosis in container environments.

[0008] In one implementation, a first working mode is executed based on the real-time status and the number of restarts, or a second working mode is executed based on the number of restarts and the termination reason. Specifically, the first working mode is executed when the real-time status indicates that the target container group is in a running state and the number of restarts is a first preset value; the second working mode is executed when the number of restarts is a second preset value and the termination reason is memory overflow or crash loop; wherein the first preset value is less than the second preset value.

[0009] One implementation also includes: collecting system-level profiling data, application-level performance profiling data, timing monitoring data, and metadata of the container orchestration system; and automatically aligning and correlating the system-level profiling data, application-level performance profiling data, timing monitoring data, and metadata of the container orchestration system using container group IDs and time windows.

[0010] One implementation also includes: collecting historical trend data and metadata of the container orchestration system; generating an intermediate state report based on the metadata, real-time status, historical trend data, and process-level performance data; and analyzing the CPU hotspots and memory leak trends corresponding to the intermediate state report based on the large language model and outputting the repair results.

[0011] The above-described fault diagnosis method for container environments collects historical trend data and metadata from the container orchestration system, and generates intermediate reports by combining real-time status and process-level performance data. Based on this, a large language model is used to perform in-depth analysis and reasoning on the CPU hotspots and memory leak trends corresponding to the intermediate reports. This not only accurately reveals the root causes of anomalies but also directly outputs targeted repair results, significantly improving the automated diagnostic capabilities of container environments in complex fault scenarios. Furthermore, this application eliminates the need for manual command execution by the user, reducing technical requirements and improving analysis efficiency.

[0012] In one implementation, an alarm message is triggered based on the status of the monitoring metric. Specifically, an alarm message is triggered when the monitoring metric exceeds the alarm threshold for a continuous preset time period.

[0013] Secondly, this application provides a fault diagnosis device based on a container environment, comprising: a monitoring module configured to monitor preset monitoring indicators and trigger alarm information according to the status of the monitoring indicators; and a processing module configured to: determine the real-time status, number of restarts, and termination reason of a target container group according to the alarm information; execute a first working mode according to the real-time status and number of restarts, or execute a second working mode according to the number of restarts and termination reason; the first working mode is used to collect process-level performance data and cache it in memory when the target container group is alive; the second working mode is used to read process-level performance data from memory and perform replay analysis when the target container group restarts or crashes; wherein the monitoring indicators include CPU utilization and / or memory usage.

[0014] In one implementation, the processing module is configured to: execute a first working mode when the real-time status indicates that the target container group is in a running state and the number of restarts is a first preset value; and execute a second working mode when the number of restarts is a second preset value and the termination reason is memory overflow or a crash loop; wherein the first preset value is less than the second preset value.

[0015] In one implementation, it also includes: a collection module configured to collect system-level profiling data, application-level performance profiling data, time-series monitoring data, and metadata of the container orchestration system; and a processing module configured to automatically align and correlate the system-level profiling data, application-level performance profiling data, time-series monitoring data, and metadata of the container orchestration system using container group IDs and time windows.

[0016] Thirdly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described container-based fault diagnosis methods.

[0017] Fourthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described container-based fault diagnosis methods.

[0018] Compared with the prior art, the present invention has at least one of the following beneficial effects: 1. After triggering alarms through real-time monitoring metrics, the system dynamically selects either a first or second working mode based on intelligent judgment of real-time status, restart count, and termination reason. The first working mode is used to collect cached process-level performance data when the container is alive, while the second working mode is used to read and analyze cached data when the container restarts or crashes, thus achieving rapid response to different scenarios. This method ensures real-time collection of performance data of the running container group, guarantees that critical data is not lost at the moment of failure, effectively reconstructs the code-level root cause at the moment of crash, and improves the success rate of container environment fault diagnosis.

[0019] 2. By collecting historical trend data and metadata from the container orchestration system, and combining it with real-time status and process-level performance data, intermediate reports are generated. Based on this, a large language model is used to perform in-depth analysis and reasoning on the CPU hotspots and memory leak trends corresponding to the intermediate reports. This not only accurately reveals the root causes of anomalies but also directly outputs targeted remediation results, significantly improving the automated diagnostic capabilities of container environments in complex fault scenarios. Furthermore, this application eliminates the need for manual command execution by the user, reducing technical requirements and improving analysis efficiency. Attached Figure Description

[0020] The preferred embodiments will now be described in a clear and easy-to-understand manner, in conjunction with the accompanying drawings, to further explain the above-mentioned characteristics, technical features, advantages, and implementation methods of the present invention.

[0021] Figure 1 An architecture diagram of a fault diagnosis system based on a container environment provided in an embodiment of this application is shown; Figure 2 A flowchart of a fault diagnosis method based on a container environment provided in an embodiment of this application is shown; Figure 3 This application provides a flowchart for determining a repair result according to an embodiment of the present application; Figure 4 This paper shows a structural block diagram of a fault diagnosis device based on a container environment provided in an embodiment of this application. Detailed Implementation

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the specific implementation methods of the present invention will be described below with reference to the accompanying drawings. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings and other implementation methods can be obtained based on these drawings without any creative effort.

[0023] To keep the drawings concise, each figure only schematically shows the parts relevant to the invention, and these do not represent the actual structure of the product. Furthermore, to facilitate understanding, in some figures, only one of components with the same structure or function is schematically depicted, or only one is labeled. In this document, "one" not only means "only one," but can also mean "more than one."

[0024] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0025] In this document, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0026] Furthermore, in the description of this application, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0027] It should be noted that the above embodiments can be freely combined as needed. The above are merely preferred embodiments of the present invention. It should be pointed out that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

[0028] With the widespread adoption of microservice architectures, Kubernetes has become the de facto standard for container orchestration. However, the dynamic scheduling mechanism of Kubernetes makes the lifecycle of container groups (Pods, or application instances) highly short-lived and unpredictable.

[0029] This characteristic poses a significant challenge to traditional monitoring systems: for example, traditional monitoring methods based on fixed IP addresses or hostnames are gradually becoming ineffective. Especially in scenarios where container groups are forcibly terminated and restarted due to performance anomalies triggering system protection mechanisms, existing monitoring methods can only capture limited log records and cannot effectively reconstruct the code-level root cause at the moment of the crash. Furthermore, traditional troubleshooting solutions require users to master low-level tools such as Perf, Pprof, and GDB and manually execute commands, placing extremely high technical demands on users and resulting in low analysis efficiency. Moreover, traditional single-monitoring strategies cannot distinguish between container group survival and container group restart or crash scenarios, thus lacking targeted data acquisition methods for different scenarios.

[0030] Therefore, there is an urgent need for a fault diagnosis method based on container environments to solve the technical problems existing in traditional monitoring.

[0031] The following explanation is based on the accompanying diagram: Reference Appendix Figure 1 This illustrates an architecture diagram of a container-based fault diagnosis system provided in an embodiment of this application. Figure 1As shown, it includes: a monitoring and alarm system, an intelligent workflow orchestration engine, Agent1 (or first agent), Agent2 (or second agent), Agent3 (or third agent), Agent4 (or fourth agent), a large language model, and a memory (attached). Figure 1 This shows high-speed temporary storage.

[0032] The monitoring and alarm system continuously monitors preset monitoring metrics and issues alarms when these metrics meet preset conditions. Simultaneously, it sends a Webhook request to the intelligent workflow orchestration engine, carrying alarm information such as the container group name, prom_ql, and trigger_value. The intelligent workflow orchestration engine is an automated process control system integrating large language model capabilities, used to connect alarm reception, logical judgment, and data summarization.

[0033] Agent1 is used to collect metadata (Events) from the container orchestration system and the real-time status of container groups. Agent2 is used to collect process-level performance data (or field data) and write it to storage as a snapshot. Agent3 is used for summary analysis. Agent4 is used to pull historical trend data (or Prometheus historical trend data).

[0034] Large language models can be established models available on the market, and this application does not restrict the type of large language model.

[0035] The memory can be volatile memory, including but not limited to memory with memory. Figure 1 High-speed temporary storage, random access memory (RAM). RAM includes, for example, static random-access memory (SRAM) or dynamic random-access memory (DRAM). DRAM includes, for example, synchronous dynamic RAM (SDRAM) or double-data-rate SDRAM (DDR).

[0036] The memory can also be non-volatile memory (NVM), such as, but not limited to, semiconductor non-volatile memory, disk storage, or optical storage. Semiconductor non-volatile memory includes, but is not limited to, read-only memory (ROM) or flash memory, such as mask ROM, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), NAND flash memory, or NOR flash memory.

[0037] Reference Appendix Figure 2 The document illustrates a flowchart of a fault diagnosis method based on a container environment provided in an embodiment of this application, as shown below. Figure 2 As shown, this method is applied to the aforementioned fault diagnosis system, including: The S200 monitors preset monitoring indicators and triggers alarm information based on the status of the monitoring indicators.

[0038] S210, based on the alarm information, determine the real-time status of the target container group, the number of restarts, and the reason for termination.

[0039] S220, based on real-time status and number of restarts, executes either the first working mode or the second working mode based on the number of restarts and the termination reason. The first working mode collects process-level performance data and caches it in memory while the target container group is alive. The second working mode reads process-level performance data from memory and performs replay analysis when the target container group restarts or crashes. The monitored metrics include CPU utilization and / or memory usage.

[0040] The monitoring and alarm system continuously monitors preset monitoring metrics and triggers alarm messages based on the status of these metrics (e.g., when a monitoring metric exceeds an alarm threshold for a continuous preset period). Simultaneously, it sends a Webhook request carrying the alarm information to the intelligent workflow orchestration engine. The intelligent workflow orchestration engine uses a large language model to perform semantic analysis on the alarm information, thereby extracting the real-time status, restart count, and termination reason of the target container group corresponding to the alarm. Additionally, it can extract information such as service name, namespace, and fault level from the alarm information.

[0041] The intelligent workflow orchestration engine determines whether to execute a first working mode (or real-time online diagnostic mode) or a second working mode (or post-mortem analysis mode) based on the real-time status, number of restarts, and termination reasons of the target container group. For example, when the real-time status indicates that the target container group is running and the number of restarts is a first preset value (i.e., the target container group is alive at this time), the first working mode is executed. In this case, Agent2 collects process-level performance data (e.g., Perf / Profile) and writes the collected process-level performance data into memory in the form of a snapshot, thereby leaving evidence for possible future crashes or restarts.

[0042] For example, when the number of restarts reaches the second preset value and the termination reason is memory overflow (OOMKilled) or crash loop (CrashLoopBackOff) (i.e., the target container group restarts or crashes at this time), the second working mode is executed. At this time, the intelligent workflow orchestration engine reads process-level performance data in snapshot form from the memory through Agent3 and performs replay analysis. The first and second preset values ​​can be set according to the actual situation. For example, the first preset value can be 0, and the second preset value can be any value greater than the first preset value. This application does not limit this.

[0043] This application embodiment dynamically selects either a first or second working mode after triggering an alarm through real-time monitoring metrics, combined with intelligent judgment of real-time status, restart count, and termination reason. The first working mode is used to collect cached process-level performance data when the container is alive, while the second working mode is used to read and analyze cached data when the container restarts or crashes, thereby achieving rapid response to different scenarios. This method ensures real-time collection of performance data of the running container group, guarantees that critical data is not lost at the moment of failure, effectively restores the code-level root cause at the moment of crash, and also improves the success rate of container environment fault diagnosis.

[0044] Reference Appendix Figure 3 This illustrates a flowchart of a real-time example for determining a repair result, as provided in this application. Figure 3 As shown, it includes: S300 collects historical trend data and metadata for container orchestration systems.

[0045] S310 generates intermediate reports based on metadata, real-time status, historical trend data, and process-level performance data.

[0046] S320 analyzes the CPU hotspots and memory leak trends corresponding to the intermediate state report based on the large language model and outputs the repair results.

[0047] Agent1 collects metadata from the container orchestration system, while Agent4 collects historical trend data. The metadata, real-time status, historical trend data, and process-level performance data from the container orchestration system are aggregated into Agent3, generating an intermediate report containing JSON data and raw logs. The intelligent workflow orchestration engine sends the intermediate report returned by Agent3 to the large language model. The large language model analyzes the CPU hotspots and memory leak trends corresponding to the intermediate report and outputs remediation results, thereby generating a Markdown format report. Finally, the Markdown format report is pushed to the relevant development groups via DingTalk.

[0048] This application generates intermediate reports by collecting historical trend data and metadata from the container orchestration system, combined with real-time status and process-level performance data. Based on this, a large language model is used to perform in-depth analysis and reasoning on the CPU hotspots and memory leak trends corresponding to the intermediate reports. This not only accurately reveals the root causes of anomalies but also directly outputs targeted remediation results, significantly improving the automated diagnostic capabilities of container environments in complex fault scenarios. Furthermore, this application eliminates the need for manual command execution by the user, reducing technical requirements and improving analysis efficiency.

[0049] In one embodiment of this application, the method further includes: collecting system-level profiling data, application-level performance profiling data, timing monitoring data, and metadata of the container orchestration system; and automatically aligning and correlating the system-level profiling data, application-level performance profiling data, timing monitoring data, and metadata of the container orchestration system using container group IDs and time windows.

[0050] This application embodiment uses container group ID and time window to automatically align and correlate system-level profiling data (Perf), application-level performance profiling data (or application-level profiling), time-series monitoring data (e.g., Prometheus / CMS), and metadata of container orchestration systems, enabling automated correlation analysis of multi-source heterogeneous data.

[0051] Reference Appendix Figure 4 This diagram illustrates a structural block diagram of a fault diagnosis device based on a container environment, as provided in an embodiment of this application. Figure 4As shown, the device 400 includes: a monitoring module 410 configured to monitor preset monitoring indicators and trigger alarm information based on the status of the monitoring indicators; and a processing module 420 configured to: determine the real-time status, number of restarts, and termination reason of the target container group based on the alarm information; execute a first working mode based on the real-time status and number of restarts, or execute a second working mode based on the number of restarts and termination reason; the first working mode is used to collect process-level performance data and cache it in memory when the target container group is alive; the second working mode is used to read process-level performance data from memory and perform replay analysis when the target container group restarts or crashes; wherein, the monitoring indicators include CPU utilization and / or memory usage.

[0052] This application embodiment dynamically selects either a first or second working mode after triggering an alarm through real-time monitoring metrics, combined with intelligent judgment of real-time status, restart count, and termination reason. The first working mode is used to collect cached process-level performance data when the container is alive, while the second working mode is used to read and analyze cached data when the container restarts or crashes, thereby achieving rapid response to different scenarios. This method ensures real-time collection of performance data of the running container group, guarantees that critical data is not lost at the moment of failure, effectively restores the code-level root cause at the moment of crash, and also improves the success rate of container environment fault diagnosis.

[0053] In one embodiment of this application, the processing module 420 is configured to: execute a first working mode when the real-time status indicates that the target container group is in a running state and the number of restarts is a first preset value; and execute a second working mode when the number of restarts is a second preset value and the termination reason is memory overflow or crash loop; wherein the first preset value is less than the second preset value.

[0054] In one embodiment of this application, the appendix Figure 4 It also includes: a collection module 430, configured to collect system-level profiling data, application-level performance profiling data, time-series monitoring data, and metadata of the container orchestration system; and a processing module 420, configured to automatically align and correlate the system-level profiling data, application-level performance profiling data, time-series monitoring data, and metadata of the container orchestration system using container group IDs and time windows.

[0055] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the fault diagnosis method based on a container environment according to any of the above embodiments.

[0056] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the fault diagnosis method based on a container environment according to any of the above embodiments.

[0057] It should be noted that the above embodiments can be freely combined as needed. The above are merely preferred embodiments of the present invention. It should be pointed out that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A fault diagnosis method based on a container environment, characterized in that, include: Monitor preset monitoring indicators and trigger alarm information based on the status of the monitoring indicators; Based on the alarm information, determine the real-time status of the target container group, the number of restarts, and the reason for termination; Based on the real-time status and the number of restarts, execute the first working mode, or based on the number of restarts and the termination reason, execute the second working mode; The first working mode is used to collect process-level performance data and cache it in memory while the target container group is alive; The second working mode is used to read the process-level performance data from the memory and perform replay analysis when the target container group restarts or crashes; The monitoring metrics include CPU utilization and / or memory usage.

2. The fault diagnosis method based on a container environment according to claim 1, characterized in that, The step of executing a first working mode based on the real-time status and the number of restarts, or executing a second working mode based on the number of restarts and the termination reason, specifically includes: When the real-time status indicates that the target container group is in a running state, and the number of restarts is a first preset value, the first working mode is executed; When the number of restarts is a second preset value and the termination reason is memory overflow or crash loop, the second working mode is executed. Wherein, the first preset value is less than the second preset value.

3. The fault diagnosis method based on a container environment according to claim 1, characterized in that, Also includes: Collect system-level profiling data, application-level performance profiling data, timing monitoring data, and metadata from container orchestration systems; The system-level profiling data, application-level performance profiling data, time-series monitoring data, and metadata of the container orchestration system are automatically aligned and correlated using container group IDs and time windows.

4. The fault diagnosis method based on a container environment according to claim 1, characterized in that, Also includes: Collect historical trend data and metadata of container orchestration systems; An intermediate report is generated based on the metadata, the real-time status, the historical trend data, and the process-level performance data. Based on the large language model, the CPU hotspots and memory leak trends corresponding to the intermediate state report are analyzed and repair results are output.

5. The fault diagnosis method based on a container environment according to any one of claims 1-4, characterized in that, The step of triggering alarm information based on the status of the monitoring indicators specifically includes: An alarm is triggered when the monitored indicator exceeds the alarm threshold for a continuous preset time period.

6. A fault diagnosis device based on a container environment, characterized in that, include: The monitoring module is configured to monitor preset monitoring indicators and trigger alarm information based on the status of the monitoring indicators. The processing module is configured to: determine the real-time status, number of restarts, and termination reason of the target container group based on alarm information; execute a first working mode based on the real-time status and the number of restarts, or execute a second working mode based on the number of restarts and the termination reason; the first working mode is used to collect process-level performance data and cache it in memory when the target container group is alive; the second working mode is used to read the process-level performance data from the memory and perform replay analysis when the target container group restarts or crashes; wherein, the monitoring indicators include CPU utilization and / or memory usage.

7. The fault diagnosis device based on a container environment according to claim 6, characterized in that, The processing module is configured to: execute a first working mode when the real-time status indicates that the target container group is in a running state and the number of restarts is a first preset value; and execute a second working mode when the number of restarts is a second preset value and the termination reason is memory overflow or crash loop; wherein the first preset value is less than the second preset value.

8. The fault diagnosis device based on a container environment according to claim 6, characterized in that, Also includes: The acquisition module is configured to collect system-level profiling data, application-level performance profiling data, time-series monitoring data, and metadata from container orchestration systems. The processing module is configured to automatically align and correlate the system-level profiling data, the application-level performance profiling data, the time-series monitoring data, and the metadata of the container orchestration system using container group IDs and time windows.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the fault diagnosis method based on the container environment as described in any one of claims 1-5.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the fault diagnosis method based on the container environment as described in any one of claims 1-5.

Images