Application fault recovery method, system and equipment based on large model and medium
By combining cluster event monitoring and large-scale model analysis, faults in the cloud-native environment are captured in real time, and the second-level discovery and automatic repair of faults are achieved, improving operation and maintenance efficiency and system stability.
Patent Information
- Application Number
- CN202510593588.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-09
- Publication Date
- 2025-08-01
AI Technical Summary
Traditional fault recovery solutions are inefficient in cloud-native environments, difficult to integrate multi-source heterogeneous data, manual retrieval is time-consuming and labor-intensive, and static recovery strategies lack dynamic adaptability, making it difficult to deal with complex failures.
The cluster event listening and polling detection are combined to capture operational status abnormalities in real time, analyze multi-dimensional data through large models, generate fault recovery tasks and automatically repair them.
The second-level discovery and hidden risk warning of faults have been realized, which has significantly improved the intelligence level of fault handling and operation and maintenance efficiency, and has reduced the need for manual intervention.
Smart Images
Figure CN120407334A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of operation and maintenance technology of cloud native environments, and more specifically relates to an application fault recovery method, system, device and medium based on a large model. Background Art
[0002] With the widespread adoption of cloud-native technologies, Kubernetes-based containerized deployment and microservices architectures have become the mainstream form of distributed applications. This architecture improves system scalability through elastic resource scheduling and service decomposition, but it also complicates the causes of failures. Problems such as resource contention, network fluctuations, and dependency conflicts frequently occur, and the impact of failures can quickly spread through the service call chain. The dynamic nature of application runtimes and the complexity of component interactions in cloud-native environments place higher demands on automated fault detection, location, and recovery capabilities.
[0003] Traditional fault recovery solutions rely on operational experience and pre-set rules, using monitoring tools to collect metrics and analyze logs to pinpoint issues. However, these approaches are inefficient at integrating heterogeneous data from multiple sources, making it difficult to correlate disparate information such as cluster events, container logs, and performance metrics. Manual search or keyword-matching analysis of complex error descriptions in unstructured logs is time-consuming and laborious, making fault tracing particularly difficult in microservice call chain scenarios. Static recovery strategies lack adaptability to dynamic fault scenarios, making it difficult to address individual issues such as resource surges and version incompatibilities, limiting recovery efficiency and success rates.
[0004] In recent years, the advantages of big models in multimodal data processing and semantic understanding have provided a new path to solving these problems. They can efficiently integrate multi-source data such as logs, events, and metrics, identify potential fault correlations through deep semantic analysis, and generate targeted action suggestions. However, the current technical solutions for applying big models to cloud-native fault recovery are still in the exploratory stage. Key aspects such as data collection standardization, model interaction mechanism design, and automated recovery strategy generation lack a systematic framework, making it difficult to meet the real-time and precision requirements of engineering implementation. Summary of the Invention
[0005] In response to the above problems, the purpose of the present invention is to provide an application fault recovery method, system, equipment and medium based on a large model, which solves the problems of difficult data integration, low analysis efficiency, and rigid policies in traditional solutions, provides systematic guarantees for the stable operation of cloud-native applications, and significantly improves the intelligence level of fault handling and operation and maintenance efficiency.
[0006] To achieve the above-mentioned purpose, the present invention is implemented through the following technical solutions: In a first aspect, an embodiment of the present application provides an application failure recovery method based on a large model, comprising: Continuously monitor the running status of the application by using cluster event listening or polling detection, and determine whether the running status of the application is abnormal; After detecting that the running status of the application is abnormal, create an exception tracking stream to comprehensively collect the running environment data and running log data of the faulty application; Sort out the collected running environment data and running log data, form a prompt word instruction and send it to the large model, and receive and save the fault cause and handling suggestions returned by the large model; According to the fault cause and handling suggestions analyzed by the large model, combined with the pre-set fault recovery processing strategy, generate a fault recovery task and try to repair the fault, and monitor the fault recovery process.
[0007] In an optional implementation manner, the continuously monitoring the running status of the application by using cluster event listening or polling detection and determining whether the running status of the application is abnormal includes: By configuring the event listening mechanism of Kubernetes, capture the abnormal events of the application running status in real time. When an abnormal event of the application running status is monitored, determine that the application running status is abnormal; The abnormal events of the application running status include: BackOff event, Unhealthy event.
[0008] In an optional implementation manner, the continuously monitoring the running status of the application by using cluster event listening or polling detection and determining whether the running status of the application is abnormal includes: Use a timed task or an automated script to periodically send a data query request to the application through an HTTP interface to verify whether the function of the application to obtain data from the database is normal; if it is not normal, determine that the application running status is abnormal; Use a timed task or an automated script to execute a storage speed measurement command in the application container to check whether the storage read and write performance of the application meets the standard; if it does not meet the standard, determine that the application running status is abnormal.
[0009] In an optional implementation manner, the creating an exception tracking stream to comprehensively collect the running environment data and running log data of the faulty application after detecting that the running status of the application is abnormal includes: Create an exception tracking stream through an automated tool, and collect the running status data of the container to which the faulty application belongs, the Metrics monitoring data of the faulty application, the console log data of the container to which the faulty application belongs, and the custom log data output during the running of the faulty application.
[0010] In an alternative embodiment, creating an exception tracking flow through an automated tool to collect the running status data of the container to which the faulty application belongs, the Metrics monitoring data of the faulty application, the console log data of the container to which the faulty application belongs, and the custom log data output during the runtime of the faulty application, including: Utilize the APIs provided by the operating system or container management tool to obtain in real-time the instant CPU usage data, memory usage data, disk I / O status data, and network status data of the container to which the faulty application belongs; Utilize a monitoring tool to periodically pull the monitoring endpoint data exposed by the faulty application, store it in a time series database, and obtain the Metrics monitoring data of the faulty application within a specified time period through a query statement. The Metrics monitoring data includes CPU usage data, memory usage data, disk I / O status data, and network status data; Collect the console log records output by the container to which the faulty application belongs through the log viewing interface provided by the container management platform; Collect the custom log data output during the runtime of the faulty application according to the log file path configured by the application.
[0011] In an alternative embodiment, organizing the collected running environment data and running log data to form a prompt instruction and sending it to a large model, receiving and saving the fault cause and handling suggestions returned by the large model, including: Organize the collected running environment data and running log data to form a prompt instruction; the prompt instruction includes: data source and content description information, result requirement description information, data format requirement information, and fault cause classification information; Interact with the large model through an API call and send the prompt instruction to the large model; Receive the fault cause and handling suggestions returned by the large model and save them to a database or file system.
[0012] In an alternative embodiment, generating a fault recovery task and attempting to repair the fault based on the fault cause and handling suggestions analyzed by the large model, and monitoring the fault recovery process, including: Based on the fault cause and handling suggestions provided by the large model, match the corresponding handling strategy from a pre-set fault recovery handling strategy library to generate a fault recovery task; the handling strategies include core adjustment strategy, memory capacity adjustment strategy, disk capacity adjustment strategy, and fault notification strategy; Suspend the monitoring of the application running status of the faulty application, execute the fault recovery task, and repair the application fault; After the failure recovery task is executed, restart the application running status monitoring. If the corresponding application running status monitoring shows normal, end the abnormal tracking flow. If the application running status is still abnormal, send an abnormal notification message to the mobile terminal of the preset operation and maintenance management personnel.
[0013] In a second aspect, an embodiment of the present application further provides an application failure recovery system based on a large model, including: An application running status monitoring module, configured to continuously monitor the application running status by using cluster event monitoring or polling detection, and determine whether the application running status is abnormal; A failure data collection module, configured to create an abnormal tracking flow after detecting that the application running status is abnormal, and comprehensively collect the running environment data and running log data of the failed application; A failure data analysis module, configured to format and classify the collected running environment data and running log data, form a prompt word instruction and send it to the large model, and receive and save the failure cause and handling suggestions returned by the large model; An application failure recovery module, configured to generate a failure recovery task and attempt to repair the failure according to the failure cause and handling suggestions analyzed by the large model, in combination with the preset failure recovery processing strategy, and monitor the failure recovery process.
[0014] In a third aspect, an embodiment of the present application further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the application failure recovery method based on a large model as described in any one of the above are implemented.
[0015] In a fourth aspect, an embodiment of the present application further provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the application failure recovery method based on a large model as described in any one of the above are implemented.
[0016] From the above technical solutions, it can be seen that the present invention has the following advantages: The large-model-based application fault recovery method provided in this application builds an intelligent, fully closed-loop application fault self-healing system. Through a composite detection mechanism combining active monitoring and passive detection, it captures container lifecycle anomalies and the health status of dependent components in real time. Combined with multi-dimensional operational data collection, it forms a complete fault context portrait. Leveraging the natural language processing capabilities of the large model, the system can perform in-depth correlation analysis of container resource metrics, application monitoring data, console logs, and business logs to accurately locate the root cause of the fault and generate remediation recommendations. This complete closed-loop process of "intelligent diagnosis - policy matching - automatic remediation - effect verification" enables self-repair of most faults through standardized operations such as parameter tuning and resource elastic scaling, while also providing practical solutions for complex scenarios. This innovative model, which deeply integrates artificial intelligence with operational knowledge, upgrades the operational paradigm from fault response to preventive maintenance. While ensuring business continuity, it significantly reduces manual troubleshooting costs and enhances system self-healing capabilities, providing an intelligent solution for application stability management in cloud-native environments.
[0017] This application builds a three-dimensional detection system by integrating Kubernetes native event monitoring and active detection mechanisms. It can not only capture container lifecycle anomalies in real time, but also actively verify key indicators such as database connectivity and storage performance, achieving fault detection in seconds and hidden risk warnings.
[0018] This application is based on automated tracking flow technology, and synchronously collects four types of data: container resource indicators, application monitoring data, console logs, and business logs to form a complete fault operation portrait, avoiding the blind spots of traditional single-dimensional troubleshooting and providing a data foundation for accurate root cause analysis.
[0019] This application combines the natural language processing capabilities of large models to perform correlation analysis on multi-source data, automatically generate fault causes and repair suggestions, and match the preset strategy library to perform standardized operations such as parameter tuning and resource scaling, realizing automated fault repair and forming a complete closed loop of "diagnosis-decision-execution-verification".
[0020] This application upgrades the traditional passive response operation and maintenance to an active prevention system. Through continuous monitoring and intelligent analysis, it can identify potential risks such as dependent component anomalies and resource bottlenecks in advance, significantly reduce the probability of business interruption, and improve the system's self-healing capabilities.
[0021] This application reduces the need for manual intervention through intelligent means, reduces the average repair time for faults, and ensures business continuity, allowing the operation and maintenance team to focus on high-value optimization work and achieve the dual value of reducing costs and increasing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] To more clearly illustrate the technical solution of the present invention, the accompanying drawings required for description will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other accompanying drawings can be obtained based on these drawings without creative efforts.
[0023] Figure 1 It is a schematic flowchart of the application fault recovery method based on a large model provided by this application.
[0024] Figure 2 It is a schematic structural diagram of the application fault recovery system based on a large model provided by this application.
[0025] Figure 3 It is a schematic structural diagram of the electronic device provided by this application. Specific embodiments
[0026] In the following detailed description of the specific steps of the application fault recovery method based on a large model, various embodiments of the present disclosure will be more comprehensively described. The present disclosure may have various embodiments and adjustments and changes can be made therein. However, it should be understood that there is no intention to limit the various embodiments of the present disclosure to the specific embodiments disclosed herein, but the present disclosure should be understood to cover all adjustments, equivalents, and / or alternative solutions falling within the spirit and scope of the various embodiments of the present disclosure.
[0027] In the following, the term "including" or "may include" that can be used in various embodiments of the present disclosure indicates the presence of the disclosed functions, operations, or elements, and does not limit the addition of one or more functions, operations, or elements. In addition, as used in various embodiments of the present disclosure, the terms "including", "having" and their cognates are only intended to indicate specific features, numbers, steps, operations, elements, components, or combinations of the foregoing items, and should not be construed as precluding the existence or addition of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing items first.
[0028] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the protection scope of the present invention.
[0029] Please refer to Figure 1 The following is a method flowchart of an application fault recovery method based on a large model in a specific embodiment. The method includes: S1: Continuously monitor the application running status by using cluster event listening or polling detection, and determine whether the application running status is abnormal.
[0030] In the specific implementation, the following two methods can be used to monitor the application running status. Specifically as follows: By configuring the event listening mechanism of Kubernetes, capture the application running status events in real time, such as BackOff event (indicating that the container continuously retries after startup failure), Unhealthy event (indicating that the application health check fails), etc. When these abnormal events are detected, it can be determined that there is a problem with the application running status, and the fault analysis and recovery process is immediately started.
[0031] Use a scheduled task or an automated script to periodically detect the functions of the application. For example, send a data query request to the application through an HTTP interface to verify whether the function of the application to obtain data from the database is normal; execute a storage speed measurement command in the application container to check whether the storage read and write performance of the application meets the standard. If the function detection is abnormal, it is also regarded as the application running status being abnormal. When the application running status is detected as abnormal by any one of the above two methods, the application fault analysis and fault recovery process is immediately triggered.
[0032] S2: After detecting that the application running status is abnormal, create an exception tracking stream to comprehensively collect the running environment data and running log data of the faulty application.
[0033] In the specific implementation, create an exception tracking stream through an automated tool to collect the running status data of the container to which the faulty application belongs, the Metrics monitoring data of the faulty application, the console log data of the container to which the faulty application belongs, and the custom log data output during the running of the faulty application.
[0034] Exemplary: Use the APIs provided by the operating system or container management tools to obtain the instant CPU usage, memory usage, disk I / O status, network status, etc. data of the container to which the faulty application belongs in real time. For example, in the Linux system, relevant information can be obtained through commands such as top, free, iostat, netstat or the corresponding system call interfaces.
[0035] With the help of monitoring tools such as Prometheus, query the Metrics monitoring data of the application in the past period of time, covering indicators such as CPU usage, memory usage, disk I / O status, network status, etc. Prometheus stores the data obtained by periodically pulling the monitoring endpoints exposed by the application in a time series database, and relevant data within a specified time period can be obtained through PromQL query statements.
[0036] Since the console logs output by the container to which the application belongs record the key information and error prompts during the application running process. Through the log viewing interface provided by the container management platform, such as the docker logs command of Docker or the kubectl logs command of Kubernetes, obtain the console log records output by the container to which the faulty application belongs.
[0037] The custom log files output during the application running process contain the detailed running information of specific application functions. Therefore, in this step, according to the log file path configured by the application, through file system operations, collect the custom log data output during the faulty application running process. For example, use the cp command to copy the log file to the specified storage location.
[0038] S3: Organize the collected running environment data and running log data to form a prompt instruction and send it to the large model, receive and save the fault cause and handling suggestions returned by the large model.
[0039] In the specific implementation manner, first, preliminarily organize the collected running environment data and running log data, and form a Prompt instruction applicable to the large model. Then, send a data analysis instruction to the large model, receive the processing result returned by the large model, and save the returned data.
[0040] For example, first organize the collected running environment data and running log data to form a prompt instruction; the prompt instruction includes: data source and content description information, result requirement description information, data format requirement information, and fault cause classification information. Then, interact with the large model through API calls, send the prompt instruction to the large model, wait for the large model to analyze and process the data, and receive the processing result returned by the large model. Finally, receive the fault cause and handling suggestions returned by the large model, and save them to the database or file system for subsequent fault recovery operations.
[0041] It should be particularly noted that the following content is described in the Prompt instruction applicable to the large model: Source and content description: Clearly state the source of the collected data in the Prompt instruction, such as from Kubernetes cluster events, Prometheus monitoring data, container console logs, etc.; list in detail the specific content of the data that needs to be analyzed by the large model, including various running environment indicators and log information.
[0042] Result requirement description: Clearly inform the large model that the result to be analyzed is the cause of the application fault and the corresponding handling suggestions.
[0043] Data format requirements: Specify that the large model returns the analysis results in a specific data format, such as JSON format, to facilitate subsequent parsing and processing of the results.
[0044] Fault cause classification: Require the large model to classify the analyzed fault causes according to dimensions such as "CPU shortage", "memory shortage", "disk space shortage", etc. It should be emphasized that these classification bases are not fixed and can be adjusted according to actual usage. The reason for classification is to facilitate presetting corresponding processing strategies for the problems that occur. Therefore, the classification dimensions can be flexibly adjusted according to actual needs to facilitate presetting corresponding processing strategies.
[0045] S4: According to the fault causes and handling suggestions obtained from the analysis of the large model, combined with the preset fault recovery processing strategies, generate a fault recovery task and attempt to repair the fault, and monitor the fault recovery process.
[0046] In the specific implementation, first, according to the fault causes and handling suggestions provided by the large model, match the corresponding processing strategies from the preset fault recovery processing strategy library to generate a fault recovery task. Then, pause the monitoring of the application running status of the faulty application and execute the fault recovery task to repair the application fault.
[0047] When the fault recovery task is completed, restart the monitoring of the application running status; if the corresponding application running status monitoring shows normal, end the abnormal tracking flow; if the application running status is still abnormal, send an abnormal notification message to the mobile terminal of the preset operation and maintenance management personnel. For example, during the process of generating a fault recovery task and attempting to recover the fault, the fault monitoring of the application is in a paused state until the fault recovery task is completed, and then the monitoring of the application running status will start again. If the application status monitoring status is normal, end the abnormal tracking flow and the process ends.
[0048] Among them, the processing strategies include core adjustment strategy, memory capacity adjustment strategy, disk capacity adjustment strategy, and fault notification strategy. The preset fault recovery processing strategies can be continuously supplemented or modified and maintained according to needs. Examples are as follows: Core adjustment strategy: When the CPU is insufficient, adjust the number of CPU cores by +2, with an upper limit of 8 cores.
[0049] Memory capacity adjustment strategy: When the memory is insufficient, adjust the memory capacity by +2G, with an upper limit of 16G.
[0050] Disk capacity adjustment strategy: When the disk capacity is insufficient, adjust the disk capacity by +20G, with an upper limit of 1T.
[0051] Fault notification strategy: When an unknown problem occurs, notify the operation and maintenance management personnel.
[0052] In this embodiment, an intelligent operation and maintenance system is constructed through large model technology. By combining real-time cluster status monitoring and multi-dimensional data correlation analysis, rapid and accurate handling of faults and proactive prevention of risks are achieved. Ultimately, the system recovery time is significantly shortened, the operation and maintenance manpower input is reduced, and the overall application stability in the cloud-native environment is promoted.
[0053] As Figure 2 shown, the following is an embodiment of an application fault recovery system based on a large model provided by an embodiment of the present disclosure. This system and the application fault recovery method based on a large model in the above embodiments belong to the same inventive concept. Details not described in detail in the embodiment of the application fault recovery system based on a large model can refer to the embodiment of the application fault recovery method based on a large model.
[0054] An application fault recovery system based on a large model includes: an application running status monitoring module, a fault data collection module, a fault data analysis module, and an application fault recovery module.
[0055] The application running status monitoring module is used to continuously monitor the application running status by using cluster event monitoring or polling detection, and determine whether the application running status is abnormal.
[0056] The fault data collection module is used to create an exception tracking stream after detecting that the application running status is abnormal, and comprehensively collect the running environment data and running log data of the faulty application.
[0057] The fault data analysis module is used to format and classify the collected running environment data and running log data, form a prompt word instruction and send it to the large model, and receive and save the fault cause and handling suggestions returned by the large model.
[0058] The application fault recovery module is used to generate a fault recovery task and attempt to repair the fault according to the fault cause and handling suggestions analyzed by the large model, combined with the pre-set fault recovery processing strategy, and monitor the fault recovery process.
[0059] The application fault recovery system provided in this embodiment constructs an intelligent operation and maintenance system based on large model technology. It realizes second-level fault perception through real-time monitoring of cluster events and active detection of component status, accurately locates the root cause through in-depth correlation analysis of multi-dimensional operation data, and automatically triggers repair strategies such as parameter tuning and resource scaling. This solution significantly shortens the fault recovery cycle, greatly reduces the manual troubleshooting cost, and at the same time realizes proactive risk defense through the prediction of the health status of dependent components and elastic strategies, promoting the upgrade of the operation and maintenance mode from passive response to intelligent self-healing, and providing high availability guarantee for cloud-native applications.
[0060] Figure 3Schematic diagram of the hardware structure of an electronic device for implementing various embodiments of the present invention.
[0061] The application fault recovery method based on a large model provided by the embodiments of the present application can be applied to an electronic device. Those skilled in the art can understand that the structure of the electronic device involved in the embodiments of the present invention does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements. In the embodiments of the present invention, the electronic device includes, but is not limited to, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as, a personal digital processor, a cellular phone, a smart phone, a wearable device, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of the present application described herein and / or claimed.
[0062] The electronic device may include a processor, an external memory interface, an internal memory, a universal serial bus (USB) interface, a charging management module, a power management module, a battery, a wireless communication module, an audio module, a speaker, a microphone, a sensor module, a key, a camera, a display screen, and a SIM card interface, etc.
[0063] The processor may include one or more processing units. For example, the processor may include a central processing unit (CPU), etc., an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.
[0064] Among them, the processor may be the nerve center and command center of the electronic device. The controller may generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching instructions and executing instructions.
[0065] A memory can also be set in the processor for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. This memory can hold the instructions or data that the processor has just used or recycled. If the processor needs to use the instruction or data again, it can directly call it from this memory. This avoids repeated accesses, reduces the waiting time of the processor, and thus improves the system efficiency.
[0066] The external memory interface can be used to connect an external memory card, such as a MicroSD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor through the external memory interface to implement the data storage function. For example, files such as music and videos are saved in the external memory card.
[0067] The internal memory can be used to store computer-executable program codes, and the computer-executable program codes include instructions. The processor executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory. The internal memory can include a program storage area and a data storage area. The internal memory can include a high-speed random access memory and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.
[0068] The wireless communication function of the electronic device can be implemented through an antenna, a wireless communication module, a modulation and demodulation processor, a baseband processor, etc.
[0069] The wireless communication module can provide solutions for wireless communications applied to the electronic device, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite systems (GNSSs), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc.
[0070] The electronic device can implement audio functions, etc. through an audio module, a speaker, a receiver, a microphone, a headphone interface, an application processor, etc.
[0071] The electronic device can implement a shooting function through an ISP, a camera, a video codec, a GPU, a display screen, an application processor, etc.
[0072] An electronic device can implement a display function through a GPU, a display screen, an application processor, etc.
[0073] The GPU is a microprocessor for image processing, connecting the display screen and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor may include one or more GPUs, which execute program instructions to generate or change display information.
[0074] The display screen is used to display images, videos, etc. The display screen includes a display panel.
[0075] The above electronic device implements the large model-driven intelligent operation and maintenance system of the application fault recovery method based on the large model in this application. Combining real-time cluster event monitoring, active component detection, and multi-dimensional operation data correlation analysis, it realizes the rapid perception of faults and the accurate root cause location, and automatically triggers repair strategies, achieving the beneficial effects of shortening the fault recovery cycle, reducing the cost of manual intervention, building the ability of active risk defense, promoting the intelligent upgrade of the operation and maintenance mode, and improving the availability of cloud-native applications.
[0076] In the storage medium provided in this application, there is a program product that can implement the application fault recovery method based on the large model.
[0077] The application fault recovery method based on the large model includes: Continuously monitor the application running status by using cluster event monitoring or polling detection. Once an abnormal status is detected, start the fault analysis and recovery process; After detecting the abnormal application running status, create an abnormal tracking stream to comprehensively collect the running environment data and running log data of the faulty application; Sort out the collected running environment data and running log data, form a prompt instruction and send it to the large model, and receive and save the fault cause and handling suggestions returned by the large model; According to the fault cause and handling suggestions analyzed by the large model, combined with the pre-set fault recovery processing strategy, generate a fault recovery task and attempt to repair the fault, and monitor the fault recovery process. In some possible implementation manners, the application fault recovery method based on the large model of the present disclosure may be implemented in the form of a program product, which includes program code, when the program product runs on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the "Exemplary Method" section of this specification.
[0078] The storage medium of the present disclosure may employ any combination of one or more readable media. The readable media may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0079] The foregoing description of the disclosed embodiments enables those skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. An application fault recovery method based on a large model, characterized in that, Including: Continuously monitor the running status of the application by using cluster event listening or polling detection, and determine whether the running status of the application is abnormal; After detecting that the running status of the application is abnormal, create an abnormal tracking stream to comprehensively collect the running environment data and running log data of the faulty application; Sort out the collected running environment data and running log data, form a prompt instruction and send it to the large model, and receive and save the fault cause and handling suggestions returned by the large model; According to the fault cause and handling suggestions analyzed by the large model, combined with the pre-set fault recovery processing strategy, generate a fault recovery task and attempt to repair the fault, and monitor the fault recovery process.
2. The application fault recovery method based on a large model according to claim 1, wherein The continuously monitoring the running status of the application by using cluster event listening or polling detection and determining whether the running status of the application is abnormal includes: By configuring the event listening mechanism of Kubernetes, capture the abnormal events of the running status of the application in real time. When an abnormal event of the running status of the application is detected, determine that the running status of the application is abnormal; The abnormal events of the running status of the application include: BackOff event, Unhealthy event.
3. The application fault recovery method based on a large model according to claim 1, wherein, The continuously monitoring the running status of the application by using cluster event listening or polling detection and determining whether the running status of the application is abnormal includes: Use a scheduled task or an automated script to periodically send a data query request to the application through the HTTP interface to verify whether the function of the application to obtain data from the database is normal; if it is not normal, determine that the running status of the application is abnormal; Use a scheduled task or an automated script to execute a storage speed measurement command in the application container to check whether the storage read and write performance of the application meets the standard; if it does not meet the standard, determine that the running status of the application is abnormal.
4. The application fault recovery method based on a large model according to claim 1, wherein The creating an abnormal tracking stream to comprehensively collect the running environment data and running log data of the faulty application after detecting that the running status of the application is abnormal includes: Create an abnormal tracking stream through an automated tool to collect the running status data of the container to which the faulty application belongs, the Metrics monitoring data of the faulty application, the console log data of the container to which the faulty application belongs, and the custom log data output during the running of the faulty application.
5. The application fault recovery method based on a large model according to claim 4, characterized in that, The creating an abnormal tracking stream through an automated tool to collect the running status data of the container to which the faulty application belongs, the Metrics monitoring data of the faulty application, the console log data of the container to which the faulty application belongs, and the custom log data output during the running of the faulty application includes: Use the API provided by the operating system or container management tool to obtain the instant CPU usage data, memory usage data, disk IO status data, and network status data of the container to which the faulty application belongs in real time; Use a monitoring tool to periodically pull the monitoring endpoint data exposed by the faulty application, store it in a time series database, and obtain the Metrics monitoring data of the faulty application within a specified time period through a query statement. The Metrics monitoring data includes CPU usage data, memory usage data, disk IO status data, and network status data; Collect the console log records output by the container to which the faulty application belongs through the log viewing interface provided by the container management platform; Collect custom log data output during the runtime of a faulty application based on the log file path configured for the application.
6. The application fault recovery method based on a large model according to claim 5, wherein Organize the collected runtime environment data and runtime log data to form a prompt instruction and send it to the large model, and receive and save the fault cause and handling suggestions returned by the large model, including: Organize the collected runtime environment data and runtime log data to form a prompt instruction; the prompt instruction includes: data source and content description information, result requirement description information, data format requirement information, and fault cause classification information; Interact with the large model through API calls and send the prompt instruction to the large model; Receive the fault cause and handling suggestions returned by the large model and save them in the database or file system.
7. The application fault recovery method based on a large model according to claim 6, wherein Based on the fault cause and handling suggestions analyzed by the large model, combine with the pre-set fault recovery processing strategy to generate a fault recovery task and attempt to repair the fault, and monitor the fault recovery process, including: Based on the fault cause and handling suggestions provided by the large model, match the corresponding processing strategy from the pre-set fault recovery processing strategy library to generate a fault recovery task; the processing strategy includes core adjustment strategy, memory capacity adjustment strategy, disk capacity adjustment strategy, and fault notification strategy; Suspend the monitoring of the application runtime status of the faulty application, execute the fault recovery task, and repair the application fault; After the fault recovery task is completed, restart the monitoring of the application runtime status; if the corresponding application runtime status monitoring shows normal, end the abnormal tracking flow; if the application runtime status is still abnormal, send an abnormal notification message to the mobile terminal of the preset operation and maintenance management personnel.
8. An application fault recovery system based on a large model, characterized in that, The system adopts the large model-based application fault recovery method described in any one of claims 1 to 7; The system includes: An application runtime status monitoring module for continuously monitoring the application runtime status by using cluster event monitoring or polling detection to determine whether the application runtime status is abnormal; A fault data collection module for creating an abnormal tracking flow and comprehensively collecting the runtime environment data and runtime log data of the faulty application after detecting that the application runtime status is abnormal; A fault data analysis module for formatting and classifying the collected runtime environment data and runtime log data to form a prompt instruction and sending it to the large model, and receiving and saving the fault cause and handling suggestions returned by the large model; An application fault recovery module for generating a fault recovery task and attempting to repair the fault based on the fault cause and handling suggestions analyzed by the large model, and combining with the pre-set fault recovery processing strategy, and monitoring the fault recovery process.
9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, When the processor executes the program, it implements the steps of the large model-based application fault recovery method described in any one of claims 1 to 7.
10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the large model-based application fault recovery method described in any one of claims 1 to 7.