System fault emergency processing method, device, equipment, medium and product

Abstract

The invention provides a system fault emergency processing method and device, equipment, a medium and a product. The method comprises the following steps: firstly, collecting application state data of an application system; wherein the application state data comprises basic environment data; judging whether the application system has a fault based on the application state data, and if the application system has the fault, generating fault information; a knowledge vector library is retrieved according to the fault information, cue words with fault context information and associated emergency strategies are constructed, and the cue words are sent to the large model; the large model is used for generating an emergency strategy according to prompt words; emergency processing is initiated according to an emergency strategy generated by the large model, and a processing result is generated; wherein the processing result is a scalar value of a combination of the system success rate, the transaction delay rate, the transaction processing amount per second and the transaction processing failure amount per second.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a method, device, equipment, medium and product for emergency handling of system failures. Background Art

[0002] With the rapid development of departmental business lines and the widespread application of distributed technologies, data centers have begun to undertake more and more operation and maintenance work for cross-regional multi-cluster application systems. Faced with the increasing number of heterogeneous systems and the continuous expansion of the scale of the basic environment, when a system fails, it is difficult to quickly determine the source and cause of the failure.

[0003] The current method for emergency decision-making for application system failures uses the expert experience-knowledge graph method. The monitoring platform generates an application system failure alarm, analyzes the failure phenomenon manually or semi-automatically, and makes emergency strategy decisions based on historical expert experience or knowledge graphs. However, this method cannot achieve automated emergency response in the event of an application system failure. Summary of the invention

[0004] The embodiments of the present application provide a method, device, equipment, medium and product for emergency handling of system failures, so as to achieve the effect of automated emergency handling in the event of an application system failure.

[0005] In a first aspect, an embodiment of the present application provides a method for emergency handling of system faults, comprising: collecting application status data of an application system; wherein the application status data includes basic environment data; judging whether the application system is faulty based on the application status data, and generating fault information if the application system fails; retrieving a knowledge vector library based on the fault information, constructing prompt words with fault context information and associated emergency strategies, and sending the prompt words to a large model; the large model is used to generate emergency strategies based on the prompt words; initiating emergency handling based on the emergency strategies generated by the large model, and generating a processing result; wherein the processing result is a scalar value of a combination of system success rate, transaction delay rate, transaction processing volume per second, and transaction processing failure volume per second.

[0006] In some examples, after determining whether the application system fails based on the processed application status data, the method further includes: if the application system does not fail, inputting the processed application status data into a knowledge vector library to update the knowledge vector library.

[0007] In some examples, after emergency processing is initiated according to the emergency strategy generated by the large model and the processing result is generated, it also includes: calculating a reward value using an objective function according to fault information, emergency strategy, emergency processing and processing result; comparing the reward value and the scalar value; if the reward value is greater than the scalar value, storing the fault emergency information corresponding to the reward value in the knowledge vector library; wherein the fault emergency information includes fault information, emergency strategy and emergency processing.

[0008] In some examples, fault information, emergency strategies, and emergency processing are characterized and input into a knowledge vector library.

[0009] In some examples, the knowledge vector library is used to: parse, differentiate, and vectorize received application status data; and update the knowledge vector library in a dynamic update manner based on the processed data.

[0010] In some examples, the application status data also includes private domain knowledge; the knowledge vector library is also used to: embed fault information and private domain knowledge using a vector model, and convert the fault information and private domain knowledge into vector form.

[0011] In some examples, a knowledge vector library is retrieved based on fault information to construct prompt words with fault context information and associated emergency strategies, specifically including: using a retrieval enhancement generation framework to retrieve the knowledge vector library to obtain emergency-related knowledge information; mapping the emergency-related knowledge information with the knowledge vector library to obtain associated emergency strategies; and constructing prompt words with fault context information and associated emergency strategies based on the fault information and the associated emergency strategies.

[0012] In some examples, before determining whether an application system fails based on application status data, the process also includes: characterizing the application status data to obtain feature values ​​of different dimensions; and normalizing the feature values ​​of different dimensions.

[0013] In some examples, determining whether an application system is faulty is based on application status data, specifically including: comparing basic environment data in the processed application status data with a preset indicator range; if the indicator of the basic environment data exceeds the preset indicator range, the application system is faulty; if the indicator of the basic environment data does not exceed the preset indicator range, the application system is not faulty.

[0014] In a second aspect, an embodiment of the present application provides a device for emergency handling of system faults, comprising: a collection module for collecting application status data of an application system; wherein the application status data includes basic environment data; a judgment module for judging whether the application system is faulty based on the application status data, and generating fault information if the application system fails; a generation module for retrieving a knowledge vector library based on the fault information, constructing prompt words with fault context information and associated emergency strategies, and sending the prompt words to a large model; the large model is used to generate emergency strategies based on the prompt words; a processing module for initiating emergency handling based on the emergency strategies generated by the large model, and generating a processing result; wherein the processing result is a scalar value of a combination of system success rate, transaction delay rate, transaction processing volume per second, and transaction processing failure volume per second.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0016] Memory stores computer-executable instructions;

[0017] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0019] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0020] The embodiment of the present application provides a method, device, equipment, medium and product for emergency handling of system failures. First, application status data of the application system is collected; wherein the application status data includes basic environment data; based on the application status data, whether the application system fails is determined, and if the application system fails, fault information is generated; according to the fault information, a knowledge vector library is retrieved to construct prompt words with fault context information and associated emergency strategies, and the prompt words are sent to a large model; the large model is used to generate emergency strategies according to the prompt words; emergency handling is initiated according to the emergency strategies generated by the large model, and a processing result is generated; wherein the processing result is a scalar value of a combination of system success rate, transaction delay rate, transaction processing volume per second, and transaction processing failure volume per second. This solution determines the fault phenomenon of the application system by collecting application status data in real time, and retrieves the knowledge vector library to construct prompt words with fault context information and associated emergency strategies, so that the large model can generate more accurate emergency strategies and realize automated emergency handling. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0022] Figure 1 A schematic diagram of a method for emergency handling of system failures provided in this application Figure 1 ;

[0023] Figure 2 A schematic diagram of a method for emergency handling of system failures provided in this application Figure 2 ;

[0024] Figure 3 A flow chart of a method for emergency handling of system failures Figure 3 ;

[0025] Figure 4 A schematic diagram of the structure of a device for emergency treatment of system failures provided in this application;

[0026] Figure 5 A schematic diagram of the structure of an electronic device provided in this application.

[0027] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0028] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0029] The terms "first", "second", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise indicated. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, for example, they can be implemented in an order other than those given in the diagrams or descriptions of the embodiments of this application.

[0030] In addition, the terms "including" and "having" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a series of components is not necessarily limited to those components explicitly listed, but may include other components not explicitly listed or inherent to these products or devices. The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware and / or software code that can perform the functions associated with the element.

[0031] With the rapid development of departmental business lines and the widespread application of distributed technologies, data centers have begun to undertake more and more operation and maintenance work for cross-regional multi-cluster application systems. Faced with the increasing number of heterogeneous systems and the continuous expansion of the scale of the basic environment, when a system fails, it is difficult to quickly determine the source and cause of the failure.

[0032] The current method for emergency decision-making for application system failures uses the expert experience-knowledge graph method. The monitoring platform generates an application system failure alarm, analyzes the failure phenomenon manually or semi-automatically, and makes emergency strategy decisions based on historical expert experience or knowledge graphs. However, this method cannot achieve automated emergency response in the event of an application system failure.

[0033] The embodiment of the present application provides a method, device, equipment, medium and product for emergency handling of system failures. First, application status data of the application system is collected; wherein the application status data includes basic environment data; based on the application status data, whether the application system fails is determined, and if the application system fails, fault information is generated; according to the fault information, a knowledge vector library is retrieved to construct prompt words with fault context information and associated emergency strategies, and the prompt words are sent to a large model; the large model is used to generate emergency strategies according to the prompt words; emergency handling is initiated according to the emergency strategies generated by the large model, and a processing result is generated; wherein the processing result is a scalar value of a combination of system success rate, transaction delay rate, transaction processing volume per second, and transaction processing failure volume per second. This solution determines the fault phenomenon of the application system by collecting application status data in real time, and retrieves the knowledge vector library to construct prompt words with fault context information and associated emergency strategies, so that the large model can generate more accurate emergency strategies and realize automated emergency handling.

[0034] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0035] Embodiment 1

[0036] Figure 1A schematic diagram of a method for emergency handling of system failures provided in this application Figure 1 ,like Figure 1 As shown, the method includes:

[0037] S101, collecting application status data of the application system; wherein the application status data includes basic environment data;

[0038] S102, judging whether the application system fails based on the application status data, and generating failure information if the application system fails;

[0039] S103, searching the knowledge vector library according to the fault information, constructing prompt words with fault context information and associated emergency strategies, and sending the prompt words to the large model; the large model is used to generate emergency strategies according to the prompt words;

[0040] S104, initiating emergency processing according to the emergency strategy generated by the large model, and generating processing results; wherein the processing results are scalar values ​​of the combination of system success rate, transaction delay rate, transaction processing volume per second, and transaction processing failure volume per second.

[0041] Among them, in addition to basic environment data, application status data also includes: transaction data, application data, process data, action data; transaction data includes cluster system success rate, transaction delay rate, transaction tps, failed tps, etc.; among them, transaction tps means the transaction processing volume per second, and failed tps means the transaction processing failure volume per second; basic environment data includes cluster network traffic, access layer node status, business layer node status, database status, etc.; application data includes cluster change status, application exception log, application alarm, etc.; process data includes production plan change, downtime maintenance plan, external fault reporting data, etc.; action data includes emergency action timeliness, emergency action impact transaction number, etc. The application status data will be collected in real time, which can warn of failures in advance, and when the application system fails, the real-time collected data is like a "black box" record, which can help operation and maintenance personnel quickly determine the cause of the failure.

[0042] Basic environment data refers to the data of the application system and its infrastructure; environmental characteristics include the current characteristics and memory characteristics of the application system; among them, the current characteristics include system success rate, transaction delay rate, transaction tps, failed tps, network traffic, access layer node status, business layer node status, database status, change status and other application status; memory characteristics include historical action sequences and application status within a certain period of time.

[0043] Among them, emergency treatment is initiated according to the emergency strategy generated by the large model. Specifically, emergency actions are executed according to the emergency strategy generated by the large model to perform emergency treatment on the faulty application system. Emergency actions include isolating clusters, restarting virtual machines, rebuilding containers, F5 switching, etc. Executing the above emergency actions can achieve the effect of automatically handling faults and quickly restoring transactions. For example, the emergency action of executing F5 switching, specifically, when F5 detects that a backend server has a fault (for example, the server is unresponsive, returns an error status code, etc.), it will automatically switch the transaction traffic originally sent to the faulty server to other healthy servers according to the pre-configured load balancing algorithm and switching strategy; after F5 performs traffic switching, transactions can continue on other servers and will not be interrupted due to the failure of a certain server, thereby achieving rapid recovery of transactions.

[0044] When an application system changes from a normal state to a warning state or an abnormal state, the state of the application system is actually a time-series change process. Factors that cause the state of the application system to change include changes in network traffic and third-party service failures. In addition, emergency processing caused by emergency actions will also cause state changes. These changes may be positive, restoring the system to a normal state, or they may be complex, bringing new problems. For example, adding server resources may improve system performance in a short period of time, but if improperly configured, it may cause resource competition or compatibility issues, which in turn may cause new anomalies.

[0045] In actual applications, application status data of the application system is collected; based on the application status data, it is determined whether the application system is faulty. If the application system fails, fault information is generated; the knowledge vector library is retrieved based on the fault information, prompt words with fault context information and associated emergency strategies are constructed, and the prompt words are sent to the large model; the large model is used to generate emergency strategies based on the prompt words; emergency processing is initiated based on the emergency strategies generated by the large model, and processing results are generated; the processing results are scalar values ​​of the system success rate, transaction delay rate, transaction processing volume per second, and transaction processing failure volume per second. By collecting application status data in real time to determine the fault phenomenon of the application system, searching the knowledge vector library, and constructing prompt words with fault context information and associated emergency strategies, the large model can generate more accurate emergency strategies and realize automated emergency handling.

[0046] In some examples, after determining whether the application system fails based on the processed application status data, the method further includes: if the application system does not fail, inputting the processed application status data into a knowledge vector library to update the knowledge vector library.

[0047] The knowledge vector library refers to a structured knowledge vector library composed of the enterprise's internal application system emergency strategy, CMDB data, architecture relationship data, production change plan, historical event review, application alarm list, application monitoring data, etc. Among them, CMDB (Configuration Management Database) is a database that stores and manages various configuration items (CI - Configuration Item) information in the enterprise IT environment.

[0048] The knowledge vector library can integrate application status data with other related knowledge. For example, application status data includes changes in system performance indicators under different circumstances. After entering this data into the knowledge vector library, it can be associated with existing knowledge about system architecture, software module functions, hardware configuration, etc. In addition, the knowledge vector library uses vectors to represent knowledge, which facilitates efficient retrieval and can provide accurate knowledge recommendations for different application states based on the content of the knowledge vector library.

[0049] Application status data is updated in real time or periodically, and entering it into the knowledge vector library can keep the knowledge vector library up to date. As application systems continue to change and develop, new application status data is constantly generated, and these data can timely update the relevant knowledge vectors in the knowledge vector library to ensure that the knowledge vector library can reflect the latest system status and problems.

[0050] Figure 2 A schematic diagram of a method for emergency handling of system failures provided in this application Figure 2 ,like Figure 2 As shown, in some examples, after initiating emergency processing according to the emergency strategy generated by the large model and generating the processing result, it also includes:

[0051] S201, calculating a reward value using an objective function according to fault information, emergency strategy, emergency processing and processing results;

[0052] S202, comparing the reward value and the scalar value; if the reward value is greater than the scalar value, storing the fault emergency information corresponding to the reward value into the knowledge vector library; wherein the fault emergency information includes fault information, emergency strategy, and emergency processing.

[0053] Among them, the emergency strategy refers to the emergency response strategy for specific or general application failure scenarios; the reward refers to whether the application state brought about by the emergency action is restored to normal, which is defined as the problem scalar feedback signal given by the environment. In actual applications, the emergency actions executed are finite steps, and for different application system states, the emergency actions executed are continuous actions or discrete actions.

[0054] If the reward value is greater than the scalar value, it means that the emergency response initiated this time is positive. The fault emergency information corresponding to the reward value is stored in the knowledge vector library, which can continuously accumulate emergency strategy experience and improve the emergency response capability.

[0055] In this example, the reinforcement learning process is to use the objective function to calculate the reward value, compare the reward value with the scalar value, and store the fault emergency information corresponding to the reward value into the knowledge vector library.

[0056] In practical applications, for example, stable-baselines is used to implement reinforcement learning induction calculations for emergency strategies. Stable-Baselines is a set of high-quality, easy-to-use libraries for reinforcement learning. It encapsulates common reinforcement learning algorithms and provides a unified interface to facilitate developers to quickly build reinforcement learning models and apply them to actual scenarios. Using Stable-Baselines to implement reinforcement learning induction calculations for emergency strategies can help developers mine complex relationships in emergency response with its powerful algorithm encapsulation and easy-to-use features, learn more effective emergency strategies, and improve the scientificity and efficiency of emergency decision-making.

[0057] In some examples, fault information, emergency strategies, and emergency processing are characterized and input into a knowledge vector library.

[0058] After the emergency response is completed, in addition to reinforcement learning, the fault information, emergency strategy, and emergency response data will also be collected in real time, processed, and entered into the knowledge vector library. Specifically, the key features of this part of the data are extracted and entered into the knowledge vector library to expand the knowledge of the knowledge vector library to better serve fault handling. For example, after the key features of the emergency strategy are extracted and stored in the knowledge vector library, the library not only contains knowledge related to the fault information, but also stores the corresponding strategic knowledge for dealing with these faults. When it is necessary to find a strategy for dealing with a specific fault, the appropriate emergency strategy can be quickly obtained by associating and retrieving the fault information feature vector.

[0059] In some examples, the knowledge vector library is used to:

[0060] Parse, differentiate, and vectorize the received application status data;

[0061] Based on the processed data, the knowledge vector library is updated in a dynamic updating manner.

[0062] Among them, since application status data often has a variety of presentation forms, and the format of the application status data after characterization is not necessarily the same as the storage format of the knowledge vector library, the received application status data must be parsed, differentiated, and vectorized. Specifically, the received application status data must first be parsed to identify the key information elements contained therein; the parsed application status data must be classified and summarized according to different dimensions or categories; on the basis of classification, key feature information must be further extracted from each type of data; using specific mathematical models and algorithms, the differentiated and extracted feature information must be converted into vector form so that it can be represented in vector space.

[0063] The application system itself is in a state of continuous development and change. Its functions will be updated and iterated, user behavior patterns will change, and the operating environment may also change. Correspondingly, the application status data will also continue to change. In order to make the knowledge vector library always fit the actual situation of the application and accurately provide effective knowledge support for analyzing application status and solving application problems, it is necessary to adopt a dynamic update method to timely integrate new data information into the knowledge base.

[0064] In some examples, the application status data also includes private domain knowledge; the knowledge vector library is also used to: embed fault information and private domain knowledge using a vector model, and convert the fault information and private domain knowledge into vector form.

[0065] Private domain is a concept relative to public domain, and mainly refers to channels owned by brands or individuals, which can be freely controlled, are free, and can repeatedly reach users. Private domain knowledge covers a series of knowledge systems such as user operations, content creation, marketing promotion, etc. in these channels. For example, corporate communities, brand official apps, public accounts, etc. all belong to the private domain category. Knowledge about how to attract, retain, and convert users on these platforms is private domain knowledge.

[0066] Private domain knowledge is embedded and converted into vector form and stored in the knowledge vector library. Private domain data will not be uploaded to a third-party platform, which ensures better local security. At the same time, based on the private domain data within the enterprise, it can effectively avoid the illusion problem of large models.

[0067] In practical applications, the bge-m3 vector model is used to embed fault information and private domain knowledge.

[0068] In some examples, a knowledge vector library is retrieved based on fault information to construct prompt words with fault context information and associated emergency strategies, including:

[0069] The knowledge vector library is retrieved using the retrieval enhancement generation framework to obtain emergency related knowledge information;

[0070] According to the mapping between the emergency-related knowledge information and the knowledge vector library, the associated emergency strategy is obtained; according to the fault information and the associated emergency strategy, the prompt word with the fault context information and the associated emergency strategy is constructed.

[0071] Among them, the Retrieval-Augmented Generation (RAG) framework is a technical framework that combines information retrieval and text generation. In practical applications, the knowledge vector library may contain a huge amount of information. Using the Retrieval-Augmented Generation framework to search the knowledge vector library can quickly screen out highly relevant emergency-related knowledge information. By mapping the emergency-related knowledge information with the knowledge vector library, the existing knowledge structure and logical relationship in the knowledge vector library can be used to generate a comprehensive and detailed associated emergency strategy with scientific basis and systematicity.

[0072] By constructing prompt words with fault information and associated emergency strategies, the large model can be accurately guided to generate strategies in the direction of solving specific faults; the strategies generated by the large model can be more in line with the actual situation.

[0073] In practical applications, the big model can mine the potential correlation between fault information and related emergency strategies, and optimize the generation of emergency strategies. For example, in the emergency response of industrial equipment failures, it may discover common factors or chain reactions between different equipment failures. Therefore, when generating emergency strategies, it not only considers the repair of a single device, but also the impact on the entire production process, and proposes more comprehensive emergency strategies including coordinated recovery between equipment and prevention of secondary failures, thereby improving the quality of emergency strategies.

[0074] In some examples, before determining whether the application system fails based on the application status data, the method further includes:

[0075] Perform feature processing on the application status data to obtain feature values ​​of different dimensions;

[0076] Normalize the feature values ​​of different dimensions.

[0077] Since the scales of eigenvalues ​​of different dimensions are different, normalization can be used to reduce the storage space requirements of data, which can make more efficient use of storage resources; and normalization of eigenvalues ​​can make these data from different sources more compatible when stored, which is convenient for subsequent data integration and analysis. For example, normalize the eigenvalues ​​of different dimensions to the interval [0, 1].

[0078] In some examples, determining whether an application system fails based on application status data specifically includes:

[0079] The basic environment data in the processed application status data is compared with the preset indicator range; if the indicator of the basic environment data exceeds the preset indicator range, the application system fails; if the indicator of the basic environment data does not exceed the preset indicator range, the application system does not fail.

[0080] The pre-set indicator range provides a clear and definite reference standard for judging whether the application system is faulty; by continuously collecting the basic environment data in the application status data and comparing it with the preset indicator range in real time, potential fault hazards can be discovered in time. For example, if the preset normal bandwidth range of the application system is 100Mbps - 1000Mbps, once the real-time monitoring of the bandwidth data is lower than 100Mbps, it can be quickly determined that the system may have run jams and other faults due to insufficient network bandwidth, and then take corresponding solutions in time, such as adjusting the network configuration or troubleshooting the network fault point, to prevent the fault from further deteriorating and affecting the normal use of the system.

[0081] In practical applications, Figure 3 A flow chart of a method for emergency handling of system failures Figure 3 ,like Figure 3 As shown, environmental data collection is performed to collect application status data of the application system; the application status data is characterized by data preprocessing, and the basic environmental data in the processed application status data is compared with the preset index range; if the index of the basic environmental data exceeds the preset index range, the application system fails; then fault information is generated; the knowledge vector library is retrieved according to the fault information, and prompt words with fault context information and associated emergency strategies are constructed, and the prompt words are sent to the large model; the large model is used to generate emergency strategies according to the prompt words; emergency processing is initiated according to the emergency strategy generated by the large model, and emergency operations are performed on the application system; after the emergency processing is completed, the reward value is calculated according to the fault information, emergency strategy, emergency processing and processing results using the objective function; the size of the reward value and the scalar value is compared; if the reward value is greater than the scalar value, the fault emergency information corresponding to the reward value is stored in the knowledge vector library; in addition, the collected fault information, emergency strategy, and emergency processing are characterized and input into the knowledge vector library. If the application system does not fail, the processed application status data is input into the knowledge vector library to update the knowledge vector library.

[0082] Embodiment 2

[0083] Figure 4 A schematic diagram of a system failure emergency treatment device provided in this application, such as Figure 4 As shown, including:

[0084] The collection module 11 is used to collect application status data of the application system; wherein the application status data includes basic environment data;

[0085] A determination module 12, configured to determine whether the application system fails based on the application status data, and generate failure information if the application system fails;

[0086] The generation module 13 is used to search the knowledge vector library according to the fault information, construct prompt words with fault context information and associated emergency strategies, and send the prompt words to the large model; the large model is used to generate emergency strategies according to the prompt words;

[0087] The processing module 14 is used to initiate emergency processing according to the emergency strategy generated by the large model and generate processing results; wherein the processing results are scalar values ​​of the system success rate, transaction delay rate, transaction processing volume per second, and transaction processing failure volume per second.

[0088] Among them, in addition to basic environment data, application status data also includes: transaction data, application data, process data, action data; transaction data includes cluster system success rate, transaction delay rate, transaction tps, failed tps, etc.; among them, transaction tps means the transaction processing volume per second, and failed tps means the transaction processing failure volume per second; basic environment data includes cluster network traffic, access layer node status, business layer node status, database status, etc.; application data includes cluster change status, application exception log, application alarm, etc.; process data includes production plan change, downtime maintenance plan, external fault reporting data, etc.; action data includes emergency action timeliness, emergency action impact transaction number, etc. The application status data will be collected in real time, which can warn of failures in advance, and when the application system fails, the real-time collected data is like a "black box" record, which can help operation and maintenance personnel quickly determine the cause of the failure.

[0089] Basic environment data refers to the data of the application system and its infrastructure; environmental characteristics include the current characteristics and memory characteristics of the application system; among them, the current characteristics include system success rate, transaction delay rate, transaction tps, failed tps, network traffic, access layer node status, business layer node status, database status, change status and other application status; memory characteristics include historical action sequences and application status within a certain period of time.

[0090] Among them, emergency treatment is initiated according to the emergency strategy generated by the large model. Specifically, emergency actions are executed according to the emergency strategy generated by the large model to perform emergency treatment on the faulty application system. Emergency actions include isolating clusters, restarting virtual machines, rebuilding containers, F5 switching, etc. Executing the above emergency actions can achieve the effect of automatically handling faults and quickly restoring transactions. For example, the emergency action of executing F5 switching, specifically, when F5 detects that a backend server has a fault (for example, the server is unresponsive, returns an error status code, etc.), it will automatically switch the transaction traffic originally sent to the faulty server to other healthy servers according to the pre-configured load balancing algorithm and switching strategy; after F5 performs traffic switching, transactions can continue on other servers and will not be interrupted due to the failure of a certain server, thereby achieving rapid recovery of transactions.

[0091] For example, isolate the cluster: when the status of the primary and backup databases of an application cluster A suddenly changes to unavailable, execute cluster switching and switch the transactions of the application cluster A to cluster B; restart the virtual machine, a certain application virtual machine node is killed, and the success rate of the transaction system drops by 10%, execute virtual machine shutdown, and resume transactions; F5 switching: the directory capacity of an application cluster A database suddenly increases to 95%, and it is expected to reach 100% within 1 hour. At that time, the database is unavailable, and the database capacity of cluster B is normal. Execute cluster switching and switch cluster A to cluster B.

[0092] When an application system changes from a normal state to a warning state or an abnormal state, the state of the application system is actually a time-series change process. Factors that cause the state of the application system to change include changes in network traffic and third-party service failures. In addition, emergency processing caused by emergency actions will also cause state changes. These changes may be positive, restoring the system to a normal state, or they may be complex, bringing new problems. For example, adding server resources may improve system performance in a short period of time, but if improperly configured, it may cause resource competition or compatibility issues, which in turn may cause new anomalies.

[0093] In actual applications, application status data of the application system is collected; based on the application status data, it is determined whether the application system is faulty. If the application system fails, fault information is generated; the knowledge vector library is retrieved based on the fault information, prompt words with fault context information and associated emergency strategies are constructed, and the prompt words are sent to the large model; the large model is used to generate emergency strategies based on the prompt words; emergency processing is initiated based on the emergency strategies generated by the large model, and processing results are generated; the processing results are scalar values ​​of the system success rate, transaction delay rate, transaction processing volume per second, and transaction processing failure volume per second. By collecting application status data in real time to determine the fault phenomenon of the application system, searching the knowledge vector library, and constructing prompt words with fault context information and associated emergency strategies, the large model can generate more accurate emergency strategies and realize automated emergency handling.

[0094] This embodiment provides a system failure emergency processing device, which can execute the method provided by the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.

[0095] Figure 5 The electronic device 50 provided in this embodiment includes: at least one processor 501 and a memory 502. Optionally, the device 50 also includes a communication component 503. The processor 501, the memory 502 and the communication component 503 are connected via a bus 504.

[0096] In a specific implementation process, at least one processor 501 executes the computer-executable instructions stored in the memory 502, so that at least one processor 501 executes the above method.

[0097] The specific implementation process of the processor 501 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0098] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the invention can be directly implemented as a hardware processor, or can be implemented by a combination of hardware and software modules in the processor.

[0099] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0100] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0101] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0102] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0103] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0104] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0105] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for emergency handling of system failures, characterized in that: include: Collecting application status data of the application system; wherein the application status data includes basic environment data; Determining whether the application system fails based on the application status data, and generating failure information if the application system fails; Retrieving a knowledge vector library based on the fault information, constructing prompt words with fault context information and associated emergency strategies, and sending the prompt words to a large model; the large model is used to generate emergency strategies based on the prompt words; Initiate emergency processing according to the emergency strategy generated by the large model and generate processing results; wherein the processing results are scalar values ​​of the combination of system success rate, transaction delay rate, transaction processing volume per second, and transaction processing failure volume per second.

2. The method according to claim 1, characterized in that After determining whether the application system fails based on the processed application status data, the method further includes: If the application system does not fail, the application status data obtained through the processing is input into the knowledge vector library to update the knowledge vector library.

3. The method according to claim 1, characterized in that After initiating emergency processing according to the emergency strategy generated by the large model and generating processing results, the method further includes: Calculating a reward value using an objective function according to the fault information, the emergency strategy, the emergency treatment and the treatment result; Compare the reward value and the scalar value; if the reward value is greater than the scalar value, store the fault emergency information corresponding to the reward value into the knowledge vector library; wherein the fault emergency information includes the fault information, the emergency strategy, and the emergency processing.

4. The method according to claim 3, characterized in that Also includes: The fault information, the emergency strategy and the emergency treatment are characterized and then input into the knowledge vector library.

5. The method according to claim 2, characterized in that: The knowledge vector library is used for: Parse, differentiate, and vectorize the received application status data; Based on the processed data, the knowledge vector library is updated in a dynamic updating manner.

6. The method according to claim 5, characterized in that The application status data also includes private domain knowledge; the knowledge vector library is also used to: embed the fault information and private domain knowledge using a vector model, and convert the fault information and the private domain knowledge into a vector form.

7. The method according to claim 1, characterized in that The step of retrieving a knowledge vector library based on the fault information and constructing prompt words with fault context information and associated emergency strategies specifically includes: The knowledge vector library is retrieved using the retrieval enhancement generation framework to obtain emergency related knowledge information; According to the mapping between the emergency-related knowledge information and the knowledge vector library, an associated emergency strategy is obtained; according to the fault information and the associated emergency strategy, a prompt word with fault context information and an associated emergency strategy is constructed.

8. The method according to any one of claims 1 to 7, characterized in that: Before determining whether the application system fails based on the application status data, the method further includes: Performing feature processing on the application status data to obtain feature values ​​of different dimensions; The eigenvalues ​​of the different dimensions are normalized.

9. The method according to claim 8, characterized in that The determining whether the application system fails based on the application status data specifically includes: The basic environment data in the processed application status data is compared with a preset indicator range; if the indicator of the basic environment data exceeds the preset indicator range, the application system fails; if the indicator of the basic environment data does not exceed the preset indicator range, the application system does not fail.

10. A device for emergency handling of system failures, characterized in that: include: A collection module, used to collect application status data of the application system; wherein the application status data includes basic environment data; A judgment module, used to judge whether the application system fails based on the application status data, and generate fault information if the application system fails; A generation module, used to search the knowledge vector library according to the fault information, construct prompt words with fault context information and associated emergency strategies, and send the prompt words to the large model; the large model is used to generate emergency strategies according to the prompt words; The processing module is used to initiate emergency processing according to the emergency strategy generated by the large model and generate processing results; wherein the processing results are scalar values ​​of the combination of system success rate, transaction delay rate, transaction processing volume per second, and transaction processing failure volume per second.

11. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 9 when executed by a processor.

13. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 9 when being executed by a processor.

Images