Fault repair method and device of business system, electronic equipment, storage medium and program product

Abstract

The invention provides a fault repairing method and device of a service system, electronic equipment, a storage medium and a program product. The method comprises the following steps: performing fault detection on each service cluster to obtain a cluster fault detection result of each service cluster; determining the service cluster with the fault indicated by the cluster fault detection result as a fault service cluster; detecting service nodes in each fault service cluster to obtain fault service nodes in each fault service cluster; and determining a fault repair mode of the service system from a plurality of preset fault repair modes based on the faulty service nodes in the faulty service cluster and the number of the faulty service clusters, and performing fault repair on the service system based on the fault repair mode. According to the invention, the fault repair efficiency of the service system can be effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a business system fault repair method and device, electronic equipment, storage medium and program product. BACKGROUND

[0002] Business cluster is the most important data network system at present due to high management control efficiency and great flexibility. The business cluster is a network system composed of multiple business nodes, each of which is an independent host for performing specific tasks such as computing tasks and data transmission tasks.

[0003] In a business system including a business cluster, the failure of a certain network device may affect other network devices in the business system, so it is necessary to quickly repair the fault of the business system. In related technologies, the fault repair of the business system is often directly repaired by a preset single repair method. Due to the diversity of the fault of the business system, the efficiency of the fault repair of the business system in related technologies is low. SUMMARY

[0004] The embodiments of the present application provide a business system fault repair method, device, electronic equipment, computer readable storage medium and computer program product, which can effectively improve the fault repair efficiency of the business system.

[0005] The technical scheme of the embodiments of the present application is as follows:

[0006] The embodiments of the present application provide a business system fault repair method, the business system includes at least one business cluster, the business cluster includes multiple business nodes, comprising:

[0007] Detecting the fault of each business cluster to obtain the cluster fault detection result of each business cluster;

[0008] The cluster fault detection result indicates that the business cluster with fault is determined as the fault business cluster;

[0009] Detecting the business nodes in each fault business cluster to obtain the business nodes with fault in each fault business cluster;

[0010] Based on the business nodes with fault in the fault business cluster and the number of the fault business cluster, a fault repair method of the business system is determined from multiple preset fault repair methods, and the business system is repaired based on the fault repair method.

[0011] The embodiment of the present application provides a fault repairing device of a service system, the service system comprises at least one service cluster, the service cluster comprises a plurality of service nodes, and the service cluster comprises the following:

[0012] A cluster detection module is configured to perform fault detection on each service cluster to obtain a cluster fault detection result of each service cluster, and indicate a service cluster with a fault in the cluster fault detection result as a fault service cluster.

[0013] A node detection module is configured to perform detection on service nodes in each fault service cluster to obtain a service node with a fault in each fault service cluster.

[0014] A fault repairing module is configured to determine a fault repairing mode of the service system from a plurality of preset fault repairing modes based on the service node with a fault in the fault service cluster and the number of fault service clusters, and perform fault repairing on the service system based on the fault repairing mode.

[0015] In the above scheme, the cluster detection module is further configured to perform data detection on each service cluster in the service system to obtain a detection result of each service cluster, the detection result is used to represent a data packet loss rate of detection data after the detection data flows through the service cluster, and whether each service cluster has a fault is determined based on the detection result.

[0016] In the above scheme, the cluster detection module is further configured to perform the following processing on each service cluster: sending detection data comprising a plurality of data packets to the service cluster, so that the service cluster performs flow conversion on the detection data to output target detection data after flow conversion, obtaining a data packet number difference between the target detection data and the detection data, and determining a data packet loss rate of the service cluster based on the data packet number difference.

[0017] In the above scheme, the cluster detection module is further configured to obtain cluster prompt information corresponding to the cluster fault detection, the cluster prompt information is used as a reference basis when the cluster fault detection is performed by a fault detection model, first input information of the fault detection model is constructed based on the cluster prompt information and the detection result of each service cluster, the fault detection model is called, each service cluster is detected based on the first input information to obtain a probability that each service cluster has a fault, and whether each service cluster has a fault is determined based on the probability that each service cluster has a fault.

[0018] In the scheme, the cluster detection module is further configured to perform the following processing on each service cluster: compare a data packet loss rate corresponding to the service cluster with a packet loss rate threshold to obtain a packet loss rate comparison result; when the packet loss rate comparison result indicates that the data packet loss rate is greater than or equal to the packet loss rate threshold, determine a cluster fault detection result of the service cluster as indicating that the service cluster has the cluster fault; and when the packet loss rate comparison result indicates that the data packet loss rate is less than the packet loss rate threshold, determine the cluster fault detection result of the service cluster as indicating that the service cluster does not have the cluster fault.

[0019] In the scheme, the fault repair apparatus of the service system further includes an output module configured to output prompt information when the cluster fault detection results of each service cluster all indicate that the service clusters do not have the cluster fault, and the prompt information is used to prompt that each service node in the service system does not have a fault.

[0020] In the scheme, the node detection module is further configured to perform the following processing on each service node in each fault service cluster: obtain a running state of the service node; and detect the service node in the fault service cluster based on the running state of the service node to obtain a service node in the fault service cluster that has a fault.

[0021] In the scheme, the node detection module is further configured to obtain node prompt information corresponding to the detection, the node prompt information is used as a reference basis when a fault detection model detects a fault service node, construct second input information of the fault detection model based on the node prompt information and the running state of each service node in the fault service cluster, call the fault detection model to detect the fault service node in the fault service cluster based on the second input information, obtain a probability that each service node in the fault service cluster has a fault, and determine the service node in the fault service cluster that has a fault based on the probability that each service node in the fault service cluster has a fault.

[0022] In the scheme, the node detection module is further configured to perform the following processing on each service node in the fault service cluster: compare the running state of the service node with a state threshold to obtain a running comparison result, and determine a sub-fault detection result of the service node based on the running comparison result; and when the sub-fault detection result of the service node indicates that the service node has the fault, determine the service node as the service node in the fault service cluster that has a fault.

[0023] The operation state includes a plurality of sub-operation states, and the state threshold includes a sub-state threshold corresponding to each of the sub-operation states; the node detection module is further configured to determine that the sub-fault detection result is that the service node does not have the fault when the operation comparison result indicates that each of the sub-operation states is greater than or equal to the corresponding state threshold; and determine that the sub-fault detection result is that the service node has the fault when the operation comparison result indicates that there is a sub-operation state that is less than the corresponding state threshold.

[0024] In the scheme, the fault repair module is further configured to obtain mode prompt information corresponding to a fault repair mode, the mode prompt information being used as a reference basis for the fault detection model when determining the fault repair mode; construct third input information of the fault detection model based on the mode prompt information, the service nodes having the fault in the fault service clusters, and the number of the fault service clusters; call the fault detection model, determine probabilities of each of the preset fault repair modes based on the third input information, and determine a preset fault repair mode with a probability greater than a preset probability threshold as the fault repair mode.

[0025] In the scheme, the preset fault repair mode includes a disaster recovery system repair mode, and the fault repair module is further configured to determine the fault repair mode as the disaster recovery system repair mode when the number of the fault service clusters is greater than or equal to a cluster number threshold and the number of the service nodes having the fault in each of the fault service clusters is greater than a node number threshold, and perform fault repair on the service system by the disaster recovery system; when the number of the fault service clusters is less than the cluster number threshold, determine, for each of the fault service clusters, a cluster repair mode of the fault service cluster from a plurality of the preset fault repair modes based on the service nodes having the fault in the fault service cluster, and determine the fault repair mode of the service system based on each of the cluster repair modes.

[0026] In the scheme, the preset fault repair mode includes a cluster isolation repair mode and a node isolation repair mode, and the fault repair module is further configured to determine a cluster repair mode of the fault service cluster as the cluster isolation repair mode when the number of the service nodes having the fault in the fault service cluster is greater than the node number threshold; determine the cluster repair mode of the fault service cluster as the node isolation repair mode when the number of the service nodes having the fault in the fault service cluster is not greater than the node number threshold; and the node isolation repair mode is used to isolate the service nodes having the fault in the fault service cluster, and the cluster isolation repair mode is used to isolate the fault service cluster in the service system.

[0027] In the scheme, the fault repair module is further configured to determine the cluster repair mode as the fault repair mode of the service system when the number of the fault service clusters is one; and determine a combination of the cluster repair modes as the fault repair mode of the service system when the number of the fault service clusters is more than one.

[0028] In the scheme, the fault repair module is further configured to, when the fault repair mode is a disaster recovery system repair mode, acquire a disaster recovery system, the disaster recovery system including at least one disaster recovery service cluster, and the disaster recovery service cluster being free of the cluster fault; and migrate service data in the service system to the disaster recovery system to implement the fault repair of the service system.

[0029] The embodiment of the present application provides an electronic device, including:

[0030] A memory is configured to store computer executable instructions or computer programs.

[0031] A processor is configured to execute the computer executable instructions or computer programs stored in the memory to implement the fault repair method of the service system provided by the embodiment of the present application.

[0032] The embodiment of the present application provides a computer readable storage medium, which stores computer executable instructions, and is used to cause a processor to execute the fault repair method of the service system provided by the embodiment of the present application.

[0033] The embodiment of the present application provides a computer program product, which includes a computer program or computer executable instructions stored in a computer readable storage medium. A processor of an electronic device reads the computer executable instructions from the computer readable storage medium, and the processor executes the computer executable instructions, so that the electronic device executes the fault repair method of the service system provided by the embodiment of the present application.

[0034] The embodiment of the present application has the following beneficial effects:

[0035] By detecting the faults of the business clusters, the cluster fault detection results of the business clusters are obtained, the business cluster with the fault indication is determined as the fault business cluster, the business nodes in each fault business cluster are detected, the business nodes with faults in each fault business cluster are obtained, the fault recovery mode of the business system is determined from the plurality of preset fault recovery modes based on the business nodes with faults in the fault business cluster and the number of fault business clusters, and the fault recovery of the business system is performed based on the fault recovery mode. In this way, the fault recovery mode of the business system is determined from the plurality of preset fault recovery modes based on the business nodes with faults in the fault business cluster and the number of fault business clusters, and the fault recovery of the business system is performed based on the fault recovery mode, so that the determined fault recovery mode can adapt to the fault of the business system, thereby achieving efficient fault recovery of the business system, and effectively improving the fault recovery efficiency of the business system. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is an architecture schematic diagram of a fault recovery system of a business system provided by an embodiment of the present application;

[0037] Figure 2 is a structural schematic diagram of an electronic device for fault recovery of a business system provided by an embodiment of the present application;

[0038] Figure 3 is an architecture schematic diagram of a business system provided by an embodiment of the present application;

[0039] Figure 4 is a flowchart of a fault recovery method of a business system provided by an embodiment of the present application Figure 1 ;

[0040] Figure 5 is a flowchart of a fault recovery method of a business system provided by an embodiment of the present application Figure 2 ;

[0041] Figure 6 is a model structure schematic diagram of a fault detection model provided by an embodiment of the present application;

[0042] Figure 7 is a flowchart of a fault recovery method of a business system provided by an embodiment of the present application Figure 3 ;

[0043] Figure 8 is a flowchart of a fault recovery method of a business system provided by an embodiment of the present application Figure 4 ;

[0044] Figure 9 is a flowchart of a fault recovery method of a business system provided by an embodiment of the present application Figure 5;

[0045] Figure 10 is a flow diagram of a fault repair method of a service system provided by an embodiment of the present application Figure 6 ;

[0046] Figure 11 is a flow diagram of a fault repair method of a service system provided by an embodiment of the present application Figure 7 ;

[0047] Figure 12 is a schematic diagram of a gateway cluster provided by an embodiment of the present application

[0048] Figure 13 is a schematic diagram of health detection of a gateway cluster provided by an embodiment of the present application

[0049] Figure 14 is a schematic diagram of fault repair provided by an embodiment of the present application Figure 1 ;

[0050] Figure 15 is a schematic diagram of fault repair provided by an embodiment of the present application Figure 2 ;

[0051] Figure 16 is a schematic diagram of fault repair provided by an embodiment of the present application Figure 3 ;

[0052] Figure 17 is a schematic diagram of a fault self-recovery intelligent agent provided by an embodiment of the present application

[0053] Figure 18 is a schematic diagram of an execution flow of a fault self-recovery intelligent agent of a gateway cluster provided by an embodiment of the present application

[0054] Figure 19 is a schematic diagram of application of a prompt engineering template provided by an embodiment of the present application

[0055] Figure 20 is a schematic diagram of fault discovery decision provided by an embodiment of the present application

[0056] Figure 21 is a schematic diagram of fault diagnosis decision provided by an embodiment of the present application

[0057] Figure 22 is a schematic diagram of fault repair decision provided by an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings. The described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0059] In the following description, "some embodiments" are referred to, which describe a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0060] In the following description, the terms "first\second\third" are only to distinguish similar objects, and do not represent a specific order of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0062] Before the embodiments of the present application are further described in detail, the terms and phrases involved in the embodiments of the present application are explained, and the terms and phrases involved in the embodiments of the present application are applicable to the following explanations.

[0063] 1) Artificial Intelligence (AI): is a comprehensive discipline involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include, such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-training model technology, operation / interaction system, mechatronics, etc. Among them, the pre-training model is also called large model, basic model, which can be widely applied to downstream tasks in various directions of artificial intelligence after fine-tuning. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0064] 2) Machine Learning (ML): is a multi-disciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithmic complexity theory, etc. It is a specialized study of how computers simulate or implement human learning behavior to acquire new knowledge or skills, reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent, and its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and rule-based learning. Pre-trained models are the latest development in deep learning, integrating the above technologies.

[0065] 3) Convolutional Neural Networks (CNN): is a class of feed-forward neural networks (FNN) containing convolutional computation and having a deep structure, which is one of the representative algorithms of deep learning (Deep Learning). Convolutional neural networks have representation learning (Representation Learning) capabilities and can perform shift-invariant classification (Shift-Invariant Classification) on input images according to their hierarchical structure.

[0066] 4) In response to: used to represent the conditions or states on which the operations performed depend, when the dependent conditions or states are met, the one or more operations performed can be real-time or have a set delay; in the absence of special instructions, there is no restriction on the execution order of multiple operations performed.

[0067] 5) Large Language Model (LLM): also known as large language model, large model, large language model refers to a deep learning model trained using a large amount of text data, containing billions of parameters or even more, which can be used to generate natural language text and understand the meaning of natural language text. Through training, the model can learn the statistical rules and semantic associations of language to build a large language knowledge base, and then simulate human language understanding and generation capabilities.

[0068] 6) game program: the game program can be any one of a massive multiplayer online role-playing game (MMORPG), a first-person shooting game (FPS), a third-person shooting game, a multiplayer online battle arena game (MOBA), a virtual reality application program, a three-dimensional map program, a simulation program, or a multiplayer gun battle type survival game.

[0069] In the implementation of the embodiments of the present application, the applicant finds that the related art has the following problems:

[0070] In a service system including a service cluster, failure of a certain network device can cause other network devices in the service system to be affected, and therefore it is necessary to quickly repair the failure of the service system. In the related art, for the repair of the failure of the service system, a preset single repair mode is often directly adopted for failure repair. Due to the diversity of the failure of the service system, the repair efficiency of the failure of the service system in the related art is low.

[0071] The embodiments of the present application provide a service system failure repair method and device, an electronic device, a computer readable storage medium, and a computer program product, which can effectively improve the failure repair efficiency of the service system. The following describes an exemplary application of the service system failure repair system provided by the embodiments of the present application.

[0072] Referring to Figure 1 , Figure 1 FIG. 1 is an architecture schematic diagram of a service system failure repair system 100 provided by the embodiments of the present application. A terminal (exemplarily shown as a terminal 400) is connected to a server 200 through a network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0073] The terminal 400 is used for a user to use a client 410, and a normal prompt information is displayed on a graphical interface 410-1 (exemplarily shown as a graphical interface 410-1). The terminal 400 and the server 200 are connected to each other through a wired or wireless network.

[0074] In some embodiments, the server 200 can be a stand-alone physical server, a server cluster composed of multiple physical servers, or a business system, and can also be a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal 400 can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart television, a smart watch, a vehicle-mounted terminal, and the like, but is not limited thereto. The electronic device provided by the embodiments of the present application can be implemented as a terminal or a server. The terminal and the server can be connected directly or indirectly through wired or wireless communication, and the embodiments of the present application do not make any limitation.

[0075] In some embodiments, the server 200 performs fault detection on each business cluster to obtain cluster fault detection results of each business cluster; indicates a business cluster with a fault in the cluster fault detection results to determine a fault business cluster; performs node fault detection on each fault business cluster to obtain a business node with a fault in each fault business cluster; determines a fault recovery mode of the business system based on the business node with a fault in the fault business cluster and the number of fault business clusters, and sends the fault recovery mode to the terminal 400; and the terminal 400 performs fault recovery on the business system based on the fault recovery mode.

[0076] In some embodiments, the terminal 400 performs fault detection on each business cluster to obtain cluster fault detection results of each business cluster; indicates a business cluster with a fault in the cluster fault detection results to determine a fault business cluster; performs node fault detection on each fault business cluster to obtain a business node with a fault in each fault business cluster; determines a fault recovery mode of the business system based on the business node with a fault in the fault business cluster and the number of fault business clusters, and sends the fault recovery mode to the server 200; and the server 200 performs fault recovery on the business system based on the fault recovery mode.

[0077] In some embodiments, the embodiments of the present application can be implemented by means of cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network to realize the calculation, storage, processing, and sharing of data in a wide area network or a local area network.

[0078] Cloud technology is a general term encompassing network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form resource pools, allowing for on-demand use with flexibility and convenience. Cloud computing technology will become a crucial support. The backend services of cloud computing systems require substantial computing and storage resources.

[0079] See Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device 500 for fault repair of a business system provided in an embodiment of this application, wherein, Figure 2 The electronic device 500 shown can be Figure 1 Server 200 or terminal 400 in the middle, Figure 2 The illustrated electronic device 500 includes at least one processor 430, a memory 450, and at least one network interface 420. The various components in the electronic device 500 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general designated all buses as Bus System 440.

[0080] Processor 430 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0081] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 430.

[0082] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0083] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0084] The operating system 451 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, and the like, for implementing various basic services and processing hardware-based tasks;

[0085] The network communication module 452 is configured to communicate with other electronic devices via one or more (wired or wireless) network interfaces 420, such as Bluetooth, Wireless Fidelity (WiFi), and Universal Serial Bus (USB), and the like.

[0086] In some embodiments, the business system failure repair apparatus provided by the embodiments of the present application can be implemented in a software manner, Figure 2 The business system failure repair apparatus 455 stored in the memory 450 is shown, which can be software in the form of programs and plug-ins, and includes the following software modules: a cluster detection module 4551, a node detection module 4552, and a failure repair module 4553. These modules are logical, and thus can be combined or further split according to the implemented functions. The functions of each module will be described below.

[0087] In some embodiments, the business system failure repair apparatus provided by the embodiments of the present application can be implemented in a software manner, As an example, the business system failure repair apparatus provided by the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the business system failure repair method provided by the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can use one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic elements.

[0088] In some embodiments, the terminal or server can implement the fault repairing method of the service system provided by the embodiments of the present application by running a computer program or computer executable instructions. For example, the computer program can be a native program (for example, a dedicated fault repairing program) in the operating system or a software module, for example, a fault repairing module that can be embedded in any program (such as an instant messaging client, a photo album program, an electronic map client, a navigation client); for example, it can be a native (Native) application program (APP, Application), that is, a program that needs to be installed in the operating system to run. In summary, the above computer program can be any form of application program, module or plug-in.

[0089] The fault repairing method of the service system provided by the embodiments of the present application will be described in conjunction with the exemplary application and implementation of the server or terminal provided by the embodiments of the present application.

[0090] Referring to Figure 3 , Figure 3 is a schematic diagram of the architecture of the service system provided by the embodiments of the present application, which includes at least one service cluster, and each service cluster includes a plurality of service nodes, for example, as Figure 3 shown, the service system includes service cluster A and service cluster B, service cluster A includes service node A, service node B, service node C and service node D, and service cluster B includes service node E, service node F, service node G and service node H.

[0091] As an example, in the business scenario of data analysis processing, the service nodes in each service cluster in the service system are responsible for processing a part of data, and the processed data can come from different data sources, which can be social media, Internet of Things devices, sensors, etc.

[0092] As an example, in the business scenario of social media data analysis, the service nodes in each service cluster in the service system are responsible for processing a part of social media data, and the processed social media data can come from different data sources, which can be different social parties.

[0093] As an example, in the business scenario of financial analysis, the service nodes in each service cluster in the service system are responsible for processing a part of financial data, and the processed financial data can come from different data sources, which can be different financial data parties.

[0094] ​​​​​​​As an example, in the business scenario of a game server, for example, for online multiplayer games such as massively multiplayer online role-playing games (MMORPG). This game needs to handle a large number of player interactions, including movement, attack, chat, etc. In order to ensure the smoothness and response speed of the game experience, the game service is designed as a distributed architecture, containing multiple business clusters and business nodes. Business cluster: the business cluster of the game server is responsible for processing requests from players around the world. Servers in the cluster may be distributed in different geographical locations to better serve players in different regions. For example, one business cluster may be responsible for handling players in the Asian region, another cluster for Europe, and so on. Business node: in each business cluster, there are many business nodes, each node is an independent server, responsible for processing a part of the player's request. For example, a business node may be responsible for tasks such as creating a character, loading a game map, updating a character's state, etc. When a player performs an operation in the game, such as moving to a new location, their request will first be sent to the nearest business node. After the business node processes the request, it may query the cache to obtain the character state, or perform complex calculations to update the game world state. After processing is complete, the business node returns the result to the player so that the player can see the actions of their character or other players. In order to ensure high availability and failover, each business cluster will have multiple business nodes, and may be configured with primary and backup nodes. If a node fails, the backup node can immediately take over to maintain the continuity of the service. In addition, in order to handle a large number of concurrent requests and data, the business system may also use load balancers to distribute traffic, use database clusters to store player data, and use cache systems to speed up data access speed.

[0095] Referring to Figure 4 , Figure 4 is a flowchart of a fault repair method of a business system provided by an embodiment of the present application Figure 1 The business system includes at least one business cluster, and the business cluster includes a plurality of business nodes. The steps 101 to 104 shown in Figure 4 will be described. The fault repair method of the business system provided by the embodiment of the present application can be implemented by a server or a terminal alone, or by a server and a terminal cooperatively. In the following, a case in which the server is implemented alone will be described.

[0096] In step 101, the fault detection of each business cluster is performed to obtain the cluster fault detection result of each business cluster.

[0097] In some embodiments, cluster fault detection is used to detect faults in a service cluster composed of multiple service nodes. It can automatically identify and report whether service nodes, services, or network connections in the service cluster have failed, thereby ensuring the healthy operation of the cluster. Fault detection of each service cluster involves continuous or periodic monitoring to determine whether the service cluster is functioning correctly. Obtaining the cluster fault detection results for each service cluster results in a cluster health status report. If the cluster fault detection results indicate a fault, it means that a part of the cluster has a problem and requires further diagnosis and repair.

[0098] As an example, see Figure 3 Fault detection is performed on business cluster A to obtain the cluster fault detection result of business cluster A. Fault detection is performed on business cluster B to obtain the cluster fault detection result of business cluster B.

[0099] In some embodiments, the above cluster fault detection results are used to characterize whether the corresponding service cluster has a fault, and the above cluster fault detection results correspond one-to-one with the service clusters.

[0100] In some embodiments, see Figure 5 , Figure 5 This is a flowchart illustrating the fault repair method for a business system provided in the embodiments of this application. Figure 2 , Figure 4 Step 101 shown can be achieved through Figure 5 Steps 1011 to 1012 shown are implemented.

[0101] In step 1011, data probing is performed on each business cluster in the business system to obtain the probing results of each business cluster.

[0102] In some embodiments, the detection results are used to characterize the data packet loss rate after the detection data stream passes through the service cluster.

[0103] In some embodiments, data probing is a network diagnostic technique used to detect and analyze data transmission across various business clusters within a business system. During this process, data probing examines each business cluster to understand the transmission efficiency and quality of data within these clusters. The probing results for each business cluster constitute network diagnostic data, reflecting the data transmission status within the business clusters. The probing results characterize the packet loss rate of the probed data flow through the business clusters, representing the proportion of data packets lost during data transmission. By analyzing the probing results, the performance of the business clusters can be evaluated, potential network problems can be identified, and measures can be taken to optimize data transmission quality.

[0104] In some embodiments, the above-mentioned data packet loss rate refers to the ratio of data packets lost due to various reasons during data transmission. A data packet is the basic unit of transmission in a network and contains a certain amount of information. During data transmission, due to network congestion, signal interference, hardware failure, etc., some data packets may not successfully reach the destination, which is called data packet loss. The data packet loss rate is the ratio of the number of data packets that do not successfully reach the destination to the total number of data packets sent. The higher the data packet loss rate, the lower the transmission efficiency and reliability of the service cluster, the greater the possibility of failure of the service cluster, and the greater the impact on user experience. Therefore, reducing the data packet loss rate is one of the important goals of improving network communication quality.

[0105] In some embodiments, data detection is performed on each service cluster in the service system to obtain the detection results of each service cluster, which can be achieved by the following method: for each service cluster, the following processing is performed: sending detection data including a plurality of data packets to the service cluster to make the service cluster process the detection data and output the target detection data after processing; obtaining the difference in the number of data packets between the target detection data and the detection data, and determining the data packet loss rate of the service cluster based on the difference in the number of data packets.

[0106] In some embodiments, specific detection data is sent to the service cluster, and the detection data includes data packets. These data packets can be UDP packets, HTTP requests, etc., which are used to simulate the data transmission process of normal users. After receiving these detection data packets, the service cluster will process them as if they were normal user requests. This may include data packet forwarding, routing, processing, etc. After processing within the service cluster, the target detection data packets after processing are output. These data packets may contain response information of the service cluster, such as the arrival time of the data packet, whether it is successfully processed, etc. The response from the service cluster, i.e. the target detection data packet, is received. Based on the received target detection data packet and the original detection data packet, the data transmission of the service cluster is analyzed. This usually involves calculating the data packet loss rate, i.e. the proportion of data packets lost during transmission to the total number of data packets sent. By comparing the number of original detection data packets and target detection data packets, the monitoring system can calculate the data packet loss rate, which represents the loss of detection data during the flow through the service cluster. The final detection result, especially the data packet loss rate, will be fed back to the network administrator or operation and maintenance team. This can help them diagnose network problems, optimize network performance, and ensure good user experience.

[0107] In some embodiments, the processing of the detection data by the service cluster refers to the processing process of the detection data in the service cluster, including data transmission, processing and return.

[0108] As an example, referring to Figure 13 , the access point probe server A sends the probe data to the service cluster (the gateway cluster A shown in Figure 13 ) to make the service cluster (the gateway cluster A shown in Figure 13 ) flow the probe data and output the target probe data after flow; the cloud server A receives the target probe data and determines the probe result of the service cluster (the gateway cluster A shown in Figure 13 ) based on the target probe data and the probe data.

[0109] Taking the above example, for the service cluster (the gateway cluster A shown in Figure 13 ), for example, the gateway cluster A includes service node A1, service node A2 and service node A3, the service cluster (the gateway cluster A shown in Figure 13 ) flows the probe data and outputs the target probe data after flow, which can be that the service node A1 receives the probe data and sends the first sub-probe data to the service node A2, the service node A2 receives the first sub-probe data and sends the second sub-probe data to the service node A3, and the service node A3 receives the second sub-probe data and outputs the target probe data, wherein the data packet quantity of the first sub-probe data is less than or equal to the data packet quantity of the probe data, and the data packet quantity of the second sub-probe data is less than or equal to the data packet quantity of the first sub-probe data.

[0110] As an example, referring to Figure 13 , the access point probe server B sends the probe data to the service cluster (the gateway cluster B shown in Figure 13 ) to make the service cluster (the gateway cluster B shown in Figure 13 ) flow the probe data and output the target probe data after flow; the cloud server A receives the target probe data and determines the probe result of the service cluster (the gateway cluster B shown in Figure 13 ) based on the target probe data and the probe data.

[0111] As an example, referring to Figure 13 , the access point probe server B sends the probe data to the service cluster (the gateway cluster B shown in Figure 13 ) to make the service cluster (the gateway cluster B shown in Figure 13 ) flow the probe data and output the target probe data after flow; the cloud server B receives the target probe data and determines the probe result of the service cluster (the gateway cluster B shown in Figure 13 ) based on the target probe data and the probe data.

[0112] In some embodiments, the determination of the data packet loss rate of the service cluster based on the data packet quantity difference can be achieved by dividing the data packet quantity difference by the data packet quantity of the target probe data.

[0113] As an example, the expression of the probe result can be:

[0114]

[0115] wherein T represents the data packet loss rate of the service cluster, S1-S2 represents the data packet quantity difference, S1 represents the data packet quantity of the probe data, and S2 represents the data packet quantity of the target probe data.

[0116] In this way, by sending probe data to each service cluster, the running state and performance indicators of each service cluster can be detected in real time, ensuring the normal operation of each cluster. By comparing the changes before and after the service cluster processes the probe data, abnormal situations or performance bottlenecks in the service processing process can be discovered in a timely manner. Through analysis of the target probe data and the probe result of the service cluster, potential risk points can be predicted, so that measures can be taken in advance to avoid or reduce possible problems. Based on the received target probe data and the flow process, the processing flow of the service cluster can be optimized to improve processing efficiency and accuracy. By analyzing the probe result of the service cluster, computing resources can be allocated more reasonably to avoid resource waste and ensure that the resource needs of critical services are met. Through continuous probing and analysis, security threats can be discovered and handled in a timely manner to ensure the data security and stability of the service cluster.

[0117] In step 1012, based on the probe result, it is determined whether each service cluster has a fault.

[0118] In some embodiments, the probe result represents the data packet loss rate after the probe data flows through the service cluster. The higher the data packet loss rate indicated by the probe result of the service cluster, the greater the possibility that the cluster fault detection result of the service cluster indicates that the service cluster has a fault. The smaller the data packet loss rate indicated by the probe result of the service cluster, the smaller the possibility that the cluster fault detection result of the service cluster indicates that the service cluster has a fault.

[0119] In some embodiments, the step 1012 can be achieved by: obtaining cluster prompt information corresponding to the cluster fault detection; based on the cluster prompt information and the probe result of each service cluster, constructing first input information of the fault detection model; calling the fault detection model, based on the first input information, performing fault detection on each service cluster to obtain the probability that each service cluster has a fault, and determining whether each service cluster has a fault based on the probability that each service cluster has a fault.

[0120] In some embodiments, the determination of whether each service cluster has a fault based on the probability of each service cluster having a fault can be implemented by determining a service cluster as a service cluster having a fault if the probability is greater than a cluster probability threshold, and determining a service cluster as a service cluster not having a fault if the probability is not greater than the cluster probability threshold.

[0121] In some embodiments, the cluster prompt information is used as a reference basis for fault detection by the fault detection model.

[0122] In some embodiments, the cluster prompt information can be fault discovery knowledge, which can be analyzed for data commonality based on the exploration data of the cluster, and attempt to classify the fault as a cluster fault or a non-cluster fault, and the cluster fault will be diagnosed for a cluster fault.

[0123] As an example, see Figure 19 The cluster prompt information can be a role setting: assume you are an intelligent agent for self-healing of a cluster fault, do not fabricate or fabricate facts, and answer the question according to [reference knowledge]. Reference knowledge 1: when all the packet loss data of the cluster detection data is concentrated in a certain cluster, classify it as a single cluster fault; when there is packet loss in multiple clusters, classify it as a non-single cluster fault, and notify human intervention for processing; when there is no packet loss, classify it as no fault. Reference knowledge 2: [real-time cluster detection data]. Answer the question: Is there a fault? Answer format: return in json format, no explanation needed. Answer example: ( "output" : "single cluster fault (fault cluster: cluster x)", "next_step" : "fault diagnosis"}.

[0124] As an example, see Figure 19 The detection results of each service cluster can be: [cluster A: 1.1.1.1, 2.2.2.1, 100%], [cluster A, 1.1.1.2, 2.2.2.2, 100%], [cluster A, 1.1.1.3, 2.2.2.3, 100%], [cluster A, 1.1.1.4, 2.2.2.4, 100%], [cluster B, 1.1.1.1, 2.2.2.1, 0%], [cluster B, 1.1.1.2, 2.2.2.2, 0%], [cluster B, 1.1.1.3, 2.2.2.3, 0%], [cluster B, 1.1.1.4, 2.2.2.4, 0%], wherein the data format is [cluster A, source IP, destination IP, data packet loss rate].

[0125] In some embodiments, the first input information includes cluster prompt information and detection results of the service clusters, the first input information is obtained by fusing the cluster prompt information and the detection results of the service clusters and formatting, the first input information is an input format that can be accepted and processed by the fault detection model, the first input information is the basis for learning and reasoning of the fault detection model, and is the basis for identifying and predicting faults of the fault detection model.

[0126] Continuing the above example, referring to Figure 19 , based on the cluster prompt information and the detection results of the service clusters (service cluster A and service cluster B), the first input information of the fault detection model is constructed, the fault detection model is called, based on the first input information, the fault detection of the service clusters (service cluster A and service cluster B) is performed, the cluster fault detection results of the service clusters (service cluster A and service cluster B) are obtained, and the fault service cluster is output: Answer: ("output": "single cluster fault (fault cluster: cluster A)", "next_step": "fault diagnosis").

[0127] In some embodiments, the fault detection model can be a large language model. The large language model, also known as a large model, is a deep learning model trained using a large amount of text data, containing billions of parameters or even more, which can be used to generate natural language text and understand the meaning of natural language text. Through training, the model can learn the statistical rules and semantic associations of language to establish a large language knowledge base, and then simulate human language understanding and generation capabilities.

[0128] As an example, referring to Figure 6 , Figure 6 is a model structure diagram of the fault detection model provided by the embodiments of the present application, Figure 6 The fault detection model shown in FIG. 8 includes a feature extraction layer 1 and a fault detection layer 2. The calling of the fault detection model based on the first input information and the fault detection of the service clusters to obtain the cluster fault detection results of the service clusters can be implemented in the following manner: calling the feature extraction layer 1 of the fault detection model to perform feature extraction on the first input information to obtain cluster detection features, calling the fault detection layer 2 of the fault detection model based on the cluster detection features to perform fault detection on the service clusters to obtain the cluster fault detection results of the service clusters.

[0129] In this way, in combination with the cluster prompt information and the detection results of each service cluster, input data of a more accurate and comprehensive fault detection model can be constructed, and the detection accuracy can be improved. By calling the fault detection model, automatic detection of each service cluster can be realized, manpower resources can be saved, and detection efficiency can be improved. The real-time of the cluster fault detection can ensure that the detection result can be obtained immediately once a fault occurs, so that the problem can be quickly responded and measures can be taken to solve the problem.

[0130] In some embodiments, the step 1012 can also be implemented by performing the following processing for each service cluster: comparing the data packet loss rate corresponding to the service cluster with the packet loss rate threshold to obtain a packet loss rate comparison result; when the packet loss rate comparison result indicates that the data packet loss rate is greater than or equal to the packet loss rate threshold, determining that the cluster fault detection result of the service cluster is that the service cluster has a fault; and when the packet loss rate comparison result indicates that the data packet loss rate is less than the packet loss rate threshold, determining that the cluster fault detection result of the service cluster is that the service cluster has no fault.

[0131] In some embodiments, the data packet loss rate corresponding to the service cluster refers to the ratio of the number of lost data packets of the service cluster in the test to the number of transmitted data groups, and the expression of the data packet loss rate can be:

[0132]

[0133] In the formula, T represents the data packet loss rate, S1 represents the number of data packets in the detection data, and S2 represents the number of data packets in the target detection data.

[0134] In some embodiments, when the packet loss rate comparison result indicates that the data packet loss rate is greater than or equal to the packet loss rate threshold, it means that the data packet loss rate corresponding to the service cluster is high, and the probability that the service cluster has a fault is high, so the cluster fault detection result of the service cluster can be determined as that the service cluster has a fault. When the packet loss rate comparison result indicates that the data packet loss rate is less than the packet loss rate threshold, it means that the data packet loss rate corresponding to the service cluster is low, and the probability that the service cluster has a fault is low, so the cluster fault detection result of the service cluster can be determined as that the service cluster has no fault.

[0135] In this way, by directly comparing the data packet loss rate of the service cluster with the preset packet loss rate threshold, it can be quickly judged whether the service cluster has a fault, and the fault diagnosis process is simplified. Real-time detection of packet loss of the service cluster is allowed to ensure that potential problems can be discovered immediately when the packet loss rate reaches the threshold. When the packet loss rate exceeds the threshold, the service cluster fault detection can be automatically triggered, the automation of fault response is realized, and manual intervention is reduced. The packet loss rate can be used as a quantitative indicator to help evaluate the severity of the fault and provide a basis for subsequent fault handling. By detecting the data packet loss rate, signs of network congestion can be detected in advance, measures can be taken to prevent network congestion, and network service quality can be ensured.

[0136] In some embodiments, after performing fault detection on each service cluster to obtain the cluster fault detection result of each service cluster, the following processing can be performed: when the cluster fault detection result of each service cluster indicates that the service cluster has no fault, output a prompt information; the prompt information is used to prompt that each service node in the business system has no fault.

[0137] In some embodiments, when the cluster fault detection result of each service cluster indicates that the service cluster has no fault, it means that there is no fault in the entire business system. Therefore, a prompt information can be output, and the prompt information is used to represent that each service node in the business system has no fault.

[0138] In this way, when the cluster fault detection result of each service cluster indicates that the service cluster has no fault, a prompt information is output, so that the cluster fault detection can be used to troubleshoot the entire business system. By outputting the prompt information, the entire business system can be troubleshooted without subsequent node fault detection and other means, thereby effectively improving the troubleshooting efficiency of the business system. By ensuring that the detection results of all service clusters show no fault, the stability of the entire business system can be finally confirmed. The output of the prompt information helps to eliminate doubts about possible faults in the business system, and provides the user and the operation and maintenance team with a clear system health status. When all service nodes are shown to be running normally, the user's trust in the system can be improved, and the user experience can be enhanced. After confirming that the business system has no fault, the allocation of computing resources and detection resources can be optimized to avoid unnecessary resource waste. Through the fault detection process of the entire business system, false positives and repeated troubleshooting work are reduced, and the operation and maintenance efficiency is improved.

[0139] In step 102, the service cluster whose cluster fault detection result indicates a fault is determined as a fault service cluster.

[0140] In some embodiments, the step 102 can be implemented by: for each service cluster, when the cluster fault detection result of the service cluster indicates that the service cluster has a fault, determining the service cluster as a fault service cluster.

[0141] In some embodiments, the step 102 can be implemented by: when the cluster fault detection results of each service cluster do not all indicate that the service cluster has no fault, determining the service cluster with the cluster fault detection result indicating that the service cluster has a fault as a fault service cluster.

[0142] For example, the service system includes service cluster A, service cluster B and service cluster C, the cluster fault detection result of service cluster A indicates that service cluster A has a fault, and service cluster A is determined as a fault service cluster. The cluster fault detection result of service cluster B indicates that service cluster B has no fault, and service cluster B is not determined as a fault service cluster. The cluster fault detection result of service cluster C indicates that service cluster C has a fault, and service cluster C is determined as a fault service cluster.

[0143] In step 103, the service nodes in each fault service cluster are detected to obtain the service nodes with faults in each fault service cluster.

[0144] In some embodiments, the fault service cluster refers to those service clusters that have been detected to have a fault problem, and the nodes in these clusters may not work normally due to hardware failure, software error, network problem or other reasons. Node fault detection is performed on each node in the fault service cluster to determine whether they also have faults. This usually includes checking the response time, processing capacity, hardware status, software running status, etc. of the node. The output of each node fault detection process indicates whether each node is working normally. If the node has a fault, the detection result will point out the specific fault node.

[0145] In some embodiments, node fault detection is a detection and diagnosis technology for identifying and reporting service nodes with faults in service clusters of a service system. Fault service cluster refers to those service clusters that have faults due to node problems. During node fault detection, each node is checked to see if it is working normally, and data about the health status of the node is collected.

[0146] In some embodiments, referring to Figure 7 , Figure 7 is the flowchart of the fault repair method of the service system provided by the embodiments of the present application Figure 3 , Figure 4 The step 103 shown in Figure 7 is implemented by steps 1031 to 1032.

[0147] In step 1031, the running status of each service node in the fault service cluster is obtained.

[0148] In some embodiments, the running status includes multiple sub-running statuses such as CPU utilization, memory utilization, etc. of the service node. The running status refers to the current working condition of a single service node (such as a server, a virtual machine, or other devices) in the service cluster. This status usually includes multiple indicators to reflect the health and performance of the service node. The running status particularly focuses on the service nodes in the fault service cluster, which are the key components of the cluster responsible for processing service requests.

[0149] In some embodiments, the running status includes but is not limited to the following multiple sub-running statuses: CPU utilization, which measures the proportion of time that the CPU of the service node is executing tasks to the total time, to evaluate the computing load and processing capacity of the service node. Memory utilization, which reflects the usage of the memory of the service node, including the proportion of used memory and free memory, to evaluate the memory load of the service node and whether there is a memory shortage problem. Disk space utilization, which represents the usage of the disk storage space of the service node, including the proportion of used space and available space, to detect whether the disk space is sufficient. Network traffic, which represents the amount of data entering and exiting the network of the service node, to evaluate the network load and network performance of the node.

[0150] In step 1032, based on the running status of the service node, the service node in the fault service cluster is detected to obtain the service node with fault in the fault service cluster.

[0151] In some embodiments, node fault detection is used to identify and report the service nodes with fault in the service cluster of the service system. The fault service cluster refers to the service cluster that is faulty due to node problems. Node fault detection can be performed based on the running status of each service node to check whether each service node is working normally.

[0152] In some embodiments, the service node with fault in the fault service cluster is used to represent the service node with fault in the fault service cluster.

[0153] In some embodiments, the step 1032 can be implemented by: obtaining detection corresponding node prompt information; constructing second input information of the fault detection model based on the node prompt information and the running state of each service node in the fault service cluster; calling the fault detection model, detecting the fault service cluster based on the second input information, obtaining the probability of each service node in the fault service cluster being faulty, and determining the service node in the fault service cluster being faulty based on the probability of each service node in the fault service cluster being faulty.

[0154] In some embodiments, the determination of the service node in the fault service cluster being faulty based on the probability of each service node in the fault service cluster being faulty can be implemented by: determining the service node with a probability greater than a node probability threshold as the service node in the fault service cluster being faulty.

[0155] In some embodiments, the service node with a probability not greater than the node probability threshold is determined as the service node in the fault service cluster not being faulty.

[0156] In some embodiments, the second input information includes the node prompt information and the running state of each service node in the fault service cluster, and the second input information is obtained by fusing and formatting the node prompt information and the running state of each service node in the fault service cluster. The second input information is an input format that can be accepted and processed by the fault detection model, and is the basis for learning and reasoning of the fault detection model, and is the basis for identification and prediction of the fault detection model.

[0157] In some embodiments, the node prompt information is used as a reference basis for the fault detection model to detect the faulty service node.

[0158] In some embodiments, the node prompt information can be fault diagnosis knowledge, which is used to classify the cluster fault into single-machine-room single-node fault, single-machine-room multi-node fault, or multi-machine-room multi-node fault according to the index data and node attributes (machine room information) of the nodes in the cluster.

[0159] As an example, see Figure 19The node prompt information can be role setting: assume you are an intelligent agent for cluster fault self-healing, do not fabricate or falsify facts, and please answer the question according to [reference knowledge]. Reference knowledge 1: the current multiple node data of a single cluster is given, including normal nodes and abnormal nodes, according to whether the node index is too high, try to classify the cluster fault as single-machine-room single-node fault, single-machine-room multi-node fault or multi-machine-room multi-node fault; if it cannot be confirmed, notify manual intervention for processing. Reference knowledge 2: [real-time cluster node data]. Answer the question: whether an abnormal node is found? The answer format is to return in json format, and no explanation is required. Answer example: ("output": "node X", "next_step": "fault repair").

[0160] As an example, see Figure 19 The running state of the business node can be: real-time cluster node data: [node A, 30%, 30%,...]; [node B, 80%, 90%,...]; [node C, 30%, 30%,...]; [node D, 30%, 30%,...], the data format is [node name, CPU utilization, memory utilization,...].

[0161] Continuing the example, see Figure 19 , obtain the node prompt information corresponding to the node fault detection, and based on the node prompt information and the running state of each business node in the fault business cluster (such as cluster A shown in Figure 19 ), construct the second input information of the fault detection model; call the fault detection model, and based on the second input information, perform node fault detection on the fault business cluster (such as cluster A shown in Figure 19 ), obtain the fault business node in the fault business cluster of the fault business cluster, and output the fault business node in cluster A: answer: {"out put": "node B", "next_step": "fault repair"}.

[0162] In some embodiments, the fault detection model can be a large language model. A large language model, also known as a large model or a large language model, is a deep learning model trained using a large amount of text data, containing billions of parameters or even more, which can be used to generate natural language text and understand the meaning of natural language text. Through training, the model can learn the statistical rules and semantic associations of language to establish a large language knowledge base, and then simulate human language understanding and generation capabilities.

[0163] As an example, see Figure 6 , Figure 6 is a model structure diagram of the fault detection model provided by the embodiments of the present application, Figure 6The illustrated fault detection model includes a feature extraction layer 1 and a fault detection layer 2. The fault detection model is called based on the second input information to perform node fault detection on the fault service cluster to obtain the service node with the fault in the fault service cluster of the fault service cluster. This can be achieved by: calling the feature extraction layer 1 of the fault detection model to perform feature extraction on the second input information to obtain node detection features; calling the fault detection layer 2 of the fault detection model to perform node fault detection on each service cluster based on the node detection features to obtain the service node with the fault in the fault service cluster of the fault service cluster.

[0164] In this way, by obtaining the node prompt information corresponding to the node fault detection, the node that may have problems can be more accurately located. Based on the node prompt information and the running state, the fault detection model can quickly identify and judge the fault node, thereby shortening the fault diagnosis time and accelerating the fault response speed. Through the automatic fault detection model, the dependence on human experts can be reduced, especially when dealing with a large number of nodes and complex faults, the model can provide efficient fault detection results without manual checking one by one. Timely and accurate node fault detection can help system administrators isolate fault nodes in time to prevent fault propagation and maintain the stability of the entire system. By continuously detecting the running state of the node and using the fault detection model for analysis, potential faults can be predicted and preventive maintenance can be performed to avoid the impact of faults on business. The application of the fault detection model helps to ensure the continuity of key business processes, and even when facing node faults, the model can quickly identify and take measures to reduce business interruption time.

[0165] In some embodiments, the step 1032 can also be implemented by: for each service node in the fault service cluster, performing the following processing: comparing the running state of the service node with the state threshold to obtain a running comparison result, and based on the running comparison result, determining a sub-fault detection result of the service node; when the sub-fault detection result of the service node indicates that the service node has the fault, determining the service node as the service node with the fault in the fault service cluster.

[0166] In some embodiments, the running comparison result is used to represent whether the running state of the service node is greater than the state threshold.

[0167] In some embodiments, the running state includes a plurality of sub-running states, and the state threshold includes a sub-state threshold corresponding to each sub-running state.

[0168] In some embodiments, the determining of the sub-failure detection result of the service node based on the running comparison result can be implemented in the following manner: when the running comparison result indicates that each sub-running state is greater than or equal to the corresponding state threshold, the sub-failure detection result is determined as that the service node does not exist failure; when the running comparison result indicates that there exists a sub-running state less than the corresponding state threshold, the sub-failure detection result is determined as that the service node exists failure.

[0169] In some embodiments, one or more thresholds need to be set for the running state of each sub-node, which are determined based on the performance indicators under normal running conditions, such as CPU utilization, memory utilization, response time, etc. The running state data of each sub-node is collected regularly, including indicators such as CPU utilization and memory utilization. The currently collected sub-running state data is compared with the preset state threshold. According to the comparison result, if the running state of all sub-nodes is greater than or equal to the corresponding state threshold, it is considered that the service node does not exist failure. This means that all key indicators are within the normal range, and the service node should be considered healthy. If the running state of at least one sub-node is less than the corresponding state threshold, it is considered that the service node exists failure, which means that at least one key indicator is not within the normal range, and the performance of the service node may be affected.

[0170] As an example, the running state includes sub-running state A and sub-running state B, sub-running state A is greater than or equal to the corresponding state threshold, and sub-running state B is greater than or equal to the corresponding state threshold. Then the running comparison result indicates that each sub-running state is greater than or equal to the corresponding state threshold, and the sub-failure detection result is determined as that the service node does not exist failure.

[0171] In this way, by setting specific state thresholds for each sub-node, it can be more accurately determined whether the node exists failure, avoiding excessive or unnecessary failure alarms. The rapid generation of the running comparison result helps the system administrator to quickly identify the failure node, so that immediate measures can be taken, such as re-distributing the load or isolating the failure node, to prevent the failure from further affecting the service. Since the running states of multiple sub-nodes and their corresponding sub-state thresholds are considered, more detailed failure analysis can be provided to help administrators understand the specific reasons for the failure. By detecting and analyzing the running state, resource allocation can be optimized, for example, tasks can be redistributed to healthy nodes, or heavily loaded nodes can be upgraded or expanded. Timely failure detection and response help to reduce the time and frequency of service interruptions, ensuring the continuity and stability of the service.

[0172] In some embodiments, the cause of the failure of the service node in the fault service cluster can be a failure of the communication connection between the service nodes in the fault service cluster, or a failure of the service node itself. Then, the service node in the fault service cluster that has the above failure cause can be determined as the failed service node.

[0173] In step 104, based on the number of failed service nodes in the fault service cluster and the number of fault service clusters, a fault recovery mode of the service system is determined from a plurality of preset fault recovery modes.

[0174] In some embodiments, the fault recovery mode includes but is not limited to a disaster recovery system recovery mode, a node isolation recovery mode, and a cluster isolation recovery mode.

[0175] In some embodiments, the fault recovery mode refers to a series of measures and technical means taken when the service system or its components (such as service clusters or nodes) fail, aiming to restore the normal operation of the system or mitigate the impact of the failure. The selection of the fault recovery mode is usually based on the fault detection results and the specific needs of the service system. The disaster recovery system recovery mode is a backup or redundant system that takes over business processing when the main system fails. This mode usually includes data replication, mirror sites, hot backups, etc., to ensure that the business can quickly switch to the disaster recovery system for continuous operation in the event of a failure. The node isolation recovery mode can isolate the failed node to prevent the failure from spreading to other healthy nodes when one or more nodes in the service cluster fail. This may include removing the failed node from the cluster, disabling services on the failed node, or isolating it from other nodes. The cluster isolation recovery mode may be needed when the entire service cluster is affected by a failure to prevent further impact on other systems or businesses. This may involve stopping service request processing for the cluster, starting a backup cluster, or taking other necessary isolation measures.

[0176] In some embodiments, referring to Figure 8 , Figure 8 is the flowchart of the fault recovery method of the service system provided by the embodiments of the present application Figure 4 , Figure 4 The step 104 shown in Figure 8 The steps 1041A to 1043A shown in

[0177] In step 1041A, the mode prompt information corresponding to the fault recovery mode is obtained.

[0178] In some embodiments, the mode prompt information is used as a reference basis for the fault detection model in determining the fault repair mode.

[0179] In some embodiments, the above-mentioned mode prompt information can be fault repair knowledge, which is used to determine the most appropriate fault repair plan according to different cluster fault types and fault repair plans.

[0180] As an example, see Figure 19 The above-mentioned mode prompt information can be role setting: assuming that you are an intelligent agent for cluster fault self-healing, do not fabricate or fabricate facts, please answer the question according to [reference knowledge]. Reference knowledge 1: According to different cluster fault types and fault repair plans, determine the most appropriate fault repair plan, single-machine room single-node fault repair plan is device isolation, single-machine room multi-node fault repair plan is machine room isolation, and multi-machine room multi-node fault repair plan is cluster migration; if you cannot make a decision, please notify manual intervention for processing Reference knowledge 2: [real-time abnormal node data] answer the question: what is the current best repair plan? Answer format: return in json format, no explanation is required. Answer example: ("output": "device isolation", "next_step": "fault recovery confirmation").

[0181] In step 1042A, based on the mode prompt information, the service node with a fault in the fault service cluster, and the number of fault service clusters, the third input information of the fault detection model is constructed.

[0182] In some embodiments, the service node with a fault in the fault service cluster is used to represent the service node with a fault in the fault service cluster, and the number of fault service clusters is less than or equal to the total number of service clusters in the service system.

[0183] In some embodiments, the above-mentioned step 1042 can be realized by splicing the mode prompt information, the service node with a fault in the fault service cluster, and the number of fault service clusters to obtain the third input information of the fault detection model.

[0184] In some embodiments, the above-mentioned third input information includes mode prompt information, service node with a fault in the fault service cluster, and the number of fault service clusters, the third input information is obtained by fusing and formatting the mode prompt information, the service node with a fault in the fault service cluster, and the number of fault service clusters, the third input information is an input format that can be accepted and processed by the fault detection model, the third input information is the basis for the fault detection model to learn and reason, and is the basis for the fault detection model to identify and predict faults.

[0185] In step 1043A, the fault detection model is called to determine the probability of each preset fault repair method based on the third input information, and the preset fault repair method with a probability greater than a preset probability threshold is determined as the determined fault repair method.

[0186] In some embodiments, the above-mentioned fault detection model can be a large language model. The large language model, also known as a large model, is a deep learning model trained using a large amount of text data, containing billions of parameters or even more, which can be used to generate natural language text and understand the meaning of natural language text. Through training, the model can learn the statistical rules and semantic associations of language to establish a large language knowledge base, and then simulate human language understanding and generation capabilities.

[0187] As an example, refer to Figure 6 , Figure 6 is a model structure diagram of the fault detection model provided by the embodiments of the present application, Figure 6 The fault detection model shown in the figure includes a feature extraction layer 1 and a fault detection layer 2. The above-mentioned calling of the fault detection model to determine the fault repair method based on the third input information can be realized in the following way: calling the feature extraction layer 1 of the fault detection model to extract features from the third input information to obtain repair method features, and calling the fault detection layer 2 of the fault detection model to determine the fault repair method from multiple candidate modification methods based on the repair method features.

[0188] In this way, by using the fault detection model, the third input information can be automatically constructed, and the fault repair method can be determined accordingly, reducing manual intervention and improving the automation level of the repair process. Quickly analyze the fault condition and the state of the business cluster, so as to quickly determine the most suitable repair method and speed up the fault recovery speed. By timely determining the fault repair method, the fault node or cluster can be effectively isolated to prevent the fault from further spreading to other healthy nodes or systems. By quickly determining and implementing the fault repair method, the impact of the fault on business operation can be minimized to ensure the continuity and stability of the business.

[0189] In some embodiments, the preset fault repair method includes a disaster recovery system repair method, refer to Figure 9 , Figure 9 is a flowchart of the fault repair method of the business system provided by the embodiments of the present application Figure 5 , Figure 4 The step 104 shown in the figure can be implemented by Figure 9 The steps 1041B to 1043B shown in the figure.

[0190] In step 1041B, when the number of the fault service clusters is greater than or equal to the cluster number threshold, and the number of the fault service nodes in each fault service cluster is greater than the node number threshold, the fault repair mode is determined as the disaster recovery system repair mode.

[0191] In some embodiments, the disaster recovery system repair mode repairs the fault of the service system through the disaster recovery system.

[0192] In some embodiments, when the number of the fault service clusters is greater than or equal to the cluster number threshold, and the fault service nodes in each fault service cluster all indicate that the number of the fault service nodes in the fault service cluster is greater than the node number threshold, it means that each fault service cluster has fault service nodes greater than the node number threshold, which means that the fault of each fault service cluster is relatively serious. At the same time, the number of the fault service clusters is greater than or equal to the cluster number threshold, that is, the number of the fault service clusters in the service system is greater than or equal to the cluster number threshold, which means that there are fault service clusters greater than or equal to the cluster number threshold in the service system, which means that the fault of the entire service system is relatively serious. Repairing all existing faults in the service system will take a long time, so the fault repair mode is determined as the disaster recovery system repair mode, which repairs the fault of the service system through the disaster recovery system, thereby directly replacing the service system with a large number of faults through the disaster recovery system, thereby effectively improving the fault repair efficiency of the service system.

[0193] As an example, referring to Figure 16 , when the number of the fault service clusters in the service system Figure 16 (shown as the gateway cluster A) is greater than or equal to the cluster number threshold (for example, the cluster number threshold is 2, and the machine room A and the machine room B in the gateway cluster A are both fault service clusters), and the fault service nodes in each fault service cluster all indicate that the number of the fault service nodes in the fault service cluster is greater than the node number threshold (for example, the node number threshold is 2, the server A and the server B in the machine room A are fault service nodes, and the server E, the server F, and the server G in the machine room B are fault service nodes), the fault repair mode is determined as the disaster recovery system repair mode, which repairs the fault of the service system (such as Figure 16 (shown as the gateway cluster A) through the disaster recovery system (for example, the disaster recovery system such as Figure 16 (shown as the gateway cluster B)). Figure 16

[0194] In step 1042B, when the number of the fault service clusters is less than the cluster number threshold, for each fault service cluster, the cluster repair mode of the fault service cluster is determined from a plurality of preset fault repair modes based on the fault service nodes in the fault service cluster. ​

[0195] In some embodiments, when the number of fault service clusters is less than the cluster number threshold, it indicates that there are less than the cluster number threshold of fault service clusters in the service system, which indicates that the fault of the entire service system is relatively slight, and the time consumption for repairing all the faults in the service system is not too large, so the fault repair mode can not be determined as the disaster recovery system repair mode, and the cluster repair mode of the fault service cluster is determined from the plurality of preset fault repair modes based on the fault service nodes in the fault service cluster, thereby effectively improving the fault repair efficiency of the service system.

[0196] In some embodiments, the preset fault repair mode includes a cluster isolation repair mode and a node isolation repair mode, and the cluster repair mode of the fault service cluster is determined from the plurality of preset fault repair modes based on the fault service nodes in the fault service cluster, which can be implemented in the following manner: when the number of fault service nodes in the fault service cluster is greater than the node number threshold, the cluster repair mode of the fault service cluster is determined as the cluster isolation repair mode; and when the number of fault service nodes in the fault service cluster is not greater than the node number threshold, the cluster repair mode of the fault service cluster is determined as the node isolation repair mode.

[0197] In some embodiments, the node isolation repair mode is used to isolate the fault service nodes in the fault service cluster, and the cluster isolation repair mode is used to isolate the fault service cluster in the service system.

[0198] In some embodiments, when the number of fault service nodes in the fault service cluster is greater than the node number threshold, it indicates that there are many fault service nodes in the fault service cluster, and the time consumption for repairing all the fault nodes in the fault service cluster is large, so the cluster repair mode of the fault service cluster can be determined as the cluster isolation repair mode, and the cluster isolation repair mode is used to isolate the fault service cluster in the service system, so that the service can be migrated to other service clusters, thereby effectively improving the fault repair efficiency of the service system.

[0199] In step 1043B, the fault repair mode of the service system is determined based on the cluster repair modes.

[0200] In some embodiments, the cluster repair mode is used to repair the corresponding fault service cluster, the fault repair mode is used to repair each fault service cluster in the service system, and the fault repair mode includes the cluster repair mode.

[0201] In some embodiments, the method for determining the fault recovery mode of the business system based on the cluster recovery mode can be implemented as follows: when the number of fault business clusters is one, the cluster recovery mode is determined as the fault recovery mode of the business system; when the number of fault business clusters is multiple, the combination of the cluster recovery modes of each fault business cluster is determined as the fault recovery mode of the business system.

[0202] For example, when the fault business clusters in the business system are fault business cluster A, fault business cluster B, and fault business cluster C, the combination of the cluster recovery mode corresponding to fault business cluster A, the cluster recovery mode corresponding to fault business cluster B, and the cluster recovery mode corresponding to fault business cluster C is determined as the fault recovery mode of the business system: {the cluster recovery mode corresponding to fault business cluster A, the cluster recovery mode corresponding to fault business cluster B, and the cluster recovery mode corresponding to fault business cluster C}.

[0203] In this way, by setting the cluster quantity threshold and the node quantity threshold, those faults with extensive impact can be responded to preferentially, and resources and services can be allocated according to the importance and urgency of the faults. The fault recovery mode is dynamically determined according to the number of fault business clusters and the number of fault business nodes in the fault business cluster, so that the repair strategy is more flexible and adaptive. When the number of fault business clusters and the number of fault nodes exceeds the preset threshold, the disaster recovery system recovery mode is automatically switched to, the service is quickly recovered, and the business interruption time is reduced. When the number of fault business clusters is less than the cluster quantity threshold, the cluster recovery mode is determined for each fault business cluster separately, and more refined fault isolation and repair are performed. Through the repair mode selection based on the fault detection result, resource allocation can be optimized, and resources can be effectively used for key business recovery links. Through timely and accurate determination of the fault recovery mode, the continuity and availability of the business can be improved, and the stability of the key business process can be ensured. By using the disaster recovery system and the cluster isolation repair mode, the resilience of the business system to faults is enhanced, and the ability of the business system to cope with large-scale faults is improved.

[0204] In step 105, the business system is repaired based on the fault recovery mode.

[0205] In some embodiments, when the fault recovery mode includes the cluster recovery mode corresponding to each fault business cluster, the fault business cluster is repaired according to the cluster recovery mode corresponding to the fault business cluster.

[0206] As an example, the fault repair mode includes a cluster isolation repair mode corresponding to the fault service cluster A and a node isolation repair mode corresponding to the fault service cluster B. For the fault service cluster A, the fault service cluster A in the service system is isolated, so that the service taken by the fault service cluster A can be migrated to other service clusters. For the fault service cluster B, the service node with a fault in the fault service cluster B is isolated, so that the service taken by the service node with a fault in the fault service cluster B can be migrated to other service nodes in the fault service cluster B.

[0207] In some embodiments, referring to Figure 10 , Figure 10 is a flowchart of a fault repair method of a service system provided by an embodiment of the present application Figure 6 , Figure 4 The step 105 shown in the figure can be implemented by the steps 1051 to 1052 shown in the figure. Figure 10

[0208] In the step 1051, when the fault repair mode is the disaster recovery system repair mode, the disaster recovery system is acquired.

[0209] In some embodiments, the disaster recovery system includes at least one disaster recovery service cluster, and the disaster recovery service cluster does not have a fault.

[0210] As an example, referring to Figure 16 , the disaster recovery system is the gateway cluster B shown in the figure, the service system is the gateway cluster A shown in the figure, the disaster recovery system includes at least one disaster recovery service cluster (such as the machine room A and the machine room B in the gateway cluster B shown in the figure), and the disaster recovery service cluster does not have a fault. Figure 16 Figure 16 Figure 16

[0211] In the step 1052, the service data in the service system is migrated to the disaster recovery system to implement fault repair of the service system.

[0212] As an example, referring to Figure 16 , the service data in the service system (such as the gateway cluster A shown in the figure) is migrated to the disaster recovery system (such as the gateway cluster B shown in the figure) through cross-cluster service migration to implement fault repair of the service system. Figure 16 Figure 16

[0213] ​​​​​​Thus, by performing fault detection on the service clusters, cluster fault detection results of the service clusters are obtained, a service cluster with an indication of a fault is determined as a fault service cluster, service nodes in each fault service cluster are detected, service nodes with faults in each fault service cluster are obtained, based on the service nodes with faults in the fault service cluster and the number of fault service clusters, a fault repair mode of the service system is determined from a plurality of preset fault repair modes, and the service system is repaired based on the fault repair mode. In this way, by determining the fault repair mode of the service system from a plurality of preset fault repair modes based on the service nodes with faults in the fault service cluster and the number of fault service clusters, and repairing the service system based on the fault repair mode, the determined fault repair mode can adapt to the fault of the service system, thereby enabling efficient fault repair of the service system, and thus effectively improving the fault repair efficiency of the service system.

[0214] In the following, an exemplary application of the embodiment of the present application in an actual gateway cluster repair application scenario will be described.

[0215] The cloud private line product is one of the important ways for customers to access the cloud. In order to ensure the reliability of the product, the cloud private line gateway adopts a cluster deployment mode with dual-machine rooms and multiple nodes.

[0216] Based on the meta large model, the embodiment of the present application designs a fault self-healing agent for a cloud private line gateway cluster. In the fault scenarios of the gateway cluster, such as single-machine room single-node fault, single-machine room multi-node fault, and multi-machine room multi-node fault, the agent can autonomously make decisions and perform tasks based on real-time external environment data after running a preset thinking chain, so as to complete the goal of cluster fault self-healing.

[0217] The fault self-healing agent provided by the embodiment of the present application can achieve active fault discovery: the health detection system is through active PING detection. In combination with the detection data of the gateway cluster and based on the data analysis capability of the meta large model, the fault of the cluster can be discovered in a timely manner. Intelligent fault diagnosis: in combination with the feature data such as alarms, logs, traffic, and port states of the equipment, the abnormal nodes in the cluster are intelligently diagnosed based on the logical reasoning capability of the meta large model. Automatic fault repair: after the root cause is located by the intelligent fault diagnosis module above, that is, after the abnormal nodes are confirmed, the optimal fault repair mode is adopted for different cluster fault scenarios based on the decision-making capability of the meta large model. In summary, the embodiment of the present application solves the problems of slow fault discovery, low diagnosis efficiency, and slow repair speed in the single-node fault scenario of the traditional gateway cluster.

[0218] In some embodiments, referring to Figure 11 , Figure 11This is a flowchart illustrating the fault repair method for a business system provided in the embodiments of this application. Figure 7 The fault repair method for the business system provided in this application embodiment can be achieved through... Figure 11 Steps 201 to 202 shown are implemented.

[0219] In step 201, the gateway cluster detects cluster faults.

[0220] In step 202, in response to a cluster failure, the health monitoring module, intelligent diagnosis module, and fault self-healing module are invoked to perform fault recovery.

[0221] In some embodiments, cloud customers may experience slight service fluctuations.

[0222] In some embodiments, the deployment method of the gateway cluster involved in this application is as follows: Figure 12 As shown, Figure 12 This is a schematic diagram of the gateway cluster provided in the embodiments of this application. The gateway cluster includes at least 4 servers, which are deployed in 2 different data centers (e.g., Figure 12 The data centers A and B shown are configured such that data center A includes servers A, B, C, and D, while data center B includes servers E, F, G, and H, thus achieving a high-availability disaster recovery architecture across data centers.

[0223] In some embodiments, see Figure 13 , Figure 13 This is a schematic diagram illustrating the principle of health detection for a gateway cluster provided in this application embodiment. The gateway cluster (e.g., such as...) Figure 13 The gateway clusters A and B shown have a health detection system, the source of which is the detection server at the access point (e.g., ...). Figure 13 The access point detection servers A and B shown simulate a cloud customer accessing the cloud network through a cloud private line product; the destination (endpoint) is a cloud server (e.g., such as...). Figure 13 The cloud servers A and B shown are used by a health detection system to simulate cloud customers purchasing business resources on the cloud. The detection stream fully covers the network path of the cloud dedicated line. However, due to the passage through the switches and routers of the cloud infrastructure network, fluctuations in detection quality caused by basic network jitter are unavoidable. To eliminate interference, this embodiment of the application designs multiple detection streams with different sources and destinations. The health detection system provides important reference data for the fault-healing intelligent agent.

[0224] In some embodiments, the goal of the fault self-healing agent is to achieve cluster fault self-healing. This application embodiment pre-sets three cluster fault self-healing plans to deal with different cluster fault scenarios. The agent will make the best recovery plan selection and execution based on real-time cluster fault data.

[0225] In some embodiments, referring to Figure 14 , Figure 14 is a principle diagram of fault repair provided by an embodiment of the present application Figure 1 , for single-room single-node fault, in this scenario, only single-node fault (for example, server A in room A as shown in Figure 14 fails), the best recovery scheme is to isolate the faulty node (for example, server A in room A as shown in Figure 14 ), that is, device isolation, wherein room A of the gateway cluster includes server A, server B, server C and server D, and room B of the gateway cluster includes server E, server F, server G and server H, which will temporarily cause the cluster to lose 1 / N of the cluster bandwidth capacity, where N is the total number of nodes of the cluster.

[0226] In some embodiments, referring to Figure 15 , Figure 15 is a principle diagram of fault repair provided by an embodiment of the present application Figure 2 , for single-room multi-node fault (for example, server A and server C in room A as shown in Figure 15 fail), room A of the gateway cluster includes server A, server B, server C and server D, and room B of the gateway cluster includes server E, server F, server G and server H, in this scenario, multiple node abnormalities occur in a single room, which can be regarded as the whole room being unreliable, and the best recovery scheme is to isolate the entire room, that is, room isolation, which will temporarily cause the cluster to lose one-half of the cluster bandwidth capacity.

[0227] In some embodiments, referring to Figure 16 , Figure 16 is a principle diagram of fault repair provided by an embodiment of the present application Figure 3 , for double-room multi-node fault, in this scenario, node abnormalities occur in nodes in both rooms (for example, server A and server C in room A are abnormal, and server F in room B is abnormal), which can be regarded as the whole cluster (gateway cluster A as shown in Figure 16 ) being unavailable, and the best recovery scheme is to migrate the entire cluster service (service of gateway cluster A) to a new cluster (gateway cluster B). Room A of gateway cluster A includes server A, server B, server C and server D, room B of gateway cluster A includes server E, server F, server G and server H, room B of gateway cluster B includes server I, server J, server K and server L, and room B of gateway cluster B includes server M, server N, server O and server P, but this scheme requires that the existing network has a backup disaster recovery cluster (gateway cluster B as shown in Figure 16The gateway cluster B) shown.

[0228] In some embodiments, referring to Figure 17 , Figure 17 is a principle schematic diagram of a fault self-recovery intelligent agent provided by the embodiments of the present application. The fault self-recovery intelligent agent has four basic frameworks: memory, task decomposition, tool library, and action library. The memory is a module for the intelligent agent to store the state of the external system environment and knowledge, including short-term memory (context conversation) and long-term memory (business knowledge base). The business knowledge base includes quasi-real-time data of the surrounding system perceived through the tool library, such as gateway cluster detection data, server running state, business traffic, and the like. In addition, it also includes professional business definitions, such as a description of the deployment mode of the cluster and a description of the health detection data. The business knowledge base is an important reference for the intelligent agent to think and make decisions. The task decomposition is to decompose a complex task into multiple simple subtasks. The embodiments of the present application are for a specific gateway cluster fault self-recovery scenario, so the way of task decomposition is relatively fixed, that is, fault discovery, fault diagnosis, and fault repair. The intelligent agent does not need to care about how to perform task decomposition temporarily, but focuses on how to think and make decisions for each subtask. The tool library includes API calling methods of surrounding systems, such as network management and controller systems. Various system APIs form various actions in the action library, which can be executed by the intelligent agent to perceive and affect the surrounding systems. The action library is a module for the intelligent agent to perceive and affect the external system environment. The action library of the fault self-recovery intelligent agent includes three fault recovery actions: device isolation, computer room isolation, and cluster migration. In addition, it also includes actions for perceiving the surrounding systems, such as querying cluster health data, querying server running state, and the like.

[0229] In some embodiments, referring to Figure 18 , Figure 18is an execution flow diagram of a fault self-healing intelligent agent of a gateway cluster provided by the embodiment of the present application. The intelligent agent is a software program developed based on the meta large model. It can run in a periodic trigger or event trigger manner. The periodic trigger is to run the intelligent agent once every 5 minutes. The event trigger is manually triggered by an operation and maintenance personnel. The start of each running of the intelligent agent is to think about a question: whether the gateway cluster has an exception? After the intelligent agent identifies that the question is a domain problem involved by it, it triggers a fixed thinking chain to split the question into multiple sub-tasks, i.e. fault discovery (perception), fault diagnosis, and fault repair. Each sub-task, i.e. the thinking of the intelligent agent, is an intelligent decision made after the meta large model combines a tool library (Tools) (gateway API, controller API and other APIs), an action library (Actions) (including device isolation, computer room isolation, cluster migration and other actions) and a memory cache (Memory). The intelligent agent will make its thinking and decision more in line with the experience of “current network operation and maintenance” as the basic capability of the meta large model improves and the business knowledge base becomes richer. After the intelligent agent completes the running of the entire thinking chain, it will draw a conclusion according to its execution plan and current network data and return a final answer to inform the operation and maintenance personnel whether the cluster fault is completed self-healing. For example, the first sub-task of the thinking chain is fault discovery. The specific implementation is to rely on the template of the hint engineering of the meta large model. The input of the meta large model is the template of the hint engineering combined with real-time current network data. The output of the current sub-task is one of the input knowledge of the next sub-task, i.e. the next call of the meta large model.

[0230] In some embodiments, referring to Figure 18 , the question is: whether the gateway cluster has an exception? The thinking is: “is the question related to the cluster fault self-healing?” The sub-task is fault discovery. The thinking is: “query the health probe data of the gateway cluster”. The action is: “call the network management API to query the health probe data”. The observation is: “the packet loss is concentrated in cluster A, and cluster A is faulty”. The action is to call the API of the health probe system to query the exploration data of the gateway cluster. The probe data sample is: [cluster A, source IP1, destination IP1, 100%], [cluster A, source IP2, destination IP2, 100%], [cluster A, source IP3, destination IP3, 100%], [cluster A, source IP4, destination IP4, 100%].

[0231] In some embodiments, referring to Figure 18, subtask: fault diagnosis, thinking: "cluster A failed, diagnose the specific fault node", action: "call the controller API to query the node status", observation: "node A metrics are abnormal, node A is the abnormal node". Action: Call the controller API to query the server's state information such as CPU, memory, process state, etc. Node metrics data: [node A, CPU, memory,...], [node B, CPU, memory,...], [node C, CPU, memory,...], [node D, CPU, memory,...].

[0232] In some embodiments, referring to Figure 18 , subtask: fault repair, thinking: "find a node, execute the corresponding repair action", action: "call the gateway API to query node data", observation: "single-machine room single-node failure, execute device isolation". Action: Call the controller's API to operate the cluster fault repair plan. Abnormal node: [node A, machine room A,...].

[0233] In some embodiments, for the memory cache of the agent, 1. Fault discovery knowledge: based on the exploration data of the cluster, perform data commonality analysis, try to classify the fault as a cluster fault or a non-cluster fault, and perform cluster fault diagnosis for the cluster fault. Fault diagnosis knowledge: based on the various index data of the nodes in the cluster and the node attributes (machine room information), try to classify the cluster fault as single-machine room single-node fault or single-machine room multi-node fault or multi-machine room multi-node fault. Fault repair knowledge: based on different cluster fault types and fault repair plans, decide to execute the most appropriate fault repair plan. Business data knowledge: cluster health detection data, cluster abnormal node data, etc. Other knowledge.

[0234] In some embodiments, referring to Figure 19 , Figure 19 is a prompt engineering template application principle schematic diagram provided by the embodiment of the application. Real-time cluster detection data: [cluster A: 1.1.1.1, 2.2.2.1, 100%], [cluster A, 1.1.1.2, 2.2.2.2, 100%], [cluster A, 1.1.1.3, 2.2.2.3, 100%], [cluster A, 1.1.1.4, 2.2.2.4, 100%], [cluster B, 1.1.1.1, 2.2.2.1, 0%], [cluster B, 1.1.1.2, 2.2.2.2, 0%], [cluster B, 1.1.1.3, 2.2.2.3, 0%], [cluster B, 1.1.1.4, 2.2.2.4, 0%]. The data format is [cluster A, source IP, destination IP, packet loss data].

[0235] In some embodiments, referring to Figure 19, the prompt template for fault discovery: role setting: assume you are an intelligent agent for cluster fault self-healing, do not fabricate or fabricate facts, please answer the question according to [reference knowledge]. Reference knowledge 1: only when the packet loss data of cluster detection data is concentrated in a certain cluster, it is classified as single cluster fault; when multiple clusters have packet loss, it is classified as non-single cluster fault, and manual intervention is notified for processing; when there is no packet loss, it is classified as no fault, reference knowledge 2: [real-time cluster detection data], answer the question: whether the fault is found? Answer format: return in json format, no explanation is needed. Answer example: ("output": "single cluster fault (fault cluster: cluster x)", "next_step": "fault diagnosis").

[0236] In some embodiments, referring to Figure 19 , input the above fault discovery prompt template into the large model to get the answer: ("output": "single cluster fault (fault cluster: cluster A)", "next_step": "fault diagnosis"). Action: query the node data of cluster A to get real-time cluster node data: [node A, 30%, 30%,...], [node B, 80%, 90%,...], [node C, 30%, 30%,...], [node D, 30%, 30%,...]. The data format is [node name, CPU utilization, memory utilization,...].

[0237] In some embodiments, referring to Figure 19 , the prompt template for fault diagnosis: role setting: assume you are an intelligent agent for cluster fault self-healing, do not fabricate or fabricate facts, please answer the question according to [reference knowledge]. Reference knowledge 1: the current single cluster node data contains normal nodes and abnormal nodes, according to whether the node index is too high, try to classify the cluster fault as single room single node fault or single room multiple node fault or multiple room multiple node fault; if it cannot be confirmed, notify manual intervention for processing. Reference knowledge 2: [real-time cluster node data]. Answer the question: whether the abnormal node is found? Answer format: return in json format, no explanation is needed. Answer example: ("output": "node X", "next_step": "fault repair").

[0238] In some embodiments, referring to Figure 19 , input the above fault diagnosis prompt template into the large model to get the answer: {"output": "node B", "next_step": "fault repair"}. Action: query the data of abnormal node B, real-time abnormal node data: [node B, machine room A,...], data format: [node name, machine room,...].

[0239] In some embodiments, referring toFigure 19 , prompt template for fault repair; role setting: assume you are an intelligent agent for cluster fault self-healing, do not fabricate or fabricate facts, please answer the question according to [reference knowledge]. Reference knowledge 1: according to different cluster fault types and fault repair plans, decide to execute the most appropriate fault repair plan. The single-machine-room single-node fault repair plan is device isolation, the single-machine-room multi-node fault repair plan is machine room isolation, and the multi-machine-room multi-node fault repair plan is cluster migration. If you cannot decide, please notify manual intervention for processing Reference knowledge 2: [real-time abnormal node data]; Answer the question: what is the current best repair plan? Answer format: return in json format, no explanation is required; Answer example: ("output": "device isolation", "next_step": "fault recovery confirmation").

[0240] In some embodiments, referring to Figure 19 , the above prompt template for fault repair is input into the large model to obtain the output answer: ("output": "device isolation", "next_step": "fault recovery confirmation").

[0241] In some embodiments, the fault self-healing intelligent agent of the embodiments of the present application can self-heal and repair the gateway cluster fault in a specific scenario. Based on the sub-task test sample of the intelligent agent, from the test effect, the intelligent agent has a certain degree of work ability similar to network operation and maintenance personnel, and can dynamically decide and handle the live network problems based on the fixed thinking chain and real-time live network data.

[0242] In some embodiments, referring to Figure 20 , Figure 20is the principle diagram of the fault discovery decision provided by the embodiment of the application, the role setting is: assuming that you are an intelligent agent of cluster fault self-recovery, do not fabricate or fabricate facts, answer briefly and strictly, strictly according to the answer format, please answer the question according to [reference knowledge] and [answer format]. Reference knowledge: current thinking stage: fault discovery. Fault discovery knowledge: only when the packet loss data of cluster detection data of all clusters are concentrated in a certain cluster, it is classified as single cluster fault; when packet loss exists in multiple clusters, it is classified as non-single cluster fault, and manual intervention is notified: when there is no packet loss, it is classified as no fault, cluster detection data: [cluster A: 1.1.1.1, 2.2.2.1, 100%], [cluster A, 1.1.1.2, 2.2.2.2, 100%], [cluster A, 1.1.1.3, 2.2.2.3, 100%], [cluster A, 1.1.1.4, 2.2.2.4, 100%], [cluster B, 1.1.1.1, 2.2.2.1, 0%], [cluster B, 1.1.1.2, 2.2.2.2, 0%], [cluster B, 1.1.1.3, 2.2.2.3, 0%], [cluster B, 1.1.1.4, 2.2.2.4, 0%], the cluster detection data format is [cluster A, source IP, destination IP, packet loss data]. [current question]: carry out fault discovery, reply according to the answer format. [answer format]: {"output": "single cluster fault (fault cluster: cluster X) non-single cluster fault | no fault", "next_step": "fault diagnosis | fault repair | notify manual intervention | none"}. [answer example]: {"output": "single cluster fault (fault cluster: cluster A)", "next_step": "fault diagnosis"}. Return in json format, no explanation. {"output": "single cluster fault (fault cluster: cluster A)", "next_step": "fault diagnosis"}.

[0243] In some embodiments, see Figure 21 , Figure 21This is a schematic diagram illustrating the principle of fault diagnosis decision-making provided in this application embodiment. Role setting: Assume you are an intelligent agent for cluster fault self-healing. Do not fabricate or falsify facts, answer concisely and rigorously, strictly follow the answer format, and refer to the answer example. Please answer the question based on [Reference Knowledge]. [Reference Knowledge]: Current thinking stage: Fault diagnosis. Fault diagnosis knowledge: Currently, multiple node data for a single cluster are given, including normal nodes and abnormal nodes. Based on whether the node indicators are too high, determine whether the node is an abnormal node; if an abnormal node is found, perform fault repair; if it cannot be determined, notify manual intervention. Node data: [Node A, 30%, 30%, Data Center A], [Node B, 80%, 90%, Data Center A], [Node C, 30%, 30%, Data Center B], [Node D, 30%, 30%, Data Center B]. The node data format is [Node Name, CPU Utilization, Memory Utilization, Data Center]. [Current Question]: Based on the indicator status of each node in the current single cluster, determine how many abnormal nodes there may be. [Answer Format]: {"output":"Node X; Node X", "next_step":"Fault Repair] Notify manual intervention"}. [Answer Example]: {"output":"Node A; Node C", "next_step":"Fault Repair"}. Returned in JSON format, no interpretation required. {"output":"Node B","next_step":"Fault Repair"}.

[0244] In some embodiments, see Figure 22 , Figure 22{"output": "device isolation", "next_step": "fault perception"}

[0245] In this way, by detecting the faults of the business clusters, the cluster fault detection results of each business cluster are obtained, the business cluster with a fault is indicated by the cluster fault result, is determined as a fault business cluster, and the business nodes in each fault business cluster are detected to obtain the business nodes with faults in each fault business cluster. Based on the business nodes with faults in the fault business cluster and the number of fault business clusters, a fault repair method of the business system is determined from a plurality of preset fault repair methods, and the business system is repaired based on the fault repair method. In this way, by determining the fault repair method of the business system from a plurality of preset fault repair methods based on the business nodes with faults in the fault business cluster and the number of fault business clusters, and repairing the business system based on the fault repair method, the determined fault repair method can adapt to the fault of the business system, thereby achieving efficient fault repair of the business system, and effectively improving the fault repair efficiency of the business system.

[0246] It can be understood that in the embodiments of the present application, data related to probe data is involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0247] The following continues to illustrate an exemplary structure of the implementation of the service system fault repair apparatus 455 provided by the embodiments of the present application as a software module. In some embodiments, as shown in Figure 2 The software module stored in the service system fault repair apparatus 455 of the memory 450 can include, as shown in the figure, a cluster detection module for performing fault detection on each of the service clusters to obtain cluster fault detection results of each of the service clusters; indicating the service cluster with a fault indicated by the cluster fault detection result as a fault service cluster; a node detection module for detecting each of the service nodes in each of the fault service clusters to obtain a service node with a fault in each of the fault service clusters; a fault repair module for determining a fault repair mode of the service system from a plurality of preset fault repair modes based on the service node with a fault in each of the fault service clusters and the number of the fault service clusters, and performing fault repair on the service system based on the fault repair mode.

[0248] In some embodiments, the cluster detection module described above is further configured to perform data detection on each of the service clusters in the service system to obtain a detection result of each of the service clusters, the detection result being used to represent a data packet loss rate of a detection data flow after passing through the service cluster; and determine whether each of the service clusters has a fault based on the detection result.

[0249] In some embodiments, the cluster detection module described above is further configured to perform the following processing for each of the service clusters: send detection data including a plurality of data packets to the service cluster to enable the service cluster to perform flow conversion on the detection data and output target detection data after the flow conversion; obtain a data packet quantity difference between the target detection data and the detection data, and determine a data packet loss rate of the service cluster based on the data packet quantity difference.

[0250] In some embodiments, the cluster detection module described above is further configured to obtain cluster prompt information corresponding to the cluster fault detection, the cluster prompt information being used as a reference basis when performing the cluster fault detection by using a fault detection model; construct first input information of the fault detection model based on the cluster prompt information and the detection result of each of the service clusters; call the fault detection model to perform fault detection on each of the service clusters based on the first input information to obtain a probability of each of the service clusters having a fault, and determine whether each of the service clusters has a fault based on the probability of each of the service clusters having a fault.

[0251] In some embodiments, the cluster detection module is further configured to perform the following processing for each of the service clusters: compare a data packet loss rate corresponding to the service cluster with a packet loss rate threshold to obtain a packet loss rate comparison result; when the packet loss rate comparison result indicates that the data packet loss rate is greater than or equal to the packet loss rate threshold, determine a cluster fault detection result of the service cluster as indicating that the service cluster has the cluster fault; and when the packet loss rate comparison result indicates that the data packet loss rate is less than the packet loss rate threshold, determine the cluster fault detection result of the service cluster as indicating that the service cluster does not have the cluster fault.

[0252] In some embodiments, the fault repair apparatus of the service system further includes an output module configured to output prompt information when the cluster fault detection results of all the service clusters indicate that the service clusters do not have the cluster fault, wherein the prompt information is used to prompt that all the service nodes in the service system do not have faults.

[0253] In some embodiments, the node detection module is further configured to perform the following processing for each of the service nodes in the fault service cluster: obtain an operating state of the service node; and detect the service node in the fault service cluster based on the operating state of the service node to obtain a service node in the fault service cluster that has a fault.

[0254] In some embodiments, the node detection module is further configured to obtain node prompt information corresponding to the detection, wherein the node prompt information is used as a reference basis when the fault detection model detects the service node with a fault; construct second input information of the fault detection model based on the node prompt information and the operating state of each of the service nodes in the fault service cluster; invoke the fault detection model to detect the service node with a fault in the fault service cluster based on the second input information to obtain a probability that each of the service nodes in the fault service cluster has a fault, and determine the service node in the fault service cluster that has a fault based on the probability that each of the service nodes in the fault service cluster has a fault.

[0255] In some embodiments, the node detection module is further configured to perform the following processing for each of the service nodes in the fault service cluster: compare the operating state of the service node with a state threshold to obtain an operating comparison result, and determine a sub-fault detection result of the service node based on the operating comparison result; and when the sub-fault detection result of the service node indicates that the service node has the fault, determine the service node as the service node in the fault service cluster that has a fault.

[0256] In some embodiments, the running state includes a plurality of sub-running states, and the state threshold includes a sub-state threshold corresponding to each of the sub-running states; the node detection module is further configured to determine the sub-fault detection result as indicating that the service node does not have the fault when the running comparison result indicates that each of the sub-running states is greater than or equal to the corresponding state threshold; and determine the sub-fault detection result as indicating that the service node has the fault when the running comparison result indicates that there is a sub-running state that is less than the corresponding state threshold.

[0257] In some embodiments, the fault recovery module is further configured to obtain mode prompt information corresponding to a fault recovery mode, the mode prompt information being used as a reference basis for the fault detection model when determining the fault recovery mode; construct third input information of the fault detection model based on the mode prompt information, the service nodes having the fault in the fault service clusters, and the number of the fault service clusters; invoke the fault detection model, determine a probability of each of the preset fault recovery modes based on the third input information, and determine a preset fault recovery mode with a probability greater than a preset probability threshold as the fault recovery mode.

[0258] In some embodiments, the preset fault recovery mode includes a disaster recovery system recovery mode, and the fault recovery module is further configured to determine the fault recovery mode as the disaster recovery system recovery mode when the number of the fault service clusters is greater than or equal to a cluster number threshold and the number of the service nodes having the fault in each of the fault service clusters is greater than a node number threshold, the disaster recovery system recovery mode being used to recover the service system through a disaster recovery system; and determine a cluster recovery mode of each of the fault service clusters from a plurality of the preset fault recovery modes based on the service nodes having the fault in the fault service cluster when the number of the fault service clusters is less than the cluster number threshold, and determine the fault recovery mode of the service system based on each of the cluster recovery modes.

[0259] In some embodiments, the preset fault recovery mode includes a cluster isolation recovery mode and a node isolation recovery mode, and the fault recovery module is further configured to determine a cluster recovery mode of the fault service cluster as the cluster isolation recovery mode when the number of the service nodes having the fault in the fault service cluster is greater than the node number threshold; determine the cluster recovery mode of the fault service cluster as the node isolation recovery mode when the number of the service nodes having the fault in the fault service cluster is not greater than the node number threshold; and the node isolation recovery mode is used to isolate the service nodes having the fault in the fault service cluster, and the cluster isolation recovery mode is used to isolate the fault service cluster in the service system.

[0260] In some embodiments, the fault recovery module is further configured to determine the cluster recovery mode as the fault recovery mode of the service system when the number of the fault service clusters is one; and determine a combination of each of the cluster recovery modes as the fault recovery mode of the service system when the number of the fault service clusters is more than one.

[0261] In some embodiments, the fault recovery module is further configured to, when the fault recovery mode is a disaster recovery system recovery mode, acquire a disaster recovery system, the disaster recovery system including at least one disaster recovery service cluster, and the disaster recovery service cluster being free of the cluster fault; and migrate service data in the service system to the disaster recovery system to implement fault recovery of the service system.

[0262] The embodiments of the present application provide a computer program product including a computer program or computer executable instructions stored in a computer readable storage medium. A processor of an electronic device reads the computer executable instructions from the computer readable storage medium, and the processor executes the computer executable instructions to cause the electronic device to perform the fault recovery method of the service system provided by the embodiments of the present application.

[0263] The embodiments of the present application provide a computer readable storage medium storing computer executable instructions, wherein the computer executable instructions, when executed by a processor, cause the processor to perform the fault recovery method of the service system provided by the embodiments of the present application, for example, the fault recovery method of the service system as shown in the above. Figure 3

[0264] In some embodiments, the computer readable storage medium can be a Read-Only Memory (ROM), a Random Access Memory (RAM), an Erasable Programmable Read-Only Memory (EPROM), a flash memory, a magnetic surface memory, an optical disc, or a CD-ROM, etc. memory; and can also be various electronic devices including one or any combination of the above memories.

[0265] In some embodiments, the computer executable instructions can be in the form of programs, software, software modules, scripts or codes, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and can be deployed in any form, including being deployed as independent programs or being deployed as modules, components, subroutines or other units suitable for use in a computing environment.

[0266] ​As an example, computer-executable instructions can be, but need not be, related to the computer-executable instructions that implement one or more modules or units described herein.

[0267] In embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as a processing circuit or a memory) or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an integral module or unit that includes the functions of the module or unit.

[0268] As an example, computer-executable instructions can be deployed to execute on one electronic device, or on multiple electronic devices that are located at one place, or on multiple electronic devices that are distributed at multiple places and interconnected through a communication network.

[0269] In summary, the embodiments of the present application have the following beneficial effects:

[0270] (1) By detecting the faults of the service clusters, the cluster fault detection results of the service clusters are obtained, the service cluster with the fault indication is determined as the fault service cluster, the service nodes in each fault service cluster are detected, the service nodes with faults in each fault service cluster are obtained, the fault repair mode of the service system is determined from the plurality of preset fault repair modes based on the service nodes with faults in the fault service cluster and the number of fault service clusters, and the service system is repaired based on the fault repair mode. In this way, by determining the fault repair mode of the service system from the plurality of preset fault repair modes based on the service nodes with faults in the fault service cluster and the number of fault service clusters, and repairing the service system based on the fault repair mode, the determined fault repair mode can adapt to the fault of the service system, thereby achieving efficient fault repair of the service system, and effectively improving the fault repair efficiency of the service system.

[0271] (2) By sending probe data to each service cluster, the running state and performance indicators of each service cluster can be detected in real time, ensuring the normal operation of each cluster. By comparing the changes before and after the service cluster processes the probe data, abnormal situations or performance bottlenecks in the service processing process can be discovered in a timely manner. Through analysis of the target probe data and the probe results of the service cluster, potential risk points can be predicted, so that measures can be taken in advance to avoid or reduce possible problems. Based on the received target probe data and the flow process, the processing flow of the service cluster can be optimized to improve processing efficiency and accuracy. By analyzing the probe results of the service cluster, computing resources can be more reasonably allocated to avoid resource waste and ensure that the resource needs of critical businesses are met. Through continuous detection and analysis, security threats can be discovered and handled in a timely manner to ensure the data security and stability of the service cluster.

[0272] (3) When the number of fault service clusters is greater than or equal to the cluster number threshold, and each fault service cluster has a fault service node indicating that the number of fault service nodes in the fault service cluster is greater than the node number threshold, it means that each fault service cluster has more than the node number threshold of fault service nodes, indicating that the fault of each fault service cluster is relatively serious. At the same time, the number of fault service clusters is greater than or equal to the cluster number threshold, that is, the number of fault service clusters in the business system is greater than or equal to the cluster number threshold, indicating that there are more than or equal to the cluster number threshold of fault service clusters in the business system, indicating that the entire business system is relatively serious. By repairing all existing faults in the business system, the fault repair method is determined as the disaster recovery system repair method, which repairs the fault of the business system through the disaster recovery system, thereby directly replacing the entire business system with a large number of faults through the disaster recovery system, thereby effectively improving the fault repair efficiency of the business system.

[0273] (4) Combined with the cluster prompt information and the probe results of each service cluster, more accurate and comprehensive input data for the fault detection model can be constructed to improve the accuracy of detection. By calling the fault detection model, automatic detection of each service cluster can be achieved, saving human resources and improving detection efficiency. The real-time nature of cluster fault detection ensures that once a fault occurs, the detection result can be obtained immediately, so that problems can be quickly responded to and measures can be taken to solve them.

[0274] (5) By directly comparing the data packet loss rate of the service cluster with the preset packet loss rate threshold, it can quickly determine whether the service cluster has a fault, simplifying the fault diagnosis process. Real-time detection of the packet loss of the service cluster is allowed to ensure that potential problems can be discovered immediately when the packet loss rate reaches the threshold. When the packet loss rate exceeds the threshold, automatic triggering of service cluster fault detection can be achieved, realizing the automation of fault response and reducing manual intervention. The packet loss rate can be used as a quantitative indicator to help evaluate the severity of the fault and provide a basis for subsequent fault handling. By detecting the data packet loss rate, signs of network congestion can be detected in advance, and measures can be taken to prevent network congestion and ensure network service quality.

[0275] (6) When the cluster fault detection results of each service cluster indicate that the service cluster has no fault, a prompt information is output, so that the cluster fault detection can be used to troubleshoot the entire service system. By outputting the prompt information, the entire service system can be troubleshooted without subsequent node fault detection and other means, thereby effectively improving the troubleshooting efficiency of the service system. By ensuring that the detection results of all service clusters show no fault, the stability of the entire service system can be finally confirmed. The output of the prompt information helps to eliminate doubts about possible faults in the service system and provides users and operation and maintenance teams with a clear system health status. When the system shows that all service nodes are running normally, user trust in the system can be improved, enhancing user experience. After confirming that the service system has no fault, the allocation of computing resources and detection resources can be optimized to avoid unnecessary resource waste. Through the entire service system fault detection process, false positives and repeated troubleshooting work are reduced, improving operation and maintenance efficiency.

[0276] (7) By obtaining the node prompt information corresponding to the node fault detection, the node that may have a problem can be more accurately located. Based on the node prompt information and the running state, a fault detection model can be constructed to quickly identify and judge the fault nodes, thereby shortening the fault diagnosis time and speeding up the fault response speed. Through the automated fault detection model, the dependence on human experts can be reduced, especially when dealing with a large number of nodes and complex faults, the model can provide efficient fault detection results without manual checking one by one. Timely and accurate node fault detection can help system administrators isolate fault nodes in a timely manner to prevent fault propagation and maintain the stability of the entire system. By continuously detecting the running state of the nodes and using the fault detection model for analysis, potential faults can be predicted and preventive maintenance can be performed to avoid the impact of faults on business. The application of the fault detection model helps to ensure the continuity of key business processes, even when facing node faults, the model can quickly identify and take measures to reduce the time of business interruption.

[0277] (8) By setting specific state thresholds for each sub-node, the system can more accurately determine whether a node is malfunctioning, avoiding excessive or unnecessary failure alerts. The rapid generation of comparison results helps system administrators quickly identify malfunctioning nodes, allowing them to take immediate action such as redistributing loads or isolating malfunctioning nodes to prevent further impact on business. By considering the operational status of multiple sub-nodes and their corresponding sub-state thresholds, more detailed failure analysis can be provided to help administrators understand the specific causes of the failure. By detecting and analyzing operational status, resource allocation can be optimized, such as redistributing tasks to healthy nodes or upgrading or expanding heavily loaded nodes. Timely failure detection and response help reduce the time and frequency of business interruptions, ensuring business continuity and stability.

[0278] (9) By using the failure detection model, the third input information can be automatically constructed, and the failure repair method can be determined accordingly, reducing manual intervention and improving the automation level of the repair process. Rapid analysis of failure conditions and business cluster status allows for quick determination of the most appropriate repair method, speeding up the failure recovery process. By determining the failure repair method in a timely manner, the malfunctioning nodes or clusters can be effectively isolated to prevent further spread to other healthy nodes or systems. By quickly determining and implementing the failure repair method, the impact of the failure on business operations can be minimized, ensuring business continuity and stability.

[0279] (10) By setting cluster quantity thresholds and node quantity thresholds, those failures with a wide range of impact can be prioritized for response, ensuring that resources and services are allocated according to the importance and urgency of the failure. Based on the number of failure business clusters and the number of business nodes in the failure business clusters, the failure repair method is dynamically determined, making the repair strategy more flexible and adaptable. When the number of failure business clusters and the number of failure nodes exceeds the pre-set threshold, the disaster recovery system repair method is automatically switched to, quickly restoring services and reducing business interruption time. When the number of failure business clusters is less than the cluster quantity threshold, the cluster repair method is determined for each failure business cluster separately, allowing for more detailed failure isolation and repair. By selecting a repair method based on failure detection results, resource allocation can be optimized to ensure that resources are effectively used for the most critical business recovery links. By determining the failure repair method in a timely and accurate manner, business continuity and availability can be improved, ensuring the stability of critical business processes. By using the disaster recovery system and cluster isolation repair method, the resilience of the business system to failures is enhanced, improving the ability of the business system to handle large-scale failures.

[0280] (11) The fault self-healing intelligent agent provided by the embodiment of the present application can realize active fault discovery: the health detection system is through the active PING detection mode. In combination with the detection data of the gateway cluster and based on the data analysis capability of the meta large model, the faults of the cluster can be discovered in time. Intelligent fault diagnosis: in combination with the feature data such as alarms, logs, traffic and port states of the equipment, based on the logical reasoning capability of the meta large model, the abnormal nodes in the cluster are intelligently diagnosed. Automatic fault repair: after the root cause positioning is completed by the above intelligent fault diagnosis module, that is, after the abnormal node is confirmed, based on the decision capability of the meta large model, the optimal fault repair mode is adopted for different cluster fault scenarios. In summary, the embodiment of the present application solves the problems of slow fault discovery, low diagnosis efficiency and slow repair speed in the single node fault scenario of the traditional gateway cluster.

[0281] The above is only an embodiment of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement and improvement made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A method for failure recovery of a business system, characterized by, The business system comprises at least one business cluster, and each business cluster comprises a plurality of business nodes. Each business cluster is detected for failure to obtain a cluster failure detection result of each business cluster. A business cluster whose cluster failure detection result indicates that there is a failure is determined as a failure business cluster. Each business node in each failure business cluster is detected to obtain a business node with a failure in each failure business cluster. Based on the business nodes with failures in the failure business cluster and the number of the failure business cluster, a failure recovery mode of the business system is determined from a plurality of preset failure recovery modes, and the business system is recovered based on the failure recovery mode.

2. The method of claim 1, wherein, The detection of each business cluster for failure to obtain a cluster failure detection result of each business cluster comprises: Each business cluster in the business system is detected for data to obtain a detection result of each business cluster. The detection result is used to represent the data packet loss rate of the detection data after flowing through the business cluster. Based on the detection result, it is determined whether each business cluster has a failure.

3. The method of claim 2, wherein, The detection of each business cluster in the business system for data to obtain a detection result of each business cluster comprises: The following processing is performed for each business cluster: The detection data comprising a plurality of data packets is sent to the business cluster to make the business cluster flow the detection data and output target detection data after flow; The number of data packets between the target detection data and the detection data is obtained, and the data packet loss rate of the business cluster is determined based on the number of data packets.

4. The method of claim 2, wherein, Based on the detection result, it is determined whether each business cluster has a failure, comprising: Obtain the cluster prompt information corresponding to the cluster failure detection. The cluster prompt information is used as a reference basis for the cluster failure detection by the failure detection model. Based on the cluster prompt information and the detection result of each business cluster, the first input information of the failure detection model is constructed. The failure detection model is called based on the first input information to detect each business cluster for failure to obtain the probability of each business cluster having a failure, and it is determined whether each business cluster has a failure based on the probability of each business cluster having a failure.

5. The method of claim 2, wherein, Based on the detection result, it is determined whether each business cluster has a failure, comprising: The following processing is performed for each business cluster: The data packet loss rate corresponding to the business cluster is compared with a packet loss rate threshold to obtain a packet loss rate comparison result. When the packet loss rate comparison result indicates that the data packet loss rate is greater than or equal to the packet loss rate threshold, the cluster failure detection result of the business cluster is determined as the business cluster having the failure. When the packet loss rate comparison result indicates that the data packet loss rate is less than the packet loss rate threshold, the cluster failure detection result of the business cluster is determined as the business cluster not having the failure.

6. The method of claim 1, wherein, After the detecting the cluster failure of each of the business clusters, the method further comprises: outputting prompt information when the cluster failure detection result of each of the business clusters indicates that the business cluster does not exist the cluster failure; the prompt information is used to prompt that each of the business nodes in the business system does not exist the failure.

7. The method of claim 1, wherein, The detecting the business node in each of the failure business clusters, obtaining the business node existing the failure in the failure business cluster, comprises: For each of the failure business clusters, the following processing is performed respectively: obtaining the running state of each of the business nodes in the failure business cluster; based on the running state of the business node, detecting the business node in the failure business cluster, obtaining the business node existing the failure in the failure business cluster.

8. The method of claim 7, wherein, The detecting the business node in each of the failure business clusters, obtaining the business node existing the failure in the failure business cluster, comprises: obtaining the node prompt information corresponding to the detection, the node prompt information is used as a reference basis when the failure detection model detects the business node; based on the node prompt information and the running state of each of the business nodes in the failure business cluster, constructing the second input information of the failure detection model; calling the failure detection model, based on the second input information, detecting the failure business node in the failure business cluster, obtaining the probability of each of the business nodes existing the failure in the failure business cluster, and based on the probability of each of the business nodes existing the failure in the failure business cluster, determining the business node existing the failure in the failure business cluster.

9. The method of claim 7, wherein, The detecting the business node in each of the failure business clusters, obtaining the business node existing the failure in the failure business cluster, comprises: For each of the business nodes in the failure business cluster, the following processing is performed respectively: comparing the running state of the business node with the state threshold to obtain a running comparison result, and based on the running comparison result, determining the sub-failure detection result of the business node; when the sub-failure detection result of the business node indicates that the business node exists the failure, the business node is determined as the business node existing the failure in the failure business cluster.

10. The method of claim 9, wherein, The running state comprises a plurality of sub-running states, and the state threshold comprises a sub-state threshold corresponding to each of the sub-running states; based on the running comparison result, determining the sub-failure detection result of the business node, comprises: when the running comparison result indicates that each of the sub-running states is greater than or equal to the corresponding state threshold, the sub-failure detection result is determined as the business node not existing the failure; when the running comparison result indicates that there is the sub-running state less than the corresponding state threshold, the sub-failure detection result is determined as the business node existing the failure.

11. The method of claim 1, wherein, The method comprises the following steps: obtaining the mode prompt information corresponding to the fault recovery mode, wherein the mode prompt information is used as a reference basis for determining the fault recovery mode by the fault detection model; constructing third input information of the fault detection model based on the mode prompt information, the number of the faulty service nodes in the faulty service cluster, and the number of the faulty service clusters; calling the fault detection model and determining the probability of each preset fault recovery mode based on the third input information, and determining the preset fault recovery mode with a probability greater than a preset probability threshold as the fault recovery mode.

12. The method of claim 1, wherein, The preset fault recovery mode includes a disaster recovery system recovery mode, and the method comprises the following steps: when the number of the faulty service clusters is greater than or equal to a cluster number threshold and the number of the faulty service nodes in each faulty service cluster is greater than a node number threshold, the fault recovery mode is determined as the disaster recovery system recovery mode, and the disaster recovery system recovery mode is used to repair the fault of the business system by a disaster recovery system; when the number of the faulty service clusters is less than the cluster number threshold, for each faulty service cluster, a cluster recovery mode of the faulty service cluster is determined from a plurality of preset fault recovery modes based on the faulty service nodes in the faulty service cluster, and a fault recovery mode of the business system is determined based on each cluster recovery mode.

13. The method of claim 12, wherein, The preset fault recovery mode includes a cluster isolation recovery mode and a node isolation recovery mode, and the method comprises the following steps: when the number of the faulty service nodes in the faulty service cluster is greater than the node number threshold, the cluster recovery mode of the faulty service cluster is determined as the cluster isolation recovery mode; when the number of the faulty service nodes in the faulty service cluster is not greater than the node number threshold, the cluster recovery mode of the faulty service cluster is determined as the node isolation recovery mode; The node isolation recovery mode is used to isolate the faulty service nodes in the faulty service cluster, and the cluster isolation recovery mode is used to isolate the faulty service cluster in the business system.

14. The method of claim 12, wherein, The method comprises the following steps: when the number of the faulty service clusters is one, the cluster recovery mode is determined as the fault recovery mode of the business system; when the number of the faulty service clusters is multiple, a combination of each cluster recovery mode is determined as the fault recovery mode of the business system.

15. The method of claim 1, wherein, The fault repairing of the business system based on the fault repairing mode comprises: When the fault repairing mode is a disaster recovery system repairing mode, a disaster recovery system is acquired, the disaster recovery system comprises at least one disaster recovery business cluster, and the disaster recovery business cluster does not have the cluster fault; Business data in the business system is migrated to the disaster recovery system to realize the fault repairing of the business system.

16. A failure recovery apparatus of a service system, characterized by comprising: The business system comprises at least one business cluster, the business cluster comprises a plurality of business nodes, and the device comprises: A cluster detection module is configured to detect faults of each business cluster to obtain a cluster fault detection result of each business cluster, and indicate a business cluster having a fault in the cluster fault detection result to determine the business cluster as a fault business cluster; A node detection module is configured to detect business nodes in each fault business cluster to obtain a business node having a fault in each fault business cluster; A fault repairing module is configured to determine a fault repairing mode of the business system from a plurality of preset fault repairing modes based on the business node having a fault in the fault business cluster and a number of the fault business cluster, and repair faults of the business system based on the fault repairing mode.

17. An electronic device, comprising: The electronic device comprises: A memory is configured to store computer executable instructions or computer programs; A processor is configured to execute the computer executable instructions or computer programs stored in the memory to implement the fault repairing method of the business system according to any one of claims 1 to 15.

18. A computer-readable storage medium storing computer-executable instructions, wherein execution of the computer-executable instructions by one or more processors of a computing system causes the one or more processors to perform operations comprising: The computer executable instructions are executed by the processor to implement the fault repairing method of the business system according to any one of claims 1 to 15.

19. A computer program product comprising computer programs or computer executable instructions, characterized in that, The computer program or computer executable instructions are executed by the processor to implement the fault repairing method of the business system according to any one of claims 1 to 15.

Images