Fault repair method and device, electronic equipment and storage medium

Abstract

The invention discloses a fault repair method and device, electronic equipment and a storage medium, and relates to the field of cluster operation and maintaining.The method comprises the steps that in response to a detected fault repair signal of a current node, context information of the current node is obtained; according to the fault state information, performing fault repair on the current node through a predefined first repair strategy, and judging whether the first repair strategy is successfully repaired; and if it is determined that repair of the first repair strategy fails, sending the context information to a strategy recommendation model to obtain a second repair strategy through the strategy recommendation model, and performing fault repair on the current node according to the second repair strategy. According to the technical scheme provided by the embodiment of the invention, the automatic repair of the node fault is realized, the node operation and maintenance efficiency is improved, the predefined first repair strategy is combined with the second repair strategy provided by the strategy recommendation model, the comprehensive repair processing of various complex fault types is ensured, and the repair effect of the node fault is greatly improved.

Description

Technical Field

[0001] This invention relates to the field of cluster operation and maintenance, and in particular to a fault repair method, apparatus, electronic device and storage medium. Background Technology

[0002] With the continuous development of computer technology, the scale of server clusters, especially artificial intelligence (AI) computing power clusters, is also constantly expanding. How to effectively operate and maintain the node devices in the cluster has become a key issue of concern in the industry.

[0003] In existing technologies, when a node in a cluster encounters a fault (especially a hardware fault), the node's own monitoring program will issue a relevant fault repair signal, or the monitoring program of the management node in the cluster will issue a relevant fault repair signal, thereby guiding the operation and maintenance personnel to carry out fault repair. The operation and maintenance personnel will carry out operation and maintenance operations through pre-configured repair scripts and monitor the repair progress until the fault repair of the current node is completed.

[0004] However, this kind of repair method not only requires high manpower and time costs and has low operation and maintenance efficiency, but also relies on the experience of operation and maintenance personnel. The pre-configured repair strategies are often not applicable to complex fault types, resulting in poor operation and maintenance effects. Summary of the Invention

[0005] This invention provides a fault repair method, apparatus, electronic device, and storage medium to solve the problems of low operation and maintenance efficiency and poor operation and maintenance effect of nodes.

[0006] According to another aspect of the present invention, a fault repair method is provided, comprising:

[0007] In response to the detection of a fault repair signal for the current node, the context information of the current node is obtained; wherein, the context information includes a global configuration file, user command parameters, a session state file, and historical repair records; the session state file includes fault state information;

[0008] Based on the fault status information, the current node is repaired using a predefined first repair strategy, and it is determined whether the first repair strategy has been successfully repaired.

[0009] If it is determined that the first repair strategy fails, the context information is sent to the strategy recommendation model to obtain a second repair strategy, and the current node is repaired according to the second repair strategy.

[0010] The step of performing fault repair on the current node according to the second repair strategy includes: obtaining the repair execution path of the current node, and obtaining a set of repair paths with similar paths through the repair path library; and obtaining the predicted repair probability, predicted repair progress and predicted repair time of the repair execution path based on the repair results of each repair path in the repair path set.

[0011] After repairing the current node according to the second repair strategy, the method further includes: performing cyclic verification on the actual repair result of the second repair strategy according to the expected repair result corresponding to the second repair strategy, until the number of cyclic verifications reaches a preset threshold.

[0012] After repeatedly verifying the actual repair result of the second repair strategy according to the expected repair result corresponding to the second repair strategy until the number of repeated verifications reaches a preset threshold, the method further includes: determining whether the second repair strategy has been successfully repaired based on the repair execution log and the repair verification log; if it is determined that the second repair strategy has not been successfully repaired, writing the fault repair record into the session state file and sending a restart signal to the current node to guide the current node to restart.

[0013] The step of determining whether the second repair strategy was successful based on the repair execution log and the repair verification log includes: obtaining a matching classification category through a semantic classification model based on the repair execution log and the repair verification log; wherein the classification category includes repair success text and repair failure text.

[0014] The step of obtaining a matching second repair strategy through the strategy recommendation model further includes: if a repair restart identifier is detected in the second repair strategy, writing the interrupt execution step corresponding to the repair restart identifier into the session state file, so that after the current node performs a restart operation based on the repair restart identifier, the second repair strategy can continue to be executed through the interrupt execution step.

[0015] According to another aspect of the present invention, a fault repair apparatus is provided, comprising:

[0016] The context information acquisition module is used to acquire the context information of the current node in response to the detection of a fault repair signal of the current node; wherein, the context information includes a global configuration file, user command parameters, session state file and historical repair records; the session state file includes fault state information;

[0017] The fault repair execution module is used to repair the current node according to the fault status information using a predefined first repair strategy, and to determine whether the first repair strategy has been successfully repaired.

[0018] The repair strategy acquisition module is used to send the context information to the strategy recommendation model if it is determined that the first repair strategy has failed, so as to obtain a second repair strategy through the strategy recommendation model and repair the current node according to the second repair strategy.

[0019] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the fault repair method described in any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the fault repair method described in any embodiment of the present invention.

[0021] According to another aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the fault repair method described in any embodiment of the present invention.

[0022] The technical solution of this invention, in response to detecting a fault repair signal of the current node, obtains the context information of the current node; based on the fault status information, performs fault repair on the current node using a predefined first repair strategy, and determines whether the first repair strategy is successful; if it is determined that the first repair strategy has failed, the context information is sent to a strategy recommendation model to obtain a second repair strategy, and the current node is repaired according to the second repair strategy. This not only achieves automatic fault repair of nodes, improving node operation and maintenance efficiency and reducing the manpower and time costs of fault repair, but also ensures comprehensive repair of various complex fault types by combining the predefined first repair strategy with the second repair strategy provided by the strategy recommendation model, greatly improving the fault repair effect of nodes.

[0023] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart of a fault repair method provided in Embodiment 1 of the present invention;

[0026] Figure 2 This is a flowchart of another fault repair method provided in Embodiment 2 of the present invention;

[0027] Figure 3 This is a flowchart of another fault repair method provided in Embodiment 3 of the present invention;

[0028] Figure 4 This is a schematic diagram of a fault repair device according to Embodiment 4 of the present invention;

[0029] Figure 5 This is a schematic diagram of the structure of an electronic device that implements the fault repair method of the present invention. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] Example 1

[0033] Figure 1This is a flowchart of a fault repair method provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where identity verification and permission verification are completed based on security tags and access tags respectively, and physical address translation is completed through an input / output translation backup buffer. This method can be executed by the fault repair device in any embodiment of the present invention. The fault repair device can be implemented in hardware and / or software. The fault repair device can exist in the form of an Agent and be configured in a node of the cluster. The node can be an electronic device such as a server. Figure 1 As shown, the method includes:

[0034] S101. In response to detecting a fault repair signal of the current node, obtain the context information of the current node; wherein, the context information includes a global configuration file, user command parameters, session state file and historical repair records; the session state file includes fault state information.

[0035] In this embodiment of the invention, a node can be a node in a server cluster, that is, a server in the server cluster; or it can be a node in an AI computing power cluster, which is a computing entity integrating one or more computing units; wherein, the computing units may include a Tensor Processing Unit (TPU) and a Graphics Processing Unit (GPU); the current node can be a management node in the cluster used for global coordination and resource scheduling, or it can be a business node that performs specific data computing tasks.

[0036] Context information refers to the collection of dynamic environmental states and configuration data on which a node depends for its operation. When a node encounters a fault or anomaly (especially a hardware fault), the node's own monitoring program will issue relevant fault repair signals and persistently store the fault status information. Alternatively, the monitoring program of the management node may issue relevant fault repair signals to guide the current node to persistently store the fault status information. For example, the fault status information may be stored in the form of a session state file to avoid the loss of fault status after power failure.

[0037] The session state file, which records information about session state objects (SSOs), not only manages the user session state and stores temporary information during user access, but also saves the current node's system state objects (e.g., memory data mirrors, incomplete transaction logs, and process context information) on disk. Fault status information includes not only fault types (e.g., GPU card loss, disk corruption), but also various parameters such as status identifiers, resource metrics, network status, and detection mechanisms.

[0038] Global configuration files define global behavior patterns in the form of system instructions, such as whether to enable a specific mode, the maximum number of retries, etc. User command parameters are command-line arguments entered by the user, or command parameters entered by the user through an API (Application Programming Interface), to reflect the priority of manual intervention operations, such as forcibly skipping certain execution steps.

[0039] Historical repair records reflect the machine's memory function, that is, the repair records of the current node over a period of time in the past, including the repair method and the repair success rate; for example, the last successful repair method was a hard reboot, and the repair success rate of this repair method was 50%; accordingly, when the Agent detects the above-mentioned fault repair signal, it reads the context information stored in the current node.

[0040] S102. Based on the fault status information, perform fault repair on the current node using a predefined first repair strategy, and determine whether the first repair strategy has been successfully repaired.

[0041] The first repair strategy is a pre-edited fixed script used to solve known specific problems. The first repair strategy can be configured with different repair rules for different fault types. Based on the current fault status information, the fault of the current node is repaired by the corresponding repair rules in the first repair strategy. In particular, in order to ensure the ease of editing the predefined script, the predefined script can be updated by dynamic parameters passed from external input to adapt to different fault scenarios.

[0042] During the repair process of the first repair strategy, if an unknown type of error code is detected, it can be determined that the first repair strategy has failed. For example, the error code for a file system error is defined as 1, and the error code for insufficient disk space is defined as 2. If other types of error codes are detected besides the above error codes, it means that the first repair strategy has failed. At the same time, if the script of the first repair strategy is forcibly terminated or fails to generate an exit code due to a syntax error, it also means that the first repair strategy has failed.

[0043] Furthermore, if key indicators fail to return to the expected state after the first repair strategy is executed—for example, if the service port fails to listen, the critical process is not alive, or the configuration file is not updated—it also indicates that the first repair strategy has failed. Similarly, if abnormal resource status occurs after the first repair strategy is executed—for example, if the CPU usage is too high, the memory usage is too high, or the disk I / O (Input / Output) is blocked—it also indicates that the first repair strategy has failed.

[0044] S103. If it is determined that the first repair strategy fails, the context information is sent to the strategy recommendation model to obtain the second repair strategy through the strategy recommendation model, and the current node is repaired according to the second repair strategy.

[0045] The strategy recommendation model can be a large language model trained on large-scale operational data. It is based on existing large language models in the network platform (e.g., large language models based on the Transformer architecture), trained using labeled strategy training samples, learning the correlation between fault modes and repair strategies, and combining multimodal fusion methods such as time series analysis, natural language processing, and knowledge graphs to generate personalized repair strategies. Alternatively, the strategy recommendation model can be a traditional model, such as a recommendation model pre-trained based on techniques like Random Forest and Convolutional Neural Network (CNN). Optionally, in this embodiment of the invention, the model architecture of the strategy recommendation model is not specifically limited.

[0046] Historical repair records characterize the historical operation and maintenance characteristics of the current node. By incorporating these records into the input information of the strategy recommendation model, the model is guided to reduce the recommendation priority of repair strategies with high failure rates, thereby improving the accuracy of the strategy recommendation results. For example, if the current node has failed in the past three soft reset operations and only succeeded in hard reboot, the strategy recommendation model will reduce the priority of soft reset during inference and give more emphasis to recommending hard reboot.

[0047] Optionally, in this embodiment of the invention, the step of performing fault repair on the current node according to the second repair strategy includes: obtaining the repair execution path of the current node, and obtaining a set of repair paths with similar paths through the repair path library; and obtaining the predicted repair probability, predicted repair progress and predicted repair time of the repair execution path based on the repair results of each repair path in the repair path set.

[0048] Specifically, the repair path library records the repair paths executed during each fault repair. Each repair path records whether the repair result was successful or unsuccessful, the repair time of each execution step in the repair process, and the total repair time. Regardless of whether the first repair strategy or the second repair strategy is executed, the repair execution path under the current repair strategy can be compared with each repair path in the repair path library, and repair paths with similar paths can be combined into a repair path set.

[0049] Then, the repair success rate of each repair path in the repair path set is used as the predicted repair probability of the repair execution path, and the average repair time of each repair path in the repair path set is used as the predicted repair time of the repair execution path. At the same time, based on the remaining execution steps under the current repair strategy and the repair time of each execution step of each repair path in the repair path set, the predicted remaining time of the remaining execution steps is determined. After calculating the ratio of the predicted remaining time to the predicted repair time, the difference between the value 1 and the ratio can be used as the predicted repair progress.

[0050] For example, there are 100 similar repair trajectories in history, 80 of which were successful. Based on this, the current progress is shown to the user as attempting a hard reboot, with a high probability (80%) of successful repair, an expected repair time of 5 minutes, and an expected remaining time of 3 minutes. Therefore, the repair progress is approximately 40%. By maintaining the repair path library, the real-time prediction of this repair process is ensured, answering the two most important questions for operations and maintenance personnel: "Can it be repaired successfully?" and "How long will it take to complete the repair?", significantly improving the human-computer interaction experience.

[0051] Optionally, in this embodiment of the invention, after repairing the current node according to the second repair strategy, the method includes: performing cyclic verification on the actual repair result of the second repair strategy according to the expected repair result corresponding to the second repair strategy, until the number of cyclic verifications reaches a preset threshold.

[0052] Specifically, after executing the repair action through the second repair strategy, it does not assume that the repair is successful on the first attempt. Instead, based on the expected repair result, it actively and repeatedly checks whether the repair behavior is truly effective. That is, it judges whether the actual repair result matches the expected repair result. For example, whether the target state parameter remains stably within the expected repair result range. And all operation traces are completely recorded throughout the process. The loop verification stops only when the number of loop verifications reaches a preset threshold, thereby ensuring the reliability and durability of the repair result and avoiding accidental success or short-term recovery phenomena.

[0053] Optionally, in this embodiment of the invention, obtaining a matching second repair strategy through the strategy recommendation model further includes: if a repair restart identifier is detected in the second repair strategy, writing the interrupt execution step corresponding to the repair restart identifier into the session state file, so that after the current node performs a restart operation based on the repair restart identifier, the second repair strategy can continue to be executed through the interrupt execution step.

[0054] Specifically, during the repair process of the first and second repair strategies, the strategy itself may include a restart operation instruction. For example, the node may be restarted to update some configuration information, and then the next repair instruction may be executed based on the updated configuration information. If a repair restart flag is detected in the first or second repair strategy, the interrupted execution steps corresponding to the repair restart flag are written to the session state file. This allows the Agent to continue executing the original repair strategy (i.e., the first or second repair strategy) through the interrupted execution steps after the current node performs a restart operation based on the repair restart flag, so as to restore the execution steps before the interruption. This ensures the continuity and integrity of the repair process.

[0055] The technical solution of this invention, in response to detecting a fault repair signal of the current node, obtains the context information of the current node; based on the fault status information, performs fault repair on the current node using a predefined first repair strategy, and determines whether the first repair strategy is successful; if it is determined that the first repair strategy has failed, the context information is sent to a strategy recommendation model to obtain a second repair strategy, and the current node is repaired according to the second repair strategy. This not only achieves automatic fault repair of nodes, improving node operation and maintenance efficiency and reducing the manpower and time costs of fault repair, but also ensures comprehensive repair of various complex fault types by combining the predefined first repair strategy with the second repair strategy provided by the strategy recommendation model, greatly improving the fault repair effect of nodes.

[0056] Example 2

[0057] Figure 2 This is a flowchart of a fault repair method provided in Embodiment 2 of the present invention. The relationship between this embodiment and the above embodiments is that, after executing the second repair strategy, it is also necessary to determine whether the second repair strategy has been successfully repaired based on the repair execution log and the repair verification log. Figure 2 As shown, the method includes:

[0058] S201. In response to detecting a fault repair signal for the current node, obtain the context information of the current node; wherein, the context information includes a global configuration file, user command parameters, a session state file, and historical repair records; the session state file includes fault state information.

[0059] S202. Based on the fault status information, perform fault repair on the current node using a predefined first repair strategy, and determine whether the first repair strategy has been successfully repaired.

[0060] S203. If it is determined that the first repair strategy fails, the context information is sent to the strategy recommendation model to obtain the second repair strategy through the strategy recommendation model, and the current node is repaired according to the second repair strategy.

[0061] S204. Based on the expected repair result corresponding to the second repair strategy, the actual repair result of the second repair strategy is repeatedly verified until the number of repeated verifications reaches a preset threshold.

[0062] S205. Based on the repair execution log and the repair verification log, determine whether the second repair strategy was successfully repaired.

[0063] S206. If it is determined that the second repair strategy has failed to repair, write the fault repair record to the session state file and send a restart signal to the current node to guide the current node to restart.

[0064] Obtain the repair execution logs generated throughout the entire repair process, including logs generated during the execution phases of the first and second repair strategies, as well as repair verification logs generated during the cyclic verification phase. These logs record which commands were executed during the repair and verification process, the output value, return value, and error information of each command, and other relevant system metrics and diagnostic data.

[0065] The rule engine can be used to determine whether the second repair strategy was successful. If a fatal error is detected in the log, it means that the repair was unsuccessful and a restart is required immediately. If multiple consecutive verifications fail and the status code remains unchanged, it means that the node has entered a deadlock detection state, which also indicates that the repair was unsuccessful and a restart is required. If there are no such abnormalities in the log, it is determined that the second repair strategy was successful.

[0066] Specifically, if historical repair records show that the success rate of repairing similar faults via restart is high, for example, greater than a preset probability threshold (e.g., 80%), even if the above-mentioned anomalies are not present in the logs, it can be determined that the second repair strategy has failed to repair, and the node is guided to perform a restart operation to enhance the repair effect of the second repair strategy. This ensures both effective verification of the repair results of the second repair strategy and that the failure of the second repair strategy is handled by restarting the current node.

[0067] Optionally, in this embodiment of the invention, determining whether the second repair strategy was successfully repaired based on the repair execution log and the repair verification log includes: obtaining a matching classification category through a semantic classification model based on the repair execution log and the repair verification log; wherein the classification category includes repair success text and repair failure text.

[0068] Specifically, the semantic classification model can be a large language model based on deep learning, used to understand and classify the deep meaning of input text. It performs intelligent classification by understanding the context and semantic relationships of the text. In the embodiments of this invention, the semantic classification model can be an existing large language model in the network platform. It is trained by annotated classification training samples to learn the relationship between text content and classification categories. Based on the repair execution log and the repair verification log, the semantic categories of the repair execution log and the repair verification log are divided to determine whether the text is successfully repaired or failed to be repaired.

[0069] Furthermore, the semantic classification model can also be a traditional model, such as a pre-trained classification model built based on techniques like Support Vector Machine (SVM) and Recurrent Neural Network (RNN). This semantic classification model's text classification of repair execution logs and repair verification logs ensures the accuracy of analysis results for complex logs and improves the effective repair of node failures.

[0070] The technical solution of this invention, after iteratively verifying the actual repair results of the second repair strategy according to the expected repair results corresponding to the second repair strategy, determines whether the second repair strategy was successful based on the repair execution log and the repair verification log. If it is determined that the second repair strategy was unsuccessful, the fault repair record is written to the session state file, and a restart signal is sent to the current node to guide the current node to restart. This ensures both effective verification of the repair results of the second repair strategy and that the failure of the second repair strategy is handled by a restart operation to address the fault anomaly of the current node.

[0071] Example 3

[0072] Figure 3 This is a flowchart of a fault repair method provided in Embodiment 3 of the present invention, as follows: Figure 3 As shown, the method includes:

[0073] S301. Obtain the context information of the current node; wherein, the context information includes a global configuration file, user command parameters, session state file, and historical repair records; the session state file includes fault state information.

[0074] S302. Perform fault repair on the current node according to the first repair strategy and the second repair strategy.

[0075] If the first repair strategy fails, the context information is sent to the strategy recommendation model to obtain the second repair strategy, and the current node is repaired according to the second repair strategy.

[0076] S303. Based on the expected repair result, the actual repair result is repeatedly verified until the number of repeated verifications reaches a preset threshold.

[0077] If the first repair strategy succeeds, there is no need to obtain the second repair strategy. Instead, the actual repair result of the first repair strategy is cyclically verified based on the expected repair result of the first repair strategy. If the first repair strategy fails, after executing the second repair strategy, the actual repair result of the second repair strategy is cyclically verified based on the expected repair result of the second repair strategy.

[0078] S304. Obtain the repair execution log and repair verification log.

[0079] S305. Based on the repair execution log and the repair verification log, determine whether the second repair strategy was successfully repaired; if not, proceed to S306; if yes, proceed to S308.

[0080] S306. Write the fault repair record to the session state file.

[0081] S307. Send a restart signal to the current node to guide the current node to restart.

[0082] S308, End.

[0083] In particular, the Agent can also monitor the above-mentioned repair process by configuring a monitoring process. On the one hand, it can calculate and obtain the predicted repair probability, predicted repair progress, and predicted repair time, and display these parameters to the operation and maintenance personnel so that they can clearly and intuitively understand the node operation and maintenance status. On the other hand, it can statistically analyze the repair time. When the repair time exceeds the preset time threshold, it indicates that the current repair process takes too long and is regarded as an abnormal repair behavior, and then the current fault repair behavior will be terminated.

[0084] The technical solution of this invention, in response to detecting a fault repair signal of the current node, obtains the context information of the current node; based on the fault status information, performs fault repair on the current node using a predefined first repair strategy, and determines whether the first repair strategy is successful; if it is determined that the first repair strategy has failed, the context information is sent to a strategy recommendation model to obtain a second repair strategy, and the current node is repaired according to the second repair strategy. This not only achieves automatic fault repair of nodes, improving node operation and maintenance efficiency and reducing the manpower and time costs of fault repair, but also ensures comprehensive repair of various complex fault types by combining the predefined first repair strategy with the second repair strategy provided by the strategy recommendation model, greatly improving the fault repair effect of nodes.

[0085] Example 4

[0086] Figure 4 This is a structural block diagram of a fault repair device provided in Embodiment 4 of the present invention. The device specifically includes:

[0087] The context information acquisition module 401 is used to acquire the context information of the current node in response to the detection of a fault repair signal of the current node; wherein, the context information includes a global configuration file, user command parameters, session state file and historical repair records; the session state file includes fault state information;

[0088] The fault repair execution module 402 is used to repair the current node according to the fault status information through a predefined first repair strategy, and to determine whether the first repair strategy has been successfully repaired.

[0089] The repair strategy acquisition module 403 is used to send the context information to the strategy recommendation model if it is determined that the first repair strategy has failed, so as to obtain a second repair strategy through the strategy recommendation model and repair the current node according to the second repair strategy.

[0090] The technical solution of this invention, in response to detecting a fault repair signal of the current node, obtains the context information of the current node; based on the fault status information, performs fault repair on the current node using a predefined first repair strategy, and determines whether the first repair strategy is successful; if it is determined that the first repair strategy has failed, the context information is sent to a strategy recommendation model to obtain a second repair strategy, and the current node is repaired according to the second repair strategy. This not only achieves automatic fault repair of nodes, improving node operation and maintenance efficiency and reducing the manpower and time costs of fault repair, but also ensures comprehensive repair of various complex fault types by combining the predefined first repair strategy with the second repair strategy provided by the strategy recommendation model, greatly improving the fault repair effect of nodes.

[0091] Optionally, the repair strategy acquisition module 403 is specifically used to acquire the repair execution path of the current node, and to acquire a set of repair paths with similar paths through the repair path library; based on the repair results of each repair path in the repair path set, to acquire the predicted repair probability, predicted repair progress and predicted repair time of the repair execution path.

[0092] Optionally, the fault repair device is further configured to perform cyclic verification of the actual repair result of the second repair strategy according to the expected repair result corresponding to the second repair strategy, until the number of cyclic verifications reaches a preset threshold.

[0093] Optionally, the fault repair device is also used to determine whether the second repair strategy has been successfully repaired based on the repair execution log and the repair verification log; if it is determined that the second repair strategy has not been successfully repaired, the fault repair record is written to the session state file and a restart signal is sent to the current node to guide the current node to restart.

[0094] Optionally, the fault repair device is further configured to obtain a matching classification category based on the repair execution log and the repair verification log using a semantic classification model; wherein the classification category includes repair success text and repair failure text.

[0095] Optionally, the repair strategy acquisition module 403 is specifically used to write the interrupt execution step corresponding to the repair restart identifier into the session state file if a repair restart identifier is detected in the second repair strategy, so that after the current node performs a restart operation based on the repair restart identifier, the second repair strategy can continue to be executed through the interrupt execution step.

[0096] The above-described apparatus can execute the fault repair method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the fault repair method provided in any embodiment of the present invention.

[0097] Example 5

[0098] Figure 5 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, electronic devices, blade electronic devices, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0099] like Figure 5As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0100] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0101] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as fault repair methods.

[0102] In some embodiments, the fault repair method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on a heterogeneous hardware accelerator via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by a processor, one or more steps of the fault repair method described above may be performed. Alternatively, in other embodiments, the processor may be configured to perform the fault repair method by any other suitable means (e.g., by means of firmware).

[0103] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0104] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0105] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0106] To provide user interaction, the systems and techniques described herein can be implemented on a heterogeneous hardware accelerator, which includes: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the heterogeneous hardware accelerator. Other types of devices can also be used to provide user interaction; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or haptic feedback); and input from the user can be received in any form (including sound input, voice input, or haptic input).

[0107] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0108] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0109] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0110] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A fault repair method, characterized in that, include: In response to the detection of a fault repair signal for the current node, the context information of the current node is obtained; wherein, the context information includes a global configuration file, user command parameters, a session state file, and historical repair records; the session state file includes fault state information; Based on the fault status information, the current node is repaired using a predefined first repair strategy, and it is determined whether the first repair strategy has been successfully repaired. If it is determined that the first repair strategy fails, the context information is sent to the strategy recommendation model to obtain a second repair strategy, and the current node is repaired according to the second repair strategy.

2. The fault repair method according to claim 1, characterized in that, The step of repairing the current node according to the second repair strategy includes: Obtain the repair execution path for the current node, and retrieve a set of repair paths with similar paths from the repair path library; Based on the repair results of each repair path in the repair path set, the predicted repair probability, predicted repair progress, and predicted repair time of the repair execution path are obtained.

3. The fault repair method according to claim 1, characterized in that, After performing fault repair on the current node according to the second repair strategy, the following is also included: Based on the expected repair result corresponding to the second repair strategy, the actual repair result of the second repair strategy is repeatedly verified until the number of repeated verifications reaches a preset threshold.

4. The fault repair method according to claim 3, characterized in that, After iteratively verifying the actual repair result of the second repair strategy based on the expected repair result corresponding to the second repair strategy, until the number of iterative verifications reaches a preset threshold, the process further includes: Based on the repair execution log and the repair verification log, determine whether the second repair strategy was successful. If the second repair strategy is determined to be unsuccessful, the fault repair record is written to the session state file, and a restart signal is sent to the current node to guide the current node to restart.

5. The fault repair method according to claim 4, characterized in that, The step of determining whether the second repair strategy was successful based on the repair execution log and the repair verification log includes: Based on the repair execution log and the repair verification log, a matching classification category is obtained through a semantic classification model; wherein, the classification category includes repair success text and repair failure text.

6. The fault repair method according to claim 1, characterized in that, The step of obtaining a matching second repair strategy through the strategy recommendation model further includes: If a repair restart identifier is detected in the second repair strategy, the interrupt execution step corresponding to the repair restart identifier is written into the session state file so that after the current node performs a restart operation based on the repair restart identifier, the second repair strategy can continue to be executed through the interrupt execution step.

7. A fault repair device, characterized in that, include: The context information acquisition module is used to acquire the context information of the current node in response to the detection of a fault repair signal of the current node; wherein, the context information includes a global configuration file, user command parameters, session state file and historical repair records; the session state file includes fault state information; The fault repair execution module is used to repair the current node according to the fault status information using a predefined first repair strategy, and to determine whether the first repair strategy has been successfully repaired. The repair strategy acquisition module is used to send the context information to the strategy recommendation model if it is determined that the first repair strategy has failed, so as to obtain a second repair strategy through the strategy recommendation model and repair the current node according to the second repair strategy.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the fault repair method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are used to cause a processor to execute the fault repair method according to any one of claims 1-6.

10. A computer program product comprising a computer program that, when executed by a processor, implements the fault repair method according to any one of claims 1-6.

Images