Container cloud platform fault processing method and device, equipment and storage medium

Abstract

The invention discloses a container cloud platform fault processing method and device, equipment and a storage medium, and the method comprises the steps: obtaining a first code corresponding to a first fault event of a container cloud platform, wherein the first fault event is any fault event which is to be processed at present and has occurred before; inputting the first code into a pre-trained multi-expert model system to obtain a first feature code corresponding to the first fault event; inputting the first feature code into a large model decision system obtained by pre-training to obtain a first repair strategy corresponding to the first fault event; and executing the first repair strategy. By applying the technical scheme provided by the invention, the automatic processing of the container cloud platform fault is realized, the stability and reliability of the container cloud platform are improved, the manual intervention is reduced, the operation and maintenance cost is reduced, and the fault processing efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of computer application technology, and in particular to a container cloud platform fault handling method, device, equipment and storage medium. Background Art

[0002] As the scale and number of users of container cloud platforms continue to expand, the frequency of failures in container cloud platforms has also increased, affecting the stability and reliability of container cloud platforms. Especially when dealing with large-scale cluster tasks that require long-term high-power operation, such as artificial intelligence model training, any abnormal alarm may become a factor that hinders the smooth execution of the task, which in turn has a significant impact on economic and time costs.

[0003] In related technologies, the handling of container cloud platform failures mainly relies on manual intervention, which not only has high operation and maintenance costs and low fault handling efficiency, but also is difficult to adapt to the operation and maintenance needs of large-scale clusters. Summary of the invention

[0004] The purpose of this application is to provide a container cloud platform fault handling method, device, equipment and storage medium to realize automated handling of container cloud platform faults, improve the stability and reliability of the container cloud platform, reduce manual intervention, reduce operation and maintenance costs, and improve fault handling efficiency.

[0005] In order to solve the above technical problems, this application provides the following technical solutions:

[0006] In a first aspect, a container cloud platform fault handling method is provided, including:

[0007] Obtaining a first code corresponding to a first fault event of the container cloud platform, where the first fault event is any fault event that has occurred before and is currently to be processed;

[0008] Inputting the first code into a pre-trained multi-expert model system to obtain a first feature code corresponding to the first fault event;

[0009] Inputting the first feature code into a pre-trained large model decision system to obtain a first repair strategy corresponding to the first fault event;

[0010] The first repair strategy is executed.

[0011] Optionally, the multi-expert model system includes a gating model and multiple expert models, and different expert models are used to process fault events of different fault types. The first code is input into the pre-trained multi-expert model system to obtain a first feature code corresponding to the first fault event, including:

[0012] Inputting the first code into the gating model and each expert model respectively;

[0013] assigning a weight to each expert model based on the fault type of the first fault event by the gating model;

[0014] The output result of each expert model based on the first encoding is multiplied by the corresponding weight and then added to obtain a first feature code corresponding to the first fault event.

[0015] Optionally, each expert model includes at least an event abstraction layer, a weight distribution layer, a multi-head attention layer, a fusion layer and an encoding layer, wherein the multi-head attention layer includes multiple attention modules;

[0016] The event abstraction layer is used to abstract the code corresponding to the fault event into data that meets the input requirements of the attention module;

[0017] The weight allocation layer is used to allocate weights to each attention module;

[0018] The multi-head attention layer is used to process the data output by the event abstraction layer;

[0019] The fusion layer is used to perform weighted summation on the output results of each attention module;

[0020] The encoding layer is used to encode the output result of the fusion layer and output the feature code corresponding to the fault event.

[0021] Optionally, the obtaining a first code corresponding to a first fault event of the container cloud platform includes:

[0022] Determine the failure type and occurrence frequency of a first failure event of the container cloud platform;

[0023] Based on the fault type and occurrence frequency of the first fault event, a first code corresponding to the first fault event is generated.

[0024] Optionally, executing the first repair strategy includes:

[0025] The first repair strategy is executed on a container node or a control node of the container cloud platform through a sidecar container.

[0026] Optionally, the method further comprises:

[0027] Obtaining a second fault event of the container cloud platform, where the second fault event is a fault event that has not occurred before;

[0028] Obtaining, by tuning the system, a second code corresponding to the second fault event, a second feature code, and a second repair strategy associated with the second feature code;

[0029] Adding the second code and the second feature code to the first training set, so as to train the multi-expert model system based on the updated first training set;

[0030] The second feature code and the second repair strategy associated with the second feature code are added to a second training set, so as to train the large model decision system based on the updated second training set.

[0031] Optionally, after obtaining the second code, the second feature code, and the second repair strategy associated with the second feature code corresponding to the second fault event through the tuning system, the method further includes:

[0032] The second repair strategy is executed.

[0033] In a second aspect, a container cloud platform fault handling device is provided, including:

[0034] An acquiring unit, configured to acquire a first code corresponding to a first fault event of the container cloud platform, where the first fault event is any fault event that has occurred before and is currently to be processed;

[0035] A first processing unit, configured to input the first code into a pre-trained multi-expert model system to obtain a first feature code corresponding to the first fault event;

[0036] A second processing unit is used to input the first feature code into a pre-trained large model decision system to obtain a first repair strategy corresponding to the first fault event;

[0037] An execution unit is used to execute the first repair strategy.

[0038] Optionally, the multi-expert model system includes a gating model and multiple expert models, and different expert models are used to process fault events of different fault types. The first processing unit is specifically used to:

[0039] Inputting the first code into the gating model and each expert model respectively;

[0040] assigning a weight to each expert model based on the fault type of the first fault event by the gating model;

[0041] The output result of each expert model based on the first encoding is multiplied by the corresponding weight and then added to obtain a first feature code corresponding to the first fault event.

[0042] Optionally, each expert model includes at least an event abstraction layer, a weight distribution layer, a multi-head attention layer, a fusion layer and an encoding layer, wherein the multi-head attention layer includes multiple attention modules;

[0043] The event abstraction layer is used to abstract the code corresponding to the fault event into data that meets the input requirements of the attention module;

[0044] The weight allocation layer is used to allocate weights to each attention module;

[0045] The multi-head attention layer is used to process the data output by the event abstraction layer;

[0046] The fusion layer is used to perform weighted summation on the output results of each attention module;

[0047] The encoding layer is used to encode the output result of the fusion layer and output the feature code corresponding to the fault event.

[0048] Optionally, the acquisition unit is specifically used to:

[0049] Determine the failure type and occurrence frequency of a first failure event of the container cloud platform;

[0050] Based on the fault type and occurrence frequency of the first fault event, a first code corresponding to the first fault event is generated.

[0051] Optionally, the execution unit is specifically used to:

[0052] The first repair strategy is executed on a container node or a control node of the container cloud platform through a sidecar container.

[0053] Optionally, the container cloud platform fault processing device further includes a third processing unit, configured to:

[0054] Obtaining a second fault event of the container cloud platform, where the second fault event is a fault event that has not occurred before;

[0055] Obtaining, by tuning the system, a second code corresponding to the second fault event, a second feature code, and a second repair strategy associated with the second feature code;

[0056] Adding the second code and the second feature code to the first training set, so as to train the multi-expert model system based on the updated first training set;

[0057] The second feature code and the second repair strategy associated with the second feature code are added to a second training set, so as to train the large model decision system based on the updated second training set.

[0058] Optionally, the third processing unit is further configured to:

[0059] After obtaining the second code, the second feature code, and the second repair strategy associated with the second feature code corresponding to the second fault event through the tuning system, the second repair strategy is executed.

[0060] In a third aspect, an electronic device is provided, including:

[0061] Memory for storing computer programs;

[0062] A processor is used to implement the steps of the container cloud platform fault handling method as described in the first aspect when executing the computer program.

[0063] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the container cloud platform fault handling method as described in the first aspect are implemented.

[0064] In a fifth aspect, a computer program product is provided, which includes computer instructions, which are stored in a computer-readable storage medium and are suitable for being read and executed by a processor, so that a computer device having the processor performs the steps of the container cloud platform fault handling method as described in the first aspect.

[0065] By applying the technical solution provided in the embodiment of the present application, after obtaining the first code corresponding to the first fault event of the container cloud platform, the first code is input into the pre-trained multi-expert model system to obtain the first feature code corresponding to the first fault event, and then the first feature code is input into the pre-trained large model decision system to obtain the first repair strategy corresponding to the first fault event, and then the first repair strategy is executed, thereby realizing the automated processing of container cloud platform faults, helping to improve the stability and reliability of the container cloud platform, reduce manual intervention, reduce operation and maintenance costs, and improve fault handling efficiency.

[0066] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0068] Figure 1 This is a flowchart of an implementation method of a container cloud platform fault handling method in an embodiment of the present application;

[0069] Figure 2 This is a schematic diagram of the structure of the expert model in the embodiment of the present application;

[0070] Figure 3 This is a schematic diagram of the structure of the attention module in the embodiment of the present application;

[0071] Figure 4 This is a schematic diagram of the structure of the fault handling platform in the embodiment of the present application;

[0072] Figure 5 This is a structural schematic diagram of a container cloud platform fault handling device in an embodiment of the present application;

[0073] Figure 6 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0074] The technical solutions in the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of the present application.

[0075] The terms "first", "second", etc. of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first" and "second" are generally of one type, and the number of objects is not limited, for example, the first object can be one or more. In addition, "or" in the present application represents at least one of the connected objects. For example, the protection scope of "A or B" covers at least three schemes, that is, scheme one: including A and excluding B; scheme two: including B and excluding A; scheme three: including both A and B. In addition, the terms "A and / or B", "at least one of A and B", and "at least one of A or B" also cover at least the above three schemes respectively. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0076] The core of this application is to provide a container cloud platform fault handling method, which can be applied to scenarios such as container cloud platform operation and maintenance. Specifically, the container cloud platform can be used to deploy and manage big data processing tasks, such as artificial intelligence model training and reasoning, and provide the necessary computing resources and environment.

[0077] After obtaining the first code corresponding to the first fault event of the container cloud platform, the embodiment of the present application inputs the first code into a pre-trained multi-expert model system, and after processing by the multi-expert model system, obtains a first feature code corresponding to the first fault event. The first feature code is then input into a pre-trained large model decision system, and after processing by the large model decision system, obtains a first repair strategy corresponding to the first fault event, and then executes the first repair strategy.

[0078] It realizes the automated handling of container cloud platform failures, which helps to improve the stability and reliability of the container cloud platform, reduce manual intervention, reduce operation and maintenance costs, and improve fault handling efficiency.

[0079] The embodiments of the present application utilize the deep learning capabilities of large models and multi-expert model technology to achieve automatic detection, classification, diagnosis and repair of container cloud platform failures, which helps to improve operation and maintenance efficiency and system stability.

[0080] See also Figure 1 As shown, it is an implementation flow chart of a container cloud platform fault handling method provided by an embodiment of the present application, and the method may include the following steps:

[0081] S110: Obtain a first code corresponding to a first failure event of the container cloud platform.

[0082] In an embodiment of the present application, a failure may occur in the container cloud platform during operation, thereby causing the interruption of tasks executed by the container cloud platform.

[0083] The main faults that lead to the interruption of container cloud platform tasks include the following:

[0084] Software / hardware damage exceptions: including server hardware failure, storage device damage, operating system or application crash, etc. These problems may cause service interruption or data loss;

[0085] Business service anomalies: Business applications or services fail to provide expected functions, such as the failure of World Wide Web (Web) services to respond to requests or database query failures.

[0086] Resource overload exception: computing resources (such as central processing unit (CPU), memory), storage resources (such as disk space) or network bandwidth have reached their upper limits, resulting in system performance degradation or service unavailability;

[0087] Network communication anomalies: network connection interruption, packet loss, increased latency, Domain Name System (DNS) resolution errors, etc. These problems affect communication between containers and access to external services;

[0088] Task scheduling anomalies: The task scheduler cannot correctly allocate resources or tasks, resulting in task execution failure or delay. For example, a task is incorrectly allocated to a node with insufficient resources.

[0089] Configuration anomalies: Configuration file errors, improper parameter settings, version compatibility issues, Access Control List (ACL) setting errors, etc. These problems may cause abnormal system behavior or service interruption;

[0090] Security anomalies: including unauthorized access, data leakage, malware infection, distributed denial of service attack (DDoS), etc. These problems threaten the security of the system and the integrity of the data.

[0091] When a first fault event occurs in the container cloud platform, a first code corresponding to the first fault event can be obtained. The first code is related to the fault type and occurrence frequency of the first fault event. The first fault event is any fault event that is currently to be processed and has occurred before.

[0092] In the embodiment of the present application, for any fault event, if the fault type of the fault event has occurred before, it is considered that the fault event has occurred before, and it is considered that a corresponding repair strategy has been established for the fault event.

[0093] S120: Inputting the first code into a pre-trained multi-expert model system to obtain a first feature code corresponding to the first fault event.

[0094] In an embodiment of the present application, a multi-expert model system can be pre-trained. The multi-expert model system is a system built using multi-expert model technology, which aims to improve the expressiveness and flexibility of the model through the combination of multiple expert models and a gating model. Each expert model focuses on processing a specific part or feature of the data, and the gating model dynamically selects the appropriate expert model for processing based on the characteristics of the input data. In this way, the multi-expert model system can improve the performance and adaptability of the model while keeping the overall scale of the model controllable.

[0095] A first training set can be obtained in advance, and each training data included in the first training set corresponds to a historical failure event of the container cloud platform, and each historical failure event has a corresponding code and a feature code. The multi-expert model system is trained based on the first training set, and the parameters of the multi-expert model system are adjusted during the training process. When the number of training times reaches the set first threshold or the multi-expert model system converges, the training can be terminated, and the trained multi-expert model system can process the code corresponding to the failure event and output the feature code corresponding to the failure event. According to the set training cycle, or when the training conditions are met, the multi-expert model system can be retrained or incrementally trained to continuously improve the stability and accuracy of the multi-expert model system.

[0096] After obtaining the first code corresponding to the first fault event, the first code may be input into a pre-trained multi-expert model system, and after being processed by the multi-expert model system, a first feature code corresponding to the first fault event may be obtained.

[0097] In the embodiment of the present application, the code corresponding to the fault event can be understood as the preliminary classification and coding of the fault event, and the feature code corresponding to the fault event can be understood as the result of further processing of the code corresponding to the fault event, representing the specific characteristics of the fault event and used to match the repair strategy.

[0098] S130: Inputting the first feature code into a pre-trained large model decision system to obtain a first repair strategy corresponding to the first fault event.

[0099] In the embodiment of the present application, a large model decision system can be pre-trained. A large model refers to a training model with a large parameter scale and complex structure, which is mainly used to process massive data and solve complex problems. This type of model has strong representation and generalization capabilities and can achieve excellent performance on a variety of tasks. Large models usually require a lot of computing resources for training. Using its deep learning capabilities, the large model can predict and analyze failure events on the container cloud platform.

[0100] A second training set can be obtained in advance, and each training data included in the second training set corresponds to a historical failure event of the container cloud platform, and each historical failure event has a corresponding feature code and an associated repair strategy. The large model decision system is trained based on the second training set, and the parameters of the large model decision system are adjusted during the training process. When the number of training times reaches the set second number threshold or the large model decision system converges, the training can be terminated. The trained large model decision system can process the feature code corresponding to the failure event and output the repair strategy corresponding to the failure event. According to the set training cycle, or when the training conditions are met, the large model decision system can be retrained or incrementally trained, or the large model decision system can learn during the application process to continuously improve the stability and accuracy of the large model decision system.

[0101] After obtaining the first feature code corresponding to the first fault event, the first feature code can be input into a pre-trained large model decision system, and after being processed by the large model decision system, a first repair strategy corresponding to the first fault event is obtained.

[0102] In the embodiment of the present application, the repair strategy can be represented in code form, including a series of scripts and execution logic required to be executed to resolve the fault.

[0103] S140: Execute the first repair strategy.

[0104] In the embodiment of the present application, after obtaining the first repair strategy corresponding to the first fault event, the first repair strategy can be executed.

[0105] After executing the first repair strategy, if the normal operation of the container cloud platform can be restored, it is considered that the first fault event is effectively handled, the fault handling is successful, and the first repair strategy corresponding to the first fault event is effective.

[0106] After executing the first repair strategy, if the normal operation of the container cloud platform cannot be restored, it is considered that the first fault event has not been effectively handled, the fault handling has failed, and the first repair strategy corresponding to the first fault event is invalid. In this case, an alarm notification can be sent. Specifically, an alarm notification can be sent to the operation and maintenance personnel in a variety of ways, such as message push, flashing lights, etc., so that the operation and maintenance personnel can troubleshoot the fault in a timely manner. Furthermore, after obtaining an effective repair strategy for the first fault event, the relevant information of the first fault event can be added to the second training set to expand the training set of the large model decision system, so that the repair strategy output by the large model decision system after retraining or incremental training is more accurate.

[0107] By applying the method provided in the embodiment of the present application, after obtaining the first code corresponding to the first fault event of the container cloud platform, the first code is input into the pre-trained multi-expert model system to obtain the first feature code corresponding to the first fault event, and then the first feature code is input into the pre-trained large model decision system to obtain the first repair strategy corresponding to the first fault event, and then the first repair strategy is executed, thereby realizing the automated processing of container cloud platform faults, helping to improve the stability and reliability of the container cloud platform, reduce manual intervention, reduce operation and maintenance costs, and improve fault handling efficiency.

[0108] In some embodiments of the present application, obtaining a first code corresponding to a first fault event of a container cloud platform may include the following steps:

[0109] Determine the failure type and occurrence frequency of a first failure event of the container cloud platform;

[0110] Based on the fault type and occurrence frequency of the first fault event, a first code corresponding to the first fault event is generated.

[0111] For the convenience of description, the above steps are combined for explanation.

[0112] In an embodiment of the present application, when a first fault event occurs in the container cloud platform, the fault type and occurrence frequency of the first fault event may be determined first.

[0113] Exemplarily, the fault type of the first fault event may include at least one of the following:

[0114] Abnormal software / hardware damage;

[0115] Business service abnormality;

[0116] Resource full load exception;

[0117] Network communication is abnormal;

[0118] Task scheduling is abnormal;

[0119] Configuration anomaly;

[0120] Security exception.

[0121] It should be noted that the above-mentioned fault types are only examples. In actual applications, fault types can also be divided based on different dimensions and different granularities, and the embodiments of the present application do not limit this.

[0122] The occurrence frequency of the first fault event can be understood as the number of times the first fault event occurs within a set time period. Within a time period, fault events of the same fault type may occur multiple times.

[0123] Based on the fault type and occurrence frequency of the first fault event, a first code corresponding to the first fault event can be generated. Optionally, a portrait of the first fault event can be generated based on the fault type and occurrence frequency of the first fault event, and then based on the portrait of the first fault event, a first code corresponding to the first fault event can be generated.

[0124] Optionally, the first code corresponding to the first fault event may include three parts, the first part is a fault type code, the second part is an occurrence frequency code, and the third part is a random code.

[0125] The first code corresponding to the first fault event is related to the fault type and occurrence frequency of the first fault event, which can improve the effectiveness of the first code corresponding to the first fault event.

[0126] In some embodiments of the present application, the multi-expert model system includes a gating model and multiple expert models, and different expert models are used to process fault events of different fault types. Inputting the first code into the multi-expert model system obtained by pre-training to obtain a first feature code corresponding to the first fault event may include the following steps:

[0127] inputting the first code into the gating model and each expert model respectively;

[0128] assigning a weight to each expert model based on the fault type of the first fault event through the gating model;

[0129] The output result of each expert model based on the first encoding is multiplied by the corresponding weight and then added to obtain a first feature code corresponding to the first fault event.

[0130] For the convenience of description, the above steps are combined for explanation.

[0131] In an embodiment of the present application, the multi-expert model system may include a gating model and multiple expert models, and different expert models are used to handle fault events of different fault types.

[0132] After obtaining the first code corresponding to the first fault event of the container cloud platform, the first code can be input into the gating model and each expert model respectively. The gating model assigns weights to each expert model based on the fault type of the first fault event. For example, the multi-expert model system includes a gating model and three expert models. After training, expert model 1 is used to process fault events of network communication abnormality type, expert model 2 is used to process fault events of task scheduling abnormality type, and expert model 3 is used to process fault events of configuration abnormality type. If the fault type of the first fault event is a network communication abnormality type, the gating model can assign 80% weight to expert model 1 and 10% weight to expert model 2 and expert model 3 respectively. In this way, each expert model can obtain the corresponding weight. Each expert model performs processing based on the first code to obtain the corresponding output result. The output result of each expert model is multiplied by the corresponding weight and added together to obtain the first feature code corresponding to the first fault event.

[0133] Through the gating model and multiple expert models included in the multi-expert model system, the first code corresponding to the first fault event can be processed to obtain the first feature code corresponding to the first fault event, thereby providing a guarantee for obtaining a subsequent repair strategy.

[0134] The multi-expert model system is obtained through pre-training. During the training process, the codes corresponding to each training data in the first training set can be respectively input into the gating model and each expert model. The gating model can randomly assign weights to each expert model, and adjust the parameters based on the output result of each expert model multiplied by the corresponding weight and added together, so that the multi-expert model system can converge after a period of training.

[0135] Optionally, the fault events can be pre-screened by the fault event pre-processing system. For example, the basic event analysis logic can be used to screen out the fault events that meet the processing requirements. Then the relevant information of the screened fault events is stored in the database, and the database is maintained. The fault events that meet the processing requirements can be considered as fault events that can be processed by the multi-expert model system and the large model decision system provided in the embodiment of the present application. The first training set based on the training of the multi-expert model system can be obtained through the database.

[0136] In the embodiment of the present application, the first fault event can be considered to be a fault event after preliminary screening.

[0137] In some embodiments of the present application, Figure 2 As shown, each expert model at least includes an event abstraction layer, a weight distribution layer, a multi-head attention layer, a fusion layer and an encoding layer, and the multi-head attention layer includes multiple attention modules, such as attention module 1, attention module 2, ..., attention module M.

[0138] Among them, the event abstraction layer is used to abstract the encoding corresponding to the fault event into data that meets the input requirements of the attention module;

[0139] The weight distribution layer is used to assign weights to each attention module. During the training phase, the weight distribution layer can randomly assign weights to different attention modules and send the data output by the event abstraction layer to each attention module. When the training is completed, the parameters corresponding to the attention module and the weight distribution layer will converge to obtain an expert model for the corresponding classification.

[0140] The multi-head attention layer is used to process the data output by the event abstraction layer;

[0141] The fusion layer is used to weighted average the output results of each attention module;

[0142] The encoding layer is used to encode the output results of the fusion layer and output the feature code corresponding to the fault event.

[0143] Optionally, each expert model may further include a label filtering layer, which is located before the event abstraction layer and is used to filter based on the codes corresponding to the fault events, such as cleaning out garbled codes and codes that do not comply with the specifications.

[0144] like Figure 3 The figure shows the structure of the attention module, which includes a self-attention layer, a first normalization layer, a feedforward network layer, and a second normalization layer connected in sequence. The self-attention layer and the first normalization layer are combined as the first level of the attention module to enhance the performance of neural network training, and the feedforward network layer and the second normalization layer are combined as the second level to serve the model training of the neural network, so that the network parameters can be passed forward.

[0145] In some embodiments of the present application, executing the first repair strategy may include the following steps:

[0146] The first repair strategy is executed on the container node or control node of the container cloud platform through the sidecar container.

[0147] In the embodiment of the present application, the first repair strategy corresponding to the first fault event can be obtained through the large model decision system. After obtaining the first repair strategy, the first repair strategy can be executed on the container node or control node of the container cloud platform through the sidecar container. The sidecar container can be attached to the cloud container, non-intrusive to the business scenario, and can monitor and execute related operations in real time.

[0148] Optionally, after obtaining the first repair strategy, the first repair strategy may be executed by a script executor.

[0149] In some embodiments of the present application, the method may further include the following steps:

[0150] Obtain a second fault event of the container cloud platform, where the second fault event is a fault event that has not occurred before;

[0151] Obtaining, by tuning the system, a second code corresponding to the second fault event, a second feature code, and a second repair strategy associated with the second feature code;

[0152] adding the second code and the second feature code to the first training set, so as to train the multi-expert model system based on the updated first training set;

[0153] The second feature code and the second repair strategy associated with the second feature code are added to the second training set, so as to train the large model decision system based on the updated second training set.

[0154] For the convenience of description, the above steps are combined for explanation.

[0155] In an embodiment of the present application, during the operation of the container cloud platform, a fault event generated may be one that has occurred before or one that has not occurred before. A fault event that has not occurred before can be understood as a new fault event, and the fault type of the fault event is different from the fault type of a fault event that has occurred before.

[0156] A second fault event of the container cloud platform is obtained, where the second fault event is a fault event that has not occurred before. The second code, the second feature code, and the second repair strategy associated with the second feature code corresponding to the second fault event can be obtained by tuning the system. The second repair strategy associated with the second feature code can be determined by the operation and maintenance personnel.

[0157] The second code and the second feature code are added to the first training set, and the first training set is updated. In this way, the multi-expert model system can be trained based on the updated first training set to improve the adaptability of the multi-expert model system.

[0158] The second feature code and the second repair strategy associated with the second feature code are added to the second training set, so that the large model decision system can be trained based on the updated second training set to improve the adaptability of the large model decision system.

[0159] In some embodiments of the present application, after obtaining the second code, the second feature code, and the second repair strategy associated with the second feature code corresponding to the second fault event through the tuning system, the method may further include the following steps:

[0160] Execute the second repair strategy.

[0161] In an embodiment of the present application, the second fault event is a fault event that has not occurred before, and after obtaining the second repair strategy corresponding to the second fault event, the second repair strategy can be executed. Optionally, the second repair strategy can be executed on a container node or a control node of the container cloud platform through a sidecar container or a script executor.

[0162] After executing the second repair strategy, if the normal operation of the container cloud platform cannot be restored, it is considered that the second fault event has not been effectively handled, the fault handling has failed, and the second repair strategy corresponding to the second fault event is invalid. In this case, an alarm notification can be sent. Specifically, an alarm notification can be sent to the operation and maintenance personnel in a variety of ways, such as message push, flashing lights, etc., so that the operation and maintenance personnel can troubleshoot in a timely manner. Furthermore, after obtaining an effective repair strategy for the second fault event, the information related to the second fault event can be added to the second training set to expand the training set of the large model decision system, so that the repair strategy output by the large model decision system after retraining or incremental training is more accurate.

[0163] The technical solution provided in the embodiment of the present application can be run on the fault handling platform. Figure 4 As shown, the fault handling platform may include a fault event preprocessing system, a multi-expert model system, a large model decision system, a tuning system, and an instruction execution system.

[0164] When a fault occurs on the container cloud platform, a fault handling request can be sent to the fault handling platform.

[0165] After receiving the fault processing request, the fault event preprocessing system can filter the fault events. For example, it can use the basic event analysis logic to filter out the fault events that do not meet the processing requirements, and then count the fault events to obtain the frequency of occurrence of the fault events, and then encode the fault events, such as encoding the fault events based on the fault type and frequency of occurrence, and finally perform fault analysis records, store the fault events in the database, and maintain the database.

[0166] The multi-expert model system sends the input to N expert models according to the codes corresponding to the fault events transmitted from the north. Each expert model is pre-trained. Before training, a gating model randomly assigns weights to each expert model. During the training process, the parameters are adjusted so that after a period of training, the expert model and the gating model converge. After training, different expert models will focus on different fault types. After the multiplier and adder, the multi-expert model system outputs the feature code corresponding to the fault event to the south.

[0167] The large model decision system can pass in two parameters. One is the feature code corresponding to the newly appeared fault event output by the southbound optimization system (such as Figure 4The feature code is associated with the corresponding repair strategy, which includes the relevant processing methods in the fault automatic repair strategy library and the manually annotated script execution strategy; the other is the feature code corresponding to the fault event passed in by the multi-expert model system. If the fault event has occurred before, the tuning system will not output the feature code.

[0168] Relevant information about new fault events can also be input into the multi-expert model system (such as Figure 4 When the training conditions are met, the multi-expert model system can be retrained or incrementally trained.

[0169] The fault diagnosis big model included in the big model decision system is used to process the correspondence between the feature code corresponding to the fault event and the repair strategy.

[0170] During training, after the fault diagnosis model obtains the feature code corresponding to the fault event, it will combine the feature code to formulate a strategy to solve the fault. After the training converges, it should adaptively output various automatic fault repair strategies to simulate the operational thinking of cluster operation and maintenance personnel.

[0171] During this process, the fault diagnosis model will learn various automatic fault repair strategies, and the final result is to output a string of code, which represents the logic and code block codes of a series of scripts that need to be executed to solve the fault.

[0172] The large model decision system and the multi-expert model system can be jointly trained, and the final output product is the repair strategy, which can be considered as an automated execution script and execution sequence logic. The repair strategy execution instructions are issued through the instruction execution system.

[0173] The repair strategy output by the large model decision system combined with the fault container and the script image library can generate a complete automatic execution script, which is automatically executed on the container node or control node through the sidecar container or script executor to automatically handle the fault, restore the normal operation of the container cloud platform or send an alarm notification.

[0174] The sidecar container will be attached to the cloud container, without intrusion into the business scenario, and will monitor and execute related operations in real time.

[0175] For example, the N expert models are E1, E2, …, E N , the parameters of each expert model are θ1, θ2, …, θ N , the input data is x, and there is the following formula relationship:

[0176]

[0177] P=M(y)(1-4)

[0178] Among them, formula 1-2 represents the weight generated by the gating model for the i-th expert model, G represents the gating model, represents the parameters of the gating model;

[0179] Formula 1-3 ensures the normalization of weights;

[0180] In formula 1-4, the input data is y, the fault diagnosis model is M, and the final output automatic repair strategy is P.

[0181] The embodiment of the present application uses the deep learning capability of the big model to analyze and organize the fault types of the container cloud platform, forming an end-to-end fault automatic repair self-closed loop system that integrates the big model. Combining multiple expert models with a gating model enhances the output capability of the fault repair strategy and enhances the comprehensive response capability of the big model in the fault automatic repair scenario.

[0182] The embodiments of the present application can realize a fully automatic process from fault occurrence to fault repair, which helps to reduce the investment of operation and maintenance personnel, effectively save operation and maintenance costs, improve operation and maintenance efficiency, and improve the stability of the container cloud platform. It can also specialize corresponding automated fault repair capabilities for specific scenarios in the environment, covering the operation and maintenance scenarios of most production environments.

[0183] Corresponding to the above method embodiment, the embodiment of the present application also provides a container cloud platform fault handling device. The container cloud platform fault handling device described below and the container cloud platform fault handling method described above can refer to each other.

[0184] See also Figure 5 As shown, the container cloud platform fault processing device 500 includes the following units:

[0185] An acquiring unit 510 is configured to acquire a first code corresponding to a first fault event of the container cloud platform, where the first fault event is any fault event that has occurred before and is currently to be processed;

[0186] A first processing unit 520 is used to input the first code into a pre-trained multi-expert model system to obtain a first feature code corresponding to the first fault event;

[0187] The second processing unit 530 is used to input the first feature code into the pre-trained large model decision system to obtain a first repair strategy corresponding to the first fault event;

[0188] The execution unit 540 is configured to execute the first repair strategy.

[0189] By using the device provided in the embodiment of the present application, after obtaining the first code corresponding to the first fault event of the container cloud platform, the first code is input into the pre-trained multi-expert model system to obtain the first feature code corresponding to the first fault event, and then the first feature code is input into the pre-trained large model decision system to obtain the first repair strategy corresponding to the first fault event, and then the first repair strategy is executed, thereby realizing the automated processing of container cloud platform faults, helping to improve the stability and reliability of the container cloud platform, reduce manual intervention, reduce operation and maintenance costs, and improve fault handling efficiency.

[0190] In some embodiments of the present application, the multi-expert model system includes a gating model and multiple expert models, and different expert models are used to process fault events of different fault types. The first processing unit 520 is specifically used to:

[0191] inputting the first code into the gating model and each expert model respectively;

[0192] assigning a weight to each expert model based on the fault type of the first fault event through the gating model;

[0193] The output result of each expert model based on the first encoding is multiplied by the corresponding weight and then added to obtain a first feature code corresponding to the first fault event.

[0194] In some embodiments of the present application, each expert model includes at least an event abstraction layer, a weight distribution layer, a multi-head attention layer, a fusion layer and an encoding layer, and the multi-head attention layer includes multiple attention modules;

[0195] Among them, the event abstraction layer is used to abstract the code corresponding to the fault event into data that meets the input requirements of the attention module;

[0196] The weight assignment layer is used to assign weights to each attention module;

[0197] The multi-head attention layer is used to process the data output by the event abstraction layer;

[0198] The fusion layer is used to weighted average the output results of each attention module;

[0199] The encoding layer is used to encode the output results of the fusion layer and output the feature code corresponding to the fault event.

[0200] In some embodiments of the present application, the acquisition unit 510 is specifically configured to:

[0201] Determine the failure type and occurrence frequency of a first failure event of the container cloud platform;

[0202] Based on the fault type and occurrence frequency of the first fault event, a first code corresponding to the first fault event is generated.

[0203] In some embodiments of the present application, the execution unit 540 is specifically configured to:

[0204] The first repair strategy is executed on the container node or control node of the container cloud platform through the sidecar container.

[0205] In some embodiments of the present application, the container cloud platform fault processing device 500 further includes a third processing unit, which is used to:

[0206] Obtain a second fault event of the container cloud platform, where the second fault event is a fault event that has not occurred before;

[0207] Obtaining, by tuning the system, a second code corresponding to the second fault event, a second feature code, and a second repair strategy associated with the second feature code;

[0208] adding the second code and the second feature code to the first training set, so as to train the multi-expert model system based on the updated first training set;

[0209] The second feature code and the second repair strategy associated with the second feature code are added to the second training set, so as to train the large model decision system based on the updated second training set.

[0210] In some embodiments of the present application, the third processing unit is further configured to:

[0211] After obtaining the second code, the second feature code, and the second repair strategy associated with the second feature code corresponding to the second fault event through the tuning system, the second repair strategy is executed.

[0212] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0213] Corresponding to the above method embodiment, the present application embodiment further provides an electronic device, including:

[0214] Memory for storing computer programs;

[0215] A processor is used to implement the steps of the above-mentioned container cloud platform fault handling method when executing a computer program.

[0216] like Figure 6 , which is a schematic diagram of the composition structure of an electronic device, the electronic device may include: a processor 10, a memory 11, a communication interface 12 and a communication bus 13. The processor 10, the memory 11 and the communication interface 12 all communicate with each other through the communication bus 13.

[0217] In the embodiment of the present application, the processor 10 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic devices, etc.

[0218] The processor 10 may call a program stored in the memory 11. Specifically, the processor 10 may execute operations in an embodiment of the container cloud platform fault handling method.

[0219] The memory 11 is used to store one or more programs, which may include program codes, and the program codes include computer operation instructions. In the embodiment of the present application, the memory 11 at least stores programs for implementing the following functions:

[0220] Obtain a first code corresponding to a first fault event of the container cloud platform, where the first fault event is any fault event that has occurred before and is currently to be processed;

[0221] Inputting the first code into a pre-trained multi-expert model system to obtain a first feature code corresponding to the first fault event;

[0222] Inputting the first feature code into a pre-trained large model decision system to obtain a first repair strategy corresponding to the first fault event;

[0223] Execute the first repair strategy.

[0224] In a possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function, etc.; the data storage area may store data created during use.

[0225] In addition, the memory 11 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0226] The communication interface 12 may be an interface of a communication module, and is used to connect to other devices or systems.

[0227] Of course, it should be noted that Figure 6 The structure shown does not constitute a limitation on the electronic device in the embodiment of the present application. In actual applications, the electronic device may include Figure 6 More or fewer components than shown, or combinations of certain components.

[0228] Corresponding to the above method embodiment, the embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned container cloud platform fault handling method are implemented.

[0229] In addition, it should be noted that: the embodiment of the present application also provides a computer program product or a computer program, which may include computer instructions, and the computer instructions may be stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor can execute the computer instructions so that the computer device executes the description of the container cloud platform fault handling method in the corresponding embodiment of the foregoing text, so it will not be repeated here. In addition, the description of the beneficial effects of using the same method will not be repeated. For technical details not disclosed in the computer program product or computer program embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0230] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0231] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises one..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0232] Through the description of the above implementation methods, those skilled in the art can also clearly understand that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0233] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein can be directly implemented using hardware, a software module executed by a processor, or a combination of the two. The software module can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a hard disk, a removable disk, a compact disc read-only memory (CD-ROM), or any other form of storage medium known in the art, including several instructions to execute the methods described in the various embodiments of the present application.

[0234] The embodiments of the present application are described above in conjunction with the accompanying drawings, and the description of the above embodiments is only used to help understand the technical solution and its core idea of ​​the present application. It should be pointed out that the present application is not limited to the above-mentioned specific implementation methods, which are merely illustrative and not restrictive. For ordinary technicians in this field, many forms of implementation methods can be made without departing from the scope of protection of the purpose of the present application and the claims, and the present application can also be improved and modified in a number of ways, and these implementation methods, improvements and modifications are all within the scope of protection of the present application.

Claims

1. A method for handling container cloud platform failures, characterized in that: include: Obtaining a first code corresponding to a first fault event of the container cloud platform, where the first fault event is any fault event that has occurred before and is currently to be processed; Inputting the first code into a pre-trained multi-expert model system to obtain a first feature code corresponding to the first fault event; Inputting the first feature code into a pre-trained large model decision system to obtain a first repair strategy corresponding to the first fault event; The first repair strategy is executed.

2. The method according to claim 1, characterized in that The multi-expert model system includes a gating model and multiple expert models, and different expert models are used to process fault events of different fault types. The first code is input into the multi-expert model system obtained by pre-training to obtain a first feature code corresponding to the first fault event, including: Inputting the first code into the gating model and each expert model respectively; assigning a weight to each expert model based on the fault type of the first fault event by the gating model; The output result of each expert model based on the first encoding is multiplied by the corresponding weight and then added to obtain a first feature code corresponding to the first fault event.

3. The method according to claim 2, characterized in that Each expert model includes at least an event abstraction layer, a weight distribution layer, a multi-head attention layer, a fusion layer and an encoding layer, wherein the multi-head attention layer includes multiple attention modules; The event abstraction layer is used to abstract the code corresponding to the fault event into data that meets the input requirements of the attention module; The weight allocation layer is used to allocate weights to each attention module; The multi-head attention layer is used to process the data output by the event abstraction layer; The fusion layer is used to perform weighted average of the output results of each attention module; The encoding layer is used to encode the output result of the fusion layer and output the feature code corresponding to the fault event.

4. The method according to claim 1, characterized in that: The obtaining a first code corresponding to a first fault event of the container cloud platform includes: Determine the failure type and occurrence frequency of a first failure event of the container cloud platform; Based on the fault type and occurrence frequency of the first fault event, a first code corresponding to the first fault event is generated.

5. The method according to claim 1, characterized in that The executing the first repair strategy includes: The first repair strategy is executed on a container node or a control node of the container cloud platform through a sidecar container.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Obtaining a second fault event of the container cloud platform, where the second fault event is a fault event that has not occurred before; Obtaining, by tuning the system, a second code corresponding to the second fault event, a second feature code, and a second repair strategy associated with the second feature code; Adding the second code and the second feature code to the first training set, so as to train the multi-expert model system based on the updated first training set; The second feature code and the second repair strategy associated with the second feature code are added to a second training set, so as to train the large model decision system based on the updated second training set.

7. The method according to claim 6, characterized in that After obtaining the second code, the second feature code, and the second repair strategy associated with the second feature code corresponding to the second fault event through the tuning system, the method further includes: The second repair strategy is executed.

8. A container cloud platform fault handling device, characterized in that: include: An acquiring unit, configured to acquire a first code corresponding to a first fault event of the container cloud platform, where the first fault event is any fault event that has occurred before and is currently to be processed; A first processing unit, configured to input the first code into a pre-trained multi-expert model system to obtain a first feature code corresponding to the first fault event; A second processing unit is used to input the first feature code into a pre-trained large model decision system to obtain a first repair strategy corresponding to the first fault event; An execution unit is used to execute the first repair strategy.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the container cloud platform fault handling method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the container cloud platform fault handling method as described in any one of claims 1 to 7 are implemented.

Images