Platform operation and maintenance method and device based on generative AI technology

Abstract

The invention relates to a platform operation and maintenance method based on a generative AI technology, and the method is characterized in that the method comprises the steps: collecting operation data in real time, and training a prediction model; processing the operation data and extracting data features; judging the system state according to the data characteristics by using an AI decision model; evaluating and auditing the judgment result by using an AI supervision model; if the judgment result passes the audit and the system has no fault, predicting the accuracy of the user model by using the prediction model; if the judgment result passes the audit and the system has a fault or is about to have a fault, triggering a fault operation; and feeding back an audit result to the AI decision model, and carrying out optimization training on the AI decision model. Compared with the prior art, the method has the advantages that the operation and maintenance cost is reduced by automatically executing the operation and maintenance task through the generative AI technology, meanwhile, accuracy prediction is conducted on the model which is being trained by the user, and the accuracy and stability of the user model are improved.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a platform operation and maintenance method and device based on generative AI technology. Background Art

[0002] Traditional IT operation and maintenance mainly relies on manual configuration of alarms and conventional automated cleanup scripts. Although K8s can automatically restart services, isolate hosts and other routine operations, it requires a lot of manpower in areas with high requirements for accuracy of facts, timeliness of information, complexity of problems and security of data. Simple automated scripts are difficult to keep up with the pace of technology.

[0003] At present, traditional platform operation and maintenance solutions mainly rely on manual operations, such as Figure 1 As shown in the figure, the operation and maintenance engineer evaluates the fault situation based on his experience, determines the platform operation and maintenance recovery plan, and then manually handles the problem and performs recovery. However, this method has obvious disadvantages, the first of which is low processing efficiency. Relying on the experience and judgment of individual engineers may lead to a long response time, which slows down the speed at which the system recovers from the fault. In addition, since manual operation is easily affected by subjective factors, the accuracy of problem handling also faces certain risks. More importantly, traditional manual operation of fault handling operations has the potential risk of operational errors, which may lead to data loss or business interruption. Even experienced engineers can hardly avoid tension and pressure in emergency situations, which increases the possibility of operational errors.

[0004] At the same time, in traditional operation and maintenance, there is not much interaction between the operation and maintenance platform and the user platform. The operation and maintenance focuses on responding to failures when the platform fails, and when users use the platform functions normally, it does not assist in their daily behavior.

[0005] The existing technical solutions mainly have the following problems:

[0006] High cost: It requires a lot of manpower and physical resources, resulting in high operation and maintenance costs.

[0007] Inefficiency: Reliance on manual labor, such as manual monitoring, troubleshooting, and system upgrades, is time-consuming, error-prone, and inefficient.

[0008] Lack of flexibility: It is difficult to quickly adapt to changes in business needs, and it takes a long time to deploy new applications or upgrade systems.

[0009] Poor scalability: When business volume grows, traditional IT infrastructure is often difficult to expand quickly to meet additional demand.

[0010] Difficult to achieve refined management: Due to limitations in data analysis and monitoring methods, it is difficult to accurately manage and optimize the system's performance.

[0011] Lack of auxiliary functions: The platform is fault-free, and users do not receive any assistance when using the platform functions normally. Summary of the invention

[0012] The purpose of the present invention is to provide a platform operation and maintenance method and device based on generative AI technology. By utilizing generative AI technology, daily operation and maintenance tasks can be automatically performed, targeted operation and maintenance can be performed on different servers, and while responding to failures, the accuracy of the user's model being trained can be predicted to improve the accuracy and stability of the user's model, while reducing operation and maintenance costs and increasing the reliability and stability of the system.

[0013] The purpose of the present invention can be achieved by the following technical solutions:

[0014] A platform operation and maintenance method based on generative AI technology, the method comprising:

[0015] Collect operation data in real time, including platform data and user model training status;

[0016] The user model training state is used to form a training state data set, and the training state set is used to train the prediction model;

[0017] Process and filter the running data to extract data features;

[0018] Use AI decision-making models to judge system status based on data characteristics;

[0019] Use AI supervision models to evaluate and review judgment results;

[0020] If the judgment result passes the review and the judgment result is that the system has no fault, the prediction model is used to predict the accuracy of the user model;

[0021] If the judgment result passes the review and the judgment result is that the system has failed or is about to fail, the fault operation is triggered;

[0022] Feed the audit results back to the AI ​​decision-making model to optimize and train the AI ​​decision-making model.

[0023] Furthermore, the platform data includes: CPU occupancy, memory usage, disk I / O, network bandwidth, log information, user access volume, application request volume, error code returned by the platform, exception information returned by the platform, system load, disk space usage and network latency.

[0024] Furthermore, the training process of the prediction model includes:

[0025] Extract the training status of the model regularly;

[0026] The model training state at the previous moment is matched with the model training state at the next moment one by one to establish a training state data set;

[0027] The prediction model is trained by using the training state data set and constructing multiple decision trees through the random forest algorithm.

[0028] Furthermore, the processing of the operating data includes removing outliers, filling in missing data and standardizing the data format.

[0029] Furthermore, the training process of the AI ​​decision model includes:

[0030] According to the server type connected to the system, network search and historical database search are performed to build an adaptive training data set for the server type;

[0031] Use adaptive training data sets to train AI decision-making models.

[0032] Furthermore, the process of the AI ​​decision model judging the system state includes:

[0033] Determine whether the system status is faulty based on data characteristics;

[0034] If it is determined that the system status is faulty, the system fault category is further analyzed.

[0035] Furthermore, the system failure categories include resource scheduling problems, container optimization problems, monitoring and early warning problems, fault tolerance and fault recovery problems, distributed training problems, model parallelization problems, and data parallelism problems.

[0036] Furthermore, the fault operation includes:

[0037] According to the fault category, adopt historical solutions to solve the fault;

[0038] Generate an alarm notification to notify relevant operation and maintenance personnel and teams to resolve the fault.

[0039] Furthermore, the AI ​​supervision model evaluates the judgment result by learning from human experience and historical failures. If the judgment result fails the review, an alarm notification is generated to notify relevant operation and maintenance personnel and teams to perform maintenance.

[0040] A platform operation and maintenance device based on generative AI technology, the device comprising:

[0041] Data collection module, which collects operation data in real time, including platform data and user model training status;

[0042] Data processing module, which processes and filters the running data and extracts data features;

[0043] Model training module, which collects operation data, historical data and network data, and trains prediction models, AI decision models and AI supervision models;

[0044] The decision-making module uses the AI ​​decision-making model to judge the system status based on data characteristics;

[0045] The supervision module uses the AI ​​supervision model to evaluate and review the judgment results;

[0046] The prediction module uses the prediction model to predict the accuracy of the user model if the judgment result passes the review and the judgment result is that the system has no faults;

[0047] The fault response module triggers the fault operation if the judgment result passes the review and the judgment result is that the system has failed or is about to fail;

[0048] The model optimization module feeds the audit results back to the AI ​​decision-making model to optimize the training of the AI ​​decision-making model.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] 1. The present invention uses generative AI technology to make self-decision-making, self-evaluation and self-optimization during platform operation and maintenance, which improves operation and maintenance efficiency while reducing operation and maintenance costs. In the face of faults and potential faults, it can automatically take measures and alarm based on historical data, thereby increasing the reliability and stability of the system. When no fault is detected, it can also continuously predict the quality of the model trained by the user in the platform, thereby improving the accuracy and stability of the user model;

[0051] 2. The present invention uses generative AI technology and AI models to replace manual operation and maintenance, which can effectively solve the problem of manual misoperation, greatly reduce the risk of faulty operation, and is adaptive to faulty operations in different scenarios, thus lowering the threshold for operation and maintenance;

[0052] 3. The present invention can train a targeted decision model according to the type of server connected to the system, so that the platform operation and maintenance has adaptability, and can make decisions more accurately during operation and maintenance, reduce missed and misjudgment of faults, and realize refined management. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a structural diagram of the existing platform operation and maintenance method;

[0054] Figure 2 is a flow chart of the method of the present invention;

[0055] Figure 3 This is a flow chart of information collection in an embodiment of the present invention;

[0056] Figure 4 Flowchart of the prediction model training method. DETAILED DESCRIPTION

[0057] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0058] Example 1

[0059] This embodiment discloses a platform operation and maintenance method based on generative AI technology, wherein the platform is a k8s platform. When a user uses the platform, the user connects the personal server used for training the model to the k8s platform for operation and maintenance. The purpose of this platform operation and maintenance method is to monitor the platform in real time, and to issue fault alarms and take countermeasures when faults are predicted or occur. When the platform operates normally without any faults, it can also assist the user's normal usage behavior.

[0060] The platform operation and maintenance method comprises:

[0061] S1, real-time collection of operation data, including platform data and user model training status;

[0062] S2, using the user model training state to form a training state data set, and using the training state set to train the prediction model;

[0063] S3, processes and filters the running data to extract data features;

[0064] S4, using AI decision-making models to judge the system status based on data characteristics;

[0065] S5, using the AI ​​supervision model to evaluate and review the judgment result. If the judgment result passes the review and the judgment result is that the system has not failed, the prediction model is used to predict the accuracy of the user model. If the judgment result passes the review and the judgment result is that the system has failed or is about to fail, the fault operation is triggered;

[0066] S6, feeds the audit results back to the AI ​​decision-making model to optimize the training of the AI ​​decision-making model.

[0067] In step S1, the method for collecting operation data is as follows: Figure 3As shown, the operation data of the platform and application system are collected regularly and input into the LMM (Large Multimodal Model) large language model for processing. The operation data includes the key performance indicators of the platform, system and resource indicators, log information, platform traffic indicators, and platform errors and exception indicators.

[0068] Among them, performance indicators are used to determine whether the platform faces performance bottlenecks or excessive resource consumption. It includes information such as CPU usage, memory usage, disk I / O, and network bandwidth;

[0069] System and resource indicators are mainly used to understand the status of the platform system environment, including system load, disk space usage, network latency and other information;

[0070] Log information can be used to analyze potential problems and failures, including kubeapiserver, kubecontrollermanger, kubelet, kubeproxy, kubescheduler error logs, etc.

[0071] Platform traffic indicators mainly reflect the usage and load distribution of the platform, including user visits, application requests, etc.

[0072] Platform error and exception indicators mainly reflect the health status and potential failures of the platform, including error codes and exception information returned by the platform.

[0073] In step S2, the training process of the prediction model is as follows: Figure 4 As shown, including:

[0074] Regularly extract the training status of the model that the user puts into the platform for training;

[0075] The model training state at the previous moment is matched with the model training state at the next moment, features are constructed and weights and standard keywords are assigned to establish a training state data set;

[0076] The prediction model is trained by using the training state data set and constructing multiple decision trees through the random forest algorithm.

[0077] In step S3, the processing of the running data is performed by the LMM large language model, and the processing operations include removing outliers, filling missing data and standardizing the data format.

[0078] In this embodiment, chartgptweb is used as an AI decision model, and the training process of the AI ​​decision model includes:

[0079] According to the server type connected to the system, network search and historical database search are performed to build an adaptive training data set for the server type;

[0080] According to the adaptive training data set, the chartglm2\codelamma algorithm is used for learning and training to obtain the AI ​​decision-making model.

[0081] In step S4, the process of the AI ​​decision model judging the system state includes:

[0082] Determine whether the system status is faulty based on data characteristics;

[0083] If it is determined that the system status is faulty, the system fault category is further analyzed.

[0084] Among them, the system failure categories include resource scheduling problems, container optimization problems, monitoring and early warning problems, fault tolerance and fault recovery problems, distributed training problems, model parallelization problems and data parallel problems.

[0085] The specific manifestations and solutions of the system failure categories include:

[0086] Resource scheduling issues manifest themselves as model training tasks being unable to obtain sufficient computing resources, such as GPUs. The solution is to use the K8s scheduler function to dynamically schedule based on resource utilization and task priority to ensure stable operation of training tasks. You can also reduce startup time and memory usage by customizing container images and optimizing the loading method of deep learning frameworks.

[0087] The failure of container optimization problems manifests itself in slow container startup and excessive memory usage. The solution is to optimize container images, reduce unnecessary dependencies and files, and use GPU optimization technologies such as GPU Direct and NCCL libraries to improve communication efficiency between GPUs.

[0088] The fault manifestation of monitoring and early warning problems is that problems in the training process are difficult to discover and solve in time. The solution is to deploy a real-time monitoring and early warning system, provide rich monitoring indicators and visual interfaces, help users quickly understand the training status and performance bottlenecks, and automatically send notifications when abnormalities occur;

[0089] Fault tolerance and fault recovery problems manifest themselves as training interruptions caused by node or network failures in large-scale GPU cluster environments. The solution is to implement automatic fault detection and recovery functions, quickly reschedule tasks, and use data redundancy and checkpoint technology to ensure data security and training recoverability.

[0090] The failure of distributed training problems manifests itself as problems with model synchronization and gradient updates in a distributed environment. The solution is to use a distributed training framework such as Horovod, which supports data parallelism and simplifies the development process of multi-GPU or multi-node distributed training, achieves efficient inter-GPU communication through a ring topology, and uses NCCL for AllReduce operations;

[0091] The failure of model parallelization is manifested as the model is too large to be effectively trained on a single GPU. The solution is to use model parallelization technology to distribute different parts of the model to different hardware devices for calculation and train them through pipeline or data parallelism.

[0092] The failure of the data parallel problem is that the data set is too large to be efficiently trained on a single machine. The solution is to use data parallel technology to split the data set and train it on multiple computing nodes. Each node contains complete model parameters and updates the model by synchronizing gradients or parameters.

[0093] In step S5, the fault operation includes:

[0094] According to the fault category, the AI ​​decision model adopts the set solutions and historical solutions to solve the fault;

[0095] Generate an alarm notification to notify relevant operation and maintenance personnel and teams to resolve the fault.

[0096] In step S5, the AI ​​supervision model evaluates the judgment result by learning from human experience and historical failures. If the judgment result fails to pass the review, an alarm notification is generated to notify the relevant operation and maintenance personnel and team to perform maintenance.

[0097] Example 2

[0098] This embodiment discloses a platform operation and maintenance device based on generative AI technology, the device comprising:

[0099] Data collection module, which collects operation data in real time, including platform data and user model training status;

[0100] Data processing module, which processes and filters the running data and extracts data features;

[0101] Model training module, which collects operation data, historical data and network data, and trains prediction models, AI decision models and AI supervision models;

[0102] The decision-making module uses the AI ​​decision-making model to judge the system status based on data characteristics;

[0103] The supervision module uses the AI ​​supervision model to evaluate and review the judgment results;

[0104] The prediction module uses the prediction model to predict the accuracy of the user model if the judgment result passes the review and the judgment result is that the system has no faults;

[0105] The fault response module triggers the fault operation if the judgment result passes the review and the judgment result is that the system has failed or is about to fail;

[0106] The model optimization module feeds the audit results back to the AI ​​decision-making model to optimize the training of the AI ​​decision-making model.

[0107] In this embodiment, the model training module centrally trains the models used for platform operation and maintenance in this embodiment, including prediction models, AI decision models, and AI supervision models. For servers connected to the platform, targeted training is performed according to the type of server to obtain an adaptive AI model, which can more accurately perform platform operation and maintenance, user model prediction, and fault response, thereby reducing operational errors.

[0108] The specific details of the modules other than the above-mentioned model training module can be understood by referring to the relevant descriptions and effects in Example 1.

[0109] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A platform operation and maintenance method based on generative AI technology, characterized in that: The method comprises: Collect operation data in real time, including platform data and user model training status; The user model training state is used to form a training state data set, and the training state set is used to train the prediction model; Process and filter the running data to extract data features; Use AI decision-making models to judge system status based on data characteristics; Use AI supervision models to evaluate and review judgment results; If the judgment result passes the review and the judgment result is that the system has no fault, the prediction model is used to predict the accuracy of the user model; If the judgment result passes the review and the judgment result is that the system has failed or is about to fail, the fault operation is triggered; Feed the audit results back to the AI ​​decision-making model to optimize and train the AI ​​decision-making model.

2. A platform operation and maintenance method based on generative AI technology according to claim 1, characterized in that: The platform data includes: CPU occupancy, memory usage, disk I / O, network bandwidth, log information, user access volume, application request volume, error code returned by the platform, exception information returned by the platform, system load, disk space usage and network latency.

3. According to claim 1, a platform operation and maintenance method based on generative AI technology is characterized in that: The training process of the prediction model includes: Extract the training status of the model regularly; The model training state at the previous moment is matched with the model training state at the next moment one by one to establish a training state data set; The prediction model is trained by using the training state data set and constructing multiple decision trees through the random forest algorithm.

4. According to claim 1, a platform operation and maintenance method based on generative AI technology is characterized in that: The process of processing and screening the running data includes removing outliers, filling in missing data and standardizing the data format.

5. According to claim 1, a platform operation and maintenance method based on generative AI technology is characterized in that: The training process of the AI ​​decision model includes: According to the server type connected to the system, network search and historical database search are performed to build an adaptive training data set for the server type; Use adaptive training data sets to train AI decision-making models.

6. A platform operation and maintenance method based on generative AI technology according to claim 1, characterized in that: The process of the AI ​​decision model judging the system status includes: Determine whether the system status is faulty based on data characteristics; If it is determined that the system status is faulty, the system fault category is further analyzed.

7. A platform operation and maintenance method based on generative AI technology according to claim 6, characterized in that: The system failure categories include resource scheduling problems, container optimization problems, monitoring and early warning problems, fault tolerance and fault recovery problems, distributed training problems, model parallelization problems, and data parallelism problems.

8. The platform operation and maintenance method based on generative AI technology according to claim 1, characterized in that: The fault operation includes: According to the fault category, adopt historical solutions to solve the fault; Generate an alarm notification to notify relevant operation and maintenance personnel and teams to resolve the fault.

9. The platform operation and maintenance method based on generative AI technology according to claim 1, characterized in that: The AI ​​supervision model evaluates the judgment result by learning from human experience and historical failures. If the judgment result fails the review, an alarm notification is generated to notify the relevant operation and maintenance personnel and teams to conduct maintenance.

10. A platform operation and maintenance device based on generative AI technology, characterized in that: The device comprises: Data collection module, which collects operation data in real time, including platform data and user model training status; Data processing module, which processes and filters the running data and extracts data features; Model training module, which collects operation data, historical data and network data, and trains prediction models, AI decision models and AI supervision models; The decision-making module uses the AI ​​decision-making model to judge the system status based on data characteristics; The supervision module uses the AI ​​supervision model to evaluate and review the judgment results; The prediction module uses the prediction model to predict the accuracy of the user model if the judgment result passes the review and the judgment result is that the system has no faults; The fault response module triggers the fault operation if the judgment result passes the review and the judgment result is that the system has failed or is about to fail; The model optimization module feeds the audit results back to the AI ​​decision-making model to optimize the training of the AI ​​decision-making model.

Images