Intelligent operation and maintenance method and system for log data, electronic equipment and storage medium

Abstract

The invention discloses a log data intelligent operation and maintenance method, an operation and maintenance system, electronic equipment and a storage medium, and belongs to the technical field of electrical digital data processing.The method comprises the steps that log data are predicted through a prediction model based on deep learning and near-end strategy optimization, and a fault classification and repair scheme is obtained; wherein the near-end strategy optimization reward function is a weighted sum of the following indexes: a recovery gradient of a monitoring index in a preset time window for executing the repair scheme in a simulation environment, an execution cost of the repair scheme, and a confidence coefficient of the repair scheme. Text vectors, time differences and position codes are fused in the input features, so that the prediction model can sense the frequency and interval of fault occurrence, near-end strategy optimization can guide the prediction model to generate an effective maintenance scheme, and the effectiveness of the maintenance scheme is improved; and analyzing the log data based on the prediction model, predicting a fault and early warning in time, and generating a corresponding repair scheme.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing technology, specifically to intelligent operation and maintenance methods, operation and maintenance systems, electronic devices, and storage media for log data. Background Technology

[0002] Log data is a crucial component of modern cybersecurity strategies, providing a rich source of information about events occurring within an organization's digital environment. Traditional operational methods typically rely on manual log analysis, using the experience of operations personnel for fault location and resolution. However, with the dramatic increase in log data volume, manual analysis has become inefficient and error-prone.

[0003] Current log data maintenance primarily relies on regular expressions for log filtering and alert pushes. However, most technologies remain at the level of identifying known failure modes, lacking the ability to intelligently learn and predict new types of failures, and are unable to cope with complex and ever-changing system behaviors and failure scenarios. They can only perform simple statistical analysis of log data, failing to delve into the underlying information and thus unable to promptly identify bottlenecks or potential risks in the system. This maintenance approach is no longer sufficient for large-scale, complex IT environments.

[0004] Currently available technologies attempt to use models such as BERT or T5 for text classification of log data to identify faults, but they lack the dynamism and outcome-oriented nature of fault handling. Specifically, during the derivation process, the model cannot determine the effectiveness of the generated maintenance plan; it also lacks time-series encoding.

[0005] Patent CN120872672A discloses a fault root cause localization method based on a large model. It fuses the text vector and the temporal features using a cross-attention mechanism to obtain the initial fused features corresponding to the target log data. The temporal features refer to the temporal frequency of the target log data, but the calculation method for the temporal frequency is not described.

[0006] Therefore, combining the semantic understanding capabilities of large models with time-based status feedback, and developing a results-oriented intelligent operation and maintenance method for log data, has become an important development direction. Summary of the Invention

[0007] To address the aforementioned technical problems in existing technologies, this invention provides an intelligent operation and maintenance method, operation and maintenance system, electronic device, and storage medium for log data, which has a strong results-oriented approach.

[0008] The first aspect of this invention discloses an intelligent operation and maintenance method for log data, comprising the following steps: predicting log data using a prediction model based on deep learning and near-end policy optimization to obtain fault classification and repair scheme; wherein, the reward function of near-end policy optimization is a weighted sum of the following indicators: the recovery gradient of monitoring indicators within a preset time window for executing the repair scheme in a simulation environment, the execution cost of the repair scheme, and the confidence level of the repair scheme.

[0009] Preferably, the reward function R is expressed as: R = a*M recovery – b*C action +c *S confidence ; in, a , b and c These are the weighted data, M recovery To monitor the recovery gradient of key metrics within a preset time window for executing the repair plan in a simulation environment, C action The cost of implementing the repair plan, S confidence The confidence level of the repair plan; The predictive model is guided to generate maintenance plans by maximizing rewards.

[0010] Preferably, the training method for the prediction model includes: Collect historical log data; Historical log data is preprocessed to obtain a training set; The historical log data of the training set is encoded to obtain text vectors; Obtain the time difference and location encoding of historical log data; The text vector, temporal difference, and positional encoding are fused to obtain the input features; Methods based on deep learning and proximal policy optimization train and / or fine-tune large language models using training sets and their input features to obtain predictive models.

[0011] Preferably, the methods for generating input features include: Map the time difference to a high-dimensional space to obtain a time embedding vector; After performing feature concatenation on the text vector and the temporal embedding vector, they are fused with the positional encoding to obtain the input features.

[0012] Preferably, the method for obtaining the position code includes: The source nodes of log data are encoded using graph neural networks to obtain location codes; Input featuresE final Represented as: E final = Concat( E text , E time ) + E position ; In this context, Concat() represents the feature concatenation operation, and + represents the feature fusion operation. E text For text vectors, E position For position encoding, E time This is the temporal embedding vector.

[0013] Preferred input features E final Represented as: E final = E text + E position + E time ; in, E text For text vectors, E position For position encoding, E time This is the temporal embedding vector, and + indicates the feature fusion operation.

[0014] Preferably, the preprocessing includes data cleaning, noise reduction, alignment, standardization, time correction and data alignment, as well as log data parsing and field extraction; This invention also includes an alarm method: Based on the fault classification and repair plan, an alarm message is generated.

[0015] A second aspect of the present invention provides an operation and maintenance system for implementing the above-mentioned intelligent operation and maintenance method for log data, including a prediction module. The prediction module is used to predict log data through a prediction model based on deep learning and near-end strategy optimization to obtain fault classification and repair scheme.

[0016] A third aspect of the present invention provides a storage medium that stores code, which, when processed, executes the above-described intelligent operation and maintenance method for log data.

[0017] A fourth aspect of the present invention provides an electronic device, including a processor and the aforementioned storage medium.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: the input features are integrated with text vectors, time difference and location encoding, which enables the prediction model to perceive the frequency and interval of fault occurrence; the near-end strategy optimization can guide the prediction model to generate effective maintenance plans and improve the effectiveness of maintenance plans; based on the analysis of log data by the prediction model, faults and warnings can be predicted in a timely manner and corresponding repair plans can be generated. Attached Figure Description

[0019] Figure 1 This is a flowchart of the intelligent operation and maintenance method for log data of the present invention; Figure 2 This is the network logic block diagram of the T5 model; Figure 3 This is a logical block diagram of the operation and maintenance system of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] The present invention will now be described in further detail with reference to the accompanying drawings: Example 1 provides an intelligent operation and maintenance method for log data, such as Figure 1 As shown, it includes the following steps: Step S1: Collect historical log data.

[0022] Step S2: Preprocess the historical log data to obtain the training set.

[0023] Step S3: After preprocessing, the historical log data of the training set is standardized and encoded to obtain text vectors.

[0024] Step S4: Obtain the time difference and location encoding of historical log data.

[0025] Step S5: Fuse the text vector (Token embedding), temporal difference, and position encoding (Position Embedding) to obtain the input features.

[0026] Step S6: Using deep learning and proximal policy optimization (PPO) methods, train and / or fine-tune the large language model using the training set and its input features to obtain the prediction model.

[0027] The large language model mentioned above is also called a large model, which can be BERT or T5 (Text-to-Text Transfer Transformer) models, but is not limited to these.

[0028] Step S7: Use a predictive model to predict log data to obtain fault classification and repair solutions.

[0029] Step S8: Generate an alarm message based on the fault classification and repair plan.

[0030] The input features incorporate text vectors, time differences, and location encoding, enabling the prediction model to perceive the frequency and intervals of fault occurrences. Near-end strategy optimization can guide the prediction model to generate effective maintenance plans, thereby improving the effectiveness of maintenance plans. Based on the analysis of log data by the prediction model, faults and early warnings can be predicted in a timely manner, and corresponding repair plans can be generated.

[0031] Policy Gradient Optimization (PPO) is a method based on policy gradients, offering advantages in stability and efficiency, making it suitable for optimizing large-scale, complex tasks. PPO achieves efficient policy optimization by limiting the step size of policy updates, preventing large policy changes from causing training instability. Within the reinforcement learning framework, three core elements are defined: State, Action, and Reward. State represents the current operational state, including features of log data and historical fault information; Action is the prediction or decision made by the model based on the current state, such as labeling fault types or generating repair suggestions; Reward is the feedback on the model's actions, with positive rewards for correct predictions or effective repair suggestions and negative rewards for incorrect predictions.

[0032] The reward function R for near-end policy optimization is expressed as: R = a*M recovery – b*C action +c *S confidence (1); in, a , b and c These are the weighted data, M recovery To monitor the recovery gradient of key metrics within a preset time window for executing the repair plan in a simulation environment, C actionThe cost of implementing the repair plan, S confidence The confidence level of the repair plan.

[0033] The predictive model is guided to generate effective maintenance solutions by maximizing rewards. Given the same performance, solutions with fewer execution steps and lower resource consumption are given priority in rewards.

[0034] Specifically, a simulation environment is established, and the repair plan is executed in the simulation environment. Monitoring indicators are collected within a certain time window, such as CPU, memory, response time, and KPIs (such as latency and throughput) within 5-10 minutes. The recovery gradient within this time window is calculated, and the weighted sum of the recovery gradients of these indicators is used as the basis for the analysis. M recovery The recovery gradient reflects the degree and rate at which these monitored metrics return to the baseline. Execution cost reflects the resources and time required to execute the remediation plan, as well as the cost of potential risks. Operations such as clearing the cache and restarting non-core services are low-cost operations with low risk and low penalty points; restarting the core database, fully rolling back code, and formatting the disk are high-cost operations. These operations can resolve the fault, but the cost is too high, and the penalty points are greater than those for low-cost operations. High-risk remediation methods, such as recursively deleting or forcibly deleting directories and files, and unauthorized restarting of the core library, are assigned penalty points greater than those for high-cost operations. The cost of these operations can be preset using regular expressions, which will not be elaborated upon in this invention. S confidence The system is generated by a predictive model. Specifically, when generating a repair plan, the certainty of the output result is assessed. If the result is confirmed, the output is 100%, avoiding the 50% ambiguity. By introducing objective feedback on the state of the predictive model, the decision-making ability for complex faults is improved.

[0035] In the simulation environment, the system simulates the execution of repair measures, such as "restarting the web service," and provides feedback on subsequent state changes.

[0036] The introduction of reinforcement learning enables predictive models not only to understand log data through supervised learning, but also to optimize their prediction and decision-making strategies through interaction with the environment. This significantly improves the adaptability and accuracy of predictive models in complex operational scenarios, especially when facing unknown fault modes. Through continuous learning and strategy adjustment, the model can achieve more efficient fault identification and prediction.

[0037] Example 2 provides the specific training steps for the prediction model: In step S1, the log data includes operating system logs, application logs, database logs, and network device logs, covering various fault types such as system crashes, performance bottlenecks, and network latency. Log data collection is achieved through a distributed architecture, deploying lightweight log collection tools such as Filebeat and Logstash to acquire log data from various environments, including different service nodes, containers, and virtual machines. Filebeat, as a lightweight log collection tool, can run on nodes and read local log files in real time, acquiring log data by configuring different input sources (such as files, Windows event logs, etc.). Filebeat's advantages lie in its efficiency and low resource consumption, enabling stable operation in large-scale, high-frequency environments, transmitting log data to the backend log processing platform. On the other hand, Logstash, as a more powerful log processing tool, is responsible for further processing and forwarding the logs collected from tools like Filebeat. It receives data through different input plugins, uses built-in filtering plugins to clean, parse, and convert the logs, and finally outputs the processed data to the target storage system.

[0038] To ensure log data is not lost under high concurrency, Kafka is used as a message queue middleware to ensure efficient data transmission and processing between different systems. Kafka features high throughput and low latency, enabling it to handle massive log data streams. Kafka stores log data as messages and organizes and manages them through different topics, ensuring data reliability and scalability during transmission. Kafka's partitioning mechanism ensures that multiple consumers can process data in parallel, improving processing capacity and fault tolerance.

[0039] In step S2, preprocessing includes data cleaning / filtering, noise reduction, standardization, time correction and data alignment, as well as log data parsing and field extraction.

[0040] First, the raw logs are denoised and filtered to remove irrelevant information, such as debugging information, duplicate log entries, and low-level log messages. Regular expressions (Regex) can be used for fast filtering of logs, and low-priority log messages that do not affect fault diagnosis can be selectively filtered out based on business needs and system specifications. This effectively reduces interference from useless information, ensuring that subsequent analysis focuses on critical error and warning messages.

[0041] Secondly, the log data needs to be standardized. Because logs come from diverse sources and have significantly different formats, they must be structurally transformed into a unified standard format. Typically, log data is converted to JSON format, which facilitates subsequent storage and analysis.

[0042] For example, consider a raw Nginx server log entry: 192.168.1.5--[19 / Jan / 2025:10:30:00+0800] "GET / index.html HTTP / 1.1" 200 1024. It automatically matches pre-defined parsing rules (such as %{IP:client_ip} ... %{HTTPDATE:timestamp} ...), resulting in a standard JSON key-value pair structure. { "client_ip": "192.168.1.5", "timestamp_utc": "2025-01-19T02:30:00Z", "method": "GET", "url": " / index.html", "status_code": 200, "bytes_sent": 1024, "log_source": "nginx_access_log"}.

[0043] The standardized logs include fields such as timestamp, log level, source module, and error message, and are stored in a unified format for easy subsequent querying and aggregation.

[0044] Inconsistent timestamps are frequently found in log data, especially in distributed systems. Due to potential clock discrepancies across nodes, log timestamps can be inaccurate or inconsistent. Therefore, timestamp correction is a crucial step in log preprocessing. Using the Network Time Protocol (NTP) to synchronize the clocks of each node ensures that the collected log timestamps are as accurate as possible. Furthermore, correcting the timestamps of already collected logs ensures that all logs are correctly aligned in time, preventing time discrepancies from affecting subsequent analysis.

[0045] Logs often contain error codes, stack traces, and database query statements. Extracting key information from these is a core aspect of log processing. Tools such as regular expressions and Logstash plugins can be used to extract crucial information from logs, including error codes, exception types, and stack traces. For example, stack traces may contain function calls and the file location where the error occurred. Regular expressions can be used to extract this information and transform it into standardized fields for subsequent analysis.

[0046] After all preprocessing steps are completed, the processed log data is stored in a backend storage platform. Elasticsearch can be used as the storage and query platform for log data. Elasticsearch has powerful search and aggregation capabilities, supporting the storage and efficient retrieval of massive amounts of log data. Through Elasticsearch, functions such as fast log querying, aggregation statistics, and trend analysis can be achieved, providing data support for subsequent intelligent diagnostics and predictions.

[0047] In summary, the core technologies in the entire log data collection and preprocessing process include distributed log collection tools (such as Filebeat and Logstash), message queues (Kafka), log storage (Elasticsearch), timestamp synchronization (NTP protocol), and log field parsing and standardization (regular expressions and Logstash plugins). These technologies ensure efficient collection, processing, and storage of log data in complex, distributed environments, providing high-quality input data.

[0048] Multimodal input encoding technology is used in steps S4 and S5. The time difference is the time difference between the current log and the previous related log, and the time difference is mapped to a time embedding vector. E time Specifically, this refers to the time difference between two adjacent logs of the same category.

[0049] Input features E final Represented as: E final = E text + E position + E time (2); in, E text For text vectors (Token Embedding), E position Position embedding E time This is the time embedding vector, and the + sign indicates a feature fusion operation. By fusing time differences, the difference between bursts of error logs in a short period and periodic regular logs can be captured. When facing unknown faults, accurate predictions can be made based on temporal patterns rather than just keywords.

[0050] In a specific embodiment, the input features are also represented as: E final = Concat( E text , E time ) + E position (3); Here, Concat() represents the feature concatenation operation, and + represents the feature fusion operation.

[0051] Figure 2 The network logic of the T5 model is shown. The time difference is projected onto a high-dimensional space through a fully connected layer to obtain a temporal embedding vector. This is based on graph neural network node encoding (… GNN Node Embedding The method for generating position codes E position It can reflect the spatial location of each node in a distributed system. Structural modifications to the input features / input layer transform the prediction model from merely a natural language processing model into a large-scale, spatiotemporally aware operational model.

[0052] In step S6, the T5 model is used to process complex log data, and its performance is further optimized through reinforcement learning to handle complex failure modes and scenarios in modern IT systems. The T5 model is highly flexible and versatile, capable of handling various tasks including text generation, translation, summarization, and classification; it possesses powerful semantic understanding and generation capabilities, effectively capturing complex patterns and contextual relationships in log data. The T5 model's architecture is based on the standard Encoder-Decoder framework. The encoder is responsible for converting the input log text into a high-dimensional contextual representation, while the decoder generates corresponding output text (such as failure type labels or repair suggestions) based on these representations. This bidirectional encoder and decoder design enables T5 to excel in processing long texts and capturing fine-grained information. Furthermore, the T5 model effectively captures long-distance dependencies and complex semantic structures in log data through multi-layer self-attention mechanisms and feed-forward neural networks.

[0053] Log data can be precisely labeled using a combination of automated tools and manual review. Each log entry is labeled with a specific fault type, accompanied by a detailed fault description and possible remedial measures. For example, a database connection failure log might be labeled "database connection timed out" and associated with corresponding remedial suggestions, such as "check the database server's load status" or "optimize the database connection pool configuration." Data augmentation techniques can also be employed to generate diverse training samples. By transforming, translating, and expanding existing log data, more training instances are generated, covering a wider range of fault modes and abnormal scenarios. This not only improves the model's adaptability to different environments but also enhances its ability to identify and predict novel and unknown faults.

[0054] The pre-training and fine-tuning phases are the core of a natural language processing model based on the Transformer architecture. Pre-training employs self-supervised learning methods, using a large amount of text data for unsupervised training. The core training task is text-to-text generation, which converts the input text into the corresponding target text. Pre-training uses a concept similar to Masked Language Modeling (MLM), but instead of directly masking individual words in the input, it masks entire phrases or paragraphs of text. This approach allows the model to better understand context and more complex linguistic relationships. For example, given the input text "Classify: This is an example of a positive review", T5 sets the target text to "Positive" and requires the model to predict the target classification based on contextual information. T5 uses specific task settings during pre-training, mainly including text infilling, translation, and summarization. These tasks are uniformly transformed into text generation tasks, ensuring that T5 can improve its generalization ability through joint training of multiple tasks. For example, in a summary generation task, if the input is "Summarize: This is a detailed report about climate change," the model's output should be a short summary: "Climate change report."

[0055] In the fine-tuning phase of T5, the model has already learned the basic structure and knowledge of language on large-scale text data through pre-training. The next step is to further optimize it using task-specific datasets to improve its performance in real-world applications. The goal of fine-tuning is to improve T5's performance in specific downstream tasks (such as text classification, named entity recognition, machine translation, etc.). Each task in the fine-tuning process is transformed into a text generation problem. For example, the input to a text classification task might be "Classify: This is a positive review," and the output might be "Positive." The input to a machine translation task might be English text, and the output might be the corresponding translated text. During the fine-tuning phase, T5 not only performs the training tasks but also adjusts its output structure and content according to the characteristics of the downstream tasks. In fine-tuning, T5 continues to utilize the cross-entropy loss function to optimize the model. The loss function calculates the error between the model's output and the true target and updates the parameters based on backpropagation of the error. The formula is as follows: (4); in, y t Indicates the first t Target words, The representation model depends on the context x 1:t-1 Predicted vocabulary y t The probability, θ1 For the parameters of the model, T1 This represents the total number of words.

[0056] Similar to the pre-training phase, the fine-tuning phase also uses the Adam optimizer (Adaptive Moment Estimation) to update model parameters. The Adam optimizer combines momentum and adaptive learning rate adjustment, effectively accelerating model training and improving training stability. Its update rules are as follows: (5); in, m t2 and v t2 These are estimates of the gradient momentum and the squared gradient, respectively. and It is the decay rate of momentum and squared gradient. g t2 Indicates time step t2 gradient, It's the learning rate. It is a small constant used to avoid errors caused by dividing by zero. and These are the deviation correction values ​​for momentum and squared gradient, respectively. To update the step size.

[0057] In step S7, potential fault types in the logs are labeled using prediction. For example, the prediction model might label logs containing "database connection failure" as "database fault" and "memory overflow" as "memory problem." This process is accomplished through sequence-to-sequence mapping; the T5 model automatically generates labels based on the input log content. The log labeling process is based on Maximum Likelihood Estimation (MLE). When generating log labels, the T5 model determines the most likely label for each log entry based on knowledge learned during training. This can be formally expressed as: (6); in, Given input log x At that time, a label sequence is generated. y The conditional probability, For model parameters, T2 The length of the label sequence. The labeled log data will be further used for fault prediction, helping operations personnel understand the real-time operating status of IT systems and identify potential faults in advance. Log labeling is not limited to known fault types; the T5 model can also identify new and unknown fault types based on learned patterns. For example, by analyzing the log data of a specific service, the predictive model may predict that an unknown fault will occur in that service, which greatly assists operations personnel.

[0058] Furthermore, the operation and maintenance method of this invention supports personalized optimization and customization, providing tailored log analysis and fault detection solutions for different enterprises. As the complexity of enterprise IT systems continues to increase, standard fault detection models often cannot fully meet the specific business needs and technical architecture of each enterprise.

[0059] Specifically, by combining transfer learning techniques with customized design, models can be optimized and adjusted to maximize their performance in enterprise environments. Transfer learning is the core of the personalized optimization and customization module. It allows enterprises to quickly adapt to the specific needs of different businesses by fine-tuning and customizing existing pre-trained models. Specifically, enterprises can provide specific log data, including common failure modes and error messages. This labeled data is then used to fine-tune the existing model using transfer learning. In this way, the model can better learn the business characteristics and failure modes of the enterprise without having to train from scratch, thus greatly improving the efficiency of the customization process. The fine-tuned model will be able to identify the enterprise's unique failure logs and make accurate failure predictions and early warnings.

[0060] During transfer learning, the model is fine-tuned by minimizing the loss function of domain-specific data. The goal of this process is to optimize model parameters to effectively handle new log data and improve the accuracy of fault prediction. For example, log data provided by a company may contain many error messages or fault modes that differ from those of other companies. Transfer learning helps the model adapt to the company's unique fault data while maintaining its strong natural language processing capabilities. The personalized optimization and customization module not only relies on transfer learning but also incorporates the concept of multi-task learning, further enhancing adaptability. Through multi-task learning, not only fault detection can be performed, but other related tasks, such as generating fault reports and diagnosing fault causes, can be learned during the same training process. These tasks share some parameters and can mutually promote each other, improving the overall performance of the model. For example, the collaborative learning of fault report generation and fault detection tasks can automatically generate more accurate fault diagnosis reports when a fault is detected, helping operations personnel quickly locate problems.

[0061] Personalized customization also provides flexible log data processing capabilities, automatically adjusting the extraction and selection of log features according to the specific business needs of an enterprise. Due to significant differences in system architecture and business processes among different enterprises, the format, structure, and content of log data vary considerably. The personalized optimization and customization module can automatically identify key log features based on the log data provided by the enterprise and perform fault prediction and early warning based on these features. For example, in the financial industry, there may be a greater focus on logs related to transaction latency and database performance, while in the power industry, the focus may be on information related to power equipment failures. To cope with the ever-changing technological environment and business needs of enterprises, personalized optimization also supports incremental learning. As log data is updated and new failure modes emerge, the predictive model needs continuous optimization to maintain its efficiency.

[0062] Incremental learning enables predictive models to gradually update their parameters without complete retraining. This learning method ensures the real-time performance and adaptability of predictive models in real-world applications, especially in responding quickly to new failure modes and business needs when enterprise IT systems change. Furthermore, personalized optimization and customization introduce a feedback mechanism, continuously collecting new failure data and feedback from operations personnel through interaction with the actual enterprise operating environment. This feedback not only helps identify potential problems in the predictive model's output but also guides subsequent adjustments and optimizations. When the predictive model makes incorrect predictions or issues warnings, operations personnel can directly provide feedback, allowing for model correction. This feedback mechanism ensures continuous optimization and self-improvement of the predictive model over long-term use.

[0063] Personalized optimization and customization not only help enterprises customize their fault prediction and log analysis models through transfer learning in the early stages, but also maintain the efficiency and adaptability of the predictive models through continuous self-optimization and incremental learning. Enterprises can continuously improve the accuracy and efficiency of fault detection in a constantly changing technological environment, thereby ensuring the stable operation and efficient maintenance of their IT systems.

[0064] Example 3 provides an operation and maintenance system for implementing the above-described intelligent operation and maintenance method for log data, such as... Figure 3 As shown, it includes a data acquisition module 1, a preprocessing module 2, an encoding module 3, a training module 4, a prediction module 5, and an alarm module 6.

[0065] The data acquisition module 1 is used to collect historical log data.

[0066] Preprocessing module 2 is used to preprocess historical log data to obtain a training set.

[0067] Encoding module 3 is used to encode the historical log data of the training set to obtain text vectors, which are used to obtain the time difference and location encoding of the historical log data; and to fuse the text vectors, time difference and location encoding to obtain input features.

[0068] Training module 4 is used for deep learning and proximal policy optimization (PPO) methods to train and / or fine-tune a large language model using the training set and its input features to obtain a prediction model.

[0069] Prediction module 5 is used to predict log data through a prediction model to obtain fault classification and repair solutions.

[0070] The alarm module 6 is used to generate alarm messages based on the fault classification and repair plan.

[0071] In addition, the operation and maintenance system may also include a modification and customization module 7, which can be used to adjust model parameters and log analysis rules according to the needs of different enterprises.

[0072] Example 4 provides a storage medium and an electronic device. The storage medium stores code, which, when processed, implements the aforementioned intelligent log data operation and maintenance method. The electronic device includes the aforementioned storage medium and a processor, the processor being used to process the code.

[0073] This invention aims to address the inefficiencies, missed diagnoses, and inaccuracies inherent in traditional operation and maintenance (O&M) methods. By utilizing a T5 large-scale model combining reinforcement learning and deep learning technologies, it automatically captures and analyzes system logs, identifying, predicting, and issuing early warnings of potential faults in real time. This improves O&M efficiency, reduces fault response time, optimizes resource utilization, and ensures system stability and reliability. Through preprocessing, deep analysis, fault prediction, and intelligent optimization, it provides customized services to meet the needs of different enterprises, making it particularly suitable for complex IT environments such as cloud computing, big data platforms, microservice architectures, and distributed systems. It not only identifies known faults but also possesses intelligent predictive capabilities for unknown faults, making it a crucial technological solution for enhancing the automated O&M management of modern enterprise IT systems.

[0074] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An intelligent operation and maintenance method for log data, characterized in that, Includes the following steps: By using a prediction model based on deep learning and near-end strategy optimization, log data is predicted to obtain fault classification and repair solutions. The reward function for near-end strategy optimization is a weighted sum of the following metrics: The recovery gradient of monitoring indicators, the execution cost of the repair plan, and the confidence level of the repair plan are measured within a preset time window during the execution of the repair plan in the simulation environment.

2. The intelligent operation and maintenance method for log data according to claim 1, characterized in that, The reward function R is expressed as: R = a*M recovery – b*C action +c *S confidence ； in, a , b and c These are the weighted data, M recovery To monitor the recovery gradient of key metrics within a preset time window for executing the repair plan in a simulation environment, C action The cost of implementing the repair plan, S confidence The confidence level of the repair plan; The predictive model is guided to generate maintenance plans by maximizing rewards.

3. The intelligent operation and maintenance method for log data according to claim 1, characterized in that, The training methods for the prediction model include: Collect historical log data; Historical log data is preprocessed to obtain a training set; The historical log data of the training set is encoded to obtain text vectors; Obtain the time difference and location encoding of historical log data; The text vector, temporal difference, and positional encoding are fused to obtain the input features; Methods based on deep learning and proximal policy optimization train and / or fine-tune large language models using training sets and their input features to obtain predictive models.

4. The intelligent operation and maintenance method for log data according to claim 3, characterized in that, Methods for generating input features include: Map the time difference to a high-dimensional space to obtain a time embedding vector; After performing feature concatenation on the text vector and the temporal embedding vector, they are fused with the positional encoding to obtain the input features.

5. The intelligent operation and maintenance method for log data according to claim 4, characterized in that, Methods for obtaining position encoding include: The source nodes of log data are encoded using graph neural networks to obtain location codes; Input features E final Represented as: E final = Concat( E text , E time ) + E position ; In this context, Concat() represents the feature concatenation operation, and + represents the feature fusion operation. E text For text vectors, E position For position encoding, E time This is the temporal embedding vector.

6. The intelligent operation and maintenance method for log data according to claim 3, characterized in that, Input features E final Represented as: E final = E text + E position + E time ； in, E text For text vectors, E position For position encoding, E time This is the temporal embedding vector, and + indicates the feature fusion operation.

7. The intelligent operation and maintenance method for log data according to claim 3, characterized in that, The preprocessing includes data cleaning, noise reduction, alignment, standardization, time correction and data alignment, as well as log data parsing and field extraction; It also includes alarm methods: Based on the fault classification and repair plan, an alarm message is generated.

8. An operation and maintenance system, characterized in that, The method for intelligent operation and maintenance of log data as described in any one of claims 1-7 includes a prediction module. The prediction module is used to predict log data using a prediction model based on deep learning and near-end strategy optimization to obtain fault classification and repair solutions.

9. A storage medium, characterized in that, The storage medium contains code, and when the code is processed, the intelligent operation and maintenance method for log data as described in any one of claims 1-7 is executed.

10. An electronic device, characterized in that, Includes a processor and a storage medium as described in claim 9.

Images