Cluster evaluation method and device, electronic equipment and storage medium
By extracting and fine-tuning the log data of Kubernetes clusters, a risk assessment model is generated, which solves the shortcomings of the traditional log analysis method in real time and accuracy, and realizes accurate identification and automated response to potential threats and system problems. Improves the security and stability of the cluster.
Patent Information
- Application Number
- CN202411079413.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-11-12
AI Technical Summary
Traditional log analysis methods are difficult to meet the real-time and accuracy requirements of Kubernetes cluster log evaluation. Especially when facing complex network attack modes and system fault diagnosis, there is a lack of adaptive learning mechanism and it is impossible to automatically adjust the analysis strategy according to the new security situation. .
By obtaining the historical log data and real-time log data of the target cluster, performing feature extraction and preprocessing, using the initial model for pre-training and fine-tuning, and generating a target risk assessment model for risk assessment of the cluster log data and automatically adjusting security policies.
It realizes accurate identification of potential security threats and system problems, improves the automation level and accuracy of cluster log evaluation, reduces operation and maintenance difficulties and costs, and improves the security and stability of the system.
Smart Images

Figure CN118939380A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the technical field of cluster evaluation, and in particular to a cluster evaluation method, device, electronic device, and storage medium. Background Art
[0002] As a mainstream container orchestration platform, the amount of log data of Kubernetes has increased dramatically with the growth of business. In the field of Kubernetes cluster log evaluation, many existing log analysis systems rely on static rule engines to match specific log patterns, and try to identify potential security threats by observing abnormal patterns in logs. However, traditional log analysis methods have been unable to meet the requirements of real-time and accuracy, especially when facing complex network attack patterns and system fault diagnosis. They lack adaptive learning mechanisms and cannot automatically adjust their analysis strategies according to new security situations. Summary of the invention
[0003] In order to solve the above technical problems or at least partially solve the above technical problems, the embodiments of the present application provide a cluster evaluation method, device, electronic device and storage medium to solve the problem that traditional log analysis methods can no longer meet the requirements of real-time and accuracy.
[0004] In order to achieve the above purpose, the technical solutions provided by the embodiments of the present application are as follows:
[0005] In a first aspect, an embodiment of the present application provides a cluster evaluation method, the cluster evaluation method comprising: obtaining historical log data and real-time log data corresponding to a target cluster, the log data comprising at least: a container log, an API server log, and a controller management log;
[0006] Performing feature extraction on the real-time log data to obtain a target feature data set;
[0007] Pre-training the initial model through the historical log data and historical security events, and fine-tuning the pre-trained initial model through the target feature data set and target security events to obtain a target risk assessment model;
[0008] The target risk assessment model is used to perform risk assessment on the log data of the target cluster.
[0009] As an optional implementation manner, in the first aspect of the embodiment of the present application, the extracting features of the real-time log data to obtain a target feature data set includes:
[0010] Preprocessing the real-time log data to obtain preprocessed log data, wherein the preprocessing includes at least one of the following: data cleaning, data deduplication, and data formatting;
[0011] Feature extraction is performed on the preprocessed log data to obtain the target feature data set.
[0012] As an optional implementation, in the first aspect of the embodiment of the present application, the pre-training of the initial model by using the historical log data and historical security events includes:
[0013] Performing data analysis on the historical log data to obtain historical common patterns and historical abnormal features;
[0014] The initial model is pre-trained using the historical common patterns, the historical abnormal features, and the historical security events.
[0015] As an optional implementation, in the first aspect of the embodiment of the present application, the pre-trained initial model is fine-tuned by the target feature data set and the target security event to obtain a target risk assessment model, including:
[0016] Inputting the target feature data set into the pre-trained initial model to obtain an initial evaluation result;
[0017] According to the initial assessment result and the target security event, the pre-trained initial model is fine-tuned to obtain the target risk assessment model.
[0018] As an optional implementation, in the first aspect of the embodiment of the present application, the risk assessment of the log data of the target cluster by using the target risk assessment model includes:
[0019] Obtaining the log data to be tested of the target cluster in real time;
[0020] The log data to be tested is input into the target risk assessment model to obtain a target risk assessment result, wherein the target risk assessment result includes at least one item: risk level, potential threat, and system failure.
[0021] As an optional implementation, in the first aspect of the embodiment of the present application, after inputting the log data to be tested into the target risk assessment model to obtain the target risk assessment result, the method further includes:
[0022] According to the pre-stored correspondence between the risk assessment results and the security policies, a target security policy corresponding to the target risk assessment result is determined.
[0023] As an optional implementation manner, in the first aspect of the embodiments of the present application, the method further includes:
[0024] The target risk assessment model is updated and optimized through the log data and feedback information obtained in real time.
[0025] In a second aspect, an embodiment of the present application provides a cluster evaluation device, the cluster evaluation device comprising: an acquisition module, used to acquire historical log data and real-time log data corresponding to a target cluster, the log data comprising at least: a container log, an API server log, and a controller management log;
[0026] A processing module, used for extracting features from the real-time log data to obtain a target feature data set;
[0027] The processing module is further used to pre-train the initial model through the historical log data and historical security events, and fine-tune the pre-trained initial model through the target feature data set and target security events to obtain a target risk assessment model;
[0028] The processing module is further configured to perform risk assessment on the log data of the target cluster using the target risk assessment model.
[0029] As an optional implementation, in the second aspect of the embodiment of the present application, the processing module is specifically used to preprocess the real-time log data to obtain preprocessed log data, and the preprocessing includes at least one of the following: data cleaning, data deduplication, and data formatting;
[0030] The processing module is specifically used to extract features from the preprocessed log data to obtain the target feature data set.
[0031] As an optional implementation, in the second aspect of the embodiment of the present application, the processing module is specifically used to perform data analysis on the historical log data to obtain historical common patterns and historical abnormal features;
[0032] The processing module is specifically used to pre-train the initial model through the historical common patterns, the historical abnormal features and the historical security events.
[0033] As an optional implementation, in the second aspect of the embodiment of the present application, the processing module is specifically used to input the target feature data set into the pre-trained initial model to obtain an initial evaluation result;
[0034] The processing module is specifically used to fine-tune the pre-trained initial model according to the initial assessment result and the target security event to obtain the target risk assessment model.
[0035] As an optional implementation, in the second aspect of the embodiment of the present application, the acquisition module is specifically used to acquire the log data to be tested of the target cluster in real time;
[0036] The processing module is specifically used to input the log data to be tested into the target risk assessment model to obtain a target risk assessment result, and the target risk assessment result includes at least one item: risk level, potential threat, and system failure.
[0037] As an optional implementation, in the second aspect of the embodiment of the present application, the processing module is also used to determine the target security policy corresponding to the target risk assessment result based on the correspondence between the pre-stored risk assessment results and the security policy.
[0038] As an optional implementation, in the second aspect of the embodiment of the present application, the processing module is also used to update and optimize the target risk assessment model through log data and feedback information obtained in real time.
[0039] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising:
[0040] A memory storing executable program code;
[0041] a processor coupled to the memory;
[0042] The processor calls the executable program code stored in the memory to execute the cluster evaluation method in the first aspect of the embodiment of the present application.
[0043] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the cluster evaluation method in the first aspect of the embodiment of the present application. The computer-readable storage medium includes a ROM / RAM, a magnetic disk or an optical disk, etc.
[0044] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute part or all of the steps of any one of the methods of the first aspect.
[0045] In a sixth aspect, an embodiment of the present application provides an application publishing platform, which is used to publish a computer program product, wherein when the computer program product runs on a computer, the computer executes part or all of the steps of any one method of the first aspect.
[0046] Compared with the prior art, the embodiments of the present application have the following beneficial effects:
[0047] The embodiment of the present application provides a cluster assessment method, device, electronic device and storage medium, which obtains historical log data and real-time log data corresponding to the target cluster, and the log data includes at least: container log, API server log, controller management log; extracts features from the real-time log data to obtain a target feature data set; pre-trains the initial model through historical log data and historical security events, and fine-tunes the pre-trained initial model through the target feature data set and target security events to obtain a target risk assessment model; and performs risk assessment on the log data of the target cluster through the target risk assessment model. In this solution, through automated log data collection, as well as pre-training and fine-tuning of the model, it is possible to accurately identify potential security threats and system problems, improve the automation level and accuracy of cluster log assessment, reduce the difficulty and cost of system operation and maintenance, and improve the security and stability of the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0050] Figure 1 This is a schematic diagram of a cluster evaluation method provided in an embodiment of the present application. Figure 1 ;
[0051] Figure 2 This is a schematic diagram of a cluster evaluation method provided in an embodiment of the present application. Figure 2 ;
[0052] Figure 3 is a structural schematic diagram of a cluster evaluation device provided in an embodiment of the present application;
[0053] Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0054] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0055] The terms “first”, “second” and the like in the description and claims of the present application are used to distinguish different objects rather than to describe a specific order of the objects.
[0056] The terms "including" and "having" and any variations thereof in the embodiments of the present application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.
[0057] It should be noted that, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.
[0058] In the field of Kubernetes cluster log assessment, many existing log analysis systems rely on static rule engines to match specific log patterns. These rules are usually predefined by security experts based on known attack patterns. Some systems use heuristic methods to detect abnormal behavior, which attempts to identify potential security threats by observing abnormal patterns in logs. Some advanced systems have begun to use machine learning algorithms to analyze log data to automatically identify and classify security events. Automation tools are used to simplify the collection, aggregation, and preliminary analysis of log data and improve efficiency. Some cloud service providers and security vendors provide integrated log analysis solutions, which are usually tightly integrated with container orchestration platforms such as Kubernetes.
[0059] It should be noted that Kubernetes (K8s for short) cluster is an open source container orchestration platform, which is mainly used for automated deployment, expansion and management of containerized applications. A Kubernetes cluster is a collection of physical or virtual machines that are organized into a single computing resource pool and run the Kubernetes platform on it. It can usually include a master node (Master Node) and multiple worker nodes (Worker Node).
[0060] The master node is responsible for cluster management and control, including components such as API server (Kube-API Server), controller manager (Kubernetes Controller Manager), scheduler (Scheduler) and etcd (cluster state storage). Multiple worker nodes are responsible for running containerized applications, including components such as kubelet (node agent), kube-proxy (network agent) and container runtime (such as Docker).
[0061] Kubernetes clusters can hide resource management and error handling: users only need to focus on application development; services have high availability and reliability: by deploying applications on multiple nodes, fault tolerance is provided; they are also scalable: the cluster size can be easily expanded or reduced to meet different load requirements; and they are cross-platform: they can run on different cloud providers or local environments. Kubernetes clusters provide a powerful platform for automating the deployment, expansion, and management of containerized applications. By properly configuring and managing Kubernetes clusters, the availability, reliability, and scalability of applications can be significantly improved.
[0062] However, static rule engines are difficult to adapt to rapidly changing attack patterns, require frequent updates, and are prone to missing unknown threats. Existing heuristic analysis methods often lack sufficient generalization capabilities and may not be sensitive enough to certain unknown or variant attack patterns. The amount of log data generated by Kubernetes clusters is huge and in various formats. Automated tools may not be able to effectively handle all types of log data in terms of data preprocessing. Although some systems are able to achieve automated analysis, there are still challenges in real time, especially in high-load or large-scale cluster environments. The accuracy of machine learning models in log analysis is limited by the quality of their training data and the generalization ability of the model. Complex machine learning models usually require a lot of computing resources, which may limit their application in resource-constrained environments. As attack methods continue to evolve, existing log analysis systems need to be constantly updated and maintained to maintain their effectiveness. Many existing systems lack adaptive learning mechanisms and cannot automatically adjust their analysis strategies according to new security situations. The user interfaces of some automated tools may not be intuitive enough, and it may be difficult for non-technical users to understand and respond to system alerts.
[0063] In order to solve some or all of the above technical problems, the embodiments of the present application provide a cluster evaluation method, device, electronic device and storage medium, obtain historical log data and real-time log data corresponding to the target cluster, and the log data at least includes: container log, API server log, controller management log; extract features from real-time log data to obtain a target feature data set; pre-train the initial model through historical log data and historical security events, and fine-tune the pre-trained initial model through the target feature data set and target security events to obtain a target risk assessment model; perform risk assessment on the log data of the target cluster through the target risk assessment model. In this solution, through automated log data collection, as well as pre-training and fine-tuning of the model, it is possible to accurately identify potential security threats and system problems, improve the automation level and accuracy of cluster log evaluation, reduce the difficulty and cost of system operation and maintenance, and improve the security and stability of the system.
[0064] like Figure 1 As shown, Figure 1 A flow chart of a cluster evaluation method provided in an embodiment of the present application, the method may include the following steps:
[0065] 101. Obtain historical log data and real-time log data corresponding to the target cluster.
[0066] In the embodiment of the present application, the log data at least includes: container logs, API server logs, and controller management logs. The target cluster may be a kubernetes cluster.
[0067] It should be noted that in a Kubernetes environment, managing and understanding the logs of various components is essential for debugging, monitoring, and ensuring stable operation of the system. In Kubernetes, each container in a Pod generates logs. Container logs are the output of the container runtime (such as Docker or containerd), which are mainly used to monitor the running status of the application and debug problems; the API server is one of the core components of the Kubernetes cluster, responsible for processing all requests from clients, and the API server logs are mainly used to diagnose problems encountered when interacting with the cluster (such as authentication problems, permission problems, or API performance problems); Kubernetes uses a variety of controllers to manage the state of the cluster, including Deployment, StatefulSet, DaemonSet, etc. The controller management logs are mainly used to monitor how the cluster adjusts according to the defined state (such as expansion, reduction, update, etc.).
[0068] In the embodiment of the present application, the historical log data may be log data corresponding to multiple historical moments, and the time range of the multiple historical moments may be pre-set, or may be all moments that can be obtained, for example: one month may be pre-set, that is, all log data within the past month is obtained; or, all log data of the target cluster in the past is obtained. The number of the historical log data is not limited.
[0069] In some embodiments, the real-time log data may be the log data corresponding to the target cluster at the current moment. Specifically, the kubectl logs command may be used to view the container logs in the Pod. In addition, if there are multiple containers in the Pod, the container name may be specified or the -c flag may be used to view the logs of a specific container. The API server is typically deployed on the master node as a static Pod, so the API server node may be accessed to obtain the API server logs. In large clusters, log and monitoring solutions such as Elasticsearch, Fluentd, Kibana (EFK) or Prometheus, Grafana, Loki (PGL) may be used to centrally manage and query logs. Controllers and managers are typically also deployed on the master node as static Pods, so the corresponding nodes may be accessed and / var / log / kube-controller-manager.log (or other similar locations) may be used to obtain the controller management logs.
[0070] 102. Perform feature extraction on the real-time log data to obtain a target feature data set.
[0071] In the embodiment of the present application, feature extraction may be performed on the real-time log data. Specifically, feature selection and feature extraction may be performed to construct a target feature data set containing key information.
[0072] In some embodiments, feature selection and feature extraction may also be performed on the historical log data to obtain a feature data set corresponding to the historical log data, and the feature data set corresponding to the historical log data is different from the target feature data set.
[0073] 103. The initial model is pre-trained through historical log data and historical security events, and the pre-trained initial model is fine-tuned through the target feature data set and target security events to obtain the target risk assessment model.
[0074] In an embodiment of the present application, the training of the model can be divided into two parts: model pre-training through historical log data, and model fine-tuning through real-time log data, and finally obtaining a target risk assessment model.
[0075] It should be noted that the initial model can be the ChatGLM-6B model, which is a large language model for Chinese-English bilingual dialogue. It is based on the General Language Model (GLM) architecture and has significant characteristics and advantages. The ChatGLM-6B model supports both Chinese and English. After about 1T of identifiers, it has sufficient pre-training capabilities for both Chinese and English. The model has 6.2 billion parameters. Although the scale is not as large as some larger models, its efficient reasoning ability and low deployment threshold make it perform well in various application scenarios.
[0076] The ChatGLM-6B model adopts a generative pre-training method and is pre-trained on a large-scale corpus, which enables the model to learn rich language knowledge and grammatical rules, thereby generating reasonable answers or dialogues; the model has a certain context-awareness ability and can understand questions and generate corresponding answers based on the context, which enables it to perform well in complex dialogue scenarios; ChatGLM-6B has been trained on a large-scale multi-domain corpus and has good generalization capabilities, and can adapt to application scenarios in different fields, such as question-and-answer systems, intelligent customer service, robot dialogues, etc.
[0077] The ChatGLM-6B model supports batch processing and fast response, and can reduce computing resources and response time while ensuring high-quality output, which makes it have good application potential in real-time interactive scenarios; combined with model quantization technology, the ChatGLM-6B model can be deployed locally on consumer-grade graphics cards, with a minimum of 6GB of video memory (INT4 quantization level), which greatly reduces the deployment cost and makes it easier for researchers and individual developers to fine-tune and deploy the model; in addition, the ChatGLM-6B model has a wide range of applications in multiple fields, including question-answering systems, robot conversations, text generation, etc. By inputting relevant context or questions, the model can generate text that conforms to language specifications and logical coherence, providing users with accurate and coherent answers or interactive experiences.
[0078] In an embodiment of the present application, the initial model is pre-trained using historical log data and historical security events so that the initial model learns common patterns and abnormal features in a large amount of historical log data; the pre-trained initial model is then fine-tuned using real-time log data and target security events so that the final target risk assessment model can adapt to the latest security threats and system behavior changes.
[0079] In some embodiments, security incidents of Kubernetes (K8s) clusters have gradually increased in recent years. Attackers have launched various types of attacks against Kubernetes clusters, including but not limited to unauthenticated access, mining, denial of service (DoS) attacks, data leakage, etc., mainly to obtain sensitive information, steal credentials, spread malware, etc., thereby causing harm to the entire cluster.
[0080] 104. Perform risk assessment on the log data of the target cluster through the target risk assessment model.
[0081] In an embodiment of the present application, after the model training is completed to obtain the target risk assessment model, the target risk assessment model can be deployed in the target cluster, new log data corresponding to the target cluster can be received in real time, and risk assessment can be performed on the log data.
[0082] The embodiment of the present application provides a cluster assessment method, which obtains historical log data and real-time log data corresponding to the target cluster, and the log data includes at least: container log, API server log, controller management log; performs feature extraction on the real-time log data to obtain a target feature data set; pre-trains the initial model through historical log data and historical security events, and fine-tunes the pre-trained initial model through the target feature data set and target security events to obtain a target risk assessment model; performs risk assessment on the log data of the target cluster through the target risk assessment model. In this solution, through automated log data collection, as well as pre-training and fine-tuning of the model, it is possible to accurately identify potential security threats and system problems, improve the automation level and accuracy of cluster log assessment, reduce the difficulty and cost of system operation and maintenance, and at the same time improve the security and stability of the system.
[0083] like Figure 2 As shown, Figure 2 A flow chart of a cluster evaluation method provided in an embodiment of the present application, the method may also include the following steps:
[0084] 201. Obtain historical log data and real-time log data corresponding to the target cluster.
[0085] In the embodiment of the present application, for the description of step 201, please refer to the detailed description of step 101 in the above embodiment, and the embodiment of the present application will not be repeated.
[0086] 202. Preprocess the real-time log data to obtain preprocessed log data.
[0087] In an embodiment of the present application, preprocessing may be performed before feature extraction is performed on real-time log data. Specifically, the preprocessing includes at least one of the following: data cleaning, data deduplication, and data formatting.
[0088] In some embodiments, data cleaning mainly involves removing or correcting errors, inconsistencies, and irrelevant data in the logs. This usually includes: removing useless information: such as spaces before and after timestamps, comment lines (lines starting with # or / / ), repeated or irrelevant fields, etc.; correcting format errors: such as inconsistent date and time formats, incomplete JSON structures or format errors, etc.; handling abnormal values: identifying and handling abnormal or erroneous data entries, such as null values, negative values, or values that are obviously illogical.
[0089] In some embodiments, data deduplication is the process of ensuring that there are no duplicate entries in the log. This can be achieved in a variety of ways: content-based deduplication: comparing key information in the log entry (such as timestamp, event type, user ID, etc.) to determine whether it is a duplicate entry; hashing technology: generating a hash value for each log, storing and checking whether the hash value already exists to determine whether it is a duplicate.
[0090] In some embodiments, data formatting is the process of converting log data into a uniform, easily analyzable form. This may include: Standardizing timestamps: converting all timestamps into a uniform format (such as ISO 8601); Structuring data: converting unstructured log entries into structured data (such as JSON or CSV format) for easy query and analysis; Extracting key information: extracting key information from the log (such as error codes, IP addresses, user IDs, etc.) and possibly storing them as separate fields.
[0091] 203. Perform feature extraction on the preprocessed log data to obtain a target feature data set.
[0092] In the embodiment of the present application, after the log data is preprocessed, feature extraction may be performed on the preprocessed log data to obtain a target feature data set.
[0093] 204. Perform data analysis on historical log data to obtain historical common patterns and historical abnormal characteristics.
[0094] In an embodiment of the present application, data analysis can be performed on historical log data. Since historical log data is used to pre-train the initial model to learn common patterns and abnormal features in the log data, the historical log data can be analyzed in advance to obtain historical common patterns and historical abnormal features.
[0095] In some embodiments, the general mode can mainly include: the log format is unified: although different types of devices and systems may generate logs in different formats, most log data follow a certain structure, such as timestamp, log level (such as INFO, WARN, ERROR, etc.), log source (such as file name, process ID, etc.) and log message itself. In addition, the log content has regularity: in a system that operates normally, the log content often shows a certain regularity. For example, operations such as system startup, shutdown, and regular maintenance will leave specific records in the log; specific types of operations (such as database queries, file reading and writing, etc.) will also generate corresponding log entries, which are usually consistent in format and content. And, log events are periodic: many system operations are periodic, such as scheduled tasks, heartbeat detection, etc. These periodic operations will form obvious periodic event patterns in the log. At the same time, log templates are repetitive: in semi-structured log data, log templates (i.e., fixed parts of log messages) will appear frequently. These templates represent different types of log events. By extracting these templates, the log analysis process can be simplified.
[0096] In some embodiments, the abnormal characteristics may mainly include: the level of the abnormal log: the abnormal log usually has a higher log level, such as ERROR, FATAL, etc. The log messages at these levels indicate that an error or abnormal situation has occurred in the system; the content of the abnormal log: the content of the abnormal log often contains specific error codes, error descriptions or exception stack information, which is crucial for locating the cause of the problem and taking repair measures; the abnormal pattern of the log event: the abnormal log event may appear as a log entry that does not conform to the normal pattern, for example, a sudden increase in error logs, missing log entries, or abnormal field values in the log entries; the abnormality of log statistics: by counting specific events or field values in the log, the statistical characteristics of the abnormality can be discovered, for example, the number of calls to a certain API suddenly increases or decreases, the response time of a certain service suddenly becomes longer, etc.; the abnormality associated with other logs: in a complex system, an exception may be associated with multiple other log entries. By analyzing the association between these log entries, the root cause and propagation path of the abnormality can be revealed.
[0097] Common patterns and abnormal features in log data are of great significance for system monitoring, troubleshooting and performance optimization. By identifying and analyzing these patterns and features, operation and maintenance personnel can find and solve problems in a timely manner to ensure the stable operation of the system. At the same time, with the development of artificial intelligence and machine learning technologies, more and more automated tools are being applied to the field of log anomaly detection, further improving the efficiency and accuracy of log analysis.
[0098] 205. The initial model is pre-trained through historical common patterns, historical abnormal features and historical security events.
[0099] In an embodiment of the present application, when pre-training the initial model, the initial model can be pre-trained through historical common patterns, historical abnormal features and historical security events, so that the initial model can learn the common patterns and abnormal features in the log data.
[0100] 206. Input the target feature data set into the pre-trained initial model to obtain an initial evaluation result.
[0101] In an embodiment of the present application, after pre-training the initial model, a pre-trained initial model is obtained. At this time, fine-tuning needs to be performed through real-time log data. Therefore, the target feature data set corresponding to the real-time log data can be input into the pre-trained initial model to obtain an initial evaluation result. The initial evaluation result is the result output by the pre-trained initial model. The result has not been fine-tuned by the model and can be used as a basis for model fine-tuning.
[0102] 207. According to the initial assessment results and target security events, the pre-trained initial model is fine-tuned to obtain the target risk assessment model.
[0103] In an embodiment of the present application, after obtaining the initial assessment result, the initial assessment result can be used as a basis for fine-tuning the model. In combination with the target security event, the pre-trained initial model can be fine-tuned to obtain the final target risk assessment model.
[0104] In some embodiments, the current large prediction model is mainly a pre-trained large model. After training on large-scale unsupervised data, it can complete many tasks after supervised fine-tuning and alignment. Nevertheless, in the face of vertical field applications, large models still need fine-tuning to obtain better application results, and there are many ways to fine-tune large models, including: instruction fine-tuning, supervised fine-tuning, prompt engineering, etc. Among them, instruction fine-tuning is the most important method to improve the controllability of the model.
[0105] Instruction fine-tuning is a process of further training a model on a dataset consisting of instructions and outputs, where instructions represent human instructions to the model and outputs represent the expected outputs of following the instructions. Instruction fine-tuning can be considered a special form of supervised fine tuning (SFT), but SFT is a process of fine-tuning a pre-trained model using labeled data so that the model can better perform specific tasks; instruction fine-tuning is a process of further training a large language model on a dataset consisting of instructions and outputs.
[0106] 208. Obtain the log data to be tested of the target cluster in real time.
[0107] In an embodiment of the present application, after obtaining the final target risk assessment model, the target risk assessment model can be deployed in the target cluster so that the target risk assessment model performs risk assessment on the log data in the target cluster, thereby obtaining new log data to be tested in the target cluster in real time.
[0108] 209. Input the log data to be tested into the target risk assessment model to obtain the target risk assessment result.
[0109] In an embodiment of the present application, after obtaining new real-time log data to be tested, the log data to be tested can be risk assessed through a target risk assessment model, that is, the log data to be tested is input into the target risk assessment model to obtain a target risk assessment result output by the target risk assessment model, and the target risk assessment result includes at least one item: risk level, potential threat, and system failure.
[0110] It should be noted that the risk level can be used to indicate the degree of risk faced by the current target cluster, which can be a pre-set level standard; potential threats can be used to predict the risks that the target cluster may face in the future; system failures can be used to detect and warn of failures in the current system.
[0111] 210. Determine a target security policy corresponding to the target risk assessment result based on a pre-stored correspondence between the risk assessment result and the security policy.
[0112] In an embodiment of the present application, in the face of risks that the target cluster is currently facing or may face, corresponding solutions need to be made in a timely manner. Therefore, the corresponding preset corresponding strategy can be automatically triggered according to the target risk assessment result. In order to trigger the most suitable strategy that matches the target risk assessment result, the correspondence between the risk assessment result and the security strategy can be pre-stored. In this way, after obtaining the target risk assessment result, the target security policy corresponding to the target risk assessment result can be directly called.
[0113] In some embodiments, the security policy may include: isolating abnormal containers, sending alarm notifications, etc.
[0114] 211. Update and optimize the target risk assessment model through real-time log data and feedback information.
[0115] In an embodiment of the present application, the target risk assessment model is an iterative model, which can be updated and optimized based on the log data obtained in real time every day. That is to say, the target risk assessment model will continuously receive new log data every day, and analyze it to obtain new risk assessment results, and will also obtain some feedback information input by the user. Therefore, by continuously collecting new log data and feedback information, the target risk assessment model can be continuously fine-tuned and optimized to adapt to the ever-changing system environment and security threats.
[0116] The embodiment of the present application provides a cluster evaluation method, which can accurately identify potential security threats and system problems through automated log data collection, as well as pre-training and fine-tuning of the model, improve the automation level and accuracy of cluster log evaluation, reduce the difficulty and cost of system operation and maintenance, and improve the security and stability of the system.
[0117] like Figure 3 As shown, an embodiment of the present application provides a cluster evaluation device, which may include:
[0118] The acquisition module 301 is used to obtain historical log data and real-time log data corresponding to the target cluster, and the log data at least includes: container logs, API server logs, and controller management logs;
[0119] The processing module 302 is used to extract features from the real-time log data to obtain a target feature data set;
[0120] The processing module 302 is further used to pre-train the initial model through historical log data and historical security events, and fine-tune the pre-trained initial model through the target feature data set and the target security event to obtain a target risk assessment model;
[0121] The processing module 302 is further configured to perform risk assessment on the log data of the target cluster through a target risk assessment model.
[0122] In some embodiments, the processing module 302 is specifically used to preprocess the real-time log data to obtain preprocessed log data, and the preprocessing includes at least one of the following: data cleaning, data deduplication, and data formatting;
[0123] The processing module 302 is specifically used to extract features from the pre-processed log data to obtain a target feature data set.
[0124] In some embodiments, the processing module 302 is specifically used to perform data analysis on historical log data to obtain historical common patterns and historical abnormal features;
[0125] The processing module 302 is specifically used to pre-train the initial model through historical common patterns, historical abnormal features and historical security events.
[0126] In some embodiments, the processing module 302 is specifically used to input the target feature data set into the pre-trained initial model to obtain an initial evaluation result;
[0127] The processing module 302 is specifically used to fine-tune the pre-trained initial model according to the initial assessment results and the target security event to obtain a target risk assessment model.
[0128] In some embodiments, the acquisition module 301 is specifically used to acquire the log data to be tested of the target cluster in real time;
[0129] The processing module 302 is specifically used to input the log data to be tested into the target risk assessment model to obtain the target risk assessment result, which includes at least one item: risk level, potential threat, and system failure.
[0130] In some embodiments, the processing module 302 is further configured to determine a target security policy corresponding to the target risk assessment result according to a pre-stored correspondence between the risk assessment result and the security policy.
[0131] In some embodiments, the processing module 302 is further used to update and optimize the target risk assessment model through the log data and feedback information obtained in real time.
[0132] In the embodiments of the present application, each module can implement the cluster evaluation method provided in the above method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described here.
[0133] like Figure 4 As shown, an embodiment of the present application further provides an electronic device, which may include:
[0134] A memory 401 storing executable program codes;
[0135] a processor 402 coupled to the memory 401;
[0136] The processor 402 calls the executable program code stored in the memory 401 to execute the cluster evaluation method executed by the electronic device in each of the above method embodiments.
[0137] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the cluster evaluation method in the above method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0138] The embodiment of the present application also provides a computer program product, which stores a computer program. When the computer program is executed by a processor, the various processes of the cluster evaluation method in the above method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0139] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media that include computer-usable program code.
[0140] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and a part of the module, program segment or code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0141] In the present application, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0142] In this application, memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.
[0143] In this application, a person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, and the computer-readable medium includes permanent and non-permanent, removable and non-removable storage media. The storage medium can implement information storage by any method or technology, and the information can be a computer-readable instruction, a data structure, a module of a program, or other data. Examples of computer storage media include, but are not limited to, Parallel Random Access Memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Programmable Read-only Memory (PROM), Erasable Programmable Read Only Memory (EPROM), other types of Random Access Memory (RAM), Read-Only Memory (ROM), One-time Programmable Read-Only Memory (OTPROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), flash memory or other memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. According to the definition in this article, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.
[0144] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0145] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. Those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application. The above-mentioned multiple embodiments are not necessarily multiple independent embodiments. Dividing into multiple embodiments is only used to highlight the different technical features in different embodiments. Those skilled in the art should be aware that the above-mentioned multiple embodiments can also be combined in any way.
[0146] In the various embodiments of the present application, it should be understood that the size of the serial numbers of the above-mentioned processes does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0147] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0148] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0149] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-accessible memory. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several requests for a computer device (which can be a personal computer, a server or a network device, etc., specifically a processor in a computer device) to perform some or all of the steps of the above-mentioned methods of various embodiments of the present application.
[0150] The above is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A cluster evaluation method, characterized in that: The method comprises: Obtain historical log data and real-time log data corresponding to the target cluster, where the log data includes at least: container logs, API server logs, and controller management logs; Performing feature extraction on the real-time log data to obtain a target feature data set; Pre-training the initial model through the historical log data and historical security events, and fine-tuning the pre-trained initial model through the target feature data set and target security events to obtain a target risk assessment model; The target risk assessment model is used to perform risk assessment on the log data of the target cluster.
2. The method according to claim 1, characterized in that The extracting features of the real-time log data to obtain a target feature data set includes: Preprocessing the real-time log data to obtain preprocessed log data, wherein the preprocessing includes at least one of the following: data cleaning, data deduplication, and data formatting; Feature extraction is performed on the preprocessed log data to obtain the target feature data set.
3. The method according to claim 1, characterized in that The pre-training of the initial model by using the historical log data and historical security events includes: Performing data analysis on the historical log data to obtain historical common patterns and historical abnormal features; The initial model is pre-trained using the historical common patterns, the historical abnormal features, and the historical security events.
4. The method according to claim 3, characterized in that The pre-trained initial model is fine-tuned by using the target feature data set and the target security event to obtain a target risk assessment model, including: Inputting the target feature data set into the pre-trained initial model to obtain an initial evaluation result; According to the initial assessment result and the target security event, the pre-trained initial model is fine-tuned to obtain the target risk assessment model.
5. The method according to claim 1, characterized in that The step of performing risk assessment on the log data of the target cluster by using the target risk assessment model includes: Obtaining the log data to be tested of the target cluster in real time; The log data to be tested is input into the target risk assessment model to obtain a target risk assessment result, wherein the target risk assessment result includes at least one item: risk level, potential threat, and system failure.
6. The method according to claim 5, characterized in that After inputting the log data to be tested into the target risk assessment model to obtain the target risk assessment result, the method further includes: According to the pre-stored correspondence between the risk assessment results and the security policies, a target security policy corresponding to the target risk assessment result is determined.
7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: The target risk assessment model is updated and optimized through the log data and feedback information obtained in real time.
8. A cluster evaluation device, characterized in that: include: An acquisition module is used to acquire historical log data and real-time log data corresponding to the target cluster, wherein the log data includes at least: container logs, API server logs, and controller management logs; A processing module, used for extracting features from the real-time log data to obtain a target feature data set; The processing module is further used to pre-train the initial model through the historical log data and historical security events, and fine-tune the pre-trained initial model through the target feature data set and target security events to obtain a target risk assessment model; The processing module is further configured to perform risk assessment on the log data of the target cluster using the target risk assessment model.
9. An electronic device, characterized in that: include: A memory storing executable program code; and a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the cluster evaluation method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: include: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the cluster evaluation method according to any one of claims 1 to 7 is implemented.
Citation Information
Cited By
Security event log template detection method based on large language model
CN120415805A
Security event log template detection method based on large language model
CN120415805B
