Log analysis method and device, equipment and medium

Abstract

The invention discloses a log analysis method and device, equipment and a medium. The method comprises the following steps: preprocessing a historical service link log of a distributed system, and extracting at least two candidate features from the preprocessed historical service link log; performing feature optimization and feature screening on each candidate feature according to the information gain of the candidate features and the relevancy between the features to obtain a to-be-input feature; and performing decision tree construction and division according to the to-be-input features to obtain a decision tree model for distributed system anomaly detection. According to the embodiment of the invention, the capability of analyzing the service link log of a large-scale and dynamically changing distributed system can be improved.

Description

Technical Field

[0001] This invention relates to the field of financial technology, and in particular to a log analysis method, apparatus, device, and medium. Background Technology

[0002] A service chain refers to the chain of calls between different services in a distributed system when they communicate via a network connection. The request and response information of each service in the chain is recorded and formed into corresponding logs for subsequent anomaly analysis and troubleshooting.

[0003] Most existing log analysis methods rely on predefined rules to parse log data. These rules may be based on simple string matching or use regular expressions to find specific anomalies or error patterns. Sometimes, statistical methods are used to perform cluster analysis and trend detection on log data to identify potential performance bottlenecks, service latency, and other issues. A common approach is to use time series analysis, setting thresholds to determine the normal or abnormal state of the service.

[0004] These technical solutions can meet the needs of service link log analysis to some extent, but they generally suffer from problems such as difficulty in updating rules, high data quality requirements, high maintenance costs, and low accuracy. Rule-based methods are difficult to update and prone to missing new anomalies, while statistical analysis-based methods are poorly adaptable to complex anomalies. These shortcomings indicate that existing methods face certain challenges in processing service link logs of large-scale, dynamically changing distributed systems, and there is an urgent need for more efficient, flexible, and easy-to-understand solutions. Summary of the Invention

[0005] This invention provides a log analysis method, apparatus, device, and medium to improve the ability to analyze service link logs of large-scale, dynamically changing distributed systems.

[0006] According to one aspect of the present invention, a log analysis method is provided, comprising:

[0007] Preprocess the historical service link logs of the distributed system and extract at least two candidate features from the preprocessed historical service link logs;

[0008] Based on the information gain of the candidate features and the correlation between the features, feature optimization and feature filtering are performed on each candidate feature to obtain the input feature;

[0009] Based on the input features, a decision tree is constructed and partitioned to obtain a decision tree model for anomaly detection in distributed systems.

[0010] According to another aspect of the present invention, a log analysis apparatus is provided, comprising:

[0011] The candidate feature extraction module is used to preprocess the historical service link logs of the distributed system and extract at least two candidate features from the preprocessed historical service link logs.

[0012] The feature selection and optimization module is used to optimize and select the candidate features based on the information gain of the candidate features and the correlation between the features to obtain the input features.

[0013] The decision tree generation module is used to construct and divide a decision tree based on the input features to obtain a decision tree model for anomaly detection in a distributed system.

[0014] According to another aspect of the present invention, a log analysis method is provided, comprising:

[0015] Extract the input features from the target service link logs of the distributed system;

[0016] The input features are fed into the decision tree model to obtain the anomaly detection results of the target service link log;

[0017] The decision tree model is trained using the log analysis method described in any embodiment of the present invention.

[0018] According to another aspect of the present invention, a log analysis apparatus is provided, comprising:

[0019] The feature extraction module is used to extract input features from the target service link logs of the distributed system;

[0020] An anomaly detection module is used to input the features to be input into the decision tree model to obtain the anomaly detection results of the target service link log;

[0021] The decision tree model is obtained using the log analysis device described in any embodiment of the present invention.

[0022] According to another aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the log analysis method according to any embodiment of the present invention.

[0023] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the log analysis method according to any embodiment of the present invention.

[0024] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the log analysis method according to any embodiment of the present invention.

[0025] When dealing with service link logs of large-scale, dynamically changing distributed systems, this invention optimizes and filters candidate features extracted from the logs based on the information gain of candidate features and the correlation between features. This reduces the number of features that need to be referenced when constructing and dividing the decision tree, significantly reducing the difficulty of constructing and dividing the decision tree. The result is more flexible and efficient, and the tree structure of the decision tree model also makes it easier for users to understand the specific decision-making process.

[0026] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart of a log analysis method provided according to an embodiment of the present invention;

[0029] Figure 2 This is a flowchart of a log analysis method provided according to another embodiment of the present invention;

[0030] Figure 3 This is a flowchart of a log analysis method provided according to another embodiment of the present invention;

[0031] Figure 4 This is a schematic diagram of the structure of a log analysis device according to another embodiment of the present invention;

[0032] Figure 5 This is a schematic diagram of the structure of a log analysis device according to another embodiment of the present invention;

[0033] Figure 6 This is a schematic diagram of the structure of an electronic device that implements an embodiment of the present invention. Detailed Implementation

[0034] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0035] It should be noted that the terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but includes other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0036] Figure 1 This is a flowchart illustrating a log analysis method according to an embodiment of the present invention. This embodiment is applicable to situations requiring anomaly detection in a distributed system based on historical service link logs. The method can be executed by a log analysis device, which can be implemented in hardware and / or software. This device can be configured in an electronic device with corresponding data processing capabilities, such as a server used for training and building decision tree models. Figure 1 As shown, the method includes:

[0037] S110. Preprocess the historical service link logs of the distributed system and extract at least two candidate features from the preprocessed historical service link logs.

[0038] S120. Based on the information gain of the candidate features and the correlation between the features, perform feature optimization and feature selection on each candidate feature to obtain the input feature.

[0039] S130. Decision tree is constructed and partitioned based on the input features to obtain a decision tree model for anomaly detection in distributed systems.

[0040] The candidate features are mainly categorized as follows: Link features: trace_id, service name, interface name, call path; Time features: timestamp, time period, request interval, sliding window statistics; Performance features: response time, latency distribution, error rate; Business features: transaction type, amount range, account type; Environment and resource features: instance load, CPU / memory usage. Feature selection is a process in machine learning that aims to extract the most useful features for model prediction from log data. In service link log analysis, feature selection helps determine which log fields are most important for predicting service behavior or detecting anomalies.

[0041] Specifically, the decision tree algorithm is a machine learning algorithm commonly used for classification and regression. It makes decisions and predictions based on input data by constructing a tree-like decision tree model. A decision tree is represented as a binary tree, where each node represents a feature judgment, and the leaf nodes represent the final decision result. In the service log analysis of distributed systems, decision trees can be used to identify abnormal behavior in log data and output corresponding anomaly detection results.

[0042] Collect log data from service chains in the distributed system, including key request information such as request ID, timestamp, response time, and error code. Perform preprocessing operations on the raw log data, such as deduplication, missing value imputation, and standardized formatting, to ensure data quality. Extract multiple candidate features from the preprocessed logs, such as request latency, error rate, service node, and request type.

[0043] For large-scale, dynamically changing historical service logs, there are a large number of features that can be extracted. Too many features will significantly increase the difficulty of subsequent decision tree construction and partitioning. Therefore, it is necessary to optimize and filter candidate features in advance to obtain the input features, so as to reduce the number of features that need to be referenced when constructing and partitioning the decision tree. During feature optimization and feature filtering, the information gain of candidate features and the correlation between features can be referenced to determine the input features that need to be retained among the reference candidate features.

[0044] Each decision node in a decision tree model partitions the dataset based on the feature values ​​of the input feature until a stopping condition is met. The decision tree model is trained using the dataset, continuously selecting the best feature for data partitioning until the stopping condition is satisfied. Cross-validation is used to evaluate the accuracy of the decision tree model and avoid overfitting. Decision tree models may suffer from overfitting, therefore pruning is necessary. Pruning simplifies the model by removing certain branches, thereby improving its generalization ability. Common pruning methods include pre-pruning and post-pruning. This invention can choose to reduce the complexity of the tree through post-pruning, thus improving the model's generalization ability. After training, the anomaly detection capability of the decision tree model can be evaluated using common evaluation metrics: accuracy, recall, precision, and F1 score.

[0045] When dealing with service link logs of large-scale, dynamically changing distributed systems, this invention optimizes and filters candidate features extracted from the logs based on the information gain of candidate features and the correlation between features. This reduces the number of features that need to be referenced when constructing and dividing the decision tree, significantly reducing the difficulty of constructing and dividing the decision tree. The result is more flexible and efficient, and the tree structure of the decision tree model also makes it easier for users to understand the specific decision-making process.

[0046] Based on the above embodiments, optionally, the preprocessing includes at least one of the following:

[0047] Sensitive fields in the service link logs are encrypted using a one-way hash algorithm;

[0048] A bucketing coding strategy is used to map non-sensitive fields in the service link log to predefined ranges.

[0049] Specifically, sensitive and non-sensitive fields in the service logs are identified, and different processing strategies are applied to each. Sensitive fields (such as user ID, mobile phone number, and national ID number) are encrypted using a one-way hash algorithm, ensuring the encrypted sensitive fields are irreversible. Non-sensitive fields are mapped using different bucketing encoding strategies based on their type (e.g., numeric, categorical, or time-based fields) to obtain new values. Bucketing encoding reduces data granularity, further enhancing privacy protection (e.g., changing age from "28" to "20-30"). Through the use of one-way hashing and bucketing encoding strategies, precise desensitization of sensitive information in the service logs can be achieved, preventing the leakage of customer privacy.

[0050] Figure 2 This is a flowchart illustrating a log analysis method according to another embodiment of the present invention. This embodiment is an optimization and improvement upon the above embodiment. Figure 2 As shown, the method includes:

[0051] S210. Preprocess the historical service link logs of the distributed system and extract at least two candidate features from the preprocessed historical service link logs.

[0052] S220. Dimensionality reduction strategy is used to optimize the high cardinality candidate features among the candidate features to obtain new candidate features.

[0053] S230. Sort the candidate features in descending order of information gain to obtain the high-gain candidate features that are ranked first.

[0054] S240. Based on the correlation between features, high-gain candidate features are filtered to obtain low-correlation input features.

[0055] Specifically, high cardinality categorical features refer to features with a large number of unique values ​​(i.e., a large number of categories), such as user ID, device ID, and IP address. These features are prone to dimensionality explosion and may only appear a very small number of times, making model generalization difficult. Therefore, before feature selection, a dimensionality reduction strategy is needed to optimize the high cardinality candidate features, forming new candidate features to replace the original high cardinality candidate features. The information gain of each candidate feature is calculated, and the candidate features are sorted in descending order of information gain. A predetermined number of features are selected as high-gain candidate features from the top-ranked features. The remaining candidate features, due to their low information gain, are not considered during decision tree construction and partitioning. High-gain candidate features are further selected based on the correlation between features to avoid simultaneously selecting highly correlated features as input features. Only low-correlation high-gain candidate features are retained as the final input features, thereby further reducing the number of features. By selecting candidate features according to information gain, the number of features is further reduced without affecting subsequent decision tree construction and partitioning.

[0056] S250. Decision tree is constructed and partitioned based on the input features to obtain a decision tree model for anomaly detection in distributed systems.

[0057] Based on the above embodiments, optionally, the step of filtering high-gain candidate features according to the correlation between features to obtain low-correlation input features includes:

[0058] For each high-gain candidate feature, calculate the correlation between the high-gain candidate feature and other high-gain candidate features;

[0059] If there are other high-gain candidate features with a similarity greater than the threshold, then the other high-gain candidate features or the high-gain candidate feature itself will be determined as the input feature.

[0060] Specifically, for each high-gain candidate feature, its correlation with other high-gain candidate features is calculated. For other high-gain candidate features with a correlation score not exceeding a threshold, they are considered unrelated and can both be retained as input features. For other high-gain candidate features with a correlation score exceeding the threshold, they are considered related, and only one can be retained as an input feature. This secondary feature selection through correlation calculation further reduces the number of features.

[0061] Based on the above embodiments, optionally, the dimensionality reduction strategy is used to optimize the high cardinality candidate features among the candidate features to obtain new candidate features, including at least one of the following:

[0062] High-cardinality candidate features with a frequency below a threshold among the candidate features are merged into new candidate features;

[0063] The original feature values ​​of the high-cardinality candidate features in the candidate features are replaced with the statistics of the corresponding variables to obtain new candidate features;

[0064] The high-cardinality candidate features among the candidate features are mapped in a low dimension to obtain new candidate features.

[0065] Specifically, dimensionality reduction strategies mainly include grouping and classification, target encoding, and hash compression. Grouping and classification involves grouping high-cardinality candidate features with frequencies below a threshold into a single "Other" category as new candidate features, thereby reducing the total number of features and sparsity. Grouping and classification is simple and effective, and can avoid overfitting the model to rare IDs.

[0066] Target encoding refers to replacing the original feature values ​​of high-base-value features with the statistics (such as mean and median) of the target variable (such as default rate) corresponding to each high-base-value feature, thereby obtaining new candidate features as replacements for the original high-base-value candidate features.

[0067] Hash bucket compression refers to mapping high-dimensional features to a fixed-size low-dimensional space (such as 1000 buckets) using a hash function, thereby obtaining new candidate features. For example, multiple different IDs may be mapped to the same bucket during the mapping process. By preparing multiple dimensionality reduction strategies in advance, the dimensionality reduction optimization capability for high cardinality candidate features is improved.

[0068] Based on the above embodiments, optionally, before sorting the candidate features in descending order of information gain to obtain the high-gain candidate features ranked first, the method further includes:

[0069] A binning strategy is used to encode consecutive candidate features in the candidate features to obtain new candidate features.

[0070] Specifically, a business rule-based binning strategy is adopted, with domain experts setting boundary points based on experience. For example, transaction amounts are divided into "small (<100 yuan)", "medium (100–1000 yuan)", and "large (>1000 yuan)".

[0071] Based on the boundary points, the original continuous values ​​of continuous candidate features are mapped to corresponding discrete intervals (such as small, medium, and large amounts), and the mapping results are ordinal encoded to obtain new feature values ​​for the continuous candidate features, which are then used as new candidate features. Binning of continuous candidate features can improve the interpretability of subsequent detection tree models and reduce the impact of outliers in continuous candidate features.

[0072] This invention further reduces the number of features by filtering candidate features according to information gain without affecting subsequent decision tree construction and partitioning.

[0073] Figure 3 This is a flowchart illustrating a log analysis method according to another embodiment of the present invention. This embodiment is applicable to situations requiring anomaly detection in a distributed system based on historical service link logs. The method can be executed by a log analysis device, which can be implemented in hardware and / or software. This device can be configured in an electronic device with corresponding data processing capabilities, such as a server deploying a decision tree model. Figure 3 As shown, the method includes:

[0074] S310. Extract the input features from the target service link log of the distributed system.

[0075] S320. Input the features to be input into the decision tree model to obtain the anomaly detection results of the target service link log.

[0076] The decision tree model is trained using the log analysis method described in any embodiment of the present invention.

[0077] Specifically, referring to the input features used in the construction and partitioning stages of the decision tree model, the same input features are extracted from the target service link logs of the distributed system during the actual anomaly detection stage. The specific extraction process can be found in the embodiments described above, and will not be repeated here. The extracted input features are then input into the decision tree model, which analyzes the input features and outputs the corresponding prediction results.

[0078] In the face of service link logs of large-scale, dynamically changing distributed systems, this invention extracts only the features to be input from the target service link logs to input into the decision tree model to obtain anomaly detection results, thereby reducing the number of features to be extracted and improving the efficiency of anomaly detection.

[0079] Figure 4 This is a schematic diagram of a log analysis device provided in another embodiment of the present invention. Figure 4 As shown, the device includes:

[0080] The candidate feature extraction module 410 is used to preprocess the historical service link logs of the distributed system and extract at least two candidate features from the preprocessed historical service link logs.

[0081] The feature selection and optimization module 420 is used to perform feature optimization and feature selection on each candidate feature based on the information gain of the candidate features and the correlation between the features to obtain the input features;

[0082] The decision tree generation module 430 is used to construct and divide a decision tree based on the input features to obtain a decision tree model for anomaly detection in a distributed system.

[0083] The log analysis device provided in the embodiments of the present invention can execute the log analysis method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0084] Optionally, the feature selection optimization module 420 includes:

[0085] The dimensionality reduction optimization unit is used to perform dimensionality reduction optimization on the high cardinality candidate features among the candidate features using a dimensionality reduction strategy to obtain new candidate features;

[0086] The gain sorting unit is used to sort candidate features in descending order of information gain, and obtain the high-gain candidate features that are ranked first.

[0087] The correlation analysis unit is used to filter high-gain candidate features based on the correlation between features, and obtain low-relevance input features.

[0088] Optionally, the correlation analysis unit is specifically used for:

[0089] For each high-gain candidate feature, calculate the correlation between the high-gain candidate feature and other high-gain candidate features;

[0090] If there are other high-gain candidate features with a similarity greater than the threshold, then the other high-gain candidate features or the high-gain candidate feature itself will be determined as the input feature.

[0091] Optionally, the correlation analysis unit includes at least one of the following sub-units:

[0092] The grouping and classification subunit is used to merge high-cardinality candidate features with a frequency below a threshold into new candidate features;

[0093] The target encoding subunit is used to replace the original feature values ​​of the high cardinality candidate features in the candidate features with the statistics of the corresponding variables to obtain new candidate features;

[0094] The hash bucket compression subunit is used to perform low-dimensional mapping on the high cardinality candidate features among the candidate features to obtain new candidate features.

[0095] Optionally, the preprocessing includes at least one of the following:

[0096] Sensitive fields in the service link logs are encrypted using a one-way hash algorithm;

[0097] A bucketing coding strategy is used to map non-sensitive fields in the service link log to predefined ranges.

[0098] Optionally, the feature selection optimization module 420 also includes:

[0099] The feature encoding unit is used to encode consecutive candidate features in the candidate features using a binning strategy to obtain new candidate features.

[0100] The log analysis device described in further detail can also execute the log analysis method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method.

[0101] Figure 5 This is a schematic diagram of a log analysis device provided in another embodiment of the present invention. Figure 5 As shown, the device includes:

[0102] Feature extraction module 510 is used to extract input features from the target service link logs of the distributed system;

[0103] Anomaly detection module 520 is used to input the features to be input into the decision tree model to obtain the anomaly detection results of the target service link log;

[0104] The decision tree model is obtained using the log analysis device described in any embodiment of the present invention.

[0105] The log analysis device provided in the embodiments of the present invention can execute the log analysis method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0106] Figure 6A schematic diagram of an electronic device 60 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0107] like Figure 6 As shown, the electronic device 60 includes at least one processor 61 and a memory, such as a read-only memory (ROM) 62 and a random access memory (RAM) 63, communicatively connected to the at least one processor 61. The memory stores computer programs executable by the at least one processor. The processor 61 can perform various appropriate actions and processes based on the computer program stored in the ROM 62 or loaded from storage unit 68 into the RAM 63. The RAM 63 can also store various programs and data required for the operation of the electronic device 60. The processor 61, ROM 62, and RAM 63 are interconnected via a bus 64. An input / output (I / O) interface 65 is also connected to the bus 64.

[0108] Multiple components in electronic device 60 are connected to I / O interface 65, including: input unit 66, such as keyboard, mouse, etc.; output unit 67, such as various types of monitors, speakers, etc.; storage unit 68, such as disk, optical disk, etc.; and communication unit 69, such as network card, modem, wireless transceiver, etc. Communication unit 69 allows electronic device 60 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0109] Processor 61 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 61 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 61 performs the various methods and processes described above, such as log analysis methods.

[0110] In some embodiments, the log analysis method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 68. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 60 via ROM 62 and / or communication unit 69. When the computer program is loaded into RAM 63 and executed by processor 61, one or more steps of the log analysis method described above may be performed. Alternatively, in other embodiments, processor 61 may be configured to perform the log analysis method by any other suitable means (e.g., by means of firmware).

[0111] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0112] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0113] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0114] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0115] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0116] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0117] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0118] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A log analysis method, characterized in that, The method includes: Preprocess the historical service link logs of the distributed system and extract at least two candidate features from the preprocessed historical service link logs; Based on the information gain of the candidate features and the correlation between the features, feature optimization and feature filtering are performed on each candidate feature to obtain the input feature; Based on the input features, a decision tree is constructed and partitioned to obtain a decision tree model for anomaly detection in distributed systems.

2. The method according to claim 1, characterized in that, The process of optimizing and filtering candidate features based on their information gain and correlation to obtain the input features includes: A dimensionality reduction strategy is used to optimize the high cardinality candidate features among the candidate features to obtain new candidate features. Candidate features are sorted in descending order of information gain to obtain the highest-gain candidate features. High-gain candidate features are filtered based on the correlation between features to obtain low-correlation input features.

3. The method according to claim 2, characterized in that, The step of filtering high-gain candidate features based on the correlation between features to obtain low-correlation input features includes: For each high-gain candidate feature, calculate the correlation between the high-gain candidate feature and other high-gain candidate features; If there are other high-gain candidate features with a similarity greater than the threshold, then the other high-gain candidate features or the high-gain candidate features are determined as the input features.

4. The method according to claim 2, characterized in that, The dimensionality reduction strategy is used to optimize the high cardinality candidate features among the candidate features, and the new candidate features obtained include at least one of the following: High-cardinality candidate features with a frequency below a threshold among the candidate features are merged into new candidate features; The original feature values ​​of the high-cardinality candidate features in the candidate features are replaced with the statistics of the corresponding variables to obtain new candidate features; The high-cardinality candidate features among the candidate features are mapped in a low dimension to obtain new candidate features.

5. The method according to claim 2, characterized in that, Before sorting the candidate features according to their information gain from largest to smallest to obtain the highest-gain candidate features, the process also includes: A binning strategy is used to encode consecutive candidate features in the candidate features to obtain new candidate features.

6. The method according to claim 1, characterized in that, The preprocessing includes at least one of the following: Sensitive fields in the service link logs are encrypted using a one-way hash algorithm; A bucketing coding strategy is used to map non-sensitive fields in the service link log to predefined ranges.

7. A method for using a decision tree model, characterized in that, The method includes: Extract the input features from the target service link logs of the distributed system; The input features are fed into the decision tree model to obtain the anomaly detection results of the target service link log; The decision tree model is trained using the log analysis method described in claim 1.

8. A log analysis device, characterized in that, The device includes: The candidate feature extraction module is used to preprocess the historical service link logs of the distributed system and extract at least two candidate features from the preprocessed historical service link logs. The feature selection and optimization module is used to optimize and select the candidate features based on the information gain of the candidate features and the correlation between the features to obtain the input features. The decision tree generation module is used to construct and divide a decision tree based on the input features to obtain a decision tree model for anomaly detection in a distributed system.

9. A log analysis device, characterized in that, The device includes: The feature extraction module is used to extract input features from the target service link logs of the distributed system; An anomaly detection module is used to input the features to be input into the decision tree model to obtain the anomaly detection results of the target service link log; The decision tree model is obtained using the log analysis device described in claim 8.

10. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the log analysis method according to any one of claims 1-7.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the log analysis method according to any one of claims 1-7.

Images