Log abstract generation system and method based on inductive hierarchical structure

Abstract

The invention discloses a log abstract generation system and method based on an inductive hierarchical structure, and the system comprises a feature extraction module which is used for extracting key features from an original alarm log; the induction hierarchy construction module is used for pre-defining an induction hierarchy structure for each extracted feature; the dissimilarity calculation module is used for defining and calculating the dissimilarity between the alarm logs based on the inductive hierarchical structure; the self-adaptive clustering module is used for abstracting feature values upwards step by step through an iterative induction process by using a heuristic algorithm, and clustering similar alarm logs after being abstracted until a preset clustering stop condition is met; and the alarm abstract generation module is used for summarizing the clustering results and generating an alarm abstract capable of summarizing each clustering feature. The system and the method can effectively process a large number of complex alarm logs, reserve important detail information, have good interpretability, and support operation and maintenance personnel to quickly and accurately carry out fault positioning and root cause analysis.

Description

Technical Field

[0001] The present invention relates to the field of log analysis, and in particular to a log summary generation system and method based on an inductive hierarchy structure. Background Art

[0002] With the continuous development and increasing complexity of business systems, system operation and maintenance are facing increasingly severe challenges. When a system fails, the number of error logs will increase dramatically, and sometimes even "crazy error reporting" will occur. In this case, error logs will bury and affect each other, making it difficult for operation and maintenance personnel to sort out the root cause of the failure in a short period of time, affecting the efficiency of troubleshooting and repair.

[0003] The commonly used log analysis methods in the industry currently include simple text analysis methods such as keyword matching and regular expression matching, as well as cluster analysis methods based on machine learning. However, these methods still have some problems when processing large and complex alarm logs:

[0004] 1. Simple text matching methods are difficult to process logs with similar semantics but different expressions, which can easily lead to the omission of useful information;

[0005] 2. Traditional clustering algorithms such as K-Means require the number of clusters to be determined in advance, which is not suitable for scenarios where the number and type of alarm logs change dynamically;

[0006] 3. Most methods lack the extraction and utilization of fine-grained features in alarm logs and cannot retain important detail information well;

[0007] 4. Clustering results often lack interpretability, making it difficult to directly guide operation and maintenance personnel to quickly locate the root cause of the problem.

[0008] Therefore, a log clustering system and method is needed that can effectively process a large number of complex alarm logs, retain important detail information, and have good interpretability to support operation and maintenance personnel to quickly and accurately locate faults and perform root cause analysis. Summary of the invention

[0009] In order to solve the above-mentioned problems existing in the prior art, the present invention provides a log summary generation system and method based on an inductive hierarchy structure, which processes complex alarm log data through a predefined inductive hierarchy structure, so that the operation and maintenance team can manage and respond to system failures more efficiently, thereby reducing fault recovery time and improving overall system stability.

[0010] To achieve the above purpose, the present invention adopts the following technical solutions:

[0011] In one embodiment of the present invention, a log summary generation system based on an inductive hierarchical structure is proposed, the system comprising:

[0012] Feature extraction module, used to extract key features from the original alarm log, including the machine room, environment, abnormal source, key content of the alarm log text and the location of the fault;

[0013] The inductive hierarchy building module is used to predefine an inductive hierarchy for each extracted feature as the basis for upward abstraction in the subsequent clustering process;

[0014] The dissimilarity calculation module is used to define and calculate the dissimilarity between alarm logs based on the inductive hierarchy, providing a quantitative basis for clustering;

[0015] The adaptive clustering module is used to use a heuristic algorithm to gradually abstract the feature values ​​upward through an iterative induction process, and cluster alarm logs that are similar after abstraction until a preset clustering stop condition is met;

[0016] The alarm summary generation module is used to summarize the clustering results and generate an alarm summary that can summarize the characteristics of each cluster.

[0017] Furthermore, the calculation steps of dissimilarity are as follows:

[0018] For the dissimilarity of a single feature: Let x1, x2 be two different values ​​of a feature, and their dissimilarity is defined as: d(x1, x2) = the length of the shortest path connecting x1 and x2 in the inductive hierarchy;

[0019] For the overall dissimilarity of two alarm logs a1 and a2: d(a1, a2) = Σd(a1[Ai], a2[Ai])(i=1, 2, ..., n), where Ai represents the i-th feature and n is the total number of features.

[0020] Furthermore, a heuristic algorithm is used to gradually abstract the feature values ​​upward through an iterative induction process, and the alarm logs that are similar after abstraction are clustered until the preset clustering stop conditions are met, including:

[0021] Initialization: record the original alarm log set as L, set the minimum cluster size min_size, and set the clustering stop condition: the number of alarm summaries in the clustering result is ≤20 or the count value of a certain category has reached min_size;

[0022] Iterative algorithm: The iterative process is executed in a loop until the clustering stop condition set in the initialization is met; in each iteration, a feature Ai with the most uniform current value distribution is selected for processing; for the selected feature Ai, the algorithm traverses all alarm logs a; for each alarm log a, its value on feature Ai is replaced with the parent node value of the value in the predefined inductive hierarchy; after completing the processing of all alarm logs, the algorithm calls the similarity calculation module, merges the alarm logs with the same values ​​on all features, and updates the count value of each alarm log combination; then enters the next iteration and repeats the above process.

[0023] Furthermore, the feature selection strategy is as follows:

[0024] Calculate the Fi value of each feature Ai: Fi = max{the number of alarms with value v on Ai|the range of v∈Ai}; Fi is an indicator used to measure the distribution of feature Ai;

[0025] Select the feature with the smallest Fi value for induction.

[0026] In one embodiment of the present invention, a log summary generation method based on an inductive hierarchical structure is also proposed, the method comprising:

[0027] Extract key features from the original alarm log, including the computer room, environment, abnormal source, key content of the alarm log text, and the location of the fault;

[0028] Predefine an inductive hierarchy for each extracted feature as the basis for upward abstraction in the subsequent clustering process;

[0029] Based on the inductive hierarchical structure, the dissimilarity between alarm logs is defined and calculated to provide a quantitative basis for clustering;

[0030] Using a heuristic algorithm, the feature values ​​are gradually abstracted upward through an iterative induction process, and alarm logs that are similar after abstraction are clustered until the preset clustering stop condition is met;

[0031] The clustering results are summarized to generate an alarm summary that can summarize the characteristics of each cluster.

[0032] Furthermore, the calculation steps of dissimilarity are as follows:

[0033] For the dissimilarity of a single feature: Let x1, x2 be two different values ​​of a feature, and their dissimilarity is defined as: d(x1, x2) = the length of the shortest path connecting x1 and x2 in the inductive hierarchy;

[0034] For the overall dissimilarity of two alarm logs a1 and a2: d(a1, a2) = Σd(a1[Ai], a2[Ai])(i=1, 2, ..., n), where Ai represents the i-th feature and n is the total number of features.

[0035] Furthermore, a heuristic algorithm is used to gradually abstract the feature values ​​upward through an iterative induction process, and the alarm logs that are similar after abstraction are clustered until the preset clustering stop conditions are met, including:

[0036] Initialization: record the original alarm log set as L, set the minimum cluster size min_size, and set the clustering stop condition: the number of alarm summaries in the clustering result is ≤20 or the count value of a certain category has reached min_size;

[0037] Iterative algorithm: The iterative process is executed in a loop until the clustering stop condition set in the initialization is met; in each iteration, a feature Ai with the most uniform current value distribution is selected for processing; for the selected feature Ai, the algorithm traverses all alarm logs a; for each alarm log a, its value on feature Ai is replaced with the parent node value of the value in the predefined inductive hierarchy; after completing the processing of all alarm logs, the algorithm calls the similarity calculation module, merges the alarm logs with the same values ​​on all features, and updates the count value of each alarm log combination; then enters the next iteration and repeats the above process.

[0038] Furthermore, the feature selection strategy is as follows:

[0039] Calculate the Fi value of each feature Ai: Fi = max{the number of alarms with value v on Ai|the range of v∈Ai}; Fi is an indicator used to measure the distribution of feature Ai;

[0040] Select the feature with the smallest Fi value for induction.

[0041] In one embodiment of the present invention, a computer device is also proposed, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the aforementioned log summary generation method based on the inductive hierarchical structure is designed.

[0042] In one embodiment of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores a computer program for executing a log summary generation method based on an inductive hierarchical structure.

[0043] Beneficial effects:

[0044] 1. Effective clustering:

[0045] (1) The present invention can achieve effective clustering of alarm logs while retaining important details through a predefined inductive hierarchy.

[0046] (2) The present invention uses an adaptive clustering algorithm, which does not require the number of clusters to be determined in advance and is suitable for handling situations where the number and type of alarm logs change dynamically.

[0047] 2. Keep detailed information:

[0048] The present invention can extract and utilize fine-grained features in the alarm log, such as the machine room, environment, abnormal source, key content of the alarm log text and the specific location where the fault occurred, so as to retain important detailed information.

[0049] 3. Improve interpretability:

[0050] (1) The clustering results generated by the present invention have good interpretability, which makes it easy for operation and maintenance personnel to quickly understand the clustering results and locate the root cause of the problem.

[0051] (2) The alarm summary generated by the present invention includes information such as the cluster ID, computer room, environment, abnormality source, fault location, summary content and count, which facilitates operation and maintenance personnel to quickly identify problems.

[0052] 4. Improve efficiency:

[0053] (1) The present invention can effectively process a large number of complex alarm logs, reduce the number of alarm logs, and at the same time maintain a sufficient amount of information, thereby significantly improving the efficiency of fault location and root cause analysis.

[0054] (2) The present invention can reduce the types of alarms through the induction process, helping operation and maintenance personnel to locate the key areas of the problem more quickly, thereby improving the speed and accuracy of troubleshooting. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a log summary generation system architecture diagram based on the inductive hierarchical structure of the present invention;

[0056] Figure 2 It is a flow chart of a log summary generation method based on an inductive hierarchical structure of the present invention;

[0057] Figure 3 It is a schematic diagram of the computer device structure of the present invention. DETAILED DESCRIPTION

[0058] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and design the present invention, and are not intended to limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0059] Those skilled in the art will appreciate that the embodiments of the present invention may be designed as a system, system, device, method or computer program product. Therefore, the present disclosure may be specifically designed in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0060] According to an embodiment of the present invention, a log summary generation system based on an inductive hierarchy is proposed. The complex alarm log data is processed through a predefined inductive hierarchy, so that the operation and maintenance team can manage and respond to system failures more efficiently, thereby reducing the failure recovery time and improving the overall system stability.

[0061] The principle and spirit of the present invention are explained in detail below with reference to several representative embodiments of the present invention.

[0062] Figure 1 It is a log summary generation system architecture diagram based on the inductive hierarchy of the present invention.

[0063] like Figure 1 As shown, the system includes:

[0064] Feature extraction module, used to extract key features from the original alarm log, including the machine room, environment, abnormal source, key content of the alarm log text and the location of the fault;

[0065] The inductive hierarchy building module is used to predefine an inductive hierarchy for each extracted feature as the basis for upward abstraction in the subsequent clustering process;

[0066] The dissimilarity calculation module is used to define and calculate the dissimilarity between alarm logs based on the inductive hierarchy, providing a quantitative basis for clustering;

[0067] The adaptive clustering module is used to use a heuristic algorithm to gradually abstract the feature values ​​upward through an iterative induction process, and adaptively cluster similar alarm logs after abstraction until a preset clustering stop condition is met, without predetermining the number of clusters;

[0068] The alarm summary generation module is used to summarize the clustering results and generate an alarm summary that can summarize the characteristics of each cluster, thereby improving the efficiency of problem location.

[0069] The term "module" used above may be a combination of software and / or hardware designed to implement a predetermined function. Although the system described in the following embodiments is preferably designed with software, hardware, or a combination of software and hardware, is also possible and contemplated.

[0070] It should be noted that although several modules of the log summary generation system based on the inductive hierarchy are mentioned in the above detailed description, this division is only exemplary and not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more modules described above can be concretized in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules for concretization.

[0071] In order to explain the above-mentioned log summary generation system based on the inductive hierarchical structure more clearly, a specific embodiment is used for illustration below. However, it should be noted that this embodiment is only for better illustrating the present invention and does not constitute an improper limitation to the present invention.

[0072] Example:

[0073] 1. Feature extraction module

[0074] This module is responsible for extracting key features from the original alarm log. Based on actual application scenarios and operation and maintenance experience, this paper selects to extract the following five key features:

[0075] (1) Server room: indicates the physical location where the alarm log is generated, such as "server_room_a", "server_room_b", etc.

[0076] (2) Environment: Indicates the system environment where the alarm log is generated, such as "online", "offline", "prod", "staging", etc.

[0077] (3) Abnormal source: indicates the system component or service that triggered the alarm, such as "Product Center", "Cache Service", "A / B Test", etc.

[0078] (4) Key content of alarm log text: Extract the text fragments in the alarm log that best represent the essence of the problem. The extraction strategy is as follows:

[0079] First find out whether there is an exception stack, if so, extract the last exception (usually the root cause);

[0080] If there is no exception stack, check whether there is an error prompt in the format of "code=..., message=...";

[0081] If none of the above exists, extract the first line of the log content (delimited by the line break) and remove any information related to the specific case that may exist therein.

[0082] (5) Fault location: indicates the code location where the problem occurs. The extraction strategy is as follows:

[0083] First check whether there is an exception stack, if so, extract the location of the first local code;

[0084] If there is no exception stack, extract the log printing position.

[0085] The following example illustrates that there are 5 original alarm logs:

[0086] server_room_a-online01 Thrift query deal info error.com.example.service.DealService

[0087] server_room_b-offline02 Thrift get deal ProductType deal error.com.example.service.ProductService

[0088] server_room_a-online02 Thrift get deal ProductType deal error.com.example.service.ProductService

[0089] server_room_b-offline01 Pigeon query deal info error.com.example.service.DealService

[0090] server_room_a-offline01 Http query deal info error.com.example.service.DealService

[0091] Extract 5 key features of each log:

[0092] Computer room: server_room_a, Environment: online, Exception source: Thrift, Text key content: querydeal info error, Location: com.example.service.DealService

[0093] Computer room: server_room_b, Environment: offline, Exception source: Thrift, Text key content: get dealProductType deal error, Location: com.example.service.ProductService

[0094] Computer room: server_room_a, Environment: online, Exception source: Thrift, Text key content: get dealProductType deal error, Location: com.example.service.ProductService

[0095] Computer room: server_room_b, Environment: offline, Exception source: Pigeon, Text key content: querydeal info error, Location: com.example.service.DealService

[0096] Computer room: server_room_a, Environment: offline, Exception source: Http, Text key content: query dealinfo error, Location: com.example.service.DealService

[0097] 2. Inductive Hierarchical Construction Modules

[0098] This module is responsible for pre-defining an inductive hierarchical structure for each extracted feature, which will serve as the basis for upward abstraction in the subsequent clustering process. This structure is like a tree, with leaf nodes being specific feature values, and nodes becoming more abstract as they go up. The following is an example of the inductive hierarchical structure of each feature:

[0099] (1) Computer room:

[0100]

[0101] (2) Environment:

[0102] name: all environments children:

[0103] -name: online environment

[0104] children:

[0105] -name:prod

[0106] -name:pre

[0107] -name: offline environment

[0108] children:

[0109] -name:test

[0110] -name: dev

[0111] (3) Abnormal source:

[0112] name: All exception sources children:

[0113] -name: Infrastructure

[0114] children:

[0115] -name: network

[0116] -name: storage

[0117] -name: middleware

[0118] children:

[0119] -name: cache

[0120] children:

[0121] -name: Redis

[0122] -name: Memcached

[0123] -name: message queue

[0124] children:

[0125] -name: Kafka

[0126] -name: RabbitMQ

[0127] -name: Business service

[0128] children:

[0129] -name: User Center

[0130] -name: Order Center

[0131] -name: Product Center

[0132] (4) Key content of alarm log text: Due to the diversity of text content, it is difficult to predefine a complete inductive hierarchy. The following strategies can be adopted:

[0133] Predefine the inductive relationship of common exception types, such as:

[0134] name: all exceptions

[0135] children:

[0136] -name: Timeout exception

[0137] children:

[0138] -name: Connection timeout

[0139]

[0140] For text content that is not predefined, it can be summarized by gradually removing specific details, such as removing specific error codes, time values, etc.

[0141] (5) Fault location: For the code location, the package path structure can be directly used as the inductive hierarchy, such as:

[0142]

[0143] -name: UserService

[0144] -name: OrderService

[0145] -name: controller

[0146] children:

[0147] -name: UserController

[0148] -name: OrderController

[0149] 3. Dissimilarity calculation module

[0150] This module is responsible for defining and calculating the dissimilarity between alarm logs, providing a quantitative basis for clustering. The calculation of dissimilarity is based on a predefined inductive hierarchy, and the specific algorithm is as follows:

[0151] (1) For the dissimilarity of a single feature: Let x1 and x2 be two different values ​​of a feature, and their dissimilarity is defined as: d(x1, x2) = the length of the shortest path connecting x1 and x2 in the inductive hierarchy.

[0152] For example, for the computer room feature: d("server_room_a", "server_room_b") = 2 (passing through the "Beijing computer room" node) d("server_room_a", "server_room_c") = 4 (passing through the "Beijing computer room", "all computer rooms" and "Shanghai computer room" nodes).

[0153] (2) For the overall dissimilarity of two alarm logs a1 and a2: d(a1, a2) = Σd(a1[Ai], a2[Ai])(i = 1, 2, ..., n), where Ai represents the i-th feature and n is the total number of features.

[0154] Take the first and second logs as an example: d(server_room_a, server_room_b) = 2d(online, offline) = 2d(Thrift, Thrift) = 0d("query deal info error", "get dealProductType deal error") = 2d(com.example.service.DealService, com.example.service.ProductService) = 2Total dissimilarity = 2+2+0+2+2 = 8

[0155] 4. Adaptive Clustering Module

[0156] This module uses a heuristic algorithm to cluster alarm logs through an iterative induction process. The algorithm steps are as follows:

[0157] (1) Initialization:

[0158] The original alarm log set is recorded as L.

[0159] Set the minimum cluster size min_size (the recommended value is 1 / 5 of the total number of alarm logs).

[0160] Set the clustering stop condition: the number of alarm summaries of clustering results is ≤ 20 or the count value of a certain category has reached min_size.

[0161] (2) Iterative algorithm:

[0162] The iterative process is executed in a loop until the clustering stop condition set by the initialization is met.

[0163] In each iteration, one feature Ai is selected for processing. The feature selection strategy is described below. Usually, the feature with the most uniform value distribution is selected.

[0164] For the selected feature Ai, the algorithm traverses all alarm logs a.

[0165] For each alarm log a, its value on feature Ai is replaced by the value of the parent node of the value in the predefined inductive hierarchy. This step realizes the abstraction or generalization of feature values.

[0166] After processing all alarm logs, the algorithm calls the similarity calculation module to merge alarms with the same values ​​on all features. When the values ​​of two alarm logs on all features are abstracted to the same or close enough degree, they will be classified into one category and the count (count value) of each alarm combination will be updated, that is, the number of alarms in each alarm group.

[0167] Then enter the next iteration and repeat the above process.

[0168] The core idea of ​​this algorithm is to aggregate similar alarm logs together by gradually abstracting feature values. Through multiple iterations, a more reasonable clustering result can be obtained, which can reduce the number of alarms while retaining key information. Pseudo code:

[0169] While the stop condition is not met do

[0170] Select a feature Ai (see below for selection strategy)

[0171] for each alarm a in L do

[0172] Replace the Ai value of a with the value of its parent node in the induction hierarchy end for

[0173] Merge the same alarms and update the count value

[0174] end while

[0175] (3) Feature selection strategy:

[0176] Calculate the Fi value of each feature Ai: Fi = max{the number of alarms with value v on Ai|the range of v∈Ai};

[0177] Select the feature with the smallest Fi value for induction.

[0178] In this feature selection strategy, Fi is an indicator used to measure the distribution of feature Ai. Let me explain in detail:

[0179] Define Fi = max{number of alarms with value v on Ai|the range of v∈Ai}. This means that Fi is the number of occurrences of the value with the highest number of occurrences among all possible values ​​of feature Ai. Calculation process:

[0180] For feature Ai, count the number of times each possible value v appears in all alarms;

[0181] Find the maximum value among these times, and this maximum value is Fi.

[0182] Fi reflects the distribution concentration of feature Ai. The larger the Fi, the more frequently a certain value of feature Ai appears in the alarm, and the more uneven the distribution of the feature. Select the feature with the smallest Fi value for induction. Reasons for this selection strategy:

[0183] The feature with a small Fi value means that its value distribution is relatively uniform, with no particularly prominent dominant value;

[0184] Summarizing such features can more effectively reduce the types of alarms without over-generalizing to a specific value;

[0185] This helps to gradually reduce the number of alerts while maintaining information diversity.

[0186] For example, suppose there are two features:

[0187] Feature A: {value 1: 10 times, value 2: 8 times, value 3: 7 times};

[0188] Feature B: {value 1: 20 times, value 2: 3 times, value 3: 2 times};

[0189] Fi=10 for feature A and Fi=20 for feature B.

[0190] In this case, the algorithm will select feature A for induction because its Fi value is smaller, indicating that its value distribution is more uniform. Inducing feature A can better balance clustering effect and information retention.

[0191] This strategy helps to reach clustering conditions as quickly as possible while avoiding over-generalization and maintaining the diversity and representativeness of clustering results.

[0192] To illustrate this module, assume min_size = 2 and start iterative clustering:

[0193] First round: Select the abnormal source features to summarize the results:

[0194] All exception sources, server_room_a, online, query deal info error, com.example.service.DealService (count: 1)

[0195] All exception sources, server_room_b, offline, get deal ProductType deal error, com.example.service.ProductService (count: 1)

[0196] All exception sources, server_room_a, online, get deal ProductType deal error, com.example.service.ProductService (count: 1)

[0197] All exception sources, server_room_b, offline, query deal info error, com.example.service.DealService (count: 1)

[0198] All exception sources, server_room_a, offline, query deal info error, com.example.service.DealService (count: 1)

[0199] Second round: Select environmental features to summarize the results:

[0200] All exception sources, server_room_a, all environments, query deal info error, com.example.service.DealService (count: 2)

[0201] All exception sources, server_room_b, all environments, get deal ProductType deal error, com.example.service.ProductService (count: 1)

[0202] All exception sources, server_room_a, all environments, get deal ProductType deal error, com.example.service.ProductService (count: 1)

[0203] All exception sources, server_room_b, all environments, query deal info error, com.example.service.DealService (count: 1)

[0204] Round 3: Select the characteristics of the computer room and summarize the results:

[0205] All exception sources, all computer rooms, all environments, query deal info error, com.example.service.DealService (count: 3)

[0206] All exception sources, all computer rooms, all environments, get deal ProductType deal error, com.example.service.ProductService (count: 2)

[0207] At this point, the stopping condition has been met (the number of clustering results is ≤ 20 and the count value of the existing cluster is ≥ min_size), and clustering ends.

[0208] 5. Alarm summary generation module

[0209] This module is responsible for summarizing the clustering results and generating an alarm summary that can summarize the characteristics of each cluster. The alarm summary should contain the following information:

[0210] ID: unique identifier of the cluster;

[0211] Server Room: summarized server room information;

[0212] Error Source: The summarized abnormal source;

[0213] Environment: summarized environmental information;

[0214] Position: the location of the fault after induction;

[0215] Summary: The key contents of the alarm log text after summary;

[0216] Count: The number of raw alarms contained in this cluster.

[0217] For the example above, the resulting alarm summary is as follows:

[0218] ID: 1Server Room: All computer roomsError Source: All exception sourcesEnvironment: All environmentsPosition: com.example.service.DealServiceSummary: query deal info errorCount: 3

[0219] ID: 2Server Room: All computer roomsError Source: All exception sourcesEnvironment: All environmentsPosition: com.example.service.ProductServiceSummary: get deal ProductTypedeal errorCount: 2

[0220] Through this alarm summary, operation and maintenance personnel can quickly understand:

[0221] There are two main types of problems: errors in querying transaction information and errors in obtaining product types;

[0222] These problems occur in all computer rooms and in all environments;

[0223] The problem mainly involves two services: DealService and ProductService;

[0224] Errors in querying transaction information occur more frequently.

[0225] Such summary information can help operation and maintenance personnel quickly locate the key areas of the problem and improve troubleshooting efficiency.

[0226] Based on the same inventive concept, the present invention also proposes a log summary generation method based on an inductive hierarchical structure. The implementation of the method can refer to the implementation of the above system, and the repeated parts will not be repeated.

[0227] Figure 2 Schematic diagram of the log summary generation method based on the inductive hierarchical structure of the present invention. Figure 2 As shown, the method includes:

[0228] 1. Extract key features from the original alarm log, including the machine room, environment, abnormality source, key content of the alarm log text, and the location of the fault.

[0229] 2. Predefine an inductive hierarchy for each extracted feature as the basis for upward abstraction in the subsequent clustering process.

[0230] 3. Based on the inductive hierarchical structure, define and calculate the dissimilarity between alarm logs to provide a quantitative basis for clustering;

[0231] The steps for calculating dissimilarity are as follows:

[0232] For the dissimilarity of a single feature: Let x1, x2 be two different values ​​of a feature, and their dissimilarity is defined as: d(x1, x2) = the length of the shortest path connecting x1 and x2 in the inductive hierarchy;

[0233] For the overall dissimilarity of two alarm logs a1 and a2: d(a1, a2) = Σd(a1[Ai], a2[Ai])(i=1, 2, ..., n), where Ai represents the i-th feature and n is the total number of features.

[0234] 4. Use the heuristic algorithm to gradually abstract the feature values ​​upward through an iterative induction process, and cluster the alarm logs that are similar after abstraction until the preset clustering stop conditions are met, including:

[0235] Initialization: record the original alarm log set as L, set the minimum cluster size min_size, and set the clustering stop condition: the number of alarm summaries in the clustering result is ≤20 or the count value of a certain category has reached min_size;

[0236] Iterative algorithm: The iterative process is executed in a loop until the clustering stop condition set by the initialization is met; in each iteration, a feature Ai with the most uniform current value distribution is selected for processing; for the selected feature Ai, the algorithm traverses all alarm logs a; for each alarm log a, its value on feature Ai is replaced with the parent node value of the value in the predefined inductive hierarchy; after completing the processing of all alarm logs, the algorithm calls the similarity calculation module, merges the alarm logs with the same value on all features, and updates the count value of each alarm log combination; then enters the next iteration and repeats the above process;

[0237] The feature selection strategy is as follows:

[0238] Calculate the Fi value of each feature Ai: Fi = max{the number of alarms with value v on Ai|the range of v∈Ai}; Fi is an indicator used to measure the distribution of feature Ai;

[0239] Select the feature with the smallest Fi value for induction.

[0240] 5. Summarize the clustering results and generate an alarm summary that can summarize the characteristics of each cluster.

[0241] It should be noted that, although the operations of the method of the present invention are described in a specific order in the above embodiments and the accompanying drawings, this does not require or imply that the operations must be performed in the specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0242] Based on the above invention concept, Figure 3As shown, the present invention also proposes a computer device 200, including a memory 210, a processor 220, and a computer program 230 stored in the memory 210 and executable on the processor 220. When the processor 220 executes the computer program 230, the aforementioned log summary generation method based on the inductive hierarchical structure is designed.

[0243] Based on the aforementioned inventive concept, the present invention further proposes a computer-readable storage medium, which stores a computer program for executing the aforementioned log summary generation method based on an inductive hierarchical structure.

[0244] The log summary generation system and method based on the inductive hierarchical structure proposed in the present invention have the following advantages:

[0245] 1. Through the predefined induction hierarchy, effective clustering of alarm logs can be achieved while retaining important details.

[0246] 2. Adopting adaptive clustering algorithm, there is no need to predetermine the number of clusters, which is suitable for dynamically changing alarm scenarios.

[0247] 3. The clustering results are highly interpretable, making it easy for operation and maintenance personnel to quickly understand and locate the root cause of the problem.

[0248] 4. It can effectively process a large number of complex alarm logs and significantly improve the efficiency of fault location and root cause analysis.

[0249] Although the spirit and principle of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements contained in the spirit and scope of the attached claims.

[0250] Regarding the limitation of the protection scope of the present invention, those skilled in the art should understand that, based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the protection scope of the present invention.

Claims

1. A log summary generation system based on an inductive hierarchical structure, characterized in that: The system includes: Feature extraction module, used to extract key features from the original alarm log, including the machine room, environment, abnormal source, key content of the alarm log text and the location of the fault; The inductive hierarchy building module is used to predefine an inductive hierarchy for each extracted feature as the basis for upward abstraction in the subsequent clustering process; The dissimilarity calculation module is used to define and calculate the dissimilarity between alarm logs based on the inductive hierarchy, providing a quantitative basis for clustering; The adaptive clustering module is used to use a heuristic algorithm to gradually abstract the feature values ​​upward through an iterative induction process, and cluster alarm logs that are similar after abstraction until a preset clustering stop condition is met; The alarm summary generation module is used to summarize the clustering results and generate an alarm summary that can summarize the characteristics of each cluster.

2. The log summary generation system based on the inductive hierarchical structure according to claim 1 is characterized in that: The calculation steps of the dissimilarity are as follows: For the dissimilarity of a single feature: Let x1, x2 be two different values ​​of a feature, and their dissimilarity is defined as: d(x1, x2) = the length of the shortest path connecting x1 and x2 in the inductive hierarchy; For the overall dissimilarity of two alarm logs a1 and a2: d(a1, a2) = Σd(a1[Ai], a2[Ai])(i=1, 2, ..., n), where Ai represents the i-th feature and n is the total number of features.

3. The log summary generation system based on the inductive hierarchical structure according to claim 1 is characterized in that: Using a heuristic algorithm, the feature values ​​are gradually abstracted upward through an iterative induction process, and alarm logs that are similar after abstraction are clustered until the preset clustering stop conditions are met, including: Initialization: record the original alarm log set as L, set the minimum cluster size min_size, and set the clustering stop condition: the number of alarm summaries in the clustering result is ≤20 or the count value of a certain category has reached min_size; Iterative algorithm: The iterative process is executed in a loop until the clustering stop condition set in the initialization is met; in each iteration, a feature Ai with the most uniform current value distribution is selected for processing; for the selected feature Ai, the algorithm traverses all alarm logs a; for each alarm log a, its value on feature Ai is replaced with the parent node value of the value in the predefined inductive hierarchy; after completing the processing of all alarm logs, the algorithm calls the similarity calculation module, merges the alarm logs with the same values ​​on all features, and updates the count value of each alarm log combination; then enters the next iteration and repeats the above process.

4. The log summary generation system based on the inductive hierarchical structure according to claim 3 is characterized in that: The feature selection strategy is as follows: Calculate the Fi value of each feature Ai: Fi = max{the number of alarms with value v on Ai|the range of v∈Ai}; Fi is an indicator used to measure the distribution of feature Ai; Select the feature with the smallest Fi value for induction.

5. A log summary generation method based on an inductive hierarchical structure, characterized in that: The method includes: Extract key features from the original alarm log, including the machine room, environment, abnormal source, key content of the alarm log text, and the location of the fault; Predefine an inductive hierarchy for each extracted feature as the basis for upward abstraction in the subsequent clustering process; Based on the inductive hierarchical structure, the dissimilarity between alarm logs is defined and calculated to provide a quantitative basis for clustering; Using a heuristic algorithm, the feature values ​​are gradually abstracted upward through an iterative induction process, and alarm logs that are similar after abstraction are clustered until the preset clustering stop condition is met; The clustering results are summarized to generate an alarm summary that can summarize the characteristics of each cluster.

6. The log summary generation method based on the inductive hierarchical structure according to claim 5 is characterized in that: The calculation steps of the dissimilarity are as follows: For the dissimilarity of a single feature: Let x1, x2 be two different values ​​of a feature, and their dissimilarity is defined as: d(x1, x2) = the length of the shortest path connecting x1 and x2 in the inductive hierarchy; For the overall dissimilarity of two alarm logs a1 and a2: d(a1, a2) = Σd(a1[Ai], a2[Ai])(i=1, 2, ..., n), where Ai represents the i-th feature and n is the total number of features.

7. The log summary generation method based on the inductive hierarchical structure according to claim 5 is characterized in that: Using a heuristic algorithm, the feature values ​​are gradually abstracted upward through an iterative induction process, and alarm logs that are similar after abstraction are clustered until the preset clustering stop conditions are met, including: Initialization: record the original alarm log set as L, set the minimum cluster size min_size, and set the clustering stop condition: the number of alarm summaries in the clustering result is ≤20 or the count value of a certain category has reached min_size; Iterative algorithm: The iterative process is executed in a loop until the clustering stop condition set in the initialization is met; in each iteration, a feature Ai with the most uniform current value distribution is selected for processing; for the selected feature Ai, the algorithm traverses all alarm logs a; for each alarm log a, its value on feature Ai is replaced with the parent node value of the value in the predefined inductive hierarchy; after completing the processing of all alarm logs, the algorithm calls the similarity calculation module, merges the alarm logs with the same values ​​on all features, and updates the count value of each alarm log combination; then enters the next iteration and repeats the above process.

8. The log summary generation method based on the inductive hierarchical structure according to claim 7 is characterized in that: The feature selection strategy is as follows: Calculate the Fi value of each feature Ai: Fi = max{the number of alarms with value v on Ai|the range of v∈Ai}; Fi is an indicator used to measure the distribution of feature Ai; Select the feature with the smallest Fi value for induction.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 5 to 8 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the method according to any one of claims 5 to 8.

Images