A log analysis method, device, equipment and readable storage medium
By extracting and classifying feature words from logs, the systematization problem of log analysis is solved, enabling fast and accurate log querying and classification, improving operational efficiency, and providing a corpus for automated operations and maintenance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- JINAN INSPUR DATA TECH CO LTD
- Filing Date
- 2022-07-27
- Publication Date
- 2026-05-29
AI Technical Summary
In existing technologies, log analysis lacks systematic analysis methods, which makes it time-consuming and laborious for operation and maintenance personnel to review and use monitoring logs.
By collecting logs, extracting feature words to form a feature word set, and calculating the mapping and distance between the feature word set and the seed set, the logs are classified into the corresponding seed classes. The aggregation and classification of logs are achieved by using hash string calculation and centroid optimization.
It enables fast and accurate classification and querying of logs, reducing the time and effort required for operations and maintenance personnel to review logs, improving operational efficiency, and providing corpus support for automated operations and maintenance.
Smart Images
Figure CN115186663B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of log analysis technology, and more specifically, to a log analysis method, apparatus, device, and readable storage medium. Background Technology
[0002] In today's era of big data and cloud computing, data centers and server rooms are of paramount importance. Monitoring server rooms is crucial for ensuring data security. Traditional server room monitoring includes both proactive and reactive methods, such as daily inspections, automated script-based maintenance, cloud platform monitoring, and monitoring log collection. Among these, daily inspections, automated maintenance, and platform monitoring have become increasingly mature, with many established industry standards and specifications.
[0003] However, monitoring logs not only contain fault information that operations and maintenance personnel care about, but also a lot of useless or low-priority information. Therefore, there is currently a lack of systematic analysis and use of monitoring logs, which makes it time-consuming and laborious for operations and maintenance personnel to review them.
[0004] In conclusion, how to analyze logs so that they can be reviewed and used by operations and maintenance personnel is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0005] In view of this, the purpose of this application is to provide a log analysis method, apparatus, device and readable storage medium for analyzing logs so that operation and maintenance personnel can view and use them.
[0006] To achieve the above objectives, this application provides the following technical solution:
[0007] A log analysis method, comprising:
[0008] Log collection;
[0009] Feature words are extracted from each of the logs to obtain the feature words in each of the logs;
[0010] The feature words in each log are used to construct a feature word set for each log.
[0011] Based on the feature word set of each log and the seed set containing multiple seeds, the seed corresponding to each log is determined, and each log is assigned to the class where the seed corresponding to each log belongs.
[0012] Preferably, based on the feature word set of each log and a seed set containing multiple seeds, the seed corresponding to each log is determined, and each log is assigned to the class of the seed corresponding to that log, including:
[0013] Map the feature word set of each log to the first hash string corresponding to each log, and map each seed in the seed set to the second hash string corresponding to each seed;
[0014] Calculate the distance between each log and each seed based on the first hash string corresponding to each log and the second hash string corresponding to each seed.
[0015] The seed closest to each log is determined as the seed corresponding to the corresponding log, and each log is assigned to the class of the corresponding seed.
[0016] Preferably, after determining the seed closest to each log as the seed corresponding to the corresponding log, and classifying each log into the class of the corresponding seed, the method further includes:
[0017] Based on the first hash string corresponding to the log contained in the class where each seed is located, calculate the third hash string of the corresponding center point of the class.
[0018] Based on the first hash string corresponding to each log and the third hash string of each center point, calculate the distance between each log and each center point;
[0019] Determine whether each of the center points corresponds to an existing seed in the seed set; if there is a center point that does not correspond to an existing seed in the seed set, then determine a new seed based on the third hash string of the center point and add the new seed to the seed set.
[0020] The seed corresponding to the center point closest to each log is determined as the seed of the corresponding log, and each log is assigned to the class of the corresponding seed.
[0021] Determine whether the classification result of the current classification of each log is consistent with the classification result of the previous classification. If they are inconsistent, return to the step of calculating the third hash string of the corresponding center point based on the first hash string of the log contained in the class of each seed, until the classification result of the current classification of each log is consistent with the classification result of the previous classification.
[0022] Preferably, feature words are extracted from each of the logs to obtain the feature words in each of the logs, including:
[0023] Each log entry is segmented into words to obtain the words contained in each log entry.
[0024] Calculate the word frequency and importance of each word contained in each of the logs;
[0025] The weight of each word in each log is calculated based on the word frequency and importance of each word contained in each log.
[0026] Words with a weight greater than the weight threshold in each log are used as feature words in the corresponding log.
[0027] Preferably, calculating the importance of each of the words contained in each of the logs includes:
[0028] use Calculate the importance I of the i-th word contained in the log. i ;
[0029] Where D is the total number of logs collected in this instance, D j Let j be the j-th log entry collected this time, where {j:i∈D} j} represents the number of logs in this collection that contain the i-th word.
[0030] Preferably, the weight of each word in each log is calculated based on the word frequency and importance of each word contained in each log, including:
[0031] Use IF i =F i *I i Calculate the weight IF of the i-th word contained in the log. i Among them, F i Let be the word frequency of the i-th word contained in the log.
[0032] Preferably, after assigning each log to the class containing the seed corresponding to that log, the method further includes:
[0033] The logs, the feature word set of the logs, and the classification results of the logs are stored in the database.
[0034] A log analysis device, comprising:
[0035] The data collection module is used to collect logs.
[0036] The extraction module is used to extract feature words from each of the logs to obtain the feature words in each of the logs;
[0037] A composition module is used to construct a feature word set for each log by corresponding feature words in each log.
[0038] The determination module is used to determine the seed corresponding to each log based on the feature word set of each log and the seed set containing multiple seeds, and to classify each log into the class where the seed corresponding to each log belongs.
[0039] A log analysis device, comprising:
[0040] Memory, used to store computer programs;
[0041] A processor for executing the computer program to implement the steps of the log analysis method as described in any of the preceding claims.
[0042] A readable storage medium storing a computer program that, when executed by a processor, implements the steps of the log analysis method as described in any of the preceding claims.
[0043] This application provides a log analysis method, apparatus, device, and readable storage medium. The method includes: collecting logs; extracting feature words from each log to obtain feature words in each log; constructing a feature word set for each log by corresponding feature words in each log; determining the seed corresponding to each log based on the feature word set of each log and a seed set containing multiple seeds, and classifying each log into the class where the seed corresponding to each log belongs.
[0044] The technical solution disclosed in this application extracts feature words from the collected logs to obtain the feature words in each log, and constructs a feature word set for each log based on the keywords in each log, thereby extracting key information from the logs. Then, based on the feature word set of each log and a seed set containing multiple seeds located in different classes, the seed corresponding to each log is determined, and each log is assigned to the class of its corresponding seed, thereby achieving aggregation and classification of the collected logs. Through the above process, key information and the category to which the logs belong can be obtained from the collected logs, enabling operation and maintenance personnel to query and use logs faster and better, reducing the time and effort spent by operation and maintenance personnel in querying and using logs, and improving the efficiency and accuracy of daily operation and maintenance. In addition, the results of log analysis can also serve as a corpus for machine learning related to automated operation and maintenance. Attached Figure Description
[0045] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0046] Figure 1 A flowchart of a log analysis method provided in an embodiment of this application;
[0047] Figure 2 A flowchart illustrating another log analysis method provided in this application embodiment;
[0048] Figure 3 A flowchart for classifying logs provided in this application embodiment;
[0049] Figure 4 This is a schematic diagram of the structure of a log analysis device provided in an embodiment of this application;
[0050] Figure 5 This is a schematic diagram of the structure of a log analysis device provided in an embodiment of this application. Detailed Implementation
[0051] The core of this application is to provide a log analysis method, apparatus, device, and readable storage medium for analyzing logs so that maintenance personnel can review and use them.
[0052] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0053] See Figure 1 and Figure 2 ,in, Figure 1 The flowchart shown is a log analysis method provided in an embodiment of this application. Figure 2 A flowchart of another log analysis method provided in an embodiment of this application is shown. The log analysis method provided in this application may include:
[0054] S11: Collect logs.
[0055] In this application, logs from various devices can be collected first. Specifically, the devices mentioned here can be data center devices (i.e., devices in the data center that need log analysis) or devices in other scenarios that need log analysis.
[0056] During log collection, the log analysis device can either actively collect logs or passively collect them (for example, the device can send logs, and then the log analysis device collects them). Furthermore, the log analysis device can collect logs in real-time or at regular intervals and store the collected logs.
[0057] For log analysis devices, a data acquisition module can be deployed to execute step S11. This data acquisition module can be deployed in various ways and with different components. Specifically, rsyslog+elasticsearch or rsyslog+logstash+elasticsearch can be deployed for log collection. rsyslog is a Linux-based logging tool (specific configuration files can be configured within rsyslog to collect logs according to the specified format, thus performing preliminary processing and formatting). Elasticsearch is a distributed full-text search engine and database (used here to store the collected logs). Logstash is a pipeline with real-time data transmission capabilities, commonly used for log collection. If retrieval of the collected logs is required, Kibana (an Elasticsearch visualization tool) can be added. Furthermore, for log collection of different scales, single-node or distributed collection clusters can be deployed in a distributed manner.
[0058] S12: Extract feature words from each log entry to obtain the feature words in each log entry.
[0059] Based on step S11, the log analysis device can extract feature words from each log to obtain the feature words in each log. Specifically, the log analysis device can periodically obtain a batch of logs from the collected logs and extract feature words from each log in this batch. That is, the log analysis device processes and analyzes a batch of logs collected within a preset time length (corresponding to the periodicity) each time. In other words, the log analysis device can maintain a task queue to periodically obtain logs from the data collection module for processing and analysis.
[0060] Before extracting feature words from each log entry, the log analysis device can also clean the logs to remove incomplete, erroneous, or duplicate data. This prevents dirty data from affecting feature word extraction and improves its accuracy. For example, if a log entry is missing key information, using the missing field for feature word extraction might lead to inaccurate or even erroneous calculations, so it needs to be removed. Similarly, if a log entry was collected twice, it also needs to be removed.
[0061] The log analysis device may include a topic analysis module, which cleans, performs semantic analysis (the purpose of semantic analysis is to extract feature words) and extracts feature words from the collected logs to obtain the feature words in the logs.
[0062] S13: Construct a feature word set for each log by matching the feature words in each log.
[0063] Based on step S12, the log analysis device can construct a feature word set for each log entry by correspondingly constructing the feature words in each log entry.
[0064] For example, suppose there is a log entry with the original content: "Server memory failure! Source: 192.168.0.1, memory name: DIMM1, slot: slot1, cause of failure: The server memory detected an uncorrectable anomaly!", then after step S12, the characteristic words in this log entry are: server, memory, failure, and these characteristic words in the log entry can be used to form the characteristic word set {"server", "memory", "failure"} of this log entry.
[0065] Steps S12 and S13 can extract key information from the device logs, so that maintenance personnel can obtain the key information from the logs without spending a lot of time and effort themselves.
[0066] S14: Based on the feature word set of each log and the seed set containing multiple seeds, determine the seed corresponding to each log and classify each log into the class corresponding to its seed.
[0067] When performing log analysis, the log analysis device can first initialize a seed set. This seed set contains common alarm events and device information (i.e., the original alarm list of the devices), and can write this seed set into memory for log analysis. The seed set contains multiple seeds, each corresponding to a class. For example, an initialized seed set might be: {"Server CPU failure", "Server memory failure", "Server temperature exceeds threshold", "Switch under network attack", "Switch port DOWN"}. Here, "Server CPU failure", "Server memory failure", "Server temperature exceeds threshold", "Switch under network attack", and "Switch port DOWN" are each a seed in the seed set, and each seed represents a class. Taking the seed "Server CPU failure" as an example, it represents the class "Server CPU failure".
[0068] Based on step S13, the log analysis device can determine the seed corresponding to each log in the seed set according to the feature word set of each log and the seed set containing multiple seeds, and classify each log into the class of the seed corresponding to each log, so as to realize the aggregation analysis of logs, that is, to divide logs into different classes.
[0069] Specifically, the log analysis device may include a topic aggregation module, which performs aggregation analysis on the feature word set generated by the topic analysis module, classifying the logs collected by the data acquisition module into multiple categories. During aggregation analysis, a set of keywords needs to be prepared in advance as seeds for classification. Then, semantic similarity is calculated based on these seeds, and logs are categorized into the closest seed, forming a class.
[0070] Step S14 aggregates and categorizes logs according to multiple dimensions to filter out important information. This allows operations and maintenance (O&M) personnel to query and use logs more quickly and efficiently, reducing the time and effort spent on log retrieval and improving the efficiency and accuracy of daily O&M operations. Furthermore, the results of log analysis can serve as a corpus for machine learning related to automated O&M.
[0071] As can be seen from the above process, this application extracts key information from the device logs and aggregates and classifies them according to multiple dimensions to filter out important information, so that it can be used by operation and maintenance personnel for retrieval or as a corpus for machine learning related to automated operation and maintenance.
[0072] The technical solution disclosed in this application extracts feature words from the collected logs to obtain the feature words in each log, and constructs a feature word set for each log based on the keywords in each log, thereby extracting key information from the logs. Then, based on the feature word set of each log and a seed set containing multiple seeds located in different classes, the seed corresponding to each log is determined, and each log is assigned to the class of its corresponding seed, thereby achieving aggregation and classification of the collected logs. Through the above process, key information and the category to which the logs belong can be obtained from the collected logs, enabling operation and maintenance personnel to query and use logs faster and better, reducing the time and effort spent by operation and maintenance personnel in querying and using logs, and improving the efficiency and accuracy of daily operation and maintenance. In addition, the results of log analysis can also serve as a corpus for machine learning related to automated operation and maintenance.
[0073] This application provides a log analysis method that, based on the feature word set of each log and a seed set containing multiple seeds, determines the seed corresponding to each log and assigns each log to the class containing its corresponding seed. This method may include:
[0074] Map the feature word set of each log to the first hash string corresponding to each log, and map the various sub-seeds in the seed set to the second hash string corresponding to each sub-seed.
[0075] Based on the first hash string corresponding to each log and the second hash string corresponding to each sub-log, calculate the distance between the feature word set of each log and each seed;
[0076] The seed closest to each log is determined as the seed corresponding to that log, and each log is assigned to the class corresponding to its seed.
[0077] In this application, the specific process of determining the seed corresponding to each log based on the feature word set of each log and the seed set containing multiple seeds, and classifying each log into the class containing its corresponding seed, is as follows, and details can be found in [link to relevant documentation]. Figure 3 It illustrates a flowchart of log classification provided in an embodiment of this application:
[0078] Step 1: Perform hash mapping (a general term for a type of mapping algorithm that can convert data of arbitrary length into data of fixed length) on the feature word set of each log in a batch of logs obtained periodically, to obtain the first hash string corresponding to each log; for example: suppose there is a log with the original content: "Server memory failure! Source: 192.168.0.1, memory name: DIMM1, slot: slot1, failure reason: server memory detected an uncorrectable anomaly!", extract the feature word set of this log as {"server", "memory", "failure"}, merge the feature word sets, and then perform hash mapping (the specific hash algorithm can be freely chosen), to obtain the corresponding and computable first hash string as 101001; it should be noted that the actual hash string obtained by mapping may not be such a simple 01 string, this application uses the 01 string as an example only for the purpose of simplifying the calculation and making it easier to understand;
[0079] Step 2: Map each seed in the seed set to a corresponding and computable second hash string (the specific hash algorithm is the same as the hash algorithm used in the feature word set of the mapping log above); for example, after hash mapping the seed "server memory failure", the corresponding computable second hash string is 101001;
[0080] Step 3: Based on the first hash string corresponding to each log obtained in Step 1 and the second hash string corresponding to each seed in the seed set obtained in Step 2, calculate the distance between each log and each seed in the seed set. Specifically, using... Calculate the distance between each log entry and each seed in the seed set; where X represents the seed point (i.e., the seed in the seed set), Y represents the clustered point (i.e., the log), i represents the coordinate of the i-th dimension of the hash string, and n represents the number of dimension coordinates of the hash string. Taking the second hash string corresponding to the aforementioned seed "server memory failure" as 101001, and the log's feature word set as {"server", "memory", "failure"}, and the first hash string corresponding to this log entry as 101001, the distance between this log entry and the aforementioned seeds can be obtained as follows: That is, the distance between the above log and the seed point "server memory failure" is 0;
[0081] Step 4: Based on the distance between each log and each seed calculated in Step 3, determine the seed with the closest distance to each log as the seed corresponding to that log (that is, the seed corresponding to each log is the seed with the distance to that log), and classify each log into the class of the corresponding seed; taking the log mentioned above as an example, we can find that the seed with the closest distance is "server memory failure", so we can classify the log into the class of the seed "server memory failure" (specifically, the seed "server memory failure" represents the class "server memory failure").
[0082] The above method allows each log to be categorized into the class corresponding to the seed contained in the seed set, thereby enabling the classification and aggregation of logs and making it easier for operations and maintenance personnel to intuitively obtain the category to which each log belongs.
[0083] The log analysis method provided in this application, after determining the seed closest to each log as the seed corresponding to the corresponding log, and classifying each log into the class of the corresponding seed, may further include:
[0084] Based on the first hash string corresponding to the log contained in the class of each subclass, calculate the third hash string of the corresponding center point of the class.
[0085] Calculate the distance between each log and each center point based on the first hash string corresponding to each log and the third hash string of each center point.
[0086] Determine whether each center point corresponds to an existing seed in the seed set; if there is a center point that does not correspond to an existing seed in the seed set, determine a new seed based on the third hash string of the center point and add the new seed to the seed set.
[0087] The seed corresponding to the center point closest to each log is determined as the seed of the corresponding log, and each log is assigned to the class of the corresponding seed.
[0088] Determine whether the classification result of the current classification of each log is consistent with the classification result of the previous classification. If they are inconsistent, return to the step of calculating the third hash string of the corresponding center point based on the first hash string of the logs contained in each subclass, until the classification result of the current classification of each log is consistent with the classification result of the previous classification.
[0089] In this application, after determining the seed closest to each log as the seed corresponding to the corresponding log, and classifying each log into the class containing the corresponding seed, the following process can also be performed:
[0090] Step 5: Calculate the third hash string of the corresponding center point for each class based on the first hash strings of all logs contained in each subclass. That is, for the current class, the third hash string of the center point can be calculated using the first hash strings of all logs contained in the current class. In other words, once all logs in a periodically acquired batch have been categorized, the center point can be recalculated for each class. The center point mentioned here is the new center point corresponding to the class (the old center point corresponding to the class is the corresponding seed in the seed set). Specifically, this can be achieved using... Where p is the total number of logs contained in this class. For the m-th log entry in this class, the n-th... i The coordinate values of each dimension (specifically, the nth coordinate in the first hash string corresponding to the log). i (values), For the nth center point of the calculated class i The coordinate values of each dimension are used to calculate the dimension coordinates of the center point corresponding to that class. That is, the third hash string of the center point corresponding to this class is obtained as follows:
[0091] Step 6: Based on the first hash string corresponding to each log obtained in Step 1 and the third hash string of each center point obtained in Step 5, calculate the distance between each log and each center point. The specific calculation method is the same as that in Step 3.
[0092] Step 7: Based on the third hash string of the centroids of each category calculated in Step 5, determine whether each centroid corresponds to an existing seed in the seed set. Specifically, this can be determined by checking if the third hash string of the centroids of each category calculated in Step 5 is the same as the second hash string corresponding to a seed in the seed set. If the third hash string of a centroid is the same as the second hash string corresponding to a seed in the seed set, then the centroid corresponds to an existing seed in the seed set. If the third hash string of a centroid is the same as the second hash string corresponding to any seed in the seed set, then the centroid does not correspond to an existing seed in the seed set. If it is determined that there is a centroid whose third hash string calculated in Step 5 does not correspond to an existing seed in the seed set, then based on the third hash string of the centroid and the first hash string corresponding to each log, determine the log closest to the centroid, and use the feature word set corresponding to the log closest to the centroid as a new centroid, and add the new seed to the seed set.
[0093] Step 8: After steps 7 and 8, determine the seed corresponding to the center point closest to each log as the seed of the corresponding log, and classify each log into the class of its corresponding seed.
[0094] Step 9: After classifying all logs in a periodically acquired batch, determine whether the classification result of the current classification of all logs in the batch is consistent with the classification result of the previous classification of all logs in the batch. If they are consistent, end the calculation and obtain the final classification result. If they are inconsistent, return to the step of calculating the third hash string of the corresponding center point based on the first hash string of the logs contained in each subclass. That is, return to steps 5-9 until the classification result of the current classification of all logs in the batch is consistent with the classification result of the previous classification of all logs in the batch.
[0095] As can be seen from the above process, the topic aggregation module in the log analysis module can recursively calculate each class and continuously optimize the seed until it becomes stable.
[0096] By recalculating the center point corresponding to the class as described above, new seeds can be added to the seed set, making the seed set richer. This allows for more accurate classification of subsequent batches of logs, enabling operations and maintenance personnel to obtain more accurate log classification results.
[0097] This application provides a log analysis method that extracts feature words from each log to obtain the feature words in each log, which may include:
[0098] Each log entry is segmented into words to obtain the words contained in each log entry;
[0099] Calculate the word frequency and importance of each word in each log entry;
[0100] Calculate the weight of each word in each log based on the word frequency and importance of each word.
[0101] Words with a weight greater than the weight threshold in each log are used as feature words in the corresponding log.
[0102] In this application, the specific implementation process for extracting feature words from each log is as follows: First, each log is segmented (specifically, the log text is segmented) to obtain the words contained in each log; then, for each log, the word frequency of each word contained in each log is calculated. Specifically, the word frequency of each word is obtained by dividing the number of times each word appears in the corresponding log by the total number of words in the corresponding log. Get the word frequency F of the i-th word in the current log. i , among which, T i Let C be the number of times the i-th word appears in the current log, and C be the total number of words in the current log. In addition, the importance of each word in each log is calculated. Then, for each word in each log, the weight of the word is calculated based on the word frequency and the word importance. That is, the weight of each word is calculated based on the word frequency and importance of each word in each log. Finally, the weight of each word in each log is compared with the weight threshold (the weight threshold is set in advance according to the actual situation or experience), and the words in each log with a weight greater than the weight threshold are taken as the feature words in the corresponding log.
[0103] The above process can accurately obtain the feature words in each log, so as to filter out the key information contained in the log. This makes it easier for operation and maintenance personnel to accurately obtain the key information in the log and to perform high-precision log classification based on the key information. This improves the convenience for operation and maintenance personnel to perform equipment operation and maintenance based on log analysis results, and also makes it easier to serve as a corpus for machine learning related to automated operation and maintenance.
[0104] This application provides a log analysis method that calculates the importance of each word in each log entry, which may include:
[0105] use Calculate the importance of the i-th word in the log. i ;
[0106] Where D is the total number of logs collected in this instance, D j Let j be the j-th log entry collected this time, where {j:i∈D} j} represents the number of logs in this collection that contain the i-th word.
[0107] In this application, specific uses can be made Calculate the importance of the i-th word in the log. i Where D is the total number of logs collected this time, that is, the number of log entries contained in a batch of logs obtained periodically. j Let j be the j-th log entry collected this time, which is also the j-th log entry in a batch of logs obtained periodically, where {j:i∈D} j} represents the number of log entries containing the i-th word in the current log collection. For example, for the word "server" in the current log entry, {j:i∈D} j} represents the number of logs containing the word "server" in this collection, which is also the number of logs collected using... Calculate the importance of the current words contained in the log.
[0108] The above method can accurately calculate the importance of each word in each log entry, thereby improving the accuracy of the weight calculation for each word in each log entry.
[0109] This application provides a log analysis method that calculates the weight of each word in each log based on the word frequency and importance of each word contained in each log. This method may include:
[0110] Use IF i =F i *I i Calculate the weight IF of the i-th word contained in the log. i Among them, F i Let be the word frequency of the i-th word contained in the log.
[0111] In this application, IF can be specifically used i =F i *I i Calculate the weight IF of the i-th word contained in each log. i , of which F i Let I be the word frequency of the i-th word contained in the log. i The importance of the i-th word contained in the log.
[0112] The above method can accurately calculate the weight of each word in each log, thereby facilitating the identification and extraction of feature words in each log, and enabling accurate acquisition of key information from the log.
[0113] The log analysis method provided in this application, after classifying each log into the class containing the corresponding seed, may further include:
[0114] Logs, log feature sets, and log classification results are stored in the database.
[0115] In this application, after each log is categorized into the class corresponding to its respective seed, formatted log information is obtained. At this point, the logs, the log feature word set, and the log classification results can be stored in the database to achieve persistent storage of this information. Additionally, the weight information of the feature words calculated during feature word extraction can also be stored in the database.
[0116] Specifically, the log analysis module may include a data persistence module, which can store the results of the categorized analysis into a database to achieve the persistence of this information.
[0117] In addition, the log analysis device also includes a data retrieval module, which provides a retrieval interface that allows users to search by the weight of feature words in the logs, the classification results of the logs, or the basic information of the logs (i.e., the collected logs, i.e., the original information of the logs), so that maintenance personnel can obtain the log information they need.
[0118] This application also provides a log analysis device, see [link to documentation]. Figure 4 It shows a schematic diagram of the structure of a log analysis device provided in an embodiment of this application, which may include:
[0119] Acquisition module 41 is used to collect logs;
[0120] Extraction module 42 is used to extract feature words from each log to obtain the feature words in each log.
[0121] Module 43 is used to construct a feature word set for each log by matching the feature words in each log.
[0122] The determination module 44 is used to determine the seed corresponding to each log based on the feature word set of each log and the seed set containing multiple seeds, and to classify each log into the class where the seed corresponding to each log belongs.
[0123] This application provides a log analysis device, wherein the determining module 44 may include:
[0124] The mapping unit is used to map the feature word set of each log into the first hash string corresponding to each log, and to map the various sub-seeds in the seed set into the second hash string corresponding to each sub-seed.
[0125] The first calculation unit is used to calculate the distance between the feature word set of each log and each seed based on the first hash string corresponding to each log and the second hash string corresponding to each sub-log.
[0126] The first determining unit is used to determine the seed closest to each log as the seed corresponding to the corresponding log, and to classify each log into the class to which the corresponding seed belongs.
[0127] The log analysis device provided in this application embodiment may further include a determination module 44 comprising:
[0128] The second calculation unit is used to determine the seed closest to each log as the seed corresponding to the corresponding log, and classify each log into the class to which the corresponding seed belongs, and then calculate the third hash string of the center point corresponding to the class according to the first hash string of the logs contained in each class.
[0129] The third calculation unit is used to calculate the distance between each log and each center point based on the first hash string corresponding to each log and the third hash string of each center point.
[0130] The first judgment unit is used to determine whether each center point corresponds to an existing seed in the seed set; if there is a center point that does not correspond to an existing seed in the seed set, a new seed is determined based on the third hash string of the center point and the new seed is added to the seed set.
[0131] The second determining unit is used to determine the seed corresponding to the center point closest to each log as the seed corresponding to the corresponding log, and to classify each log into the class where the corresponding seed belongs.
[0132] The second judgment unit is used to determine whether the classification result of the current classification of each log is consistent with the classification result of the previous classification. If they are inconsistent, the unit returns to the step of calculating the third hash string of the corresponding center point based on the first hash string of the logs contained in the class of each subclass, until the classification result of the current classification of each log is consistent with the classification result of the previous classification.
[0133] This application provides a log analysis device, wherein the extraction module 42 may include:
[0134] The word segmentation unit is used to segment each log entry into words to obtain the words contained in each log entry.
[0135] The fourth calculation unit is used to calculate the word frequency and importance of each word contained in each log.
[0136] The fifth calculation unit is used to calculate the weight of each word in each log based on the word frequency and importance of each word contained in each log.
[0137] As a unit, it is used to identify words in each log that have a weight greater than a weight threshold as feature words in the corresponding log.
[0138] This application provides a log analysis device, wherein the fourth computing unit may include:
[0139] The first computational subunit is used to utilize... Calculate the importance of the i-th word in the log. i ;
[0140] Where D is the total number of logs collected in this instance, D j Let j be the j-th log entry collected this time, where {j:i∈D} j} represents the number of logs in this collection that contain the i-th word.
[0141] This application provides a log analysis device, wherein the fifth computing unit may include:
[0142] The second calculation subunit is used to utilize IF i =F i *I i Calculate the weight IF of the i-th word contained in the log. i Among them, F i Let be the word frequency of the i-th word contained in the log.
[0143] The log analysis device provided in this application embodiment may further include:
[0144] The storage module is used to store the logs, the log feature word set, and the log classification results into the database after classifying each log into the class corresponding to its seed.
[0145] This application also provides a log analysis device, see [link to relevant documentation] Figure 5 It shows a schematic diagram of the structure of a log analysis device provided in an embodiment of this application, which may include:
[0146] Memory 51 is used to store computer programs;
[0147] When processor 52 executes a computer program stored in memory 51, it can perform the following steps:
[0148] Collect logs; extract feature words from each log to obtain the feature words in each log; construct the feature word set of each log by corresponding to the feature words in each log; determine the corresponding seed for each log based on the feature word set of each log and the seed set containing multiple seeds, and classify each log into the class where the corresponding seed belongs.
[0149] This application embodiment also provides a readable storage medium storing a computer program, which, when executed by a processor, can perform the following steps:
[0150] Collect logs; extract feature words from each log to obtain the feature words in each log; construct the feature word set of each log by corresponding to the feature words in each log; determine the corresponding seed for each log based on the feature word set of each log and the seed set containing multiple seeds, and classify each log into the class where the corresponding seed belongs.
[0151] The readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0152] For a description of the relevant parts of the log analysis apparatus, device and readable storage medium provided in this application, please refer to the detailed description of the corresponding parts in the log analysis method provided in the embodiments of this application, and will not be repeated here.
[0153] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that the elements inherent in a process, method, article, or apparatus that includes a list of elements are included. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Additionally, portions of the technical solutions provided in the embodiments of this application that are consistent with the implementation principles of corresponding technical solutions in the prior art have not been described in detail to avoid excessive elaboration.
[0154] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A log analysis method, characterized in that, include: Log collection; Feature words are extracted from each of the logs to obtain the feature words in each of the logs; The feature words in each log are used to construct a feature word set for each log. Based on the feature word set of each log and the seed set containing multiple seeds, determine the seed corresponding to each log and classify each log into the class corresponding to its seed. Correspondingly, based on the feature word set of each log and the seed set containing multiple seeds, the seed corresponding to each log is determined, and each log is assigned to the class containing the seed corresponding to that log, including: Map the feature word set of each log to the first hash string corresponding to each log, and map each seed in the seed set to the second hash string corresponding to each seed; Calculate the distance between each log and each seed based on the first hash string corresponding to each log and the second hash string corresponding to each seed. The seed closest to each log is determined as the seed corresponding to the corresponding log, and each log is assigned to the class of the corresponding seed.
2. The log analysis method according to claim 1, characterized in that, After determining the seed closest to each log as the seed corresponding to the corresponding log, and classifying each log into the class of the corresponding seed, the process further includes: Based on the first hash string corresponding to the log contained in the class where each seed is located, calculate the third hash string of the corresponding center point of the class. Based on the first hash string corresponding to each log and the third hash string of each center point, calculate the distance between each log and each center point; Determine whether each of the center points corresponds to an existing seed in the seed set; if there is a center point that does not correspond to an existing seed in the seed set, then determine a new seed based on the third hash string of the center point and add the new seed to the seed set. The seed corresponding to the center point closest to each log is determined as the seed of the corresponding log, and each log is assigned to the class of the corresponding seed. Determine whether the classification result of the current classification of each log is consistent with the classification result of the previous classification. If they are inconsistent, return to the step of calculating the third hash string of the corresponding center point based on the first hash string of the log contained in the class of each seed, until the classification result of the current classification of each log is consistent with the classification result of the previous classification.
3. The log analysis method according to claim 1, characterized in that, Feature words are extracted from each of the logs to obtain the feature words in each log, including: Each log entry is segmented into words to obtain the words contained in each log entry. Calculate the word frequency and importance of each word contained in each of the logs; The weight of each word in each log is calculated based on the word frequency and importance of each word contained in each log. Words with a weight greater than the weight threshold in each log are used as feature words in the corresponding log.
4. The log analysis method according to claim 3, characterized in that, Calculating the importance of each of the aforementioned words contained in each of the logs includes: use Calculate the importance of the i-th word contained in the log. ; in, This represents the total number of logs collected in this instance. This is the j-th log entry collected this time. This refers to the number of logs collected this time that contain the i-th word mentioned in the log.
5. The log analysis method according to claim 4, characterized in that, Based on the word frequency and importance of each word contained in each log, the weight of each word contained in each log is calculated, including: use Calculate the weight of the i-th word contained in the log. ;in, Let be the word frequency of the i-th word contained in the log.
6. The log analysis method according to claim 1, characterized in that, After assigning each log entry to the class containing its corresponding seed, the process further includes: The logs, the feature word set of the logs, and the classification results of the logs are stored in the database.
7. A log analysis device, characterized in that, include: The data collection module is used to collect logs. The extraction module is used to extract feature words from each of the logs to obtain the feature words in each of the logs; A composition module is used to construct a feature word set for each log by corresponding feature words in each log. The determination module is used to determine the seed corresponding to each log based on the feature word set of each log and the seed set containing multiple seeds, and to classify each log into the class where the seed corresponding to each log belongs; Correspondingly, based on the feature word set of each log and the seed set containing multiple seeds, the seed corresponding to each log is determined, and each log is assigned to the class containing the seed corresponding to that log, including: Map the feature word set of each log to the first hash string corresponding to each log, and map each seed in the seed set to the second hash string corresponding to each seed; Calculate the distance between each log and each seed based on the first hash string corresponding to each log and the second hash string corresponding to each seed. The seed closest to each log is determined as the seed corresponding to the corresponding log, and each log is assigned to the class of the corresponding seed.
8. A log analysis device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the log analysis method as described in any one of claims 1 to 6 when executing the computer program.
9. A readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, implements the steps of the log analysis method as described in any one of claims 1 to 6.