Log statement collection method, anomaly detection method and system, and related devices

Abstract

The present application relates to the field of computer technology, and in particular to a log statement collection method, anomaly detection method and system, and related devices. The log statement collection method includes: obtaining a log method in the source code of a target system, screening an associated method in the source code of the target system that is associated with the obtained log method; based on the associated method and the log method, collecting log statements of the target system; serializing the log statements to obtain log sequences, and assigning category labels to the log sequences; wherein the log sequence categories are associated with the target system analysis requirements. The use of this method can collect the computational burden of log statements in the source code and is conducive to improving the comprehensiveness of the collection, thereby realizing anomaly detection of the storage system based on the operation log of the storage system.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a log statement collection method, anomaly detection method, anomaly detection system, computer program product, computer device, and computer-readable storage medium. Background Art

[0002] During the operation of target systems such as storage systems, computers, and servers, operation logs are usually generated based on events that occur during the operation. Operation logs can usually record the operating status of the target system, the operation and intention of the service, etc. Therefore, the operation logs can be analyzed in the hope of discovering and solving potential problems of the target system.

[0003] However, there is a relative lack of public log resources. In view of this, in terms of log statement collection, it is necessary to redeploy the system and re-implement the workload for each target system, traverse all hypothetical scenarios as much as possible during the execution process, and trigger the running logs through the designed workload. This has a heavy deployment burden and trigger burden, and is prone to the risk of missing some possible scenarios. Summary of the invention

[0004] Based on this, it is necessary to provide a log statement collection method, anomaly detection method, anomaly detection system, computer program product, computer device and computer-readable storage medium to address the above technical problems, which can collect the computational burden of log statements in source code and help improve the comprehensiveness of collection, and realize anomaly detection of the storage system based on the operation log of the storage system.

[0005] On the one hand, a log statement collection method is provided, which includes: obtaining a log method in a target system source code, and screening associated methods in the target system source code that are associated with the obtained log method; based on the associated method and the log method, collecting log statements of the target system; serializing the log statements to obtain log sequences, and assigning category labels to the log sequences; wherein the log sequence categories are associated with target system analysis requirements.

[0006] In one embodiment of the present application, screening the associated methods associated with the acquired log method in the target system source code includes: treating each code method in the target system source code as a method node; wherein the code method includes a log method; parsing the target system source code to construct a call topology of the method node; identifying the method node leading to the log method node in the call topology, and treating the code method to which it belongs as the associated method.

[0007] In one embodiment of the present application, screening the associated method associated with the acquired log method in the target system source code also includes: selecting redundant nodes in the call topology; wherein the redundant nodes represent method nodes that are not log method nodes and do not lead to log method nodes; and pruning the redundant nodes in the call topology.

[0008] In one embodiment of the present application, obtaining a log method in the target system source code includes: detecting whether each code method in the target system source code uses a log interface; wherein the log interface represents a programming interface for recording application runtime information; and in response to a code method using the log interface, determining that it is a log method.

[0009] In one embodiment of the present application, collecting log statements of the target system includes: parsing the code definition parameters included in the code of the log method; in response to the code definition parameters being constant string parameters, parsing the assignment of the code definition parameters when the log method is running, and obtaining the log statement based on the assignment and the code definition parameters.

[0010] In one embodiment of the present application, the code of the log parsing method includes, after the code definition parameters, the following further includes: in response to the code definition parameters being non-constant string parameters, assigning a virtual identifier to the code definition parameters, or updating the code definition parameters to the virtual identifier so as to ignore the code definition parameters when analyzing the target system based on the virtual identifier; wherein the non-constant string parameters include dynamic parameters.

[0011] In one embodiment of the present application, serializing log statements to obtain log sequences includes: performing word segmentation on the log statements to obtain log word segmentations; performing vectorization on the log word segmentations of the log statements to obtain word segmentation vectors; and fusing the word segmentation vectors of the log statements to obtain feature vectors of the log statements as log sequences.

[0012] In one embodiment of the present application, after obtaining the log segmentation, it also includes: constructing a vocabulary set based on the log segmentation and assigning a word identifier to each log segmentation; and / or identifying whether the log segmentation is a noise word; in response to the log segmentation being a noise word, deleting the log segmentation.

[0013] In one embodiment of the present application, assigning a category label to a log sequence includes: based on the target system analysis requirements, performing a cluster analysis on the log sequence of log statements to obtain multiple log clusters; determining that log statements in the same log cluster belong to the same log sequence category, and assigning the same category label as an initial annotation.

[0014] In one embodiment of the present application, the target system analysis requirements include performing anomaly detection on the target system based on log statements; performing cluster analysis on the log sequences of the log statements to obtain multiple log clusters including: selecting two log sequences as the initial cluster centers of two log sequence categories; assigning other log sequences to log sequence categories that match them based on their similarities to the cluster centers; wherein other log sequences represent log sequences that are not used as the initial cluster centers; calculating the category means of the log sequence categories respectively, and performing cluster analysis on the log sequences using the category means as the new cluster centers until the iteration completion conditions are met.

[0015] In one embodiment of the present application, based on the similarity between each other log sequence and the cluster center, assigning it to the log sequence category that matches it includes: calculating the cluster similarity between the current sequence and the two cluster centers respectively; wherein the current sequence is the other log sequence whose log sequence category is currently to be assigned; and determining that the current sequence belongs to the log sequence category of the cluster center with a higher cluster similarity.

[0016] In one embodiment of the present application, until the iteration completion condition is met, it also includes: generating an intervention request; wherein the intervention request is used to request determination of attributes of two log sequence categories, the attributes of the log sequence categories include a normal statement category and an abnormal statement category; obtaining attribute information in response to the intervention request, and assigning category labels to the log statements based on the attribute information.

[0017] In one embodiment of the present application, assigning a category label to a log sequence also includes: identifying a log template that matches the log statement to which the current log sequence belongs as the current template; and determining the category label of the current log sequence based on the category label of the current template and the initial label of the current statement.

[0018] In one embodiment of the present application, the log statement collection method further includes: performing information analysis on the log statements, constructing a plurality of log templates from the log statements based on information similarity; and marking a category label of each log template.

[0019] In one embodiment of the present application, information is parsed on log statements, and several log templates are constructed from the log statements based on information similarity, including: segmenting the log statements to obtain several log segmentations to form a log sequence of the log statements; based on the number of segmentations of the log segmentations in the log sequence, determining a query path of the parsing topology; wherein the number of segmentations of several log groups linked by the query path matches the number of segmentations of the log statement; and adding the log statement to the log group under the query path that matches it to update the parsing topology.

[0020] In one embodiment of the present application, adding a log statement to a log group under a query path that matches it includes: searching for a log group under the query path that matches the log word segmentation as an associated log group; wherein the associated log group includes a number of classified sequences; calculating the similarity between the log sequence and each classified sequence as a parsing similarity; selecting a maximum parsing similarity value for comparison with a similarity threshold; in response to the maximum parsing similarity value being higher than the similarity threshold, adding the log sequence to an associated log group of the maximum parsing similarity value; in response to the maximum parsing similarity value being lower than the similarity threshold, creating a new log group based on the log sequence and the parsing topology.

[0021] On the other hand, an anomaly detection method is provided, which includes: obtaining an operation log of a storage system; reading a category tag of a log statement that matches the operation log, and determining whether the storage system is in an abnormal state based on the category tag; wherein the category tag of the log statement is obtained using a log statement collection method as in any of the above embodiments.

[0022] On the other hand, an anomaly detection system is provided, which includes: a collection device and a detection device; the collection device is used to implement the log statement collection method as in any of the above embodiments; the detection device is connected to the collection device to implement the anomaly detection method as in the above embodiments.

[0023] On the other hand, a computer program product is provided, including a computer program, which implements the following steps when executed by a processor: obtaining a log method in a target system source code, screening an associated method in the target system source code that is associated with the obtained log method; collecting log statements of the target system based on the associated method and the log method; serializing the log statements to obtain log sequences, and assigning category tags to the log sequences; wherein the log sequence categories are associated with target system analysis requirements; or, implementing the following steps: obtaining an operation log of a storage system; reading a category tag of a log statement that matches the operation log, and determining whether the storage system is in an abnormal state based on the category tag; wherein the category tag of the log statement is obtained using a log statement collection method as in any of the above embodiments.

[0024] On the other hand, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: obtaining a log method in a target system source code, screening an associated method in the target system source code that is associated with the obtained log method; collecting log statements of the target system based on the associated method and the log method; serializing the log statements to obtain log sequences, and assigning category labels to the log sequences; wherein the log sequence categories are associated with target system analysis requirements; or, implementing the following steps: obtaining an operation log of a storage system; reading a category label of a log statement that matches the operation log, and determining whether the storage system is in an abnormal state based on the category label; wherein the category label of the log statement is obtained using a log statement collection method as in any of the above embodiments.

[0025] On the other hand, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: obtaining a log method in a target system source code, and screening an associated method in the target system source code that is associated with the obtained log method; based on the associated method and the log method, collecting log statements of the target system; serializing the log statements to obtain log sequences, and assigning category labels to the log sequences; wherein the log sequence categories are associated with target system analysis requirements; or, implementing the following steps: obtaining a running log of a storage system; reading a category label of a log statement that matches the running log, and determining whether the storage system is in an abnormal state based on the category label; wherein the category label of the log statement is obtained using a log statement collection method as in any of the above embodiments.

[0026] The log statement collection method, anomaly detection method, anomaly detection system, computer program product, computer device and computer-readable storage medium analyze the target system source code to screen the log method and the associated method associated with the log method, so as to reduce the code method that needs to be paid attention to in the log statement collection process, and reduce the computational burden of collecting log statements in the source code. In addition, by screening the associated method associated with the log method, collecting log statements is conducive to comprehensive collection. At the same time, the collected log statements can be classified and assigned category labels. In this way, when the storage system is used as the target system, the log statements can be classified into at least two dimensions of normal statements and abnormal statements, so that when the operation log of the storage system is obtained, the log statement to which the operation log belongs can be identified, so as to determine whether the storage system is abnormal based on the category label of the log statement to which it belongs, and realize abnormal detection of the storage system based on the operation log of the storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1It is a flowchart of an embodiment of an abnormality detection method of the present application;

[0028] Figure 2 This is a flow chart of an embodiment of a method for collecting log statements of the present application;

[0029] Figure 3 It is a flow chart of another embodiment of the log statement collection method of the present application;

[0030] Figure 4 This is a flow chart of an embodiment of constructing a calling topology in the present application;

[0031] Figure 5 It is a flowchart of an embodiment of redundant node pruning processing of the present application;

[0032] Figure 6 This is a flow chart of an embodiment of log parsing of the present application;

[0033] Figure 7 It is a structural diagram of an embodiment of an abnormality detection system of the present application;

[0034] Figure 8 It is a structural diagram of an embodiment of a storage system of the present application;

[0035] Fig. 9 It is a structural diagram of an embodiment of a computer device of the present application. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0037] With the rapid development of large-scale IT (Internet Technology) systems, the demand for high-quality target system services is increasing day by day. As a result, anomaly detection of target systems has become one of the key links.

[0038] To this end, the target system can be detected for anomalies based on the event logs (i.e., operation logs) generated when the target system is running. Generally speaking, these operation logs record the operating status of the target system such as the storage system and the user's operations and intentions in the service, which is conducive to timely discovery and resolution of potential problems.

[0039] To address this challenge, AI IT operations and maintenance (AIOps) (intelligent operations and maintenance based on machine learning) may be used in related technologies to analyze and process relevant logs to identify abnormal situations by integrating advanced deep learning algorithms. However, the development of advanced data-driven methods requires the support of continuously growing data volume and quality. Although log analysis can continuously develop automated solutions for storage system maintenance, there is still a lack of effective data sets. The lack of relatively high-quality log resources as training data sets has limited the use of methods such as AIOps.

[0040] That is to say, obtaining relatively sufficient, high-quality, and representative logs for practical analysis can still be considered a challenging task.

[0041] On the one hand, industrial logs from large real-world service providers contain rich events, but may involve relatively private data, making it difficult to publish their log resources publicly. On the other hand, logs collected in a laboratory environment using simulations are public, but contain simple events generated by limited standard workloads, and therefore cannot accurately reflect the complex runtime workflows of large storage systems.

[0042] In both cases, existing log statement datasets are based on passive collection methods that retrieve log files after running applications in computing systems. The output logs of passive collection methods are completely dependent on the input values, which hinders the exploration and evaluation of models in many aspects, such as poor comprehensiveness of log events and generally lack of scalability across different systems.

[0043] To expand, log events are expressed as log statements in source code. When collecting logs passively, the diversity of log events is mainly determined by the diversity of input workloads. Even if the system has been deployed at scale, it is still impractical to trigger complete logging statements through designed workloads while traversing all hypothetical scenarios during execution. Collecting logs from new systems requires engineers to redeploy the system and reimplement the workload, which is a time-consuming and laborious task. Therefore, existing datasets are insufficient in terms of system logs, which makes it challenging to evaluate the generalization ability of the proposed log analysis techniques across different systems.

[0044] In order to solve the technical problems in related technologies that have heavy deployment burdens and triggering burdens, and are prone to the risk of missing some possible scenarios, the present application provides a log statement collection method, anomaly detection method, anomaly detection system, computer program product, computer device and computer-readable storage medium. The log statement collection method includes: obtaining the log method in the source code of the target system, screening the associated method associated with the acquired log method in the source code of the target system; based on the associated method and the log method, collecting the log statements of the target system; serializing the log statements to obtain log sequences, and assigning category labels to the log sequences; wherein the log sequence category is associated with the target system analysis requirements. In this way, the computational burden of the log statements in the source code can be collected, which is conducive to improving the comprehensiveness of the collection and realizing anomaly detection of the storage system based on the operation log of the storage system. The specific working principle of the present application is described below with examples.

[0045] See also Figure 1 , Figure 1 It is a flowchart of an embodiment of the abnormality detection method of the present application.

[0046] S101: Obtaining operation logs of a storage system.

[0047] In this embodiment, during the actual operation of the storage system, the operation logs generated during the execution of log statements in the source code of the storage system are collected.

[0048] The operation log may include a system log, an application log, etc. For example, a log statement may include LOG.error("failed to start web server."), indicating an error log generated when starting a web server fails.

[0049] S102: Read the category tag of the log statement that matches the operation log, and determine whether the storage system is in an abnormal state based on the category tag; wherein the category tag of the log statement is obtained by using a log statement collection method.

[0050] In this embodiment, the log statement collection method has good generalization ability and can be relatively easily migrated between different target systems. In addition, the log statement collection method in this embodiment can collect and classify log statements based on the log method in the storage system source code and the associated method associated with the log method. In this way, the log statements can be classified into at least two dimensions of normal statements and abnormal statements, so that when the operation log of the storage system is obtained, the log statement to which the operation log belongs can be identified, and whether the storage system is abnormal can be determined based on the category label of the log statement to which it belongs, so as to realize abnormal detection of the storage system based on the operation log of the storage system.

[0051] The specific working principle of the log statement collection method is described in detail below.

[0052] See also Figure 2 , Figure 2 It is a flow chart of an embodiment of the log statement collection method of the present application. It should be noted that the log statement collection method in this embodiment can be applied to storage systems, servers, computers, home appliances, etc., that is, the target system can be a storage system, server, computer, home appliance, etc., which is not limited here.

[0053] S201: Obtain a log method in a target system source code, and filter associated methods in the target system source code that are associated with the obtained log method.

[0054] In this embodiment, source code refers to an uncompiled text file written in accordance with certain programming language specifications. Source code usually includes several methods, namely, code methods in this article, which can be considered as code blocks that perform specific tasks.

[0055] The log method represents a way to record, track and analyze specific activities or system runtime information. In this embodiment, the characteristic system recorded by the log method is the target system. The associated method associated with the log method can be a method that has a calling relationship with the log method, and can include ancestor nodes and downstream nodes, that is, code methods that can call the log method and nodes that the log method can call the log method, or other custom association relationships, which are not limited here.

[0056] That is, the code method in the target system source code can be read and identified to obtain the log method therein, and the code method associated with the log method can be found as the associated method.

[0057] S202: Collect log statements of the target system based on the association method and the log method.

[0058] In this embodiment, we can focus on log methods and associated methods to collect log statements defined in the source code, reduce the number of code methods that are focused on during the log statement collection process, and thus help reduce the computational burden. In addition, focusing on the source code of the target system for log statement collection and extraction can have good generalization capabilities, help improve the comprehensiveness of log statement collection, and reduce the cumbersome design burden of hypothetical scenarios.

[0059] S203: Serialize the log statements respectively to obtain log sequences, and assign category labels to the log sequences; wherein the log sequence categories are associated with target system analysis requirements.

[0060] In this embodiment, the collected log statements are serialized to obtain log sequences, and are respectively assigned category labels, so as to reduce the serialization process of the specific analysis process and improve the analysis efficiency of the target system.

[0061] Among them, the log sequence category can be associated with the target system analysis requirements. For example, the target system analysis can be anomaly detection of the storage system as described above, and the log sequence can be divided into two categories, namely normal statement category and abnormal statement category. The normal statement category means that when the operation log belongs to this category, the storage system operation status can be considered normal, and the abnormal statement category means that when the operation log belongs to this category, the storage system operation status can be considered abnormal. Alternatively, the target system analysis can also be an overall analysis, task and requirement analysis, functional analysis, indicator allocation, etc., which will not be elaborated here.

[0062] It can be seen that in this embodiment, the target system source code can be analyzed to screen the log method and the associated method associated with the log method, so as to reduce the code method that needs to be paid attention to in the process of collecting log statements, and reduce the computational burden of collecting log statements in the source code. In addition, by screening the associated method associated with the log method, collecting log statements is conducive to comprehensive collection. At the same time, the collected log statements can be classified and assigned category labels. As such, when the log statement collection method is applied to the above-mentioned anomaly detection method, for example, when the storage system is used as the target system, the log statements can be classified into at least two dimensions of normal statements and abnormal statements, so that when the operation log of the storage system is obtained, the log statement to which the operation log belongs can be identified, so as to determine whether the storage system is abnormal based on the category label of the log statement to which it belongs, and realize anomaly detection of the storage system based on the operation log of the storage system.

[0063] See also Figure 3 , Figure 3 It is a flow chart of another embodiment of the log statement collection method of the present application.

[0064] S301: Parse the target system source code to construct a call topology of method nodes.

[0065] In this embodiment, the call topology may be a program / method call graph or the like.

[0066] Specifically, a standard static analysis can be performed to construct a system call graph that links methods according to their execution time relationships in a program. A call graph is a directed graph where each node identifies a method and each edge represents a pair of (caller, callee) method-level relationships. Figure 4 As shown in the example, Figure 4 This is a flow chart of an embodiment of the present application to construct a call topology. Figure 4 The code on the left draws the call topology on the right. For example, you can use call graph building tools such as Soot and JavaParser to draw the call topology.

[0067] S302: Obtain the log method in the source code of the target system.

[0068] In this embodiment, it is possible to detect whether each code method in the target system source code uses a log interface. The log interface represents a programming interface for recording application runtime information. In response to the code method using the log interface, it is determined to be a log method. The analysis based on the log interface can identify more comprehensive log interfaces and log methods by checking the inheritance relationship, which can improve the comprehensiveness of log method identification, and thus help improve the comprehensiveness of log statement collection.

[0069] Optionally, a call topology may be constructed based on the target system source code, and the log methods in the code methods may be identified and annotated, so as to facilitate subsequent processing based on the annotations of the call topology.

[0070] Specifically, you can check whether any logging API (Application Programming Interface) is used in the code method. The logging API refers to the programming interface used to record application runtime information. Common logging APIs provide a series of methods to specify the level of log messages to identify logging methods, such as DEBUG, INFO, WARN, ERROR, etc. This API-based analysis can identify more comprehensive logging APIs by checking inheritance relationships. The application's logging framework can include Log4j, SLF4J, java.util.logging, etc.

[0071] S303: Prune redundant nodes in the calling topology.

[0072] In this embodiment, it is possible to filter the associated methods associated with the log method in the target system source code.

[0073] Specifically, each code method in the target system source code may be used as a method node. The code method includes a log method. The target system source code may be parsed to construct a call topology of the method node. The method node leading to the log method node in the call topology is identified, and the code method to which it belongs is used as an associated method.

[0074] That is, the association method may directly call the log method, and / or may indirectly call the log method through other intermediate code methods, which is not limited here.

[0075] Optionally, the call topology can be a directed graph, and the direction of the direction can represent the call relationship. In this way, the intuitiveness of the call relationship between code methods can be improved based on the call topology. In addition, processing the code method based on the call topology can reduce the amount of computation required for directly identifying and processing the code.

[0076] In this way, a redundant node in the call topology can be selected. Among them, a redundant node represents a method node that is not a log method node and does not lead to a log method node, that is, a code method that does not call a log method and is not a log method is regarded as a redundant method, and the node of the redundant method in the call topology is regarded as a redundant node. The redundant nodes in the call topology are pruned. It should be noted that redundancy only means that it is considered as a redundant node during the log statement collection process in this embodiment, and does not limit the utility of the relevant code method.

[0077] That is to say, the intuitive idea to solve the execution path search problem is to build and traverse the complete execution graph (i.e., the call topology in this embodiment). However, in most cases, large-scale target systems such as storage systems and services usually contain extremely cumbersome paths, so exhaustive enumeration and other methods are prone to cause path explosion problems. Therefore, in order to improve the reliability of the log statement collection method process and reduce the traversal burden and collection burden in the embodiment, redundant nodes can be regarded as method nodes with less impact on log statement collection, so it is considered that they are pruned, which is beneficial to improve the efficiency of log statement collection and facilitate subsequent analysis. Moreover, based on the call topology, redundant nodes can be quickly located, and the reliability of redundant node selection is improved, which is beneficial to improve pruning efficiency.

[0078] like Figure 5 As shown in the example, Figure 5 It is a flowchart of an embodiment of redundant node pruning processing of the present application. Figure 5 Method nodes A, B, and D represent log method nodes. Method node C can call method node D, which is a log method node, so the code method to which it belongs is taken as an associated method. Other nodes are method nodes that do not call log method nodes, so they are treated as redundant nodes for pruning.

[0079] In layman's terms, the call topology can be a linear sequence of all vertices of a directed acyclic graph. In topological sorting, the nodes in the graph are arranged in a linear sequence so that for any directed edge (u, v), node u appears before node v in the sequence. Assuming the call order is A→B, A→C, C→D, the topological order can be A→B→C→D. If a node is neither a log method node nor appears before any log method node, indicating that it is not the ancestor node of any log method, the node is considered to be a non-log method induced node, that is, a redundant node.

[0080] S304: Collect log statements of the target system.

[0081] In this embodiment, log statements of the target system may be collected based on the association method and the log method.

[0082] Specifically, the code definition parameters included in the code of the log method may be parsed.

[0083] In this way, in response to the code definition parameter being a constant string parameter, the assignment of the code definition parameter during the log method runtime is parsed, and the log statement is obtained based on the assignment and the code definition parameter. That is, in this embodiment, the log statement is restored by the runtime parameter to provide more detailed information than the basic log statement. In addition, the log statement can be restored based on the target system source code, and the format of the log statement can be matched with the running log format of the target system based on the source code analysis, so as to facilitate the subsequent target system analysis.

[0084] Furthermore, in response to the code definition parameter being a non-constant string parameter, the code definition parameter can be assigned a virtual identifier, or the code definition parameter can be updated to a virtual identifier, so that the code definition parameter can be ignored during target system analysis based on the virtual identifier; wherein the non-constant string parameter includes a dynamic parameter. Non-constant string parameters such as dynamic parameters are variables, and it can be considered that their impact in processes such as anomaly detection is relatively small, so they are regarded as useless information and removed during the preprocessing process of target system analysis, so that the collected log statements can be streamlined and the validity of the collected log statements can be improved.

[0085] For example, the parameter msg ( Figure 4 Line 21 of the example code) from its assignment ( Figure 4 For other types of parameters (such as block, etc.), in this embodiment, they can be replaced by a virtual tag (ie, <*>, etc.).

[0086] S305: Serialize the log statements respectively to obtain log sequences.

[0087] In this embodiment, the log statements may be segmented to obtain log segmentation.

[0088] Optionally, a vocabulary set can be constructed based on the log word segmentation, and a word identifier can be assigned to each log word segmentation. That is, after the word segmentation is completed, a vocabulary table can be constructed, and a unique ID (Identity document, identification number) can be assigned to each word or subword. And / or, it can be identified whether the log word segmentation is a noise word, and in response to the log word segmentation being a noise word, the log word segmentation is deleted. That is, in the word embedding stage, each word or subword in the log sentence text can be converted into a real vector so that the computer can process and understand it. For example, stop words, punctuation marks, etc. can be used as noise words, and removing them can help reduce the noise in the corpus, which in turn helps to improve the reliability of subsequent analysis.

[0089] For example, this can be achieved through word embedding models (such as Word2Vec, GloVe), or more complex models (such as BERT, etc.) can be used to generate context-dependent word embeddings.

[0090] Furthermore, the log word segmentation of the log statement can be vectorized to obtain the word segmentation vector. The word segmentation vectors of the log statement are fused to obtain the feature vector of the log statement as the log sequence. In layman's terms, for each log entry, the average or weighted average of all its word or phrase vectors can be calculated to obtain the overall vector representation of the log entry, that is, the word segmentation vector of the fused log statement, which can be used as the feature vector of the log entry. Such a processing process can provide a more compact and semantically rich representation for text data, and provide better input for subsequent clustering analysis.

[0091] That is to say, after the log statement is restored, the category of the log statement can be judged, which can be used as the initial reference for subsequent judgment of log tags. In this stage, word embedding combined with cluster analysis is used to judge the log category. Each word can be converted into a vector representation using word embedding models.

[0092] Subsequently, these feature vectors are clustered using a clustering algorithm, etc., and the log statements are divided into two categories: normal statement category and abnormal statement category. By analyzing the clustering results, abnormal log clusters and normal log clusters can be identified. Clustering is described in detail below.

[0093] S306: Assign category labels to the log sequences.

[0094] In this embodiment, the log sequence category may be associated with the target system analysis requirement.

[0095] Specifically, based on the target system analysis requirements, the log sequence of the log statements can be clustered to obtain multiple log clusters. The log statements of the same log cluster are determined to belong to the same log sequence category and are given the same category label as the initial annotation.

[0096] Taking the target system analysis requirement including anomaly detection of the target system based on log statements as an example, two log sequences can be selected as the initial clustering centers of the two log sequence categories.

[0097] Based on the similarity between each other log sequence and the cluster center, it can be assigned to the log sequence category that matches it; wherein the other log sequence represents the log sequence that is not used as the initial cluster center. The category means of the log sequence category are calculated respectively, and the category means are used as the new cluster center to perform cluster analysis on the log sequence until the iteration completion condition is met. In this way, the initial classification automation of log statements can be realized in this embodiment, reducing the cumbersome burden of identifying and marking log statements one by one, and also helping to improve the efficiency of initial marking.

[0098] Optionally, when assigning other log sequences to the log sequence category that matches it based on similarity, the cluster similarity between the current sequence and the two cluster centers can be calculated respectively; wherein the current sequence is the other log sequence whose log sequence category is currently to be assigned. If so, it can be determined that the current sequence belongs to the log sequence category of the cluster center with a higher cluster similarity, so as to further improve the reliability of the current cluster classification.

[0099] Furthermore, until the iteration completion condition is met, an intervention request can also be generated. The intervention request is used to request the determination of the attributes of two log sequence categories, and the attributes of the log sequence categories include normal statement categories and abnormal statement categories, that is, manual intervention can be used to determine which cluster represents normal and which represents abnormal. Manual judgment of classified log statements is relatively less labor-intensive, which can reduce the manual burden, and can also be combined with manual participation in judgment to improve the accuracy of the judgment. In an alternative embodiment, this can also be determined by a computer, which is not limited here. In this way, the attribute information in response to the intervention request can be obtained, and the log statements can be assigned category labels based on the attribute information.

[0100] The specific cluster analysis process will be explained with examples in the following text and will not be elaborated here.

[0101] That is to say, taking the target system analysis requirement of detecting anomalies in the storage system as an example, the log sequence category may include a normal log statement category and an abnormal log statement category. The normal log statement category indicates that when the operation log of the storage system belongs to this category, the operation status of the storage system is normal. The abnormal log statement category indicates that when the operation log of the storage system belongs to this category, the operation status of the storage system is already abnormal or has the risk of abnormal operation.

[0102] S307: parse the log statements and construct several log templates.

[0103] In this embodiment, the log statements can be parsed and several log templates can be constructed based on information similarity. The category labels of each log template are marked respectively. In this way, this embodiment can convert the originally unstructured log data into a structured event template by extracting key information and removing unnecessary parameters, so as to reduce the workload of manual labeling.

[0104] Specifically, the log statement can be segmented to obtain several log segmentations to form a log sequence of the log statement. Based on the number of segmentations of the log segmentations in the log sequence, the query path of the parsing topology is determined; wherein the number of segmentations of several log groups linked by the query path matches the number of segmentations of the log statement. The log statement is added to the log group under the query path that matches it to update the parsing topology.

[0105] Optionally, a log group matching the log segmentation under the query path may be searched as an associated log group, wherein the associated log group includes several classified sequences, and the similarity between the log sequence and each classified sequence is calculated as the resolution similarity.

[0106] Select the maximum analytical similarity value and compare it with the similarity threshold.

[0107] In response to the maximum resolution similarity value being higher than the similarity threshold, the log sequence is added to a log group associated with the maximum resolution similarity value.

[0108] In response to the maximum parsing similarity value being lower than the similarity threshold, a new log group is created based on the log sequence and the parsing topology.

[0109] The following text will combine Figure 6 To illustrate this step, we will give an example. Figure 6 This is a flow chart of an embodiment of log parsing of the present application, which will not be described in detail here.

[0110] S308: Based on the category label of the current template and the initial label of the current statement, update the category label of the current log sequence.

[0111] In this embodiment, in order to more effectively improve the reliability of labels and reduce the workload of manual labeling. In this embodiment, the log statements restored by step S304 are parsed to extract log templates. Then, the log templates are labeled, and the final log labels are determined by comparing the cluster analysis results and the expert labeling results. In other words, the log templates can be analyzed to give accurate and reliable labels, and other log statements can be updated with labels based on the log templates to which they belong, which can reduce the amount of labeling calculations while reducing the dependence on the reliability of cluster analysis, and can further improve the reliability of log statement classification.

[0112] Specifically, a log template matching the log statement to which the current log sequence belongs may be identified as the current template, and the category label of the current log sequence may be determined based on the category label of the current template and the initial label of the current statement.

[0113] For example, after the extracted log templates are manually or automatically labeled with high accuracy, the log template can be matched for each log statement in combination with the initial label of the cluster analysis. And if the initial label of the clustering result is consistent with the category label result of the log template to which it belongs, the log is skipped. Otherwise, a confirmation request can be generated to request manual confirmation of the category of the log statement.

[0114] The following is a detailed description of the clustering analysis process for the initial annotation of log statements.

[0115] a) The k value, that is, the number of clustering categories, can be set. In this embodiment, the data can be divided into two categories, namely, normal sentence categories and abnormal sentence categories, so k=2 can be set.

[0116] b) Randomly select initial cluster centers: Randomly select the feature vectors of two log entries from the data set of log statements as the initial cluster centers.

[0117] c) Assign data points to the nearest cluster center: Take each log statement as a data point, calculate the distance between each data point and the two cluster centers (such as Euclidean distance, etc.), and assign it to the category corresponding to the nearest cluster center.

[0118] d) Update cluster centers: For each category, recalculate the mean of the feature vectors of all log entries in that category and use this mean as the new cluster center.

[0119] e) Iterative optimization: Repeat steps c and d until the iteration completion condition is met. The iteration completion condition may include reaching the maximum number of iterations; and / or the cluster center no longer changes significantly; and / or the distribution of data points no longer changes.

[0120] f) Interpret the clustering results: Analyze the two cluster centers and their corresponding log entries. One cluster center may represent the characteristics of normal logs, while the other represents the characteristics of abnormal logs. Manual intervention is required to determine which cluster represents normal and which represents abnormal.

[0121] The following will be combined Figure 6 , the specific working principle of step S308 is explained with examples.

[0122] Log statement data can be segmented. After preprocessing, log statements can be segmented according to spaces. In the segmentation process, common camel case expressions and connected compound words in log statements are considered. For camel case expressions, segmentation is performed according to uppercase letters; for compound words connected by underscores, segmentation is performed according to underscores. The results of segmentation are collected to form a log vocabulary list, which is called a token set. Each segmentation in the list can become a token.

[0123] As such, in the parsing topology, the query path of the log template to which the log statement linker belongs can be searched by log length. Specifically, the parsing topology may include a parse tree (a tree structure for describing the hierarchical structure of log messages) with pre-processed log messages. Therefore, the search can be started from the root node of the self-parse number. Wherein, the first layer of the parse number distinguishes log groups based on the number of participles, that is, the first layer node of the parse tree distinguishes log groups by the length of the log statement (the number of tokens). The path to the first layer node is selected according to the log statement length of the pre-processed log statement, because it is believed in the present embodiment that log messages with the same log event may have the same log message length.

[0124] Furthermore, further search can be performed based on token. Figure 6 The example in shows that the second-layer node uses log word segmentation to match the log group, that is, "Send", "Receive", and "Starting" are log words. The first word segmentation after the log statement is processed by word segmentation can be selected, that is, the token at the beginning of the log message. Specifically, you can start traversing from the first-layer node obtained by searching based on the number of word segments until you reach the leaf node. Specifically, select the next internal node by the token at the beginning of the log message. For example, for the log message "Receive from node 4", traverse from the first-layer node "Number of word segments: 4" (in the actual code, it can be defined as Length: 4, etc., Length indicates the length, that is, the number of word segments) to the second-layer node "Receive", because the token at the first position of the log message is "Receive". Then, continue to traverse to the leaf node connected to the internal node "Receive".

[0125] Furthermore, token similarity matching can be performed. In response to having traversed to a leaf node, the node contains a list of log groups. The log statements in these log groups conform to the internal node rules encoded along the path. For example, Figure 6The log group in has a log event "Receive from node *", where the log message contains 4 tokens and starts with "Receive". This step selects the most appropriate log group from the list of log groups. Calculate the similarity between the log statement and the log event of each log group. The calculation principle of similarity can be shown as the following formula:

[0126] Formula 1-1

[0127] Formula 1-2

[0128] Among them, seq1 and seq2 represent log sequences (sequences formed by log statements to be assigned) and log events (sequences of log times in a log group after tokenization), respectively; seq(i) is the i-th token of the sequence; n is the number of tokens; the definition of function equ is as shown in Formula 1-2; t1 and t2 represent seq1(i) and seq2(i), respectively; other indicates that t1 and t2 are different.

[0129] Furthermore, after finding the log group with the maximum similarity (i.e., the maximum similarity value), it is compared with the predefined similarity threshold. If simSeq ≥ st (similarity threshold), the group is returned as the most suitable log group. Otherwise, a prompt flag is returned, indicating that there is no suitable log group.

[0130] Update the parse tree based on the returned results. Specifically, if a suitable log group is returned in step 4, the log ID of the current log statement will be added to the log ID of the returned log group. At the same time, the log events in the returned log group will be updated. Specifically, the tokens at the same position in the log message and the log event can be scanned. If the two tokens are the same, we do not modify the token at that position. Otherwise, a wildcard (i.e. "*") is used to update the token at that position in the log event. If a suitable log group cannot be found, it will create a new log group based on the current log message, where the log ID only contains the ID of the log message, and the log event happens to be the log message. Then, the parse tree will be updated with the new log group. Intuitively, traverse from the root node to the leaf node that should contain the new log group, and add the missing internal nodes and leaf nodes accordingly along the path.

[0131] It can be seen that in this embodiment, an effective log data set is generated through four stages: the log statement acquisition stage explores all methods containing log statements to achieve comprehensive coverage of log events; the path pruning and log recovery stage prunes the call graph and recovers the log statements; the log clustering stage converts the log statements into word vectors through word embedding and clusters them into normal and abnormal categories; the label correction stage extracts the log template through log parsing, manually determines the template label, and performs label correction based on the clustering results and the template label.

[0132] Generally speaking, the first stage is the log statement acquisition stage.

[0133] First, a program call graph can be constructed through standard static analysis and other means. The graph presents the calling relationship between methods in the system in the form of a directed graph, where nodes represent methods and edges represent the relationship between callers and callees. Next, the methods are marked as log methods, and the application runtime information is recorded by checking their log APIs. The analysis not only focuses on methods that directly use the log API, but also uses the inheritance relationship of the API to identify a wider range of logging situations to capture all potential log events. This stage is committed to comprehensively mining the log events in the system to ensure the integrity of the log data set.

[0134] The second stage is the path pruning and log recovery stage.

[0135] In the face of the infinite path problem that may exist in large-scale software, this application reduces the risk of path explosion by pruning redundant nodes. The first step is to prune the call graph, aiming to eliminate irrelevant nodes to simplify analysis. By identifying the induced nodes that directly or indirectly call the log method and performing topological sorting, the non-log method induced nodes and their related edges are removed to form a more streamlined pruned graph. The second step is log statement recovery, which aims to enrich the detailed information of the logging statement. By parsing the runtime parameters, the constant strings are separated from the dynamic parameters, and the latter are replaced with virtual tags (such as <*>) to focus on the constant parts that are more valuable for anomaly detection.

[0136] The third stage is the log clustering stage.

[0137] This stage mainly uses cluster analysis to divide the recovered log statements into two categories: normal and abnormal. First, word segmentation is required to split each log statement into words. Next, each word is converted into a vector representation using a word embedding model. Then, a clustering algorithm is used to cluster these feature vectors and divide the log data into two categories: abnormal and normal. By analyzing the results of clustering, this application can identify abnormal log clusters and normal log clusters.

[0138] The fourth stage is the label correction stage.

[0139] This stage focuses on solving two problems to improve the reliability of log statement labeling. First, this application parses the recovered log statements and extracts log templates. Then, experts only need to annotate these log templates without annotating each log statement one by one, which can effectively reduce the workload of annotation. Next, compare the recovered logs with the annotated templates to further improve the accuracy of the labels. This method can more effectively improve the reliability of labels, while helping to reduce the workload of manual annotation.

[0140] In summary, this application aims to build an accurate, comprehensive and highly targeted log dataset through four stages: log statement acquisition, path pruning and log recovery, log clustering and label correction, which effectively assists in the in-depth analysis and anomaly detection of system logs and is practical and scalable.

[0141] In summary, the present application can generate log sequences by relying on source code program analysis and intelligent algorithms without actually running the target system, thus overcoming the limitations of traditional passive collection methods and reducing implementation costs and risks; by exploring the execution of log statements in the source code, the generated log sequences have comprehensive coverage and effectively capture abnormal events in the system; clustering and expert annotations are used to mark abnormal sequences to further improve the accuracy and availability of the generated log sequences. This method provides an efficient and reliable data generation method for anomaly detection in storage systems, which helps to improve the reliability and stability of the system.

[0142] It should be understood that although Figure 1-Figure 6 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1-Figure 6 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0143] See also Figure 7 , Figure 7 It is a structural diagram of an embodiment of an abnormality detection system of the present application.

[0144] In one embodiment, the anomaly detection system may include a collection device 11 and a detection device 12 .

[0145] The collection device 11 is used to implement the log statement collection method in any of the above embodiments; the detection device 12 is connected to the collection device 11 and is used to implement the anomaly detection method in the above embodiments.

[0146] See also Figure 8 , Figure 8 It is a structural diagram of an embodiment of the storage system of the present application.

[0147] In one embodiment, the storage system may include a storage system body 20 and an anomaly detection system 10. The anomaly detection system 10 may be disposed in the storage system body 20 (e.g. Figure 8 Alternatively, the anomaly detection system 10 may be disposed in a server, a terminal, etc., which is not limited here.

[0148] It should be noted that the specific limitations of the collection device can be found in the limitations of the log statement collection method above, which will not be repeated here. The various modules in the above-mentioned collection device and / or anomaly detection system can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0149] In one embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps of the log statement collection method described above or the steps of the anomaly detection method described above are implemented, which will not be described in detail here.

[0150] See also Fig. 9 , Fig. 9 It is a structural diagram of an embodiment of a computer device of the present application.

[0151] In one embodiment, the computer device may be a server, and its internal structure diagram may be as follows: Fig. 9 As shown in the example.

[0152] The computer device includes a processor, a memory, a network interface and a database connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store operation logs and / or log statement data carrying category tags. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a log statement collection method is implemented, or the anomaly detection method described in the above text is implemented.

[0153] Those skilled in the art will understand that Fig. 9 The structure shown as an example is merely a block diagram of a portion of the structure related to the present application solution, and does not constitute a limitation on the computer device to which the present application solution is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0154] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps may be implemented:

[0155] S101: Obtaining operation logs of a storage system.

[0156] S102: Read the category tag of the log statement that matches the operation log, and determine whether the storage system is in an abnormal state based on the category tag; wherein the category tag of the log statement is obtained by using a log statement collection method.

[0157] In one embodiment, the processor may further implement the following steps when executing the computer program:

[0158] S201: Obtain a log method in a target system source code, and filter associated methods in the target system source code that are associated with the obtained log method.

[0159] S202: Collect log statements of the target system based on the association method and the log method.

[0160] S203: Serialize the log statements respectively to obtain log sequences, and assign category labels to the log sequences; wherein the log sequence categories are associated with target system analysis requirements.

[0161] In one embodiment, when screening the associated methods associated with the acquired log method in the target system source code, the processor may also implement the following steps when executing the computer program: taking each code method in the target system source code as a method node; wherein the code method includes the log method; parsing the target system source code to construct a call topology of the method node; identifying the method node leading to the log method node in the call topology, and taking the code method to which it belongs as the associated method.

[0162] In one embodiment, when screening the associated methods associated with the acquired log method in the target system source code, the processor may further implement the following steps when executing the computer program: selecting redundant nodes in the call topology; wherein the redundant nodes represent method nodes that are not log method nodes and do not lead to log method nodes; and pruning the redundant nodes in the call topology.

[0163] In one embodiment, when obtaining the log method in the target system source code, the processor can also implement the following steps when executing the computer program: detect whether each code method in the target system source code uses a log interface; wherein the log interface represents a programming interface for recording application runtime information; in response to the code method using the log interface, determine that it is a log method.

[0164] In one embodiment, when collecting log statements of the target system, the processor may further implement the following steps when executing the computer program: parsing the code definition parameters included in the code of the log method; in response to the code definition parameters being constant string parameters, parsing the assignment of the code definition parameters when the log method is running, and obtaining the log statement based on the assignment and the code definition parameters.

[0165] In one embodiment, after parsing the code definition parameters included in the code of the log method, the processor may further implement the following steps when executing the computer program: in response to the code definition parameter being a non-constant string parameter, assigning a virtual identifier to the code definition parameter, or updating the code definition parameter to the virtual identifier so as to ignore the code definition parameter when analyzing the target system based on the virtual identifier; wherein the non-constant string parameter includes a dynamic parameter.

[0166] In one embodiment, when the log statements are serialized to obtain a log sequence, the processor may further implement the following steps when executing the computer program: perform word segmentation on the log statements to obtain log word segmentations; perform vectorization on the log word segmentations of the log statements to obtain word segmentation vectors; and fuse the word segmentation vectors of the log statements to obtain a feature vector of the log statements as a log sequence.

[0167] In one embodiment, after obtaining the log segmentation, the processor may further implement the following steps when executing the computer program: constructing a vocabulary set based on the log segmentation and assigning a word identifier to each log segmentation; and / or identifying whether the log segmentation is a noise word; in response to the log segmentation being a noise word, deleting the log segmentation.

[0168] In one embodiment, when assigning a category label to a log sequence, the processor may further implement the following steps when executing a computer program: based on the target system analysis requirements, perform cluster analysis on the log sequence of log statements to obtain multiple log clusters; determine that log statements in the same log cluster belong to the same log sequence category, and assign the same category label as an initial annotation.

[0169] In one embodiment, the target system analysis requirements include performing anomaly detection on the target system based on log statements; performing cluster analysis on the log sequences of the log statements to obtain multiple log clusters, and the processor can also implement the following steps when executing the computer program: selecting two log sequences as the initial cluster centers of two log sequence categories; based on the similarity between each other log sequence and the cluster center, assigning it to a log sequence category that matches it; wherein the other log sequences represent log sequences that are not used as the initial cluster centers; calculating the category means of the log sequence categories respectively, and using the category means as the new cluster centers to perform cluster analysis on the log sequences until the iteration completion conditions are met.

[0170] In one embodiment, when each other log sequence is assigned to a log sequence category that matches it based on the similarity between the other log sequences and the cluster center, the processor may further implement the following steps when executing the computer program: respectively calculating the clustering similarity between the current sequence and the two cluster centers; wherein the current sequence is another log sequence to be assigned a log sequence category; and determining that the current sequence belongs to the log sequence category of the cluster center with which the clustering similarity is higher.

[0171] In one embodiment, until the iteration completion condition is met, the processor may further implement the following steps when executing the computer program: generating an intervention request; wherein the intervention request is used to request determination of attributes of two log sequence categories, the attributes of the log sequence categories including a normal statement category and an abnormal statement category; obtaining attribute information in response to the intervention request, and assigning category labels to the log statements based on the attribute information.

[0172] In one embodiment, when assigning a category label to a log sequence, the processor may further implement the following steps when executing a computer program: identifying a log template that matches a log statement to which the current log sequence belongs as the current template; and determining the category label of the current log sequence based on the category label of the current template and the initial label of the current statement.

[0173] In one embodiment, when the processor executes the computer program, the following steps may also be implemented: performing information analysis on the log statements, constructing several log templates from the log statements based on information similarity; and marking the category labels of the log templates respectively.

[0174] In one embodiment, when information is parsed for log statements and several log templates are constructed from the log statements based on information similarity, the processor may further implement the following steps when executing the computer program: segmenting the log statements to obtain several log segmentations to form a log sequence of the log statements; determining a query path of the parsing topology based on the number of segmentations of the log segmentations in the log sequence; wherein the number of segmentations of several log groups linked by the query path matches the number of segmentations of the log statement; and adding the log statement to the log group under the query path that matches it to update the parsing topology.

[0175] In one embodiment, when a log statement is added to a log group under a query path that matches it, the processor can also implement the following steps when executing a computer program: searching for a log group under the query path that matches the log segmentation as an associated log group; wherein the associated log group includes a number of classified sequences; calculating the similarity between the log sequence and each classified sequence as a parsing similarity; selecting a maximum parsing similarity value for comparison with a similarity threshold; in response to the maximum parsing similarity value being higher than the similarity threshold, adding the log sequence to an associated log group of the maximum parsing similarity value; in response to the maximum parsing similarity value being lower than the similarity threshold, creating a new log group based on the log sequence and the parsing topology.

[0176] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps can be implemented:

[0177] S101: Obtaining operation logs of a storage system.

[0178] S102: Read the category tag of the log statement that matches the operation log, and determine whether the storage system is in an abnormal state based on the category tag; wherein the category tag of the log statement is obtained by using a log statement collection method.

[0179] In one embodiment, when the computer program is executed by a processor, the following steps may also be implemented:

[0180] S201: Obtain a log method in a target system source code, and filter associated methods in the target system source code that are associated with the obtained log method.

[0181] S202: Collect log statements of the target system based on the association method and the log method.

[0182] S203: Serialize the log statements respectively to obtain log sequences, and assign category labels to the log sequences; wherein the log sequence categories are associated with target system analysis requirements.

[0183] In one embodiment, when screening the associated methods associated with the acquired log method in the target system source code, the following steps may also be implemented when the computer program is executed by the processor: taking each code method in the target system source code as a method node; wherein the code method includes the log method; parsing the target system source code to construct a call topology of the method node; identifying the method node leading to the log method node in the call topology, and taking the code method to which it belongs as the associated method.

[0184] In one embodiment, when screening the associated methods associated with the acquired log method in the target system source code, the following steps may also be implemented when the computer program is executed by the processor: selecting redundant nodes in the call topology; wherein the redundant nodes represent method nodes that are not log method nodes and do not lead to log method nodes; and pruning the redundant nodes in the call topology.

[0185] In one embodiment, when obtaining the log method in the target system source code, the following steps can also be implemented when the computer program is executed by the processor: detecting whether each code method in the target system source code uses a log interface; wherein the log interface represents a programming interface for recording application runtime information; and in response to the code method using the log interface, determining that it is a log method.

[0186] In one embodiment, when collecting log statements of the target system, the computer program can also implement the following steps when executed by the processor: parsing the code definition parameters included in the code of the log method; in response to the code definition parameters being constant string parameters, parsing the assignment of the code definition parameters when the log method is running, and obtaining the log statement based on the assignment and the code definition parameters.

[0187] In one embodiment, after the code definition parameters included in the code of the log method are parsed, the following steps may be further implemented when the computer program is executed by the processor: in response to the code definition parameter being a non-constant string parameter, assigning a virtual identifier to the code definition parameter, or updating the code definition parameter to the virtual identifier so as to ignore the code definition parameter during target system analysis based on the virtual identifier; wherein the non-constant string parameter includes a dynamic parameter.

[0188] In one embodiment, when the log statements are serialized to obtain a log sequence, the following steps may be further implemented when the computer program is executed by the processor: performing word segmentation on the log statements to obtain log word segmentations; performing vectorization on the log word segmentations of the log statements to obtain word segmentation vectors; and fusing the word segmentation vectors of the log statements to obtain a feature vector of the log statements as a log sequence.

[0189] In one embodiment, after obtaining the log segmentation, the computer program can also implement the following steps when executed by the processor: construct a vocabulary set based on the log segmentation and assign a word identifier to each log segmentation; and / or, identify whether the log segmentation is a noise word; in response to the log segmentation being a noise word, delete the log segmentation.

[0190] In one embodiment, when assigning a category label to a log sequence, the following steps may also be implemented when the computer program is executed by the processor: based on the target system analysis requirements, a cluster analysis is performed on the log sequence of log statements to obtain multiple log clusters; it is determined that log statements in the same log cluster belong to the same log sequence category, and the same category label is assigned as an initial label.

[0191] In one embodiment, the target system analysis requirements include performing anomaly detection on the target system based on log statements; performing cluster analysis on the log sequences of the log statements to obtain multiple log clusters. When the computer program is executed by the processor, the following steps can also be implemented: selecting two log sequences as the initial cluster centers of two log sequence categories; based on the similarity between each other log sequence and the cluster center, assigning it to a log sequence category that matches it; wherein the other log sequences represent log sequences that are not used as the initial cluster centers; calculating the category means of the log sequence categories respectively, and using the category means as the new cluster centers to perform cluster analysis on the log sequences until the iteration completion conditions are met.

[0192] In one embodiment, when each other log sequence is assigned to a log sequence category that matches it based on the similarity between the other log sequences and the cluster center, the computer program can also implement the following steps when executed by the processor: respectively calculating the clustering similarity between the current sequence and the two cluster centers; wherein the current sequence is the other log sequence whose log sequence category is currently to be assigned; and determining that the current sequence belongs to the log sequence category of the cluster center with which the clustering similarity is higher.

[0193] In one embodiment, until the iteration completion condition is met, the following steps may be implemented when the computer program is executed by the processor: generating an intervention request; wherein the intervention request is used to request determination of attributes of two log sequence categories, the attributes of the log sequence categories including a normal statement category and an abnormal statement category; obtaining attribute information in response to the intervention request, and assigning category labels to the log statements based on the attribute information.

[0194] In one embodiment, when assigning a category label to a log sequence, the computer program may further implement the following steps when executed by a processor: identifying a log template that matches the log statement to which the current log sequence belongs as the current template; and determining the category label of the current log sequence based on the category label of the current template and the initial label of the current statement.

[0195] In one embodiment, when the computer program is executed by the processor, the following steps may also be implemented: performing information analysis on the log statements, constructing several log templates from the log statements based on information similarity; and marking the category labels of the log templates respectively.

[0196] In one embodiment, when information is parsed for log statements and several log templates are constructed from the log statements based on information similarity, the following steps may be implemented when the computer program is executed by the processor: the log statements are segmented to obtain several log segmentations to form a log sequence of the log statements; based on the number of segmentations of the log segmentations in the log sequence, a query path of the parsing topology is determined; wherein the number of segmentations of several log groups linked by the query path matches the number of segmentations of the log statement; and the log statement is added to the log group under the query path that matches it to update the parsing topology.

[0197] In one embodiment, when a log statement is added to a log group under a query path that matches it, the computer program can also implement the following steps when executed by the processor: searching for a log group under the query path that matches the log word segmentation as an associated log group; wherein the associated log group includes a number of classified sequences; calculating the similarity between the log sequence and each classified sequence as a parsing similarity; selecting a maximum parsing similarity value for comparison with a similarity threshold; in response to the maximum parsing similarity value being higher than the similarity threshold, adding the log sequence to the associated log group of the maximum parsing similarity value; in response to the maximum parsing similarity value being lower than the similarity threshold, creating a new log group based on the log sequence and the parsing topology.

[0198] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0199] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0200] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. A log statement collection method, characterized in that: The log statement collection method comprises: Obtaining a log method in a target system source code, and screening associated methods in the target system source code that are associated with the obtained log method; Based on the association method and the log method, collecting log statements of the target system; Serializing the log statements respectively to obtain log sequences, and assigning category labels to the log sequences; wherein the log sequence categories are associated with target system analysis requirements; There is a calling relationship between the association method and the log method; The method of screening the associated method associated with the acquired log method in the target system source code includes: Taking each code method in the target system source code as a method node; wherein the code method includes the log method; Parsing the target system source code to construct a call topology of the method node; A method node leading to a log method node in the call topology is identified, and the code method to which it belongs is used as the associated method.

2. The log statement collection method according to claim 1, characterized in that: The method of screening the association method associated with the acquired log method in the target system source code also includes: Selecting a redundant node in the call topology; wherein the redundant node represents a method node that is not the log method node and does not lead to the log method node; Pruning is performed on the redundant nodes in the calling topology.

3. The log statement collection method according to claim 1, characterized in that: The method for obtaining the log in the target system source code includes: Detecting whether each code method in the target system source code uses a log interface; wherein the log interface represents a programming interface for recording application runtime information; In response to the code method using the log interface, it is determined to be the log method.

4. The log statement collection method according to claim 1, characterized in that: The collecting of log statements of the target system includes: Parsing the code definition parameters included in the code of the log method; In response to the code definition parameter being a constant string parameter, parsing the value assigned to the code definition parameter when the log method is running, and obtaining the log statement based on the value assigned and the code definition parameter.

5. The log statement collection method according to claim 4, characterized in that: The code for parsing the log method includes the following code definition parameters: In response to the code definition parameter being a non-constant string parameter, assigning the code definition parameter a virtual identifier, or updating the code definition parameter to the virtual identifier so as to ignore the code definition parameter when analyzing the target system based on the virtual identifier; wherein the non-constant string parameter includes a dynamic parameter.

6. The log statement collection method according to claim 1, characterized in that: The serializing the log statements to obtain the log sequence comprises: Performing word segmentation processing on the log statement to obtain log word segmentation; Performing vectorization processing on the log word segmentation of the log statement to obtain a word segmentation vector; The word segmentation vectors of the log sentence are fused to obtain a feature vector of the log sentence as the log sequence.

7. The log statement collection method according to claim 6, characterized in that: After obtaining the log segmentation, the following steps are also included: Constructing a vocabulary set based on the log word segmentation, and assigning a word identifier to each log word segmentation; and / or, Identify whether the log word segment is a noise word; In response to the log segmentation being the noise word, the log segmentation is deleted.

8. The log statement collection method according to claim 1, characterized in that: The step of assigning a category label to the log sequence comprises: Based on the target system analysis requirements, cluster analysis is performed on the log sequence of the log statement to obtain multiple log clusters; It is determined that the log statements of the same log cluster belong to the same log sequence category, and the same category label is assigned as an initial label.

9. The log statement collection method according to claim 8, characterized in that: The target system analysis requirements include performing anomaly detection on the target system based on log statements; The cluster analysis of the log sequence of the log statement to obtain multiple log clusters includes: Selecting the two log sequences as initial clustering centers of two log sequence categories; Based on the similarity between each other log sequence and the cluster center, assign it to a log sequence category that matches it; wherein the other log sequence represents a log sequence that is not used as the initial cluster center; The category means of the log sequence categories are calculated respectively, and the category means are used as new cluster centers to perform cluster analysis on the log sequences until an iteration completion condition is met.

10. The log statement collection method according to claim 9, characterized in that: The step of assigning each other log sequence to a log sequence category matching the other log sequence based on the similarity between the other log sequence and the cluster center includes: Calculating the clustering similarity between the current sequence and the two cluster centers respectively; wherein the current sequence is another log sequence to be assigned a log sequence category; It is determined that the current sequence belongs to the log sequence category of the cluster center having a higher cluster similarity.

11. The log statement collection method according to claim 9, characterized in that: The step of meeting the iteration completion condition further includes: Generate an intervention request; wherein the intervention request is used to request to determine the attributes of two log sequence categories, and the attributes of the log sequence categories include a normal statement category and an abnormal statement category; Attribute information in response to the intervention request is acquired, and a category label is assigned to the log statement based on the attribute information.

12. The log statement collection method according to claim 1, characterized in that: The step of assigning a category label to the log sequence further comprises: Identify the log template that matches the log statement of the current log sequence as the current template; Based on the category label of the current template and the initial label of the current statement, the category label of the current log sequence is determined.

13. The log statement collection method according to claim 12, characterized in that: The log statement collection method further includes: Performing information analysis on the log statements, and constructing a plurality of log templates from the log statements based on information similarity; The category labels of the log templates are marked respectively.

14. The log statement collection method according to claim 13, characterized in that: The performing information parsing on the log statements and constructing a plurality of log templates from the log statements based on information similarity includes: Performing word segmentation processing on the log statement to obtain a plurality of log word segments, forming a log sequence of the log statement; Based on the number of word segments of the log word segmentation in the log sequence, determining a query path for parsing topology; wherein the number of word segments of several log groups linked by the query path matches the number of word segments of the log statement; The log statement is added to the log group under the query path that matches it, so as to update the parsing topology.

15. The log statement collection method according to claim 14, characterized in that: Adding the log statement to the log group under the query path matching the log statement includes: Searching for a log group matching the log segmentation under the query path as an associated log group; wherein the associated log group includes a plurality of classified sequences; Calculating the similarity between the log sequence and each of the classified sequences as the analytical similarity; Select the maximum value of analytical similarity and compare it with the similarity threshold; In response to the maximum parsing similarity value being higher than the similarity threshold, the log sequence is added to the associated log group of the maximum parsing similarity value; in response to the maximum parsing similarity value being lower than the similarity threshold, a new log group is created based on the log sequence in the parsing topology.

16. An anomaly detection method, characterized in that: The anomaly detection method comprises: Get the operation log of the storage system; Read the category tag of the log statement that matches the operation log, and determine whether the storage system is in an abnormal state based on the category tag; wherein the category tag of the log statement is obtained using the log statement collection method according to any one of claims 1-15.

17. An anomaly detection system, characterized in that: The anomaly detection system comprises: A collection device, used to implement the log statement collection method according to any one of claims 1 to 15; A detection device, connected to the collection device, is used to implement the abnormality detection method as claimed in claim 16.

18. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the log statement collection method described in any one of claims 1 to 15 are implemented; or, the steps of the anomaly detection method described in claim 16 are implemented.

19. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the processor implements the steps of the log statement collection method described in any one of claims 1 to 15; or, implements the steps of the anomaly detection method described in claim 16.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the log statement collection method described in any one of claims 1 to 15 are implemented; or, the steps of the anomaly detection method described in claim 16 are implemented.

Images