Detection method and device for hidden abnormal process, equipment and medium

Abstract

The invention discloses a hidden abnormal process detection method and device, equipment and a medium, and the method comprises the steps: collecting each process log produced by a monitoring process in a current detection interval, and determining a corresponding to-be-compared structured format log; matching with each log mode in a mode library, counting the number of logs which are successfully matched, and extracting included log parameters according to the matched log mode; when it is detected that the log number meets the abnormal condition of the proportion of the mode number of the adjacent detection intervals, or the log parameters meet the parameter abnormal condition, it is determined that the monitoring process is the hidden abnormal process. By structuring each process log, matching each process log with each log mode in the mode library, monitoring the log quantity and log parameters of the successfully matched logs, and determining the hidden abnormal process, the abnormal log is automatically identified and the abnormal state is found through an algorithm without presetting a monitoring alarm rule in advance; and the monitoring efficiency of the hidden abnormal process and the accuracy of abnormal alarm are improved.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, and in particular to a detection method, device, equipment and medium for concealing abnormal processes. Background Art

[0002] Process management is a core responsibility of the operating system. The smooth operation of processes is critical to maintaining system efficiency and optimizing user experience. However, in actual operation, processes may face abnormal states such as freezing or suspended animation, which will significantly affect the stability and reliability of the system.

[0003] In the prior art, network management tools can track the CPU (Central Processing Unit) and memory usage of processes and issue an alarm when the preset threshold is exceeded. However, when processes are in a zombie or pseudo-dead state, these processes may still be active in the system, but the alarm is not triggered due to low resource usage. This results in the inability of traditional network management methods to accurately identify zombie processes and pseudo-dead processes. Traditional network management methods have limitations in meeting the needs of daily monitoring and inspection, thereby reducing the stability and reliability of system operation. Summary of the invention

[0004] The present invention provides a method, device, equipment and medium for detecting hidden abnormal processes, so as to solve the problem of low stability and reliability of system operation caused by failure to accurately identify the dead or suspended state of processes in actual operation.

[0005] According to one aspect of an embodiment of the present invention, a method for detecting a hidden abnormal process is provided, comprising:

[0006] Collect each process log produced by the monitoring process in the current detection interval, and determine the structured format log to be compared corresponding to each process log; match each structured format log to be compared with each log pattern in the pattern library, wherein the pattern library is a tree structure, and each node of the tree structure corresponds to a log pattern; count the number of structured format logs to be compared that are successfully matched under each log pattern, and extract the log parameters contained in each structured format log to be compared according to the matched log pattern; when it is detected that the number of structured format logs to be compared that are successfully matched under each log pattern meets the abnormal condition of the proportion of the number of patterns in the adjacent detection interval, or the log parameters contained in each structured format log to be compared meet the parameter abnormal condition, determine that the monitoring process is a hidden abnormal process.

[0007] According to another aspect of an embodiment of the present invention, there is also provided a detection device for concealing abnormal processes, including:

[0008] A module for obtaining structured format logs to be compared is used to collect each process log generated by the monitoring process in the current detection interval, and determine the structured format logs to be compared corresponding to each process log; a log pattern matching module is used to match each structured format log to be compared with each log pattern in the pattern library; a log parameter extraction module is used to count the number of structured format logs to be compared that are successfully matched under each log pattern, and extract the log parameters contained in each structured format log to be compared according to the matched log pattern; a hidden abnormal process determination module is used to determine that the monitoring process is a hidden abnormal process when it is detected that the number of structured format logs to be compared that are successfully matched under each log pattern meets the abnormal condition of the proportion of the number of patterns in the adjacent detection interval, or the log parameters contained in each structured format log to be compared meet the parameter abnormal condition.

[0009] According to another aspect of an embodiment of the present invention, an electronic device is provided, the electronic device comprising:

[0010] at least one processor; and

[0011] a memory communicatively connected to the at least one processor; wherein,

[0012] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for detecting hidden abnormal processes described in any embodiment of the present invention.

[0013] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for detecting hidden abnormal processes described in any embodiment of the present invention when executed.

[0014] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the steps of the method described in any embodiment of the present invention are implemented.

[0015] The technical solution of the embodiment of the present invention is to collect each process log produced by the monitoring process in the current detection interval, and determine the structured format log to be compared corresponding to each process log; match each structured format log to be compared with each log pattern in the pattern library; count the number of structured format logs to be compared that are successfully matched under each log pattern, and extract the log parameters contained in each structured format log to be compared according to the matched log pattern; when it is detected that the number of structured format logs to be compared that are successfully matched under each log pattern meets the abnormal condition of the proportion of the number of patterns in the adjacent detection interval, or the log parameters contained in each structured format log to be compared meet the abnormal condition of the parameter, determine that the monitoring process is a hidden abnormal process. By structuring each process log and matching it with each log pattern in the pattern library, monitoring the number of successfully matched logs and their parameters, determining the hidden abnormal process that generates the abnormal number proportion or abnormal parameter, it is realized that there is no need to preset the monitoring alarm rules in advance, and the abnormal logs are automatically identified and the abnormal state is discovered through the algorithm, thereby improving the monitoring efficiency of the hidden abnormal process and the accuracy of the abnormal alarm.

[0016] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0018] Figure 1 This is a flow chart of a method for detecting hidden abnormal processes provided according to Embodiment 1 of the present invention;

[0019] Figure 2 is a schematic diagram of a mode tree applicable to an embodiment of the present invention;

[0020] Figure 3 is a schematic diagram of log mining in the reasoning phase applicable to the embodiment of the present invention;

[0021] Figure 4 is a schematic diagram of log mining applicable to an embodiment of the present invention;

[0022] Figure 5 It is a schematic diagram of word segmentation and clustering in the log parsing link applicable to the embodiment of the present invention;

[0023] Figure 6It is a schematic diagram of abnormality detection on a log sequence in the construction phase and the production phase applicable to the embodiment of the present invention;

[0024] Figure 7 It is a schematic diagram of a data processing flow applicable to an embodiment of the present invention;

[0025] Figure 8 is a schematic diagram showing anomaly detection results applicable to an embodiment of the present invention;

[0026] Fig. 9 This is a schematic diagram of the structure of a detection device for concealing abnormal processes provided according to Embodiment 2 of the present invention;

[0027] Fig.10 It is a structural schematic diagram of an electronic device for implementing the method for detecting hidden abnormal processes according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] Embodiment 1

[0031] Figure 1 This is a flowchart of a method for detecting hidden abnormal processes provided by the first embodiment of the present invention. This embodiment can be applied to detecting hidden abnormal processes in a system based on process logs. The method can be performed by a detection device for hidden abnormal processes. The detection device for hidden abnormal processes can be implemented in the form of hardware and / or software and can generally be configured in an electronic device. Figure 1 As shown, the method includes:

[0032] S110 , collecting each process log generated by the monitoring process in the current detection interval, and determining a structured format log to be compared corresponding to each process log.

[0033] In an embodiment of the present invention, the current detection interval refers to the time range involved in the log data that the current system is particularly concerned about or is analyzing in log analysis or system monitoring. A monitoring process refers to a process that is currently being monitored in a system or application. During the execution of the process, the monitoring process generates a log, which may contain detailed information about the process running status, performance indicators, error information, and user operations. The structured format log to be compared refers to log data that has been structured in a certain format to facilitate further analysis and comparison. Structured logs contain well-defined fields with a certain format, such as timestamps, log levels, and message content, which can be used as the basis for log analysis. When performing log analysis, the system can compare these structured logs with known normal patterns or abnormal patterns to identify potential problems or abnormal behaviors.

[0034] Specifically, each process log of the currently monitored process within the current time range is collected, and each process log is structured to obtain a structured format log to be compared. The specific method of log structuring can be: determine the target information to be extracted and clarify the meaning of each field, such as timestamp, log level, message or user name, etc., define the data type of each field, such as string, integer or Boolean value, etc., determine the log format of the structured log; extract the target information in the log, and output the structured log according to the determined log format to obtain the structured format log to be compared.

[0035] S120: Match each structured format log to be compared with each log pattern in the pattern library.

[0036] The mode library is a tree structure, and each node of the tree structure corresponds to a log mode.

[0037] In an embodiment of the present invention, a pattern library is a collection for storing and organizing different log patterns. These patterns are used to match, identify and parse templates or rules for specific types of information in log files. A log pattern refers to a specific format or structure that appears repeatedly in a log file, which can be used to identify and extract information from the log. A log pattern can be composed of a series of fixed texts and variable variable parts, that is, a structured log containing fixed features and variable features. By matching each structured format log to be compared with the log pattern recorded in the pattern library, events in the parsed log can be identified.

[0038] In the tree-structured pattern library, each node represents a log pattern, and the hierarchical relationship between nodes represents the classification and sub-classification between different log patterns. The root node can represent the collection of all log patterns, and the child nodes can represent different types of logs, such as error logs or warning logs. The deeper child nodes can represent sub-categories of specific types of logs, such as database errors or network errors. By organizing and managing a large number of log patterns in a tree structure, it is possible to quickly locate and apply specific log patterns during log analysis.

[0039] S130 , counting the number of structured format logs to be compared that are successfully matched in each log mode, and extracting log parameters contained in each structured format log to be compared according to the matched log mode.

[0040] S140: When it is detected that the number of structured format logs to be compared that are successfully matched in each log mode meets the abnormal condition of the proportion of the number of adjacent detection interval patterns, or the log parameters contained in each structured format log to be compared meet the abnormal condition of parameters, it is determined that the monitoring process is a hidden abnormal process.

[0041] In the embodiment of the present invention, the adjacent detection interval mode can be specifically understood as: in the log analysis process, the log data is divided into continuous time intervals, and the logs in a time interval of the same log mode are compared with the logs in another adjacent time interval to identify the changes or anomalies of the logs. For example, if the business activity has obvious daily periodicity, the time interval can be divided by hours or days. This method helps to find abnormal behaviors that start or end in a specific time period. The number of logs generated by a process in the current time period has changed significantly or deviated from the preset range compared with the average number of logs generated in each time period under this mode. Under normal circumstances, the number of logs generated in each time period is regular and the variation is small. When a drastic change is detected or the number of logs generated exceeds the preset range, it indicates that the system or application has a number of abnormal proportions. Parameter abnormality can be specifically understood as: some parameter values ​​in the log exceed the preset range or mode. For example, a normal HTTP request contains specific parameters and values. If the values ​​of these parameters do not conform to the preset mode, such as the parameter value exceeds the limited range or the format of the email address is incorrect, it indicates that there is a parameter abnormality.

[0042] Hidden abnormal processes refer to abnormal processes that are difficult to detect with conventional monitoring tools, including pseudo-dead processes and zombie processes. A zombie process refers to a process that, after execution, sends a child process change signal to the parent process to inform the parent process that it has finished executing, but its parent process does not reclaim its resources and status information, and the child process becomes a zombie process. A zombie process does not occupy CPU resources, but its process descriptor remains in the system, resulting in continuous occupation of system resources. A pseudo-dead process refers to a process that is unresponsive for a long time due to resource competition or deadlock during execution. A pseudo-dead process will occupy system resources, such as CPU or memory, but will not perform any effective work. Specifically, as the number of pseudo-dead processes or zombie processes increases, the CPU or memory usage of the system continues to increase. Since it does not perform any operations itself, no new logs will be generated.

[0043] Specifically, the obtained structured format logs to be compared are matched with the log patterns recorded in the pattern library, the number of successfully matched logs is counted, and the log parameters contained in the logs are extracted according to the matched log patterns. When the number of logs deviates from the preset range (such as [10, 20]) or exceeds the preset threshold value (such as 20) compared to the average number of generated logs, or the log parameters exceed the preset range or do not conform to the set pattern, the monitored process is determined to be a hidden abnormal process.

[0044] The technical solution of the embodiment of the present invention is to collect each process log produced by the monitoring process in the current detection interval, and determine the structured format log to be compared corresponding to each process log; match each structured format log to be compared with each log pattern in the pattern library; count the number of structured format logs to be compared that are successfully matched under each log pattern, and extract the log parameters contained in each structured format log to be compared according to the matched log pattern; when it is detected that the number of structured format logs to be compared that are successfully matched under each log pattern meets the abnormal condition of the proportion of the number of patterns in the adjacent detection interval, or the log parameters contained in each structured format log to be compared meet the abnormal condition of the parameter, determine that the monitoring process is a hidden abnormal process. By structuring each process log and matching it with each log pattern in the pattern library, monitoring the number of successfully matched logs and their parameters, determining the hidden abnormal process that generates the abnormal number proportion or abnormal parameter, it is realized that there is no need to preset the monitoring alarm rules in advance, and the abnormal logs are automatically identified and the abnormal state is discovered through the algorithm, thereby improving the monitoring efficiency of the hidden abnormal process and the accuracy of the abnormal alarm.

[0045] Furthermore, in an embodiment of the present invention, before matching each structured format log to be compared with each log pattern in the pattern library, the method further includes:

[0046] Acquire multiple historical process logs that match the monitoring process; perform word segmentation processing on the log content in each historical process log to obtain a word segmentation field set corresponding to each historical process log, and determine the field type of each word segmentation field in the word segmentation field set; replace the word segmentation field of the target field type in each word segmentation field set with a standardized field to obtain a structured format log corresponding to each historical process log; calculate the similarity between each word segmentation field in different structured format logs by using the Smith-Waterman algorithm, and perform clustering processing by using a hierarchical clustering algorithm to obtain a hierarchical nested clustering tree as a pattern library; wherein the clustering processing is achieved by replacing one or more word segmentation fields in each word segmentation field set with a wildcard, and a log pattern in the pattern library corresponds to a standardized word segmentation field set, and the standardized word segmentation field set contains one or more wildcards.

[0047] In the embodiment of the present invention, word segmentation refers to the process of segmenting a continuous text string into meaningful words or phrases, such as segmenting with spaces, "=" and copulas (am, is and are), and the word segmentation results are formed into a word segmentation field set. Field type is the type of data that a field can store. For example, timestamp, url (Uniform Resource Locator, i.e., the address used to identify resources such as web pages, pictures or videos), numbers and ip (Internet Protocol, Internet Protocol address, i.e., used to identify each device address on the Internet). The target field type refers to the type of log information that needs to be paid attention to in log analysis. Standardized fields refer to converting each field into a unified format or form according to the composition form of the log and its field type. For example, converting the date field into the date type and the time field into the time type. The Smith-Waterman algorithm is a sequence alignment algorithm that finds the best local alignment between two sequences by similarity. The hierarchical clustering algorithm displays the hierarchical relationship, similarity and clustering process between the best locally aligned sequence data points by constructing a clustering tree, such as a nested clustering tree. A wildcard is a character that can represent one or more characters. For example, for variable information in a log, XXX can represent three arbitrary characters, and * can represent any number of characters.

[0048] Specifically, obtain and segment the content of each historical process log to obtain a segmentation field set, determine the field types it contains, use standardized fields to unify the segmentation fields of the target field type to obtain a structured format log, calculate the similarity between the segmentation fields of each log based on the Smith-Waterman algorithm, and use a hierarchical clustering algorithm to build a clustering tree, that is, replace the variable part in the log with a wildcard. The lowest level of the tree is the original data points of different categories, and the top level of the tree is the cluster root node containing the fixed features of the log. Each log pattern has its own standardized segmentation field set, which is used to generate a clustering tree corresponding to the log pattern. Multiple clustering trees are combined to obtain a pattern library containing the fixed features and indefinite features of each log in the historical log.

[0049] Furthermore, in an embodiment of the present invention, after obtaining a plurality of historical process logs matching the monitoring process, the method further includes:

[0050] Extract event type fields from each historical process log, and form an event type field list with the extracted event type fields;

[0051] After determining the structured format log to be compared corresponding to each process log, it also includes: extracting the comparison event type field corresponding to each process log, and detecting whether each comparison event type field exists in the event type field list; if the comparison event type field of the first target process log does not exist in the event type field list, giving up matching the structured format log to be compared of the first target process log with each log pattern in the pattern library.

[0052] In an embodiment of the present invention, the event type field refers to a field that uses the application name (appname) and tag (tag) in the log to classify and mark the log entries, and the log data can be organized and clustered according to its source and specific tags. Appname refers to the name of the application or service that generates the log entry, and tag refers to the severity level of the log, such as ERROR (error log level), INFO (information log level) and DEBUG (debug log level), etc., or a specific function identifier, such as AUTH (authentication log category), DB (database log category) and NET (network log category). The first target process log refers to the structured format log to be compared corresponding to each comparison event type field that needs to be compared with the event type field list before matching with each log pattern in the pattern library.

[0053] Specifically, after obtaining multiple historical process logs that match the monitored process, the collected logs need to be labeled with appname and tag, and the tag is used as the event type field. The event type fields are combined into an event type field list and used to detect the comparison event type field corresponding to the structured format log to be compared. If the list includes the comparison event type field of the first target process log, the structured format log to be compared of the first target process log is matched with each log pattern in the pattern library. Otherwise, it means that the structured format log to be compared contains a new log type that needs to be processed manually without matching with each log pattern in the pattern library, which simplifies the process of hidden abnormal process detection and improves the efficiency of hidden abnormal process detection.

[0054] Furthermore, in an embodiment of the present invention, after matching each structured format log to be compared with each log pattern in the pattern library, the method further includes:

[0055] According to the matching result, the structured format log to be compared of the second target process log that has not successfully matched any log pattern is identified; the method also includes: when the pattern library update condition is met, the structured format log to be compared of the first target process log and / or the structured format log to be compared of the second target process log is used to update the pattern library.

[0056] In the embodiment of the present invention, the second target process log refers to each structured format log to be compared for matching with each log pattern in the pattern library. The pattern library update condition refers to detecting that the structured format log to be compared is a new log pattern not recorded in the pattern library.

[0057] Specifically, when the structured format log to be compared of the second target process log fails to successfully match any log pattern in the pattern library, it means that the structured format log to be compared is a new log pattern that needs to be manually processed, and when a new log pattern not recorded in the pattern library is detected, the structured format log to be compared in the first target process log and / or the second target process log involving the new log pattern will be used to update the pattern library, thereby realizing continuous iterative optimization of the pattern library and improving the accuracy and reliability of hidden anomaly detection.

[0058] Optionally, in the embodiment of the present invention, it is detected that the number of structured format logs to be compared that are successfully matched in each log mode meets the abnormal condition of the proportion of the number of adjacent detection interval modes, including:

[0059] In the current detection interval and the previous adjacent historical detection interval, obtain the difference in the number of structured format logs to be compared that are successfully matched in each log mode, and the difference in the proportion of the number of structured format logs to be compared that are successfully matched to the total number of structured format logs to be compared that are successfully matched in all log modes; if it is determined that the quantity difference or the proportion difference in any log mode exceeds the preset difference threshold, it is determined that an abnormal condition in the quantity proportion of adjacent detection interval modes has been detected.

[0060] Specifically, the preset difference threshold is a key parameter for determining whether the log behavior is abnormal. For the abnormal number of percentages, it can be set as the percentage of the historical average number of logs generated or the percentage of the percentage of the unit detection interval. Obtain the difference in the number of structured format logs to be compared between the current detection interval that successfully matches and the previous adjacent historical detection interval under each log mode, and the difference in the percentage of the number of generated logs to the total number of structured format logs to be compared that successfully matches under all log modes, and obtain the change in the number and percentage of logs generated in the current detection interval compared with the previous adjacent historical detection interval. If the change in the number or percentage of logs generated under any log mode exceeds the preset difference threshold, it indicates that an abnormal condition of the number and percentage of the adjacent detection interval mode is detected, and an abnormal percentage exists.

[0061] Optionally, in an embodiment of the present invention, the log parameters contained in each structured format log to be compared are extracted according to the matched log mode, including:

[0062] Obtain the field position of the variable feature field marked in the log pattern matched by each structured format log to be compared; according to the field position, extract the log parameters contained in each structured format log to be compared; wherein the log parameters include: continuous value parameters, constant parameters, discrete value parameters or high base values.

[0063] In the embodiment of the present invention, the variable feature field is the variable feature part in the log mode, and the variable feature includes a variable parameter. A continuous value parameter means that the parameter can take any value within the entire specified interval or range. A constant parameter refers to a parameter that cannot be changed during the execution of the program. A discrete value parameter means that the parameter can only take values ​​between specific, separate values. A high base value refers to a parameter value that needs to be paid attention to or a parameter value whose frequency of occurrence is higher than a preset threshold, such as IP and URL.

[0064] Specifically, by obtaining the field position of the variable feature field marked in the log pattern matching the structured format log to be compared, continuous value parameters, constant parameters, discrete value parameters or high base values ​​are extracted to obtain parameter features in the structured format log to be compared.

[0065] Optionally, in the embodiment of the present invention, detecting that the log parameters contained in each of the structured format logs to be compared meet the parameter abnormality condition includes:

[0066] If the continuous value parameter contained in at least one of the structured format logs to be compared does not conform to the preset parameter distribution law, it is determined that the parameter abnormality condition is met; if the constant parameter value contained in at least one of the structured format logs to be compared is not unique, it is determined that the parameter abnormality condition is met; if the discrete value parameter contained in at least one of the structured format logs to be compared does not have a value within the preset value set range, it is determined that the parameter abnormality condition is met; or if the high base value parameter type contained in at least one of the structured format logs to be compared is inconsistent, it is determined that the parameter abnormality condition is met.

[0067] Specifically, the parameter characteristics in the structured format log to be compared are obtained through parameter anomaly condition detection. If the continuous value parameter does not conform to the preset parameter distribution law (such as normal distribution or binomial distribution), the constant parameter value is not unique, the discrete value parameter is outside the preset value set range, or the parameter type in the high base value is inconsistent, for example, the IP address is usually a string type value in a standard format (such as 192.168.1.1), if the parameter sent by the client is not a valid IP address format, but other types of data (such as integers, floating point numbers, or strings containing illegal characters), it is determined that the parameter anomaly condition is met, that is, there is a parameter anomaly in the structured format log to be compared.

[0068] Specific application scenarios

[0069] Process management is the core of the operating system and is crucial to system performance and user experience. However, a process may encounter a deadlock or pseudo-death problem during operation, thereby affecting the stability of the system. To this end, the present invention proposes a method for detecting hidden abnormal processes, which may specifically include:

[0070] 1. Log collection and labeling

[0071] Collect massive amounts of application process or service log data, and label the collected logs by appname and tag to distinguish them.

[0072] 2. Build a pattern library

[0073] The collected original logs are segmented and typed according to the raw_message field using a hierarchical clustering algorithm, and then clustered and merged, with different parts replaced with wildcards. The similarity between data points of different categories is calculated to create a hierarchical nested clustering tree, with the original data points of different categories at the bottom of the tree and the top of the tree being the root node of a cluster.

[0074] Figure 2 A schematic diagram of a mode tree applicable to an embodiment of the present invention is shown in FIG. Figure 2As shown in the figure, 8 logs (serial numbers 1-8) are first merged into 4 log formats (serial numbers 9-12), then merged into 2 log formats (serial numbers 13 and 14), and finally merged into 1 log format (serial number 15). These 15 log formats are each used as a node and finally form a tree. The tree structure of this log format is the pattern tree. Combining multiple pattern trees is the pattern library obtained by training.

[0075] 3. Pattern Recognition

[0076] Pattern recognition refers to the reverse process of inferring the source code from the original log text. The fixed features of the log obtained by reverse engineering are called patterns, and the indefinite features of the log are called parameters. There are three processing steps in log anomaly inspection, namely log collection, log parsing and log mining. The inspection process can be:

[0077] (1) Log collection and labeling

[0078] The adaptation of the original training log is achieved by configuring the corresponding query range processing on the page, that is, collecting the original training log in the log collection stage and marking it with appname and tag.

[0079] (2) Log mining

[0080] When these labeled logs are identified, they will enter the word segmentation and clustering stage, matching to form a structured log event list and generating the corresponding event type. The event type is treated as a matching rule, and the original logs of the subsequent real-time production stage can be classified in the word segmentation and clustering matching process and anomaly detection can be performed through the anomaly detection model.

[0081] Figure 3 This is a schematic diagram of log mining in the inference phase applicable to the embodiment of the present invention. The inference phase refers to the deployment phase, using the parameters learned in the training phase to predict or classify new data through the model. Visualization refers to displaying and processing anomalies in a predefined form when anomalies occur; debugging means that the staff can adjust the clustering parameters according to the actual results to achieve the best clustering and anomaly detection effects, or to adapt to different clustering algorithms for different requirements such as data format, scale and distribution; anomaly detection correspondence / fault diagnosis refers to anomaly detection, fault prediction and fault diagnosis through the pattern library; decision-making refers to the optimization and update of the pattern library.

[0082] Figure 4 Schematic diagram of log mining applicable to the embodiment of the present invention, the discovery phase refers to extracting the event list in the training original log entity and the clustering form of the training original log, and the matching phase refers to matching the clustering form with the event type in the event list. The pattern library includes pre-trained anomaly detection models, fault prediction models and fault diagnosis models to achieve anomaly detection, fault prediction and fault diagnosis.

[0083] Among them, the system corresponding to the anomaly has not crashed, and the system corresponding to the fault has crashed. The anomaly detection model refers to matching the structured log event list of the reasoning stage with the structured log event list and event types of the training stage, and determining whether the event in the reasoning stage is abnormal and the type of abnormality when an abnormality occurs through similarity; the fault prediction model refers to collecting log data without abnormalities within a fixed time (one day / one week), judging the development trend of the future CPU or memory occupancy rate according to the historical usage rules of the CPU or memory occupancy rate recorded in the log, so as to judge whether there is a dead or suspended process, and predict whether a fault will occur. When the CPU or memory occupancy rate continues to increase and the predicted value exceeds the preset threshold value, the early warning mechanism is triggered and the fault prediction result is displayed; the fault diagnosis model refers to matching the structured log event list of the reasoning stage with the structured log event list and event types of the training stage, and determining the type of abnormality that causes system failure in the event in the reasoning stage through similarity.

[0084] For some special systems, such as systems with extremely high security requirements, systems with periodic data processing requirements, and distributed systems that require centralized processing, it is impossible to collect logs in real time for analysis and anomaly detection, so offline methods can be used. For most systems, logs can be acquired in real time, and adaptive matching of logs can be performed. After the event type is identified and processed in the matching stage of word segmentation and clustering, anomaly detection is performed by the anomaly detection model to achieve continuous training.

[0085] Regarding continuous model training, it can be achieved by adjusting the event type and associating the identified new event type with the event list, that is, updating the new event type to the event list, so as to achieve continuous optimization and improve the accuracy and reliability of the model output results.

[0086] 4. Log analysis

[0087] Log parsing is one of the links of anomaly detection. Its core idea is: currently there are many types of log outputs, and they contain multiple types of logs. However, the log output format is written by developers and has rules. Therefore, logs can be classified by using clustering algorithms after similarity calculation, so as to identify whether there are new log types, new log ratios, whether abnormal parameters appear in the logs, etc. The similarity calculation algorithm and log clustering algorithm are the core algorithms for realizing log anomaly detection. Figure 5It is a schematic diagram of word segmentation clustering in the log parsing link applicable to the embodiment of the present invention. The variable corresponds to the variable parameter in the log, that is, the variable feature. The log template is a fixed feature. The model correction refers to clustering the fixed features, abstracting the feature form, and replacing the variable features with wildcards. The model statistics refers to statistics of three types of anomalies: log type anomaly, log ratio anomaly, and log parameter anomaly.

[0088] The anomaly detection algorithm adopts a multi-composite algorithm, uses the Smith-Waterman algorithm for similarity calculation, and uses a hierarchical clustering algorithm for log clustering. At the same time, it combines flink, word segmenter, pattern tree, parameter tree and other technologies to learn historical data to generate a pattern library. The pattern library can identify fixed and variable features in log sequences, match them by event type, and perform online anomaly detection on real-time logs to discover abnormal problems such as new patterns, abnormal proportions and abnormal parameters in logs. At the same time, the detected anomalies are marked as log type anomalies, log proportion anomalies or log parameter anomalies. Figure 6 It is a schematic diagram of anomaly detection of log sequences in the construction phase and production phase applicable to the embodiment of the present invention. The construction phase refers to the process of model development and training, and the production phase refers to the stage where the model has been trained and deployed in the actual environment and begins to process real-time data. Log vectorization refers to the process of converting log data into quantifiable numerical vectors, that is, the word segmentation process. In log analysis, the original log is usually in an unstructured text form, containing information such as timestamps, log levels, and messages. In order to use machine learning models for anomaly detection, these unstructured data need to be converted into structured numerical forms. Representative log sequences refer to the organization of parsed log events into sequences in log analysis, each sequence representing a stream of log events over a period of time. These sequences can be used to train machine learning models to identify normal and abnormal behavior patterns. In anomaly detection, by comparing the newly entered log sequence with the known representative log sequence (i.e., the representative log sequence recorded in the pattern library), it can be determined whether there is abnormal behavior. In the process of cyclically checking the new representative log sequence through the pattern library, when the pattern is successfully matched and it is judged that an abnormal proportion or parameter is generated, it is judged that there is a pseudo-dead or zombie program, and the log sequence of the production phase corresponding to the new representative log sequence is classified into the historical log; when the pattern matching fails or the event type of the new representative log sequence fails to match the event type of the representative log sequence in the construction phase, it indicates that the new representative log sequence is a log type not recorded in the pattern library, and the log sequence needs to be manually processed, and the representative log sequence and its event type need to be updated to the pattern library.

[0089] For the task of detecting real-time data with existing process or server pattern libraries, considering that new patterns may be generated due to incomplete training data, version changes, etc., it is necessary to add the function of incremental training and continuously improve the model. By associating new event types with event lists to form incremental training data, the incremental training data is added to the pattern library for training, and the abnormal types of logs are identified and annotated using the optimized pattern library to achieve continuous iterative optimization of the pattern library. For example, if the system has a new zombie process, continuous model training optimization is achieved to improve the accuracy of training results and the accuracy and reliability of model anomaly detection.

[0090] 5. Log segmentation

[0091] The first step in data processing is log word segmentation, which is the process of recombining continuous log character sequences into word sequences according to certain specifications. Specifically, it can be: segmentation based on spaces; type identification based on key fields such as timestamps, URLs, numbers, and IP addresses; and parameter bit identification using symbols such as "=" and copulas (am, is, and are).

[0092] 6. Type conversion

[0093] After word segmentation, the data is converted into data types, such as date is converted into date data type, time is converted into time data type, and after the conversion is completed, it is submitted to the Smith-Waterman algorithm for processing. For each cluster after clustering, the corresponding pattern form of the cluster is generated according to the log. A cluster is a collection of similar data points. Figure 7 It is a schematic diagram of a data processing flow applicable to an embodiment of the present invention.

[0094] Word segmentation is used to break text strings into smaller units, such as by splitting based on spaces, punctuation, or other delimiters. In log analysis, word segmentation helps break log entries into separately analyzable parts, such as timestamps, log levels, and messages. Type detection identifies the data type of each field in the data, such as whether a field is a number, string, date, or other data type, and generates a clustered abstract form. Alignment arranges data into a consistent format or structure so that related data points can be matched, such as aligning the same fields of different log entries for easy comparison and analysis. Field detection identifies and extracts specific fields in log entries and replaces variable parameter features with wildcards. These fields can include timestamps, log levels, error codes, numbers, IPs, URLs, and messages. The purpose of field detection is to extract useful information from log data for further analysis and processing.

[0095] Figure 8is a schematic diagram showing the anomaly detection results applicable to the embodiment of the present invention, where NUM represents the log number, the pointer start address, the clustering parameter and the command line number in sequence. Figure 8 The data is displayed in a form so that the staff can identify the generated exceptions and handle them subsequently.

[0096] 7. Handling of abnormal results

[0097] The types of abnormal detection results in the present invention include: proportion abnormality, parameter abnormality and mode abnormality.

[0098] (1) Abnormal proportion

[0099] ① Count the number of logs in each mode in each time period. ② Assume that the number of logs in mode pt (Production Trial) of this test phase is n, and the proportion is p; the number of logs in mode pt of the previous phase is n', and the proportion is p'. ③ Quantity anomaly: refers to the anomaly when the rate of change of the number of logs in the current mode exceeds the preset threshold e1, that is, |n'-n| / n≥e1. ④ Percentage anomaly: refers to the change in the proportion of the number of logs in the current mode to the total number of logs exceeds the preset threshold e2, that is, |p'-p|≥e2. e1 and e2 can be set according to actual needs.

[0100] (2) Parameter abnormality

[0101] Parameters involved in parameter anomalies include: continuous value parameters, constant parameters, discrete value parameters and high base values. ① Continuous value parameters: detect the distribution of parameter values ​​(such as normal distribution or binomial distribution). ② Constant parameters: detect whether the parameter value is unique. ③ Discrete value parameters: detect whether the parameter is within the value set range. ④ High base value: detect whether the parameter type is consistent (such as high base value IP or URL). Users can adjust the detailed parameter range of clustering by setting whether to enable IP and URL mode replacement to achieve attention to specific address parameters.

[0102] (3) Abnormal mode

[0103] ① Align the pattern sets of the log to be tested and the training results one by one and calculate the similarity. ② If the similarity with a certain pattern exceeds the threshold range, the log pattern is considered to be matched successfully and the pattern is returned; if all patterns fail to match, the log pattern is considered abnormal. ③ Its time complexity is O(nm 2 ), n is the number of patterns, and m is the average length of the log. Table 1 is a comparison table of pattern anomalies.

[0104] The first line is the pattern line, representing the original data field, the second line is the contribution line, representing the similarity, and the third line is the specific data, that is, clustering is implemented based on the third line and similarity analysis is performed to determine whether there is anomaly. field1 to field7 represent the fields in the data set, and each field corresponds to a column of data in the data set. The pattern line shows the mode or type of the data, for example, "timestamp" represents the timestamp, "URL" represents the URL, "[number]" represents the data of the numeric type, "http" represents the protocol type, and "GET" and "SET" represent the HTTP request method. 100%, 47%, [{10-20}], {400,404,…} and 53% in the contribution line, these percentages, interval ranges and sets represent the similarity or contribution of each field in clustering and similarity analysis. For example, "100%" means that the specific data is completely consistent in this field when compared with the original data, while "53%" means that the similarity of this field is relatively low. [{10-20}] represents a range, that is, the number in the field3 field should be between 10 and 20. {400,404,…} represents a set, that is, the data in field7 can be 400, 404 and other limited values. In the specific data row, 2013-07-09 14:25:10 is the specific data of field1, indicating a timestamp. www is the specific data of field2, indicating a substring of a website address.

[20123] is the specific data of field3, indicating a number. http is the specific data of field4, indicating the type of protocol used. GET is the specific data of field5, indicating the HTTP request method used. helloworld.com is the specific data of field6, indicating the URL or domain name of the response. 200 is the specific data of field7, indicating the HTTP response status code.

[0105] By comparing the specific data row with the pattern row and the contribution row, a similarity analysis can be performed. If the value in the specific data row matches the pattern defined in the pattern row, and the contribution of each field reaches or exceeds the threshold defined in the contribution row, then this row of data can be considered normal. If the contribution of any field is lower than the threshold, or the data does not match the pattern, it indicates an anomaly. For example, if the value of field3 is

[20123] , and the range defined in the contribution row is [{10-20}], this value exceeds the defined range and is marked as an anomaly. If the value of field5 is GET, the corresponding values ​​in the contribution row are 53% and 47%, which means that the similarity of field5 is only 53% and 47%, which may not be enough to prove that the data is completely normal and requires further analysis.

[0106] Table 1

[0107]

[0108] If a system process or service is in a dead state, the abnormal number and proportion of log patterns can be merged for analysis. When the log pattern proportion suddenly changes to a certain proportion, such as a 5% sudden change, the pattern proportion alarm is triggered to quickly identify the dead process or service.

[0109] The present invention changes the mode of operation and maintenance monitoring and fault location. It does not need to preset monitoring and alarm rules in advance. It can automatically identify abnormal logs and find abnormal states through algorithms, which greatly improves the monitoring efficiency and the accuracy of abnormal alarms. When an abnormal state is identified, the relevant logs of the abnormal state are directly pushed according to the preset display form, which improves the efficiency of fault location.

[0110] Embodiment 2

[0111] Fig. 9 This is a schematic diagram of the structure of a detection device for concealing abnormal processes provided by Embodiment 2 of the present invention. Fig. 9 As shown, the device includes: a structured format log acquisition module 910 to be compared, a log pattern matching module 920, a log parameter extraction module 930 and a hidden abnormal process determination module 940.

[0112] The structured format log acquisition module 910 is used to collect each process log generated by the monitoring process in the current detection interval, and determine the structured format log to be compared corresponding to each process log;

[0113] The log pattern matching module 920 is used to match each structured format log to be compared with each log pattern in the pattern library;

[0114] The log parameter extraction module 930 is used to count the number of structured format logs to be compared that are successfully matched under each log mode, and extract the log parameters contained in each structured format log to be compared according to the matched log mode;

[0115] The hidden abnormal process determination module 940 is used to determine that the monitoring process is a hidden abnormal process when it is detected that the number of structured format logs to be compared that are successfully matched under each log mode meets the abnormal condition of the proportion of the number of adjacent detection interval patterns, or the log parameters contained in each structured format log to be compared meet the parameter abnormal condition.

[0116] The technical solution of the embodiment of the present invention is to collect each process log produced by the monitoring process in the current detection interval, and determine the structured format log to be compared corresponding to each process log; match each structured format log to be compared with each log pattern in the pattern library; count the number of structured format logs to be compared that are successfully matched under each log pattern, and extract the log parameters contained in each structured format log to be compared according to the matched log pattern; when it is detected that the number of structured format logs to be compared that are successfully matched under each log pattern meets the abnormal condition of the proportion of the number of patterns in the adjacent detection interval, or the log parameters contained in each structured format log to be compared meet the abnormal condition of the parameter, determine that the monitoring process is a hidden abnormal process. By structuring each process log and matching it with each log pattern in the pattern library, monitoring the number of successfully matched logs and their parameters, determining the hidden abnormal process that generates the abnormal number proportion or abnormal parameter, it is realized that there is no need to preset the monitoring alarm rules in advance, and the abnormal logs are automatically identified and the abnormal state is discovered through the algorithm, thereby improving the monitoring efficiency of the hidden abnormal process and the accuracy of the abnormal alarm.

[0117] Further, based on the above embodiments, the detection device for concealing abnormal processes may also include:

[0118] A historical process log acquisition module is used to acquire multiple historical process logs that match the monitoring process; a segmentation field set acquisition module is used to perform segmentation processing on the log content in each historical process log, obtain a segmentation field set corresponding to each historical process log, and determine the field type of each segmentation field in the segmentation field set; a structured format log acquisition module is used to replace the segmentation field of the target field type in each segmentation field set with a standardized field, and obtain a structured format log corresponding to each historical process log; a pattern library acquisition module is used to calculate the similarity between the segmentation fields in different structured format logs through the Smith-Waterman algorithm, and use a hierarchical clustering algorithm to perform clustering processing to obtain a hierarchical nested clustering tree as a pattern library.

[0119] Further, based on the above embodiments, the detection device for concealing abnormal processes may also include:

[0120] An event type field list acquisition module is used to extract event type fields from each historical process log respectively, and form an event type field list with the extracted event type fields; a comparison event type field detection module is used to extract the comparison event type fields corresponding to each process log respectively, and detect whether each comparison event type field exists in the event type field list; a matching abandonment module is used to abandon matching the structured format log to be compared of the first target process log with each log pattern in the pattern library if the comparison event type field of the first target process log does not exist in the event type field list.

[0121] Further, based on the above embodiments, the detection device for concealing abnormal processes may also include:

[0122] An unsuccessful matching log identification module is used to identify the structured format log to be compared of the second target process log that has not successfully matched any log pattern according to the matching result;

[0123] The pattern library update module is used to update the pattern library using the structured format log to be compared of the first target process log and / or the structured format log to be compared of the second target process log when the pattern library update condition is met.

[0124] On the basis of the above embodiments, the hidden abnormal process determination module 940 is specifically used for:

[0125] In the current detection interval and the previous adjacent historical detection interval, obtain the difference in the number of structured format logs to be compared that are successfully matched in each log mode, and the difference in the proportion of the number of structured format logs to be compared that are successfully matched to the total number of structured format logs to be compared that are successfully matched in all log modes; if it is determined that the quantity difference or the proportion difference in any log mode exceeds the preset difference threshold, it is determined that an abnormal condition in the quantity proportion of adjacent detection interval modes has been detected.

[0126] Based on the above embodiments, the log parameter extraction module 930 is specifically used to:

[0127] The field position of the variable characteristic field marked in the log mode matched by each structured format log to be compared is obtained; and the log parameters contained in each structured format log to be compared are extracted according to the field position.

[0128] Further, based on the above embodiments, the hidden abnormal process determination module 940 can also be specifically used for:

[0129] If the continuous value parameter contained in at least one of the structured format logs to be compared does not conform to the preset parameter distribution law, it is determined that the parameter abnormality condition is met; if the constant parameter value contained in at least one of the structured format logs to be compared is not unique, it is determined that the parameter abnormality condition is met; if the discrete value parameter contained in at least one of the structured format logs to be compared does not have a value within the preset value set range, it is determined that the parameter abnormality condition is met; or if the high base value parameter type contained in at least one of the structured format logs to be compared is inconsistent, it is determined that the parameter abnormality condition is met.

[0130] The device for detecting hidden abnormal processes provided by the embodiment of the present invention can execute the method for detecting hidden abnormal processes provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0131] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0132] Embodiment 3

[0133] Fig.10 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0134] like Fig.10 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0135] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0136] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as the detection method of concealing abnormal processes, that is:

[0137] Collect each process log generated by the monitoring process in the current detection interval, and determine the structured format log to be compared corresponding to each process log; match each structured format log to be compared with each log pattern in the pattern library, wherein the pattern library is a tree structure, and each node of the tree structure corresponds to a log pattern;

[0138] Count the number of structured format logs to be compared that are successfully matched under each log mode, and extract log parameters contained in each structured format log to be compared according to the matched log mode;

[0139] When it is detected that the number of structured format logs to be compared that are successfully matched in each log mode meets the abnormal condition of the proportion of the number of adjacent detection interval patterns, or when the log parameters contained in each structured format log to be compared meet the abnormal condition of the parameters, the monitoring process is determined to be a hidden abnormal process.

[0140] In some embodiments, the method for detecting hidden abnormal processes may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method for detecting hidden abnormal processes described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the method for detecting hidden abnormal processes in any other appropriate manner (e.g., by means of firmware).

[0141] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0142] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0143] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0144] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0145] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0146] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0147] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0148] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting hidden abnormal processes, characterized in that: include: Collect each process log generated by the monitoring process in the current detection interval, and determine the structured format log to be compared corresponding to each process log; Match each structured format log to be compared with each log pattern in the pattern library, wherein the pattern library is a tree structure, and each node of the tree structure corresponds to a log pattern; Count the number of structured format logs to be compared that are successfully matched under each log mode, and extract log parameters contained in each structured format log to be compared according to the matched log mode; When it is detected that the number of structured format logs to be compared that are successfully matched in each log mode meets the abnormal condition of the proportion of the number of adjacent detection interval patterns, or when the log parameters contained in each structured format log to be compared meet the abnormal condition of the parameters, the monitoring process is determined to be a hidden abnormal process.

2. The method according to claim 1, characterized in that Before matching each structured format log to be compared with each log pattern in the pattern library, the following steps are also included: Get multiple historical process logs that match the monitored process; Perform word segmentation processing on the log content in each historical process log to obtain a word segmentation field set corresponding to each historical process log, and determine the field type of each word segmentation field in the word segmentation field set; Replace the segmentation field of the target field type in each segmentation field set with a standardized field to obtain a structured format log corresponding to each historical process log; The Smith-Waterman algorithm is used to calculate the similarity between the word segmentation fields in different structured format logs, and the hierarchical clustering algorithm is used for clustering processing to obtain a hierarchical nested clustering tree as a pattern library; The clustering process is implemented by replacing one or more segmentation fields in each segmentation field set with a wildcard. A log pattern in the pattern library corresponds to a standardized segmentation field set, and the standardized segmentation field set contains one or more wildcards.

3. The method according to claim 2, characterized in that After obtaining multiple historical process logs that match the monitored process, it also includes: Extract event type fields from each historical process log, and form an event type field list with the extracted event type fields; After determining the structured format logs to be compared corresponding to each process log, the following steps are also included: Extract the comparison event type fields corresponding to each process log, and detect whether each comparison event type field exists in the event type field list; If the comparison event type field of the first target process log does not exist in the event type field list, the matching of the to-be-compared structured format log of the first target process log with each log pattern in the pattern library is abandoned.

4. The method according to claim 3, characterized in that After matching each structured format log to be compared with each log pattern in the pattern library, the following steps are also included: According to the matching result, identifying the structured format log to be compared of the second target process log that has not successfully matched any log pattern; The method further comprises: When the pattern library update condition is met, the pattern library is updated using the structured format log to be compared of the first target process log and / or the structured format log to be compared of the second target process log.

5. The method according to any one of claims 1 to 4, characterized in that: The number of structured format logs to be compared that are successfully matched in each log mode meets the abnormal ratio of the number of modes in adjacent detection intervals, including: In the current detection interval and the previous adjacent historical detection interval, obtain the difference in the number of structured format logs to be compared that are successfully matched in each log mode, and the difference in the ratio of the number of structured format logs to be compared that are successfully matched to the total number of structured format logs to be compared that are successfully matched in all log modes; If it is determined that the quantity difference or proportion difference in any log mode exceeds the preset difference threshold, it is determined that an abnormal condition of the quantity proportion of adjacent detection interval modes is detected.

6. The method according to any one of claims 1 to 4, characterized in that: According to the matched log mode, the log parameters contained in each structured format log to be compared are extracted, including: Obtain the field position of the variable feature field marked in the log mode matched by each structured format log to be compared; Extracting log parameters contained in each structured format log to be compared according to the field position; The log parameters include: continuous value parameters, constant parameters, discrete value parameters or high base values.

7. The method according to claim 6, characterized in that It is detected that the log parameters contained in each structured format log to be compared meet the parameter abnormality conditions, including: If at least one of the continuous value parameters contained in the structured format log to be compared does not conform to the preset parameter distribution law, it is determined that the parameter abnormality condition is met; If the value of the constant parameter contained in at least one of the structured format logs to be compared is not unique, it is determined that the parameter abnormality condition is met; If at least one discrete value parameter contained in the structured format log to be compared does not fall within a preset value set range, it is determined that the parameter abnormality condition is met; or If the parameter type of the high base value contained in at least one of the structured format logs to be compared is inconsistent, it is determined that the parameter abnormality condition is met.

8. A detection device for concealing abnormal processes, characterized in that: include: The module for obtaining structured format logs to be compared is used to collect each process log produced by the monitoring process in the current detection interval, and determine the structured format logs to be compared corresponding to each process log; The log pattern matching module is used to match each structured format log to be compared with each log pattern in the pattern library; The log parameter extraction module is used to count the number of structured format logs to be compared that are successfully matched under each log mode, and extract the log parameters contained in each structured format log to be compared according to the matched log mode; The hidden abnormal process determination module is used to determine that the monitoring process is a hidden abnormal process when it is detected that the number of structured format logs to be compared that successfully match each log mode meets the abnormal condition of the proportion of the number of adjacent detection interval patterns, or the log parameters contained in each structured format log to be compared meet the parameter abnormal condition.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for detecting hidden abnormal processes according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for detecting hidden abnormal processes according to any one of claims 1 to 7 when executed.

Images