Abnormal behavior detection method and device based on log, equipment and storage medium
By acquiring global features and natural language processing features from the log set, and using machine learning algorithms to train the detection model, the problem of slow anomaly detection speed in the system was solved, and automated abnormal behavior detection and unknown sample identification were achieved.
Patent Information
- Application Number
- CN202410565902.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-08
- Publication Date
- 2025-11-11
AI Technical Summary
In existing technologies, system anomaly detection is slow and cannot effectively identify abnormal behavior in large-scale systems. In particular, when modern systems generate a large number of logs every hour, manual inspection and keyword search are inefficient.
By acquiring a log set, extracting global features and natural language processing features, and using machine learning algorithms to train a detection model, abnormal behavior in the logs to be detected can be automatically detected.
It achieves automated anomaly detection, improves the speed and accuracy of anomaly detection in the system, and can identify the abnormal behavior of unknown samples.
Smart Images
Figure CN120929322A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of information security technology, and in particular to a log-based method, apparatus, device, and storage medium for detecting abnormal behavior. Background Technology
[0002] Operating systems typically generate logs that record detailed runtime information during system operation. These widely available logs serve as the primary data source for system anomaly detection. Traditional detection methods require developers to manually examine system logs or write rules based on their domain knowledge to detect anomalies, often supplemented by keyword searches such as "failure" or "anomaly," or regular expression matching.
[0003] However, this anomaly detection, which relies heavily on manual log inspection, has become impractical for large-scale systems. The scale and parallelism of modern systems make their behavior too complex for any single developer to understand, and modern systems are generating a massive amount of logs at a rate of approximately 50 gigabytes (Gb) per hour. This sheer volume of logs makes manually identifying key information from noisy data for anomaly detection extremely difficult and slow.
[0004] Therefore, the slow speed of system anomaly detection is a problem that this invention urgently needs to solve. Summary of the Invention
[0005] This application provides a log-based method, apparatus, device, and storage medium for detecting abnormal behavior, in order to solve the problem of slow system anomaly detection speed.
[0006] Firstly, this application provides a log-based method for detecting abnormal behavior, the method comprising:
[0007] Get the log set, which includes: multiple normal logs without abnormal behavior, and multiple abnormal logs with abnormal behavior;
[0008] Based on the log set, global features and natural language processing features of the log set are obtained through feature extraction. The global features are used to indicate the number of each parameter in the log set, and the natural language processing features are used to indicate the word frequency of each parameter in the log set.
[0009] Based on the global features and natural language processing features of the log set, a detection model is obtained by training the model using machine learning algorithms;
[0010] The system acquires the logs to be inspected and uses a detection model to inspect them, obtaining the inspection results. The inspection results are used to indicate whether the logs to be inspected exhibit any abnormal behavior.
[0011] In one possible design, based on the log set, global features and natural language processing features of the log set are obtained through feature extraction, including:
[0012] Based on the log set, the application interface call sequence of the log set is obtained through feature extraction, where the application interface call sequence is used to indicate various parameters of the application interface call;
[0013] Based on the application interface call sequence of the log set, the global features and natural language processing features of the log set are obtained.
[0014] In one possible design, the application interface call sequence includes: the application interface being called, the parameters passed when calling the application interface, and the return value of the application interface;
[0015] Based on the log set, the application interface call sequence of the log set is obtained through feature extraction, including:
[0016] Based on a preset time window, the log set is divided into multiple log sequences, where each log sequence includes at least one normal log and / or at least one abnormal log.
[0017] Based on multiple log sequences, feature extraction is used to obtain the application interface called for each log sequence, the parameters passed when calling the application interface, and the return value of the application interface.
[0018] In one possible design, global features and natural language processing features of the log set are obtained based on the sequence of application interface calls to the log set, including:
[0019] Based on the application interface called for each log sequence, the parameters passed when calling the application interface, and the return value of the application interface, the global features of the log set are obtained. Based on the application interface called for each log sequence and the parameters passed when calling the application interface, the natural language processing features of the log set are obtained.
[0020] In one possible design, natural language processing features include: bag-of-words features and inverse document frequency features;
[0021] Based on the application interface called for each log sequence and the parameters passed when calling the application interface, the natural language processing features of the log set are obtained, including:
[0022] Based on the application interface called for each log sequence and the parameters passed when calling the application interface, the bag-of-words feature and inverse document frequency feature of the log set are obtained.
[0023] In one possible design, the log collection is obtained, including:
[0024] By following up with the follow command, you can obtain the normal logs for each normal process and the abnormal logs for each abnormal process.
[0025] In one possible design, the log to be detected is acquired, and the log is detected using a detection model to obtain the detection results, including:
[0026] Obtain the log to be detected, and obtain the global features and natural language processing features of the log to be detected through feature extraction;
[0027] The global features and natural language processing features of the logs to be detected are imported into the detection model to obtain the detection results.
[0028] Secondly, this application provides a log-based abnormal behavior detection device, the device comprising:
[0029] The collection module is used to obtain a log set, which includes: multiple normal logs without abnormal behavior, and multiple abnormal logs with abnormal behavior;
[0030] The feature processing module is used to obtain global features and natural language processing features of the log set through feature extraction. The global features are used to indicate the number of each parameter in the log set, and the natural language processing features are used to indicate the word frequency of each parameter in the log set.
[0031] The training module is used to train the model using machine learning algorithms based on the global features and natural language processing features of the log set, so as to obtain the detection model.
[0032] The detection module is used to acquire the logs to be detected, and to detect the logs through the detection model to obtain the detection results. The detection results are used to indicate whether there is any abnormal behavior in the logs to be detected.
[0033] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;
[0034] The memory stores the instructions that the computer executes;
[0035] When a processor executes computer execution instructions stored in memory, it implements a log-based abnormal behavior detection method for the first aspect of the invention.
[0036] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement a log-based abnormal behavior detection method according to the first aspect of the invention.
[0037] Fifthly, this application provides a computer program product, including a computer program, which, when executed by a processor, is used to implement a log-based abnormal behavior detection method according to the first aspect of the invention.
[0038] This application provides a log-based method, apparatus, device, and storage medium for detecting abnormal behavior. The method involves acquiring a log set, which includes multiple normal logs without abnormal behavior and multiple abnormal logs with abnormal behavior. Based on the log set, global features and natural language processing (NLP) features of the log set are extracted. Based on the global features and NLP features of the log set, a detection model is trained using a machine learning algorithm. The method then acquires the logs to be detected and uses the detection model to detect them, obtaining the detection result. This achieves the following technical effects: by acquiring the logs to be detected and using the detection model to detect them, the problem of slow system anomaly detection speed is solved; by extracting global features and NLP features of the log set through feature extraction, the problem of being unable to identify unknown samples is solved. Attached Figure Description
[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0040] Figure 1 A flowchart illustrating a log-based abnormal behavior detection method provided in this application embodiment. Figure 1 ;
[0041] Figure 2 A flowchart illustrating a log-based abnormal behavior detection method provided in this application embodiment. Figure 2 ;
[0042] Figure 3 A schematic diagram of the structure of a log-based abnormal behavior detection device provided in an embodiment of this application;
[0043] Figure 4 This is a schematic diagram of the structure of the electronic device hardware provided in the embodiments of this application.
[0044] Figure label:
[0045] 100 - Log-based anomaly detection device;
[0046] 110 - Data Collection Module; 120 - Feature Processing Module; 130 - Training Module; 140 - Detection Module;
[0047] 200 - Electronic device; 210 - Processor; 220 - Memory; 230 - Communication component; 240 - Bus. Detailed Implementation
[0048] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0049] In the embodiments of this application, the terms "first" and "second" are used to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, nor do they necessarily imply difference. It should be noted that in the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete manner. In the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more.
[0050] It should be noted that the phrase "at...time" in the embodiments of this application can refer to the instant at which a certain situation occurs, or to a period of time after the occurrence of a certain situation; the embodiments of this application do not specifically limit this. Furthermore, the log-based abnormal behavior detection method provided in the embodiments of this application is merely an example; a log-based abnormal behavior detection method may also include more or less content.
[0051] Operating systems typically generate logs that record detailed runtime information during system operation. These widely available logs serve as the primary data source for system anomaly detection. Traditional detection methods require developers to manually examine system logs or write rules based on their domain knowledge to detect anomalies, often supplemented by keyword searches such as "failure" or "anomaly," or regular expression matching.
[0052] However, this anomaly detection, which relies heavily on manual log inspection, has become impractical for large-scale systems. The scale and parallelism of modern systems make their behavior too complex for any single developer to understand. Furthermore, modern systems generate a massive amount of logs at a rate of approximately 50 Gb per hour (about 120 million to 200 million lines). This sheer volume of logs makes manually identifying key information from noisy data for anomaly detection extremely difficult and slow.
[0053] Therefore, the slow speed of system anomaly detection is a problem that this invention urgently needs to solve.
[0054] Based on this, embodiments of this application provide a log-based method, apparatus, device, and storage medium for detecting abnormal behavior, which can be used in the field of information security technology and aims to solve the above-mentioned technical problems of the prior art.
[0055] Figure 1 A flowchart illustrating a log-based abnormal behavior detection method provided in this application embodiment. Figure 1 .like Figure 1 As shown, the method includes:
[0056] S101. Obtain the log set, wherein the log set includes: multiple normal logs without abnormal behavior, and multiple abnormal logs with abnormal behavior;
[0057] Specifically, during the training phase, multiple normal and abnormal logs are collected as training data for the abnormal behavior detection model.
[0058] S102. Based on the log set, obtain the global features and natural language processing features of the log set through feature extraction;
[0059] Specifically, global features are used to indicate the number of each parameter in the log set, and natural language processing features are used to indicate the word frequency of each parameter in the log set; the logs are sorted out to extract global features and natural language processing features from the log set.
[0060] S103. Based on the global features and natural language processing features of the log set, a detection model is obtained by training the model using machine learning algorithms.
[0061] Specifically, machine learning algorithms are used to train models on the extracted global features and natural language processing features. The trained models are then saved for use in the detection phase. Furthermore, the machine learning algorithm can be a gradient boosting decision tree model (Light Gradient Boosting Machine, LightGBM).
[0062] S104. Obtain the log to be detected, and detect the log to be detected through the detection model to obtain the detection result;
[0063] Specifically, the detection results are used to indicate whether the logs to be detected exhibit abnormal behavior. In practice, the process log data to be detected is imported into a pre-trained abnormal process log detection machine learning model. If the model detects a process log file as abnormal, it is then detected.
[0064] This application provides a log-based method, apparatus, device, and storage medium for detecting abnormal behavior. The method involves acquiring a log set, which includes multiple normal logs without abnormal behavior and multiple abnormal logs with abnormal behavior. Based on the log set, global features and natural language processing (NLP) features of the log set are extracted. Based on the global features and NLP features of the log set, a detection model is trained using a machine learning algorithm. The method then acquires the logs to be detected and uses the detection model to detect them, obtaining the detection result. This achieves the following technical effects: by acquiring the logs to be detected and using the detection model to detect them, the problem of slow system anomaly detection speed is solved; by extracting global features and NLP features of the log set through feature extraction, the problem of being unable to identify unknown samples is solved.
[0065] Figure 2 A flowchart illustrating a log-based abnormal behavior detection method provided in this application embodiment. Figure 2 This embodiment is in Figure 1 Based on the examples, the log-based abnormal behavior detection method is described in detail. For example... Figure 2 As shown, the method includes:
[0066] S201. Follow up on multiple normal processes and multiple abnormal processes using the follow command to obtain the normal logs for each normal process and the abnormal logs for each abnormal process.
[0067] Specifically, the log set includes: multiple normal logs without abnormal behavior, and multiple abnormal logs with abnormal behavior; taking the Ubuntu operating system as an example, the trace command (strace-tt-Tp) can be used to trace the process, trace the underlying application programming interface (api) call information of the normal process and the abnormal process, and collect multiple abnormal logs and multiple normal logs as training data.
[0068] S202. Divide the log set into multiple log sequences according to the preset time window;
[0069] Specifically, each log sequence includes at least one normal log and / or at least one abnormal log; valuable features are extracted from the log events, specifically the system API call sequence generated during the log parsing step, with the output being an event count matrix. To extract features, the log set first needs to be divided into different groups based on time windows, where each group represents a log sequence. For this purpose, windows are applied to divide the log dataset into finite blocks.
[0070] Furthermore, three different types of windows can be used: fixed windows, sliding windows, and session windows.
[0071] Fixed windows: Both fixed windows and sliding windows are timestamp-based, recording when each log entry occurred. Each fixed window has a size, which represents the time span or duration, such as one hour or one day. The number of fixed windows depends on the preset window size, and log entries occurring within the same window are considered a log sequence.
[0072] Sliding windows: Unlike fixed windows, sliding windows consist of two attributes: window size and step size. For example, an hourly window slides every five minutes. Typically, the step size is smaller than the window size, which can lead to overlap between different windows. The number of sliding windows is usually greater than that of fixed windows, depending primarily on the window size and step size. Logs occurring within the same sliding window are also grouped into log sequences.
[0073] Session windows: Compared to the two window types mentioned above, session windows are based on identifiers rather than timestamps. Identifiers are used to mark different execution paths within some log data. For example, a distributed file system log with unique identifiers for data blocks records the allocation, writing, replication, and deletion of certain data blocks. Logs can be grouped according to identifiers, with each session window having a unique identifier.
[0074] S203. Based on multiple log sequences, obtain the application interface called for each log sequence, the parameters passed when calling the application interface, and the return value of the application interface through feature extraction;
[0075] Specifically, the application interface call sequence is used to indicate various parameters of the application interface call; the application interface call sequence includes: the application interface being called, the parameters passed when calling the application interface, and the return value of the application interface;
[0076] By analyzing the log format, we can extract the following five parameters: file number (file_id), file tag, etc.
[0077] (label), the underlying API for file calls, the application interface return value (api_params), which is the parameter passed when calling the file API, and the application interface return value (return_value), which is the API return value. Among them, when the file label is 0, it means that the file is normal; when the file label is 1, it means that the file is abnormal.
[0078] S204. Based on the application interface called for each log sequence, the parameters passed when calling the application interface, and the return value of the application interface, obtain the global characteristics of the log set.
[0079] Specifically, global features are used to indicate the number of each parameter in the log collection; global features include the following three dimensions:
[0080] API dimension: Total number of APIs called, total number of unique APIs called;
[0081] api_params dimension: total number of api_params called, total number of unique api_params called;
[0082] ReturnValue dimensions: total number of all return values, total number of unique return values, minimum return value, maximum return value, etc.
[0083] S205. Based on the application interface called for each log sequence and the parameters passed when calling the application interface, obtain the bag-of-words feature and inverse document frequency feature of the log set.
[0084] Specifically, Natural Language Processing (NLP) features are used to indicate the word frequency of each parameter in the log set; NLP features include: bag-of-words features and; Natural Language Processing (NLP) features include:
[0085] Bag-of-words features are obtained by converting api and api_params.
[0086] Term frequency-inverse document frequency (TF-IDF) is obtained by using api and api_params.
[0087] S206. Based on the global features and natural language processing features of the log set, a detection model is obtained by training the model using machine learning algorithms.
[0088] S206 and S103 are similar, and will not be described again in this embodiment.
[0089] S207. Obtain the log to be detected, and extract the global features and natural language processing features of the log to be detected through feature extraction; import the global features and natural language processing features of the log to be detected into the detection model to obtain the detection results;
[0090] Specifically, the detection results indicate whether the log file to be detected exhibits abnormal behavior. For the log file to be detected, global features and NLP features are extracted, and the feature data of the log document to be detected is imported into the abnormal behavior detection machine learning model that has already been trained during the training phase. If the model detects a log process document as abnormal, it is then detected.
[0091] The method provided in this embodiment obtains a log set, which includes multiple normal logs without abnormal behavior and multiple abnormal logs with abnormal behavior. Based on the log set, global features and natural language processing (NLP) features of the log set are obtained through feature extraction. Based on the global features and NLP features of the log set, a machine learning algorithm is used to train a detection model. The method then obtains the logs to be detected and uses the detection model to detect them, obtaining the detection result. This achieves the following technical effects: by obtaining the logs to be detected and using the detection model to detect them, the problem of slow system anomaly detection speed is solved; by obtaining the global features and NLP features of the log set through feature extraction, the problem of being unable to identify unknown samples is solved; by extracting abnormal behavior sequences from the process logs, the extraction parameters are optimized, simplifying the steps and improving efficiency while ensuring detection effectiveness.
[0092] In this embodiment of the invention, electronic devices or main control devices can be divided into functional modules according to the above method examples. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional module. It should be noted that the module division in this embodiment of the invention is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.
[0093] Figure 3 This is a schematic diagram of a log-based abnormal behavior detection device provided in an embodiment of this application. Figure 3 As shown, the log-based abnormal behavior detection device 100 includes: a collection module 110, a feature processing module 120, a training module 130, and a detection module 140;
[0094] The collection module 110 is used to obtain a log set, wherein the log set includes: multiple normal logs without abnormal behavior, and multiple abnormal logs with abnormal behavior;
[0095] The feature processing module 120 is used to obtain global features and natural language processing features of the log set through feature extraction based on the log set. The global features are used to indicate the number of each parameter in the log set, and the natural language processing features are used to indicate the word frequency of each parameter in the log set.
[0096] Training module 130 is used to train a detection model using machine learning algorithms based on the global features and natural language processing features of the log set.
[0097] The detection module 140 is used to acquire the log to be detected, and to detect the log to be detected through the detection model to obtain the detection result. The detection result is used to indicate whether there is abnormal behavior in the log to be detected.
[0098] In one possible design, the feature processing module 120 is specifically used for:
[0099] Based on the log set, the application interface call sequence of the log set is obtained through feature extraction, where the application interface call sequence is used to indicate various parameters of the application interface call;
[0100] Based on the application interface call sequence of the log set, the global features and natural language processing features of the log set are obtained.
[0101] In one possible design, the application programming interface (API) call sequence includes: the API being called, the parameters passed when calling the API, and the API return value; the feature processing module 120 is specifically used for:
[0102] Based on a preset time window, the log set is divided into multiple log sequences, where each log sequence includes at least one normal log and / or at least one abnormal log.
[0103] Based on multiple log sequences, feature extraction is used to obtain the application interface called for each log sequence, the parameters passed when calling the application interface, and the return value of the application interface.
[0104] In one possible design, the feature processing module 120 is specifically used for:
[0105] Based on the application interface called for each log sequence, the parameters passed when calling the application interface, and the return value of the application interface, the global features of the log set are obtained. Based on the application interface called for each log sequence and the parameters passed when calling the application interface, the natural language processing features of the log set are obtained.
[0106] In one possible design, the natural language processing features include: bag-of-words features and inverse document frequency features; the feature processing module 120 is specifically used for:
[0107] Based on the application interface called for each log sequence and the parameters passed when calling the application interface, the bag-of-words feature and inverse document frequency feature of the log set are obtained.
[0108] In one possible design, the collection module 110 is specifically used for:
[0109] By following up with the follow command, you can obtain the normal logs for each normal process and the abnormal logs for each abnormal process.
[0110] In one possible design, the detection module 140 is specifically used for:
[0111] Obtain the log to be detected, and obtain the global features and natural language processing features of the log to be detected through feature extraction;
[0112] The global features and natural language processing features of the logs to be detected are imported into the detection model to obtain the detection results.
[0113] This embodiment provides a log-based abnormal behavior detection device that can execute a log-based abnormal behavior detection method described in the above embodiment. Its implementation principle and technical effects are similar, and will not be repeated here.
[0114] In the aforementioned specific implementation of a log-based abnormal behavior detection device, each module can be implemented as a processor. The processor can execute computer execution instructions stored in the memory, thereby enabling the processor to execute the aforementioned log-based abnormal behavior detection method.
[0115] Figure 4 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application. For example... Figure 4 As shown, the electronic device 200 includes at least one processor 210 and a memory 220. The electronic device 200 also includes a communication component 230. The processor 210, memory 220, and communication component 230 are connected via a bus 240.
[0116] In the specific implementation process, at least one processor 210 executes computer execution instructions stored in memory 220, causing at least one processor 210 to execute a log-based abnormal behavior detection method as executed on the electronic device side as described above.
[0117] The specific implementation process of processor 210 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.
[0118] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.
[0119] The memory may include high-speed random-access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.
[0120] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.
[0121] The above description of the functions implemented by electronic devices and main control devices has introduced the solutions provided by the embodiments of the present invention. It is understood that, in order to implement the above functions, the electronic device or main control device includes hardware structures and / or software modules corresponding to the execution of each function. By combining the units and algorithm steps of the various examples described in the embodiments of the present invention, the embodiments of the present invention can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of the embodiments of the present invention.
[0122] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the above-described log-based abnormal behavior detection method.
[0123] The aforementioned readable storage medium can be any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.
[0124] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in an electronic device or a host device.
[0125] This application also provides a computer program product, comprising: a computer program stored in a readable storage medium, wherein at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause the electronic device to perform the scheme provided in any of the above embodiments.
[0126] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.
[0127] The technical solutions of this application have been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it is readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. The above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.
Claims
1. A log-based method for detecting abnormal behavior, characterized in that, The method includes: Obtain a log set, wherein the log set includes: multiple normal logs without abnormal behavior, and multiple abnormal logs with abnormal behavior; Based on the log set, global features and natural language processing features of the log set are obtained through feature extraction, wherein the global features are used to indicate the number of each parameter in the log set, and the natural language processing features are used to indicate the word frequency of each parameter in the log set; Based on the global features and natural language processing features of the log set, a detection model is obtained by training a model using a machine learning algorithm. Obtain the log to be detected, and detect the log to be detected through the detection model to obtain the detection result, wherein the detection result is used to indicate whether the log to be detected has abnormal behavior.
2. The method according to claim 1, characterized in that, The step of obtaining global features and natural language processing features of the log set through feature extraction based on the log set includes: Based on the log set, an application interface call sequence of the log set is obtained through feature extraction, wherein the application interface call sequence is used to indicate various parameters of the application interface call; Based on the application interface call sequence of the log set, the global features and natural language processing features of the log set are obtained.
3. The method according to claim 2, characterized in that, The application interface call sequence includes: the application interface being called, the parameters passed when calling the application interface, and the return value of the application interface; The step of obtaining the application interface call sequence of the log set through feature extraction based on the log set includes: According to a preset time window, the log set is divided into multiple log sequences, wherein each log sequence includes at least one normal log and / or at least one abnormal log. Based on the multiple log sequences, the application interface called, the parameters passed when calling the application interface, and the return value of the application interface are obtained for each log sequence through feature extraction.
4. The method according to claim 3, characterized in that, The step of obtaining the global features and natural language processing features of the log set based on the application interface call sequence of the log set includes: The global features of the log set are obtained based on the application interface called for each log sequence, the parameters passed when calling the application interface, and the return value of the application interface. The natural language processing features of the log set are also obtained based on the application interface called for each log sequence and the parameters passed when calling the application interface.
5. The method according to claim 4, characterized in that, The natural language processing features include: bag-of-words features and inverse document frequency features; The step of obtaining the natural language processing features of the log set based on the application programming interface (API) invoked for each log sequence and the parameters passed when invoking the API includes: Based on the application interface called for each log sequence and the parameters passed when calling the application interface, the bag-of-words feature and inverse document frequency feature of the log set are obtained.
6. The method according to claim 1, characterized in that, The log collection to be acquired includes: By following up with the follow command, multiple normal processes and multiple abnormal processes are obtained, and normal logs corresponding to each normal process and abnormal logs corresponding to each abnormal process are obtained.
7. The method according to any one of claims 1-6, characterized in that, The process of acquiring the log to be detected and detecting the log using the detection model to obtain the detection result includes: The log to be detected is obtained, and the global features and natural language processing features of the log to be detected are obtained through feature extraction; The global features and natural language processing features of the log to be detected are imported into the detection model to obtain the detection result.
8. A log-based abnormal behavior detection device, characterized in that, The device includes: A collection module is used to acquire a log set, wherein the log set includes: multiple normal logs without abnormal behavior, and multiple abnormal logs with abnormal behavior; The feature processing module is used to obtain global features and natural language processing features of the log set through feature extraction based on the log set, wherein the global features are used to indicate the number of each parameter in the log set, and the natural language processing features are used to indicate the word frequency of each parameter in the log set; The training module is used to train a detection model using machine learning algorithms based on the global features and natural language processing features of the log set. The detection module is used to acquire the log to be detected and to detect the log to be detected through the detection model to obtain the detection result, wherein the detection result is used to indicate whether the log to be detected has abnormal behavior.
9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; When the processor executes the computer execution instructions stored in the memory, it is used to implement the log-based abnormal behavior detection method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the log-based abnormal behavior detection method as described in any one of claims 1 to 7.