Log processing method and system, electronic equipment, medium and program product

Abstract

The invention provides a log processing method and system, electronic equipment, a medium and a program product. The method comprises the following steps: acquiring real-time running logs corresponding to a plurality of devices, and obtaining a real-time log group according to the real-time running logs; an anomaly detection mode corresponding to each data category identifier is obtained, and the anomaly detection mode is used for sensing whether behavior anomaly exists in the real-time log group or not; for each data category identifier, alarm information corresponding to the real-time log group is obtained based on an anomaly detection mode corresponding to the data category identifier and the real-time log group, and the alarm information comprises real-time running logs with abnormal behaviors in the real-time log group; according to the alarm information, real-time alarm data is obtained, and the real-time alarm data is used for generating a real-time alarm instruction.

Description

Technical Field

[0001] The present application relates to the field of log management and analysis, and in particular to a log processing method, system, electronic device, medium and program product. Background Art

[0002] Log management and analysis technology involves a variety of tools and methods for collecting, storing, analyzing, and visualizing log data for troubleshooting, performance optimization, security monitoring, and compliance checking.

[0003] In the related art, a management device is often used to manage and analyze the logs of a single device in a device cluster, where the device cluster includes multiple single devices. Specifically, the management device obtains the operation log of a single device to realize log collection; then, the management device performs log analysis on each single device based on the alarm rules to realize log monitoring and obtain an alarm log.

[0004] However, for any single device in the device cluster, when the real-time alarm log of the device is similar to the historical alarm log of a non-device, a technical problem of low efficiency of the log monitoring process will occur. Summary of the invention

[0005] The present application provides a log processing method, system, electronic device, medium and program product to solve the technical problem of insufficient system operation stability.

[0006] In a first aspect, the present application provides a log processing method, comprising:

[0007] Acquire real-time operation logs corresponding to each of the multiple devices, and obtain real-time log groups according to the real-time operation logs, wherein different real-time log groups represent multiple real-time operation logs corresponding to different data categories, and the real-time log groups include the corresponding data category identifiers and the multiple real-time operation logs corresponding to the data category identifiers;

[0008] Obtaining an anomaly detection method corresponding to each of the data category identifiers, wherein the anomaly detection method is used to sense whether there is a behavioral anomaly in the real-time log group;

[0009] For each of the data category identifiers, based on the anomaly detection method corresponding to the data category identifier and the real-time log group, obtain the alarm information corresponding to the real-time log group, wherein the alarm information includes the real-time operation log with abnormal behavior in the real-time log group;

[0010] According to the alarm information, real-time alarm data is obtained, wherein the real-time alarm data is used to generate a real-time alarm instruction.

[0011] Optionally, in the method described above, the anomaly detection method includes a feature matching algorithm and an event alarm feature, and the anomaly detection method corresponding to the data category identifier and the real-time log group are obtained to obtain the alarm information corresponding to the real-time log group, including:

[0012] Based on the feature matching algorithm and the real-time log group, an alarm log group corresponding to the data category identifier is obtained, wherein the alarm log group belongs to the real-time log group, and the alarm log group includes a plurality of real-time operation logs containing the event alarm feature;

[0013] The alarm information is obtained according to the alarm log group.

[0014] Optionally, in the method described above, obtaining the alarm log group corresponding to the data category identifier based on the feature matching algorithm and the real-time log group includes:

[0015] According to the real-time log group, a plurality of user IPs and native command records corresponding to the plurality of user IPs are obtained, and according to the plurality of user IPs, a plurality of target IPs corresponding to the plurality of IP categories are obtained, wherein the IP category indicates whether the target IP corresponding to the target IP is a user IP in a blacklist or a whitelist;

[0016] According to the feature matching algorithm, the behavior feature matching methods corresponding to the multiple IP categories are obtained;

[0017] For each of the target IPs, according to the target IP, the native command record corresponding to the target IP, the real-time operation log corresponding to the target IP, and the behavior feature matching method, a sub-alarm log corresponding to the target IP is obtained;

[0018] The alarm log group is obtained according to the multiple target IPs and their corresponding sub-alarm logs.

[0019] Optionally, in the above method, obtaining the alarm information corresponding to the real-time log group based on the anomaly detection method corresponding to the data category identifier and the real-time log group includes:

[0020] Based on the anomaly detection method corresponding to the data category identifier and the real-time log group, obtaining initial alarm information corresponding to the real-time log group;

[0021] When detecting a third-party alarm log, the alarm information is obtained according to the third-party alarm log and the initial alarm information.

[0022] Optionally, the method as described above, before obtaining the real-time alarm data according to the alarm information, further comprises:

[0023] Acquire an alarm frequency parameter, and obtain a target noise reduction method according to the alarm frequency parameter, wherein the alarm frequency parameter represents the frequency and number of times the alarm information is obtained;

[0024] According to the alarm information, real-time alarm data is obtained, including:

[0025] Based on the target noise reduction method, the warning information is merged and noise reduced to obtain target information;

[0026] The real-time alarm data is obtained according to the target information.

[0027] Optionally, the method as described above further comprises:

[0028] Obtaining a processing speed deviation of each thread in the thread pool, wherein the processing speed deviation is the difference between the target processing speed and the real-time processing speed of the thread; and summing a plurality of the processing speed deviations to obtain a total speed deviation;

[0029] For each thread, according to the total speed deviation and the processing speed deviation corresponding to the thread, a thread resource change rate corresponding to the thread is obtained;

[0030] Obtaining a current resource amount corresponding to the thread, and obtaining a target resource amount corresponding to the thread according to the thread resource change rate and the current resource amount;

[0031] Resource configuration information is generated according to the target resource amount, wherein the resource configuration information is used to generate resource configuration instructions for the thread pool.

[0032] In a second aspect, the present application provides a log processing system, comprising:

[0033] A first acquisition module is used to acquire real-time operation logs corresponding to each of the multiple devices, and obtain a real-time log group according to the real-time operation logs, wherein different real-time log groups represent multiple real-time operation logs corresponding to different data categories, and the real-time log group includes a corresponding data category identifier and the multiple real-time operation logs corresponding to the data category identifier;

[0034] A second acquisition module is used to acquire an anomaly detection method corresponding to each of the data category identifiers, wherein the anomaly detection method is used to sense whether there is a behavioral anomaly in the real-time log group;

[0035] A processing module, for obtaining, for each of the data category identifiers, alarm information corresponding to the real-time log group based on an anomaly detection method corresponding to the data category identifier and a real-time log group, wherein the alarm information includes a real-time operation log with abnormal behavior in the real-time log group;

[0036] A generating module is used to obtain real-time alarm data according to the alarm information, wherein the real-time alarm data is used to generate a real-time alarm instruction.

[0037] Optionally, the anomaly detection method includes a feature matching algorithm and an event alarm feature. The processing module, when executing the anomaly detection method corresponding to the data category identifier and the real-time log group to obtain the alarm information corresponding to the real-time log group, is used to:

[0038] Based on the feature matching algorithm and the real-time log group, an alarm log group corresponding to the data category identifier is obtained, wherein the alarm log group belongs to the real-time log group, and the alarm log group includes a plurality of real-time operation logs containing the event alarm feature;

[0039] The alarm information is obtained according to the alarm log group.

[0040] Optionally, the processing module, when executing the feature matching algorithm and the real-time log group to obtain the alarm log group corresponding to the data category identifier, is used to:

[0041] According to the real-time log group, a plurality of user IPs and native command records corresponding to the plurality of user IPs are obtained, and according to the plurality of user IPs, a plurality of target IPs corresponding to the plurality of IP categories are obtained, wherein the IP category indicates whether the target IP corresponding to the target IP is a user IP in a blacklist or a whitelist;

[0042] According to the feature matching algorithm, the behavior feature matching methods corresponding to the multiple IP categories are obtained;

[0043] For each of the target IPs, according to the target IP, the native command record corresponding to the target IP, the real-time operation log corresponding to the target IP, and the behavior feature matching method, a sub-alarm log corresponding to the target IP is obtained;

[0044] The alarm log group is obtained according to the multiple target IPs and their corresponding sub-alarm logs.

[0045] Optionally, the processing module, when executing the anomaly detection method and the real-time log group corresponding to the data category identifier to obtain the alarm information corresponding to the real-time log group, is used to:

[0046] Based on the anomaly detection method corresponding to the data category identifier and the real-time log group, obtaining initial alarm information corresponding to the real-time log group;

[0047] When detecting a third-party alarm log, the alarm information is obtained according to the third-party alarm log and the initial alarm information.

[0048] Optionally, the above system further includes:

[0049] A first determination module is used to obtain an alarm frequency parameter, and obtain a target noise reduction method according to the alarm frequency parameter, wherein the alarm frequency parameter represents the frequency and number of times the alarm information is obtained;

[0050] Accordingly, the above generation module, when executing to obtain real-time alarm data according to the alarm information, is used to:

[0051] Based on the target noise reduction method, the warning information is merged and noise reduced to obtain target information;

[0052] The real-time alarm data is obtained according to the target information.

[0053] Optionally, the above system further includes:

[0054] The second determination module is used for:

[0055] Obtaining a processing speed deviation of each thread in the thread pool, wherein the processing speed deviation is the difference between the target processing speed and the real-time processing speed of the thread; and summing a plurality of the processing speed deviations to obtain a total speed deviation;

[0056] For each thread, according to the total speed deviation and the processing speed deviation corresponding to the thread, a thread resource change rate corresponding to the thread is obtained;

[0057] Obtaining a current resource amount corresponding to the thread, and obtaining a target resource amount corresponding to the thread according to the thread resource change rate and the current resource amount;

[0058] Resource configuration information is generated according to the target resource amount, wherein the resource configuration information is used to generate resource configuration instructions for the thread pool.

[0059] According to a third aspect, an electronic device includes: a processor, and a memory communicatively connected to the processor;

[0060] The memory stores computer-executable instructions;

[0061] The processor executes the computer-executable instructions stored in the memory to implement the log processing method as described in the first aspect.

[0062] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the log processing method as described in the first aspect.

[0063] In a fifth aspect, a computer program product includes a computer program, and when the computer program is executed by a processor, the application server disaster recovery method as described in the first aspect is implemented.

[0064] The log processing method, system, electronic device, medium and program product provided by the present application obtain the real-time operation logs corresponding to each of the multiple devices, and then obtain the real-time log group corresponding to at least one device according to the real-time operation logs corresponding to each of the multiple devices, thereby integrating the real-time operation logs of at least one device together; obtain the anomaly detection method corresponding to each data category identifier, thereby determining the method of sensing whether there is a behavioral abnormality in the real-time log group; for each data category identifier, based on the anomaly detection method corresponding to the data category identifier, determine the real-time operation log with behavioral abnormality from the real-time log groups corresponding to the multiple devices, thereby obtaining the alarm information corresponding to the real-time log group; obtain the real-time alarm data according to the alarm information, thereby achieving the beneficial effect of accurately generating the real-time alarm instruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0066] Figure 1 Schematic diagram of the log processing system architecture provided for this application;

[0067] Figure 2 Schematic diagram of application scenarios of the log processing method provided in this application;

[0068] Figure 3 A flow chart of the log processing method provided for this application;

[0069] Figure 4 Another flow chart of the log processing method provided by this application;

[0070] Figure 5 A schematic diagram of the structure of the log processing system provided for this application;

[0071] Figure 6 A schematic diagram of the structure of the electronic device provided in this application.

[0072] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0073] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of systems and methods consistent with some aspects of the present application as detailed in the appended claims.

[0074] First, the terms involved in this application are explained:

[0075] SIEM platform: It is the abbreviation of Security Information and Event Management. It is a system responsible for collecting security log data from enterprise security controls, host operating systems, enterprise applications and other software, and performing analysis and reporting.

[0076] Elasticsearch: A distributed search and analysis engine that can store, search, and analyze large amounts of data, hereinafter referred to as ES index.

[0077] Filebeat: A log file shipping tool / agent, also a type of beats, used for data collection and data.

[0078] Logstash: A data collection engine.

[0079] Kafka: is a high-throughput distributed publish-subscribe messaging system that can handle all the action stream data of consumers on the website. This action (web browsing, search and other user actions) is a key factor in many social functions on the modern network. This data is usually solved by processing logs and log aggregation due to throughput requirements.

[0080] The log processing method provided in this application can be applied to Figure 1 The SIEM platform system architecture diagram is shown in Figure 1. Figure 1 As shown in the figure, the SIEM platform includes: Elasticsearch, Logstash and Kibana. It should be noted that Figure 1The various technical scenarios of the SIEM platform shown in the figure include: traditional IT infrastructure such as network devices, servers and databases, among which network device monitoring is used to monitor the logs and events of network devices such as routers, switches, and firewalls to detect network attacks and abnormal traffic. Servers and operating systems are used to collect and analyze logs of operating systems such as Windows, Linux, and Unix to detect system vulnerabilities and abnormal behaviors. Database security is used to monitor database access and operation logs to prevent data leakage and unauthorized access; virtualization and containerization environments such as virtual machines and containers, among which virtual machine monitoring is used to monitor the logs and events of virtual machines (such as VMware, Hyper-V, and KVM) to ensure the security of virtualized environments, and container security is used to monitor containers ( It also includes cloud computing environments (such as public clouds, private clouds, and cloud-native applications), applications and services (such as Web applications and APIs), user behavior analysis (such as user activity and privileged user monitoring), industrial control systems (ICS) and the Internet of Things (IoT), email and communications, endpoint security (such as endpoint detection and response, anti-malware), threat intelligence integration (such as external and internal threat intelligence), and compliance and auditing (such as compliance checks and audit log management). By integrating log management, threat detection, behavior analysis, and compliance management, it helps organizations achieve comprehensive security monitoring and event management in different technical scenarios, and improves overall security situation awareness.

[0081] Figure 2 The application scenario diagram of the SIEM platform provided for this application is as follows: Figure 2 As shown, the specific application scenario of the present application is: for a device to be monitored, S101, Filebeat in the SIEM platform collects logs and sends the logs to Logstash in the SIEM platform; S102, Logstash cleans the logs and stores the cleaned data in the ES index in the SIEM platform; S103, based on the ES index and the logs in the ES index, obtain the alarm log and complete the determination of the alarm log.

[0082] Combining the above scenarios, it can be seen that in the prior art, the devices in the device cluster are all maintained by a single device. Some behaviors are normal on a single device, but when some normal behaviors are aggregated and counted as illegal or abnormal behaviors, the abnormal behavior detection results will be inaccurate, resulting in low efficiency of the log monitoring process.

[0083] The log processing method provided in this application solves the technical problem of low efficiency of the log monitoring process by utilizing the SIEM platform and the ELK stack to simultaneously determine alarm logs for multiple devices in a device cluster.

[0084] A log processing method provided in an embodiment of the present application is intended to solve the above technical problems of the prior art, and is implemented by running the above ELK stack in an appropriate hardware environment, wherein the appropriate hardware environment includes but is not limited to high-performance physical servers or virtual servers (such as VMware, Hyper-V, KVM), public cloud instances (such as AWS EC2, AzureVM, Google Cloud Compute Engine) or private cloud instances, network attached storage (NAS), storage area network (SAN) or local storage (such as hard disk or solid-state drive), high-performance computing clusters (HPC) or distributed computing nodes, load balancers, firewalls and security devices, as well as high-performance workstations for development and testing and user terminal devices for accessing the Kibana dashboard (such as PCs, laptops and mobile devices) and other hardware environments, which can provide the ELK stack with the necessary computing power, storage and network resources to ensure that the ELK stack can efficiently collect, store, analyze and visualize log data, and the embodiments of the present application are not specifically limited.

[0085] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0086] like Figure 3 As shown, a log processing method may specifically include steps S201, S202, S203 and S205, wherein:

[0087] S201. Acquire real-time operation logs corresponding to each of the multiple devices, and obtain real-time log groups based on the real-time operation logs, wherein different real-time log groups represent multiple real-time operation logs corresponding to different data categories, and the real-time log group includes its corresponding data category identifier and multiple real-time operation logs corresponding to the data category identifier, and the multiple real-time operation logs are real-time operation logs corresponding to at least one device.

[0088] The multiple devices are devices in the device cluster to be monitored corresponding to the SIEM platform, and the real-time operation logs include but are not limited to security logs such as identity authentication logs, access control logs, firewall and intrusion detection logs of each device. It should be noted that the data categories of this application include identity authentication, access control, firewall and intrusion detection; the data category identifier can achieve the purpose of distinguishing data categories, and this embodiment will not be specifically limited.

[0089] Specifically, Filebeat in the SIEM platform collects the operation logs corresponding to each of the above devices, and sends the operation logs to Logstash in the SIEM platform; Logstash performs data cleaning on the above operation logs to obtain available operation logs, wherein, available operation logs; create an audit* index for the above available operation logs in the ES index, and store the above available operation logs in the audit* index to obtain an available audit* index including real-time operation logs, wherein, the real-time operation logs are available operation logs after being stored in the audit* index, and the real-time operation logs are multiple groups, each group of real-time operation logs corresponds to a unique data category. It should be noted that the query of the real-time operation log can be performed based on the data category identifier and the above available audit* index.

[0090] S202: Obtain an anomaly detection method corresponding to each data category identifier, wherein the anomaly detection method is used to sense whether there is a behavioral anomaly in the real-time log group.

[0091] The methods for identifying whether logs of different data categories are abnormal are different. Determining appropriate anomaly detection methods for logs of different data categories can improve the efficiency of the anomaly detection process.

[0092] Specifically, select the corresponding data type in the preset alarm rules, and write a DSL or EQL statement for each data category (external statements can also be imported), configure cron expressions, alarm frequency, time range, alarm level, notification module, recipient and other information, and save it as an alarm method; obtain the query rules corresponding to the data type, wherein the query rules represent the query conditions of the abnormal log, and the above query conditions can be abnormal features; use the query rules and alarm methods corresponding to the data type as abnormality detection methods.

[0093] S203. For each data category identifier, based on the anomaly detection method corresponding to the data category identifier and the real-time log group, obtain the alarm information corresponding to the real-time log group, wherein the alarm information includes the real-time operation log with abnormal behavior in the real-time log group.

[0094] Specifically, for each data category identifier, based on the query conditions, the log corresponding to the data category identifier is queried in the audit* index to obtain the query return result. When the query return result is empty, it indicates that there is no alarm log for the data category. Otherwise, it indicates that there is an alarm log for the data category. The returned alarm log is used as alarm information, and the above alarm information is written into the alarm eventlog index.

[0095] S205. Obtain real-time alarm data according to the alarm information, wherein the real-time alarm data is used to generate a real-time alarm instruction.

[0096] Specifically, while writing the alarm information into the alarm eventlog index, the SIEM platform sends an email, text message, or DingTalk message to the recipient's corresponding terminal to notify the person in charge to conduct analysis and processing. The corresponding message to the recipient is real-time alarm data.

[0097] Among them, the person in charge of the alarm log can process the alarm log in the threat perception module and can ignore or judge it as a hidden danger or event, thereby completing the analysis and judgment of the alarm log.

[0098] The log processing method provided in the embodiment of the present application obtains the real-time operation logs corresponding to each of the multiple devices, and then obtains the real-time log group corresponding to at least one device according to the real-time operation logs corresponding to each of the multiple devices, thereby integrating the real-time operation logs of at least one device together; obtains the anomaly detection method corresponding to each data category identifier, thereby determining the method of sensing whether there is a behavioral abnormality in the real-time log group; for each data category identifier, based on the anomaly detection method corresponding to the data category identifier, determines the real-time operation log with the behavioral abnormality from the real-time log groups corresponding to the multiple devices, thereby obtaining the alarm information corresponding to the real-time log group; obtains the real-time alarm data according to the alarm information, thereby achieving the beneficial effect of accurately generating the real-time alarm instruction.

[0099] In one possible implementation, the anomaly detection method includes a feature matching algorithm and an event alarm feature. The above S203 obtains the alarm information corresponding to the real-time log group based on the anomaly detection method corresponding to the data category identifier and the real-time log group, which may specifically include S203-a1 and S203-a2, wherein:

[0100] S203-a1. Based on a feature matching algorithm and a real-time log group, an alarm log group corresponding to the data category identifier is obtained, wherein the alarm log group belongs to the real-time log group, and the alarm log group includes several real-time operation logs containing event alarm features.

[0101] It should be noted that the above “belongs to” refers to the logical relationship between sets.

[0102] In one possible implementation, the feature matching algorithm can be implemented based on keyword query or the like.

[0103] In another implementable manner, the feature matching algorithm includes: specifically, replaying the operations according to the IP of the user of a single device to obtain the native command record of the single device; judging whether the behavior corresponding to each cleaned log is abnormal according to the single device user IP, the native command record and the cleaned log, wherein if the single device user IP is a blacklist IP or a whitelist IP, no alarm is generated for operations generated by the specified whitelist IP or account, and an alarm is generated for operations generated by the specified blacklist IP or account.

[0104] S203-a2. Obtain alarm information according to the alarm log group.

[0105] Specifically, determine whether the alarm log group is empty; if so, it indicates that no exception of the data category has occurred; if not, it indicates that an exception of the data type has occurred. At this time, write each real-time running log in the alarm log group into the eventlog index to obtain the alarm information corresponding to the data category identifier.

[0106] In an implementable manner, the above S203-a1, based on the feature matching algorithm and the real-time log group, obtains the alarm log group corresponding to the data category identifier, which may specifically include S203-a1-1 to S203-a1-4, wherein:

[0107] S203-a1-1. According to the real-time log group, multiple user IPs and the native command records corresponding to each of the multiple user IPs are obtained, and according to the multiple user IPs, target IPs corresponding to each of the multiple IP classifications are obtained, wherein the IP classification indicates whether the corresponding target IP is a user IP in the blacklist or the whitelist.

[0108] The user IP can be extracted from the real-time log group, and the user IP is the user IP of a single device.

[0109] S203-a1-2. According to the feature matching algorithm, the behavior feature matching methods corresponding to the multiple IP categories are obtained.

[0110] All IP categories include: in the whitelist, in the blacklist, and neither in the whitelist nor in the blacklist. Multiple IP categories include one or more of all IP categories.

[0111] Specifically, among the behavior feature matching methods corresponding to all IP categories, the behavior feature matching methods of the IP categories that appear are screened out, and the behavior feature matching methods of the IP categories that do not appear are screened out, wherein the feature matching algorithm includes the behavior feature matching methods corresponding to all IP categories.

[0112] S203-a1-3. For each target IP, obtain the sub-alarm log corresponding to the target IP according to the target IP, the native command record corresponding to the target IP, the real-time operation log corresponding to the target IP, and the behavior feature matching method.

[0113] When the IP classification is in the whitelist, the sub-alarm log is set to empty; when the IP classification is in the blacklist, the sub-alarm log is set to non-empty, and all real-time operation logs corresponding to the user IP are returned as the result; when the IP classification is neither in the whitelist nor in the blacklist, based on the target IP, the native command record corresponding to the target IP, and the real-time operation log corresponding to the target IP, the real-time operation log corresponding to the target IP is queried.

[0114] S203-a1-4. Obtain an alarm log group according to multiple target IPs and their corresponding sub-alarm logs.

[0115] The log processing method provided in the embodiment of the present application realizes determining the alarm log based on the black and white lists by matching different feature matching methods for different user IPs.

[0116] In an implementable manner, the above S203, based on the abnormality detection method corresponding to the data category identifier and the real-time log group, obtains the alarm information corresponding to the real-time log group, which may specifically include S203-b1 and S203-b2, wherein:

[0117] S203 - b1 . Based on the anomaly detection method corresponding to the data category identifier and the real-time log group, obtain initial alarm information corresponding to the real-time log group.

[0118] S203-b2: When detecting the third-party alarm log, obtain alarm information according to the third-party alarm log and the initial alarm information.

[0119] When third-party vendors access the SIEM platform, they can send alarm logs to Kafka according to the log specifications. After the platform receives the message, it will write the alarm log eventlog index. They can also send audit logs to Kafka. After the platform receives the message, it will write the audit log audit* index, and then configure the alarm rules for matching.

[0120] The log processing method provided in the embodiment of the present application realizes real-time update of the operation log on the SIEM platform by utilizing Kafka.

[0121] In an implementable manner, before obtaining the real-time alarm data according to the alarm information in S205, the method further includes:

[0122] S204, obtaining an alarm frequency parameter, and obtaining a target noise reduction method according to the alarm frequency parameter, wherein the alarm frequency parameter represents the frequency and number of times the alarm information is obtained.

[0123] Specifically, when the alarm frequency parameter is greater than the preset alarm frequency, the target noise reduction mode is determined as merging noise reduction, otherwise, the target noise reduction mode is determined as non-merging noise reduction.

[0124] Accordingly, the above S205 obtains real-time alarm data according to the alarm information, which may specifically include S205-1 and S205-2, wherein:

[0125] S205-1. Based on the target noise reduction method, the warning information is merged and noise reduced to obtain target information.

[0126] S205-2. Obtain real-time warning data based on target information.

[0127] Specifically, obtain custom conditions such as the noise reduction time range, merge and reduce the noise of the alarm data based on the custom conditions such as the noise reduction time range, and then generate the alarm log. The data before merging is written into the detail log eventhistorylog index.

[0128] The log processing method provided in the embodiment of the present application can achieve the beneficial effect of reducing the amount of real-time alarm data and even improving the stability of the alarm process by merging and denoising the alarm information when the alarm is frequent.

[0129] In the related art, each thread has sub-threads, and the electronic device allocates thread resources to each thread, including sub-thread resources. However, when the query process is in a high concurrency state, the corresponding thread of Elasticsearch will experience problems such as reduced speed when processing tasks.

[0130] In an implementable manner, the log processing method provided in the embodiment of the present application further includes S200-1 to S200-4, wherein:

[0131] S200-1. Obtain a processing speed deviation of each thread in the thread pool, wherein the processing speed deviation is a difference between a target processing speed and a real-time processing speed of the thread; and sum up multiple processing speed deviations to obtain a total speed deviation.

[0132] Specifically, the sub-real-time processing speed of the sub-thread is obtained, and the real-time processing speed of the thread is obtained according to the sub-real-time processing speed of the sub-thread, wherein the thread processing speed is a value such as an average value and a median value that can characterize the central tendency of the sub-real-time processing speed of the sub-thread, and this embodiment is not specifically limited; the preset processing speed corresponding to each thread is obtained, and the preset processing speed is used as the target processing speed corresponding to the thread; according to the target processing speed and the real-time processing speed, the processing speed deviation corresponding to the thread is obtained, and the processing speed deviation = the target processing speed - the real-time processing speed. Multiple processing speed deviations are summed to obtain the total speed deviation.

[0133] The target processing speed is an interval, and the speed within the interval is uniformly recorded as 0. A negative processing speed deviation indicates that the real-time speed is too low, and a positive processing speed deviation indicates that the real-time speed is too high.

[0134] S200 - 2 . For each thread, obtain the thread resource change rate corresponding to the thread according to the total speed deviation and the processing speed deviation corresponding to the thread.

[0135] The adjusted speed change value = total deviation value / number of threads. By calculating the average of the processing speed deviation, the influence of different normal speed ranges of different threads on the adjusted speed can be avoided.

[0136] Thread resource change rate = |processing speed deviation - adjusted speed change value| ÷ processing speed deviation.

[0137] S200-3. Obtain the current resource amount corresponding to the thread, and obtain the target resource amount corresponding to the thread according to the thread resource change rate and the current resource amount.

[0138] Thread resource change rate = |thread deviation - adjusted speed change value| ÷ thread deviation.

[0139] Adjusted thread resources = (1 + thread resource change rate) × current resource amount.

[0140] S200-4. Generate resource configuration information according to the target resource amount, wherein the resource configuration information is used to generate resource configuration instructions for the thread pool.

[0141] It should be noted that S200 - 1 to S200 - 4 can be implemented based on DSL / EQL language.

[0142] The log processing method provided in the embodiment of the present application configures appropriate resources for different threads to configure the thread pool of Elasticsearch to improve the concurrent processing capability.

[0143] The log processing method provided in the above embodiment is described below with reference to a specific example. Figure 4 As shown, the following steps are included:

[0144] S301. Obtain the processing speed deviation of each thread in the thread pool, wherein the processing speed deviation is the difference between the target processing speed and the real-time processing speed of the thread; and generate resource configuration information based on the above processing speed deviation, wherein the resource configuration information is used to generate resource configuration instructions for the thread pool.

[0145] Specifically, the processing speed deviation of each thread in the thread pool is obtained, wherein the processing speed deviation is the difference between the target processing speed of the thread and the real-time processing speed; and multiple processing speed deviations are summed to obtain a total speed deviation; for each thread, the thread resource change rate corresponding to the thread is obtained based on the total speed deviation and the processing speed deviation corresponding to the thread; the current resource amount corresponding to the thread is obtained, and the target resource amount corresponding to the thread is obtained based on the thread resource change rate and the current resource amount; and resource configuration information is generated based on the target resource amount, wherein the resource configuration information is used to generate resource configuration instructions for the thread pool.

[0146] S302. Based on the resource configuration instruction, obtain the real-time operation logs corresponding to each of the multiple devices, and obtain a real-time log group according to the real-time operation logs, wherein different real-time log groups represent multiple real-time operation logs corresponding to different data categories, and the real-time log group includes its corresponding data category identifier and multiple real-time operation logs corresponding to the data category identifier, and the multiple real-time operation logs are real-time operation logs corresponding to at least one device.

[0147] Specifically, resources are allocated to the thread pools corresponding to Elasticsearch, Logstash, and Kibana in the ELK stack based on the resource configuration instructions; after the resources are allocated, the Logstash obtains the real-time running logs corresponding to each of the multiple devices based on the allocated resources.

[0148] S303: Obtain an anomaly detection method corresponding to each data category identifier, wherein the anomaly detection method is used to sense whether there is a behavioral anomaly in the real-time log group.

[0149] S304. For each data category identifier, based on the anomaly detection method corresponding to the data category identifier and the real-time log group, obtain the alarm information corresponding to the real-time log group, wherein the alarm information includes the real-time operation log with abnormal behavior in the real-time log group.

[0150] In one implementation, the anomaly detection method includes a feature matching algorithm and an event alarm feature. Based on the anomaly detection method corresponding to the data category identifier and the real-time log group, the alarm information corresponding to the real-time log group is obtained. Specifically, it can include: based on the feature matching algorithm and the real-time log group, the alarm log group corresponding to the data category identifier is obtained, wherein the alarm log group belongs to the real-time log group, and the alarm log group includes several real-time operation logs containing event alarm features; according to the alarm log group, the alarm information is obtained.

[0151] Furthermore, based on the feature matching algorithm and the real-time log group, the alarm log group corresponding to the data category identifier is obtained, including: according to the real-time log group, multiple user IPs and the native command records corresponding to each of the multiple user IPs are obtained, and according to the multiple user IPs, the target IPs corresponding to each of the multiple IP classifications are obtained, wherein the IP classification characterizes whether the corresponding target IP is a user IP in the blacklist or the whitelist; according to the feature matching algorithm, the behavioral feature matching method corresponding to each of the multiple IP classifications is obtained; for each target IP, the sub-alarm log corresponding to the target IP is obtained according to the target IP, the native command record corresponding to the target IP, the real-time operation log corresponding to the target IP, and the behavioral feature matching method; according to the multiple target IPs and their respective corresponding sub-alarm logs, an alarm log group is obtained.

[0152] In another implementable manner, based on the anomaly detection method corresponding to the data category identifier and the real-time log group, alarm information corresponding to the real-time log group is obtained, which may specifically include: based on the anomaly detection method corresponding to the data category identifier and the real-time log group, initial alarm information corresponding to the real-time log group is obtained; when detecting a third-party alarm log, alarm information is obtained based on the third-party alarm log and the initial alarm information.

[0153] In another possible implementation, the anomaly detection method includes a feature matching algorithm and an event alarm feature, and obtains alarm information corresponding to the real-time log group based on the anomaly detection method corresponding to the data category identifier and the real-time log group. Specifically, it can include: obtaining initial alarm information corresponding to the real-time log group based on the anomaly detection method corresponding to the data category identifier and the real-time log group; obtaining a third-party alarm log, and obtaining the above-mentioned alarm information based on the initial alarm information and the third-party alarm log.

[0154] Further, based on the anomaly detection method corresponding to the data category identifier and the real-time log group, the initial alarm information corresponding to the real-time log group is obtained, which can specifically include: based on the feature matching algorithm and the real-time log group, the alarm log group corresponding to the data category identifier is obtained, wherein the alarm log group belongs to the real-time log group, and the alarm log group includes a number of real-time operation logs containing event alarm features; according to the alarm log group, the initial alarm information is obtained. It should be noted that the process of obtaining the initial alarm information according to the alarm log group is similar to steps S203-a1 and S203-a2, and only the alarm information is replaced, which will not be repeated in this embodiment.

[0155] S305 , obtaining an alarm frequency parameter, and obtaining a target noise reduction method according to the alarm frequency parameter, wherein the alarm frequency parameter represents the frequency and number of times the alarm information is obtained.

[0156] S306: Based on the target noise reduction method, the warning information is merged and noise reduced to obtain target information; and real-time warning data is obtained according to the target information.

[0157] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0158] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0159] Figure 5 The structural diagram of the log processing system provided for this application is as follows: Figure 5 As shown, the log processing system 40 provided in this embodiment includes:

[0160] The first acquisition module 401 is used to acquire the real-time operation logs corresponding to each of the multiple devices, and obtain a real-time log group according to the real-time operation logs, wherein different real-time log groups represent multiple real-time operation logs corresponding to different data categories, and the real-time log group includes the corresponding data category identifier and the multiple real-time operation logs corresponding to the data category identifier;

[0161] The second acquisition module 402 is used to acquire an anomaly detection method corresponding to each data category identifier, wherein the anomaly detection method is used to sense whether there is a behavioral anomaly in the real-time log group;

[0162] The processing module 403 is used to obtain, for each data category identifier, alarm information corresponding to the real-time log group based on the abnormal detection method corresponding to the data category identifier and the real-time log group, wherein the alarm information includes the real-time operation log with abnormal behavior in the real-time log group;

[0163] The generating module 404 is used to obtain real-time alarm data according to the alarm information, wherein the real-time alarm data is used to generate a real-time alarm instruction.

[0164] Optionally, the anomaly detection method includes a feature matching algorithm and an event alarm feature. The processing module 403, when executing the anomaly detection method corresponding to the data category identifier and the real-time log group to obtain the alarm information corresponding to the real-time log group, is used to:

[0165] Based on the feature matching algorithm and the real-time log group, an alarm log group corresponding to the data category identifier is obtained, wherein the alarm log group belongs to the real-time log group, and the alarm log group includes a number of real-time operation logs containing event alarm features;

[0166] Get the alarm information according to the alarm log group.

[0167] Optionally, the processing module 403, when executing the feature matching algorithm and the real-time log group to obtain the alarm log group corresponding to the data category identifier, is used to:

[0168] According to the real-time log group, multiple user IPs and the native command records corresponding to the multiple user IPs are obtained, and according to the multiple user IPs, the target IPs corresponding to the multiple IP classifications are obtained, wherein the IP classification indicates whether the corresponding target IP is a user IP in the blacklist or the whitelist;

[0169] According to the feature matching algorithm, the behavior feature matching methods corresponding to multiple IP categories are obtained;

[0170] For each target IP, obtain the sub-alarm log corresponding to the target IP according to the target IP, the native command record corresponding to the target IP, the real-time operation log corresponding to the target IP, and the behavior feature matching method;

[0171] According to the multiple target IPs and their corresponding sub-alarm logs, an alarm log group is obtained.

[0172] Optionally, the processing module 403, when executing the anomaly detection method corresponding to the data category identifier and the real-time log group to obtain the alarm information corresponding to the real-time log group, is used to:

[0173] Based on the anomaly detection method corresponding to the data category identifier and the real-time log group, initial alarm information corresponding to the real-time log group is obtained;

[0174] When detecting the third-party alarm log, the alarm information is obtained according to the third-party alarm log and the initial alarm information.

[0175] Optionally, the above system 40 further includes:

[0176] The first determination module is used to obtain an alarm frequency parameter and obtain a target noise reduction method according to the alarm frequency parameter, wherein the alarm frequency parameter represents the frequency and number of times the alarm information is obtained;

[0177] Accordingly, the generating module 404, when executing to obtain real-time alarm data according to the alarm information, is used to:

[0178] Based on the target noise reduction method, the warning information is merged and noise-reduced to obtain the target information;

[0179] Based on the target information, real-time alarm data is obtained.

[0180] Optionally, the above system 40 further includes:

[0181] The second determination module is used for:

[0182] Obtaining a processing speed deviation of each thread in the thread pool, wherein the processing speed deviation is the difference between the target processing speed and the real-time processing speed of the thread; and summing up multiple processing speed deviations to obtain a total speed deviation;

[0183] For each thread, the thread resource change rate corresponding to the thread is obtained according to the total speed deviation and the processing speed deviation corresponding to the thread;

[0184] Obtain the current resource amount corresponding to the thread, and obtain the target resource amount corresponding to the thread based on the thread resource change rate and the current resource amount;

[0185] Resource configuration information is generated according to the target resource amount, wherein the resource configuration information is used to generate resource configuration instructions for the thread pool.

[0186] The log processing system provided in this embodiment can execute the method provided in the above method embodiment, and its implementation principle and technical effect are similar, which will not be described in detail in this embodiment.

[0187] It should be understood that the above system embodiments are only illustrative, and the system of the present application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0188] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0189] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to a transistor, a memristor, etc.

[0190] Figure 6 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 6 As shown, the electronic device 50 provided in this embodiment includes: at least one processor 501 and a memory 502. Optionally, the electronic device 50 also includes a communication component 503. The processor 501, the memory 502 and the communication component 503 are connected via a bus 504.

[0191] In a specific implementation process, at least one processor 501 executes the computer-executable instructions stored in the memory 502, so that at least one processor 501 executes the above method.

[0192] The specific implementation process of the processor 501 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0193] If not otherwise specified, the processor 501 may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. If not otherwise specified, the memory 502 may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc.

[0194] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0195] An embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above redundant field updating method for a distributed system is implemented.

[0196] An embodiment of the present application also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the above redundant field updating method for a distributed system.

[0197] In the above embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0198] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0199] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A log processing method, characterized in that: include: Acquire real-time operation logs corresponding to each of the plurality of devices, and obtain a real-time log group according to the real-time operation logs, wherein different real-time log groups represent a plurality of real-time operation logs corresponding to different data categories, the real-time log group includes a corresponding data category identifier and the plurality of real-time operation logs corresponding to the data category identifier, and the plurality of real-time operation logs are real-time operation logs corresponding to at least one device; Obtaining an anomaly detection method corresponding to each of the data category identifiers, wherein the anomaly detection method is used to sense whether there is a behavioral anomaly in the real-time log group; For each of the data category identifiers, based on the anomaly detection method corresponding to the data category identifier and the real-time log group, obtain the alarm information corresponding to the real-time log group, wherein the alarm information includes the real-time operation log with abnormal behavior in the real-time log group; According to the alarm information, real-time alarm data is obtained, wherein the real-time alarm data is used to generate a real-time alarm instruction.

2. The log processing method according to claim 1, characterized in that: The anomaly detection method includes feature matching algorithm and event alarm feature, The abnormality detection method and the real-time log group corresponding to the data category identifier are used to obtain the alarm information corresponding to the real-time log group, including: Based on the feature matching algorithm and the real-time log group, an alarm log group corresponding to the data category identifier is obtained, wherein the alarm log group belongs to the real-time log group, and the alarm log group includes a plurality of real-time operation logs containing the event alarm feature; The alarm information is obtained according to the alarm log group.

3. The log processing method according to claim 2, characterized in that: The obtaining, based on the feature matching algorithm and the real-time log group, an alarm log group corresponding to the data category identifier includes: According to the real-time log group, a plurality of user IPs and native command records corresponding to the plurality of user IPs are obtained, and according to the plurality of user IPs, a plurality of target IPs corresponding to the plurality of IP categories are obtained, wherein the IP category indicates whether the target IP corresponding to the target IP is a user IP in a blacklist or a whitelist; According to the feature matching algorithm, the behavior feature matching methods corresponding to the multiple IP categories are obtained; For each of the target IPs, according to the target IP, the native command record corresponding to the target IP, the real-time operation log corresponding to the target IP, and the behavior feature matching method, a sub-alarm log corresponding to the target IP is obtained; The alarm log group is obtained according to the multiple target IPs and their corresponding sub-alarm logs.

4. The log processing method according to claim 1, characterized in that: The abnormality detection method and the real-time log group corresponding to the data category identifier are used to obtain the alarm information corresponding to the real-time log group, including: Based on the anomaly detection method corresponding to the data category identifier and the real-time log group, obtaining initial alarm information corresponding to the real-time log group; When detecting a third-party alarm log, the alarm information is obtained according to the third-party alarm log and the initial alarm information.

5. The log processing method according to claim 1, characterized in that: Before obtaining real-time alarm data according to the alarm information, the method further includes: Acquire an alarm frequency parameter, and obtain a target noise reduction method according to the alarm frequency parameter, wherein the alarm frequency parameter represents the frequency and number of times the alarm information is obtained; According to the alarm information, real-time alarm data is obtained, including: Based on the target noise reduction method, the warning information is merged and noise reduced to obtain target information; The real-time alarm data is obtained according to the target information.

6. The log processing method according to any one of claims 1 to 5, characterized in that: Also includes: Obtaining a processing speed deviation of each thread in the thread pool, wherein the processing speed deviation is the difference between the target processing speed and the real-time processing speed of the thread; and summing a plurality of the processing speed deviations to obtain a total speed deviation; For each thread, according to the total speed deviation and the processing speed deviation corresponding to the thread, a thread resource change rate corresponding to the thread is obtained; Obtaining a current resource amount corresponding to the thread, and obtaining a target resource amount corresponding to the thread according to the thread resource change rate and the current resource amount; Resource configuration information is generated according to the target resource amount, wherein the resource configuration information is used to generate resource configuration instructions for the thread pool.

7. A log processing system, characterized in that: include: A first acquisition module is used to acquire real-time operation logs corresponding to each of the multiple devices, and obtain a real-time log group according to the real-time operation logs, wherein different real-time log groups represent multiple real-time operation logs corresponding to different data categories, and the real-time log group includes a corresponding data category identifier and the multiple real-time operation logs corresponding to the data category identifier; A second acquisition module is used to acquire an anomaly detection method corresponding to each of the data category identifiers, wherein the anomaly detection method is used to sense whether there is a behavioral anomaly in the real-time log group; A processing module, for obtaining, for each of the data category identifiers, alarm information corresponding to the real-time log group based on an anomaly detection method corresponding to the data category identifier and a real-time log group, wherein the alarm information includes a real-time operation log with abnormal behavior in the real-time log group; A generating module is used to obtain real-time alarm data according to the alarm information, wherein the real-time alarm data is used to generate a real-time alarm instruction.

8. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images