A flow-based real-time log analysis alarm method, device and related equipment

Abstract

This invention discloses a stream-based real-time log analysis and alerting method, apparatus, and related equipment. The method includes: collecting log data generated by different security devices; adding a flag field to each collected log data; formatting the log data according to the flag field; normalizing the formatted log data using a streaming processing framework; caching the normalized log data according to a preset matching scheme; determining whether the number of cached logs of the same statistical dimension reaches a preset threshold; if so, determining whether the generation time of all log data reaching the preset threshold exceeds a preset observation time range; and generating an alert if so. This method can solve the problem of log consumption capacity being less than log generation speed, significantly improving the analysis efficiency of attack log data and achieving more efficient log processing and alerting.

Description

Technical Field

[0001] This invention relates to the field of security information technology, and in particular to a stream-based real-time log analysis and alarm method, apparatus, and related equipment. Background Technology

[0002] In network attack detection scenarios, detection systems generate a large number of alarm logs. As various business systems become increasingly complex and scaled up, the number of attack logs generated by different security devices also continues to grow, significantly increasing the pressure on attack event analysis, filtering, and processing. The analysis and processing of large volumes of alarm logs has long been a challenge for security operations and analysis personnel. Summary of the Invention

[0003] To increase the selection space for log analysis alarm methods and improve the efficiency of log analysis alarms, this invention provides a stream-based real-time log analysis alarm method, apparatus, and related equipment.

[0004] In a first aspect, embodiments of the present invention provide a stream-based real-time log analysis and alarm method, comprising:

[0005] Collect log data generated by different security devices;

[0006] Add a flag field to each piece of log data collected;

[0007] The log data is formatted based on the flag fields to obtain formatted log data.

[0008] The formatted log data is normalized using a streaming processing framework.

[0009] For the normalized log data, log data that meets the preset matching scheme is cached according to the preset matching scheme;

[0010] Determine whether the number of cached logs for the same statistical dimension has reached a preset threshold.

[0011] If so, determine whether the generation time of all log data that have reached the preset quantity threshold exceeds the preset observation time range;

[0012] If so, generate an alarm.

[0013] In one or more optional embodiments, the real-time log analysis alarm method further includes:

[0014] Add a unique identifier to each log data item collected.

[0015] In one or more optional embodiments, the real-time log analysis alarm method further includes:

[0016] After an alarm is generated, information carrying the unique identifier is saved in all log data that led to the alarm generation.

[0017] In one or more optional embodiments, the step of formatting the log data according to the flag field of the log data to obtain formatted log data includes:

[0018] The type of log data is determined based on the flag field of the log data;

[0019] Based on the type of log data, the corresponding filter plugin is used to uniformly format the log data of each type into a preset format, resulting in formatted log data.

[0020] In one or more optional embodiments, adding a flag field to each piece of collected log data includes:

[0021] Configure flag fields in different Filebeats based on the source and type of the collected log data;

[0022] Log data from different security devices is transmitted using the corresponding Filebeat.

[0023] In one or more optional embodiments, before caching log data that satisfies the preset matching scheme according to the preset matching scheme for the normalized log data, the method further includes:

[0024] Based on the preset whitelist rules, safe log data is filtered out after normalization.

[0025] Secondly, embodiments of the present invention provide a real-time log analysis and alarm device, comprising:

[0026] The data acquisition module is used to collect log data generated by different security devices;

[0027] Add a module to add a flag field for each piece of log data collected;

[0028] The first processing module is used to format the log data according to the flag field of the log data to obtain formatted log data;

[0029] The second processing module is used to normalize the formatted log data through a streaming processing framework.

[0030] The caching module is used to cache log data that meets the preset matching scheme after normalization.

[0031] The first judgment module is used to determine whether the number of cached logs of the same statistical dimension has reached a preset number threshold.

[0032] The second judgment module is used to determine whether the generation time of all log data that have reached the preset quantity threshold exceeds the preset observation time range.

[0033] The generation module is used to generate alarms.

[0034] Thirdly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, characterized in that, when the program is executed by a processor, it implements the real-time log analysis and alarm method as described in the first aspect.

[0035] Fourthly, embodiments of the present invention provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the program to implement the real-time log analysis and alarm method as described in the first aspect.

[0036] Fifthly, embodiments of the present invention provide a computer program product containing instructions, characterized in that, when the computer program product is run on a computer device, it causes the computer device to execute the real-time log analysis and alarm method as described in the first aspect.

[0037] The beneficial effects of the above-described technical solutions provided in the embodiments of the present invention include at least the following:

[0038] The real-time log analysis and alarm method based on streams provided in this invention adds a flag field to the log data to ensure the identification of the log data's source and type. This enables the orderly formatting and normalization of the collected log data, improving the efficiency of subsequent log data analysis. By temporarily caching log data that meets the matching scheme and sequentially performing real-time quantity and observation time checks on the cached log data, alarms are generated. Compared to traditional batch processing, which consumes a large amount of resources at once and whose memory usage only increases after the entire batch of data is processed, this method offers higher real-time performance through real-time judgment and alarm generation. It effectively solves the problem of log consumption capacity being less than log data generation speed, significantly improving the efficiency of analyzing attack log data under a large number of attack events, and achieving more efficient and reliable log processing and alarms.

[0039] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0040] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0041] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0042] Figure 1 This is a flowchart illustrating the real-time log analysis and alarm method based on streams provided in this embodiment of the invention.

[0043] Figure 2 This is a schematic diagram of a matching scheme provided in an embodiment of the present invention;

[0044] Figure 3 This is a schematic diagram of a matching scheme combination provided in an embodiment of the present invention;

[0045] Figure 4 This is a structural block diagram of the stream-based real-time log analysis and alarm device provided in an embodiment of the present invention.

[0046] Figure 5 This is a structural block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0047] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0048] The inventors discovered that traditional attack log analysis methods, which involve periodically fetching logs from data storage services in batches for a specific time period, suffer from problems such as processing latency, data loss, and high resource consumption. In practice, this method of periodically fetching and analyzing logs in batches often results in situations where the log volume is large, the analysis efficiency of a single log entry is poor, and the processing time is long. In other words, the processing capacity is less than the log generation rate. The continuous accumulation of log analysis tasks consumes a large amount of server resources, eventually leading to the exhaustion of business system or even server resources and system crashes.

[0049] Based on this, embodiments of the present invention provide a stream-based real-time log analysis and alarm method, apparatus, and related equipment.

[0050] Example 1

[0051] This invention provides a stream-based real-time log analysis and alarm method, referring to... Figure 1 As shown, it includes:

[0052] S101: Collect log data generated by different security devices;

[0053] S102: Add a flag field to each piece of log data collected;

[0054] S103: Format the log data according to the flag field of the log data to obtain formatted log data;

[0055] S104: Normalize the formatted log data using a streaming processing framework;

[0056] S105: For normalized log data, cache the log data that meets the preset matching scheme according to the preset matching scheme;

[0057] S106: Determine whether the number of logs in the same statistical dimension in the cache has reached the preset number threshold. If not, proceed directly to the analysis of the next log. If yes, execute step S107.

[0058] S107: Determine whether the generation time of all log data that have reached the preset quantity threshold exceeds the preset observation time range. If not, proceed directly to the analysis of the next log. If yes, execute step S108.

[0059] S108: Generate an alarm.

[0060] In one specific embodiment, log data generated by different security devices can be collected using any type of log collector. Step S101: Collecting log data generated by different security devices may specifically include:

[0061] Based on the Syslog protocol, a log collector gathers log data generated by different security devices and stores the collected log data into text files in a storage device, such as Redis.

[0062] Log data from security information management platforms comes from diverse sources, such as common security devices like WAF and IPS. Because the log formats generated by various security devices differ, and log processors have specific formatting and parsing logic for different types of logs, the log data collector needs to include a flag field when transmitting log data to the log processor to distinguish the type and source of the log data. In this embodiment of the invention, the collected raw log data can generally be transmitted via Filebeat, and log data from different security devices are transmitted using independent Filebeats. Therefore, a flag field needs to be configured in different Filebeats to distinguish the device source and type of the raw log data. Thus, step S102: adding a flag field to each collected log data, specifically may include:

[0063] Different flag fields are configured in different Filebeats depending on the source and type of the collected log data; optionally, the flag field can be the log_type field, which makes it easier to distinguish the device source and type of the raw log data. For example, the Filebeat configuration for WAF logs has the log_type value set to waf.

[0064] Log data from different security devices is transmitted to the log processor via the corresponding Filebeat, so that each log data includes a flag field.

[0065] In this embodiment of the invention, since the original log data from different sources vary in data type, it is necessary to use the log processor's filter plugin to uniformly format various log files into JSON format. Furthermore, different types of log data use different filter plugins. For example, WAF and IPS may both require the Grok plugin, but the Grok regular expressions for these two types of logs are not consistent. Therefore, the processing logic differs for different types of log data. Accordingly, step S103: Based on the log data's flag field, the log data is formatted to obtain formatted log data, which may specifically include:

[0066] The type of log data is determined based on the flag field of the log data;

[0067] Based on the type of log data, the corresponding filter plugin is used to uniformly format the log data of each type into a preset format, resulting in formatted log data.

[0068] In an optional implementation, LogStash can act as a receiver for log data transmitted by Filebeat, and it can interface with multiple Filebeat devices. Various log files can be uniformly formatted into JSON format using methods such as Grok, time processing, and custom code processing. The formatted JSON log data will also include a flag field to distinguish the source of the log data.

[0069] In this embodiment of the invention, the formatted JSON log data is transmitted to a streaming processing framework in preparation for log data consumption and analysis. Specifically, before consuming and analyzing the log data, a streaming log analysis channel needs to be built using the streaming processing framework. In this channel, for each log data entry received, the streaming processing framework will perform further processing on the JSON-formatted log data according to its type. Optionally, the streaming processing framework can be Kafka Stream, Flink, etc.

[0070] In this embodiment of the invention, because fields with the same meaning may have different names in the JSON bodies parsed from different log data—for example, the attacker identifier for a WAF is "attcker," while the attacker identifier for an IPS is "attack_ip"—it is necessary to normalize the log data to parse different log data into a unified model. Step S104: Normalize the formatted log data using a streaming processing framework, which may specifically include:

[0071] Map some fields of JSON format log data to a normalized data model according to business logic.

[0072] It should be noted that the normalized data model can include fields such as attacker, source IP, destination IP, source port, destination port, and attack name. The JSON formatted log data after normalization includes the original log fields parsed during the formatting process, as well as the fields included in the normalized model after normalization based on business logic.

[0073] In this embodiment of the invention, one matching scheme can only correspond to one log type. For example, Figure 2 This example shows a matching scheme for WAF logs. This scheme only processes WAF logs; other types of log data in JSON format will be filtered out. The keywords represent the matching conditions for WAF logs. Figure 2The `match` parameter specifies that if any value corresponding to a key in the JSON data contains the attack field, the log data meets the matching criteria for this scheme. Logs that do not meet the matching criteria are filtered out. Multiple keywords can be set, and the keyword combination algorithm represents the relationship between these keywords in the matching criteria. The match count represents the number of logs required to generate an alert. The statistical dimension represents the aggregation condition; only logs with the same statistical dimension value in the JSON data are aggregated together to calculate the statistical match count. Furthermore, multiple matching schemes can be used. Figure 3 The algorithm for combining two matching schemes is shown. The algorithm for combining matching schemes refers to the logical relationship between multiple matching schemes. The values ​​can be AND or OR, which represent the relationship of AND or OR.

[0074] In this embodiment of the invention, reference is made to Figure 3 As shown, the observation period duration indicates that the time interval between the first log data and the last log data that meets the matching conditions in the matching scheme cannot exceed the preset observation time range. If it does, the log data that exceeds the preset observation time range needs to be filtered out and the number of matching times needs to be recalculated to see if the threshold is reached.

[0075] In this embodiment of the invention, reference is made to Figure 3 As shown, the alarm interval time refers to the interval between the generation of alarms for the same statistical dimension by the same alarm rule, which is also the alarm silence time. Setting the alarm silence time can effectively avoid duplicate alarms.

[0076] In this embodiment of the invention, after filtering out the log data that meets the matching conditions according to the matching scheme, the log data that meets the matching conditions can be stored in a cache array first, that is, step S105 is executed: for the normalized log data, the log data that meets the preset matching scheme is cached according to the preset matching scheme.

[0077] In this embodiment of the invention, after each log data is matched, it is determined whether the log data under the statistical dimension to which the log belongs meets the preset quantity threshold (reaching the preset number of matches). That is, step S106 is executed: determining whether the number of cached logs of the same statistical dimension has reached the preset quantity threshold. If not, step S107 is executed: determining whether the generation time of all log data that has reached the preset quantity threshold exceeds the preset observation time range. If it exceeds, an alarm can be generated; if it does not exceed, the analysis of the next log is directly initiated. Furthermore, if after filtering out log data outside the preset observation time range, the number of log data under the statistical dimension does not reach the preset quantity threshold, the analysis of the next log is also directly initiated.

[0078] For example, an alert rule can be defined as generating an alert if the same attacker's IP generates 10 WAF logs within ten minutes. When matching log data, the alert rule filters out WAF logs. Logs generated by different attackers can be placed into a key in Redis or a high-performance local cache like Caffeine Cache. When the amount of log data corresponding to a key meets the threshold of 10 entries, it checks if the cached log data is all from within the last ten minutes. If so, an alert is generated; otherwise, the log data temporarily stored under the key is updated.

[0079] It should be noted that the above description of the alarm rules is only for illustration using a simple example and does not limit the invention. This invention is related to the field of security information; therefore, in the actual business implementation of alarm rules, methods including but not limited to keyword filtering, field value type filtering, statistical dimension filtering composed of multiple fields, and combinations of multiple log type matching rules can be used to analyze and issue alarms for security event logs.

[0080] The real-time log analysis and alarm method based on streams provided in this invention adds a flag field to the log data to ensure the identification of the log data's source and type. This enables the orderly formatting and normalization of the collected log data, improving the efficiency of subsequent log data analysis. By temporarily caching log data that meets the matching scheme and sequentially performing real-time quantity and observation time checks on the cached log data, alarms are generated. Compared to traditional batch processing, which consumes a large amount of resources at once and whose memory usage only increases after the entire batch of data is processed, this method offers higher real-time performance through real-time judgment and alarm generation. It effectively solves the problem of consumption capacity being less than the log data generation speed, significantly improving the efficiency of analyzing attack log data under a large number of attack events, and achieving more efficient and reliable log processing and alarms.

[0081] In an optional implementation, before step S105: for the normalized log data, before caching the log data that satisfies the preset matching scheme according to the preset matching scheme, the method further includes:

[0082] For normalized log data, secure log data is filtered out according to preset whitelist rules. For example, if the attacker's field value in normalized log data is in the whitelist, that log data is considered secure and will not be processed further, proceeding directly to the analysis of the next log data. Clearly, this whitelist function can be selected for its necessity based on the actual business scenario.

[0083] In an optional implementation, the stream-based real-time log analysis alarm method may further include: adding a unique identifier to each collected log data to facilitate querying all information of the log data that generated the alarm after the alarm is generated.

[0084] In an optional implementation, the stream-based real-time log analysis and alerting method further includes: after generating an alert, saving information carrying a unique identifier from all log data that led to the alert generation. The log analysis and alerting module does not directly store logs, nor is it suitable for querying and retrieving massive amounts of log data. After the logs are analyzed and alerted, only a portion of the log information is stored. Therefore, to query all information from the log data that generated the alert, a unique identifier, such as a UUID, can be included when the log data is transmitted to the log data processor. When detailed alert information needs to be viewed, this unique identifier can be used to query all log information from log storage services, such as ElasticSearch or other search and analysis engines.

[0085] For example, when using Elasticsearch for log querying and retrieval services, in order to perform a correlation query between the information of the log data recorded when the alarm is generated and all the information of the log data stored in Elasticsearch, a fingerprint can be added to the Filebeat configuration as a unique identifier for the correlation query.

[0086] The real-time log analysis and alarm method based on streams provided in this invention enables the correlation query of alarm logs in the log storage system by adding a unique identifier to each log data, ensuring that the log data that caused the alarm can be accurately queried, which helps to improve the efficiency of subsequent alarm handling.

[0087] In optional implementations, after an alarm is generated, corresponding alarm handling is required. For example, the alarm may be notified to the responsible party. Depending on the specific business requirements, firewall actions may also be implemented to block attacking IPs. When dealing with large log volumes and heavy analysis loads, post-alarm handling can be implemented asynchronously to further improve log analysis efficiency. For instance, keyword analysis and statistical dimension analysis logic can be placed within a KafkaStream Processor. This means the alarm generation process is handled in real-time by Kafka Stream. Since firewall actions and other log analysis-related operations do not conflict with the alarm handling logic, and these actions are typically time-consuming, a separate thread can be started at the end of the Processor's analysis logic to handle the alarms. This asynchronous processing avoids blocking log analysis, improves efficiency, and has no impact on actual business logic.

[0088] Example 2

[0089] Based on the same inventive concept, embodiments of the present invention also provide a real-time log analysis and alarm device, referring to... Figure 4 As shown, it includes:

[0090] The acquisition module 101 is used to collect log data generated by different security devices;

[0091] Add module 102 to add a flag field for each piece of log data collected;

[0092] The first processing module 103 is used to format the log data according to the flag field of the log data to obtain formatted log data;

[0093] The second processing module 104 is used to normalize the formatted log data through a streaming processing framework.

[0094] The caching module 105 is used to cache log data that meets the preset matching scheme according to the normalized log data.

[0095] The first judgment module 106 is used to determine whether the number of cached logs of the same statistical dimension has reached a preset number threshold.

[0096] The second judgment module 107 is used to determine whether the generation time of all log data that have reached the preset quantity threshold exceeds the preset observation time range.

[0097] Generation module 108 is used to generate alarms.

[0098] Example 3

[0099] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the real-time log analysis and alarm method as described in Embodiment 1.

[0100] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: electrical connections having one or more wires; portable computer disks; hard disks; random access memory (RAM); read-only memory (ROM); erasable programmable read-only memory (EPROM); registers; hard disks; optical fibers; portable compact disc read-only memory (CD-ROM); optical storage devices; magnetic storage devices; or any suitable combination thereof; or any other form of computer-readable storage medium known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium may also be a component of the processor. The processor and the storage medium may reside in an application-specific integrated circuit (ASIC). In the embodiments of this application, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0101] Example 4

[0102] Based on the same inventive concept, this embodiment of the invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the real-time log analysis and alarm method as described in Embodiment 1.

[0103] Figure 5 A possible structural diagram of the computer device involved in the above embodiments is shown. The computer device includes a processor 1002 and a communication interface 1003. The processor 1002 is used to control and manage the operation of the computer device, for example, executing the alarm association method described above, and / or other processes of the technology described herein. The communication interface 1003 is used to support communication between the computer device and other network entities, for example, executing the steps performed by the communication unit 902 described above. The computer device may also include a memory 1001 and a bus 1004, the memory 1001 being used to store the program code and data of the computer device.

[0104] The memory 1001 may be a memory in a computer device, and the memory may include: volatile memory, such as random access memory; the memory may also include: non-volatile memory, such as read-only memory, flash memory, hard disk or solid-state drive; the memory may also include: a combination of the above types of memory.

[0105] The processor 1002 described above can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0106] Bus 1004 can be an Extended Industry Standard Architecture (EISA) bus, etc. Bus 1004 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0107] Example 5

[0108] Based on the same inventive concept, this embodiment of the invention also provides a computer program product containing instructions, characterized in that, when the computer program product is run on a computer device, it causes the computer device to execute the real-time log analysis and alarm method as described in Embodiment 1.

[0109] The principles by which the above-described apparatus, client, medium, related equipment, and system in this embodiment solve the problem are similar to those of the aforementioned method. Therefore, their implementation can refer to the implementation of the aforementioned method, and repeated details will not be repeated.

[0110] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0111] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0112] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0113] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0114] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A stream-based real-time log analysis and alarm method, characterized in that, include: Collect log data generated by different security devices; Add a flag field to each piece of log data collected; The log data is formatted based on the flag fields to obtain formatted log data. The formatted log data is normalized using a streaming processing framework. For the normalized log data, log data that meets the preset matching scheme is cached according to the preset matching scheme; Determine whether the number of cached logs for the same statistical dimension has reached a preset threshold. If so, determine whether the generation time of all log data that have reached the preset quantity threshold exceeds the preset observation time range; If so, generate an alarm.

2. The real-time log analysis and alarm method according to claim 1, characterized in that, Also includes: Add a unique identifier to each log data item collected.

3. The real-time log analysis and alarm method according to claim 2, characterized in that, Also includes: After an alarm is generated, information carrying the unique identifier is saved in all log data that led to the alarm generation.

4. The real-time log analysis and alarm method according to claim 1, characterized in that, The step of formatting the log data according to the flag field to obtain formatted log data includes: The type of log data is determined based on the flag field of the log data; Based on the type of log data, the corresponding filter plugin is used to uniformly format the log data of each type into a preset format, resulting in formatted log data.

5. The real-time log analysis and alarm method according to claim 1, characterized in that, The step of adding a flag field to each piece of collected log data includes: Configure flag fields in different Filebeats based on the source and type of the collected log data; Log data from different security devices is transmitted using the corresponding Filebeat.

6. The real-time log analysis and alarm method according to claim 1, characterized in that, Before caching log data that satisfies the preset matching scheme according to the normalized log data, the process further includes: Based on the preset whitelist rules, safe log data is filtered out after normalization.

7. A real-time log analysis and alarm device, characterized in that, include: The data acquisition module is used to collect log data generated by different security devices; Add a module to add a flag field for each piece of log data collected; The first processing module is used to format the log data according to the flag field of the log data to obtain formatted log data; The second processing module is used to normalize the formatted log data through a streaming processing framework. The caching module is used to cache log data that meets the preset matching scheme after normalization. The first judgment module is used to determine whether the number of cached logs of the same statistical dimension has reached a preset number threshold. The second judgment module is used to determine whether the generation time of all log data that have reached the preset quantity threshold exceeds the preset observation time range. The generation module is used to generate alarms.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the real-time log analysis and alarm method as described in any one of claims 1-6.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the real-time log analysis and alarm method as described in any one of claims 1-6.

10. A computer program product containing instructions, characterized in that, When a computer program product is run on a computer device, the computer device performs the real-time log analysis and alarm method as described in any one of claims 1-6.

Images