Log processing method, log processing system, and computer-readable recording medium

Abstract

A log processing method, a log processing system and a computer readable recording medium; the log processing method comprises the following steps: continuously obtaining a plurality of original log data; the original log data comes from a plurality of sources; clustering the original log data; sequentially inputting the original log data into a log analysis engine to output a plurality of normalized log data; in the same group, the normalized log data records the same security level and the same security description; and storing the normalized log data into a log search engine.

Description

Technical Field

[0001] This disclosure relates to a digital processing method and a digital processing system, and more particularly to a log processing method, a log processing system, and a computer-readable recording medium. Background Technology

[0002] In the current field of cybersecurity, log analysis is a crucial step in maintaining system stability and security. However, traditional log analysis processes often face the following challenges.

[0003] Multi-source logs require manual format parsing. Different systems generate logs with varying formats. Modern log management software primarily addresses log formatting needs by establishing log processing rules and combining them with different log processing plugins, ultimately producing a regularized format. This method requires significant manpower and time to understand the source log formats and to combine plugins and rules specifically for those formats before a log format suitable for analysis can be generated.

[0004] Security Information and Event Management (SIEM) systems often suffer from numerous false security alerts. Existing log analysis systems lose some log information during processing and struggle to flexibly handle logs from multiple sources with varying formats and to correlate logs from multiple sources. This forces cybersecurity personnel to spend a significant amount of time sifting through a large number of false security alerts to find underlying security incidents. Summary of the Invention

[0005] This disclosure relates to a log processing method, log processing system, and computer-readable recording media, which utilizes clustering and normalization techniques to enable multi-source logs to be searched and analyzed hierarchically according to security levels, significantly improving log processing efficiency and consistency, and reducing false alarm rates while enhancing cybersecurity analysis capabilities.

[0006] According to one aspect of this disclosure, a log processing method is proposed. The log processing method includes the following steps: Continuously acquiring several raw log data entries. These raw log data entries come from several sources. Clustering these raw log data entries. Sequentially inputting these raw log data entries into a log parsing engine to output several normalized log data entries. Within the same cluster, these normalized log data entries record the same security level and the same security description. Storing these normalized log data entries in a log search engine.

[0007] In one embodiment of the above log processing method, in the step of clustering the multiple original log data, the multiple original log data that trigger the clustering come from the same source.

[0008] In one embodiment of the above log processing method, in the step of clustering the multiple original log data, a cluster triggering condition includes:

[0009] An urgent keyword has appeared.

[0010] In one embodiment of the above log processing method, in the step of clustering the multiple original log data, a cluster triggering condition includes:

[0011] One memory usage rate is higher than a predetermined percentage.

[0012] In one embodiment of the above log processing method, in the step of clustering the multiple original log data, a cluster triggering condition includes:

[0013] The number of entries in the queue of this log parsing engine is less than a predetermined number.

[0014] In one embodiment of the above log processing method, in the step of clustering the multiple original log data, a cluster triggering condition includes:

[0015] The waiting time for multiple raw log data entries is longer than a predetermined time.

[0016] In one embodiment of the log processing method described above, the predetermined time is dynamically adjusted.

[0017] In one embodiment of the above log processing method, the predetermined time is dynamically adjusted based on a baseline waiting time, a flow rate coefficient, and a priority coefficient.

[0018] In one embodiment of the above log processing method, in the step of clustering the multiple original log data, a cluster triggering condition includes:

[0019] The number of waiting records for multiple raw log data entries is higher than the predetermined number.

[0020] In one embodiment of the above log processing method, the predetermined quantity is dynamically adjusted.

[0021] In one embodiment of the above log processing method, the predetermined quantity is dynamically adjusted based on a baseline batch size, a flow rate coefficient, and a memory pressure coefficient.

[0022] In one embodiment of the above log processing method, it further includes:

[0023] The log search engine performs tiered searches based on this security level.

[0024] According to another aspect of this disclosure, a log processing system is proposed. The log processing system includes an input unit, a clustering unit, a log parsing engine, and a log search engine. The input unit continuously acquires several raw log data entries. These raw log data entries come from several sources. The clustering unit clusters these raw log data entries. These raw log data entries are sequentially input into the log parsing engine to output several normalized log data entries. Within the same cluster, these normalized log data entries record the same security level and the same security description. These normalized log data entries are stored in the log search engine.

[0025] In one embodiment of the log processing system described above, the multiple raw log data that trigger the cluster come from the same source.

[0026] In one embodiment of the above log processing system, the cluster triggering conditions adopted by the cluster unit include:

[0027] An urgent keyword has appeared.

[0028] In one embodiment of the above log processing system, the cluster unit uses a cluster triggering condition, which includes:

[0029] One memory usage rate is higher than a predetermined percentage.

[0030] In one embodiment of the above log processing system, the cluster unit uses a cluster triggering condition, which includes:

[0031] The number of entries in the queue of this log parsing engine is less than a predetermined number.

[0032] In one embodiment of the above log processing system, the cluster unit uses a cluster triggering condition, which includes:

[0033] The waiting time for multiple raw log data entries is longer than a predetermined time.

[0034] In one embodiment of the log processing system described above, the cluster unit dynamically adjusts the predetermined time.

[0035] In one embodiment of the log processing system described above, the cluster unit dynamically adjusts the predetermined time based on a baseline waiting time, a traffic rate coefficient, and a priority coefficient.

[0036] In one embodiment of the above log processing system, the cluster unit uses a cluster triggering condition, which includes:

[0037] The number of waiting records for multiple raw log data entries is higher than the predetermined number.

[0038] In one embodiment of the log processing system described above, the cluster unit dynamically adjusts the predetermined number.

[0039] In one embodiment of the log processing system described above, the cluster unit dynamically adjusts the predetermined quantity based on a baseline batch size, a traffic rate coefficient, and a memory pressure coefficient.

[0040] In one embodiment of the log processing system described above, the log search engine performs hierarchical searches based on the security level.

[0041] According to another aspect of this disclosure, a computer-readable recording medium is proposed. The computer-readable recording medium records program instructions. When executed by a processor, the program instructions cause the processor to perform the following steps: Continuously acquire a plurality of raw log data entries. These raw log data entries come from several sources. Cluster these raw log data entries. Sequentially input these raw log data entries into a log parsing engine to output a plurality of normalized log data entries. Within the same cluster, these normalized log data entries record the same security level and the same security description. Store these normalized log data entries in a log search engine.

[0042] To provide a better understanding of the above and other aspects of this disclosure, specific embodiments are described below in conjunction with the accompanying drawings: Attached Figure Description

[0043] Figure 1 A schematic diagram of a log analysis system and several electronic devices according to an embodiment of the present disclosure is shown.

[0044] Figure 2 A block diagram of a log analysis system according to an embodiment of the present disclosure is shown.

[0045] Figure 3 A flowchart illustrating a log processing method according to an embodiment of the present disclosure is shown.

[0046] Figure 4 The example illustrates the dynamic adjustment of the reservation time and reservation quantity.

[0047] Figures 5A to 5C illustrate detailed flowcharts of step S120 according to an embodiment of the present disclosure.

[0048] Figure 6 A detailed flowchart illustrating step S130 according to an embodiment of the present disclosure is shown.

[0049] Figure 7 The example illustrates the format of normalized log data.

[0050] Figure 8 Draw the hierarchical structure of the normalized log data.

[0051] In the attached figures, the following labels are used:

[0052] 100: Log Analysis System

[0053] 110: Input Unit

[0054] 120: Cluster Unit

[0055] 130: Log parsing engine

[0056] 150: Log search engine

[0057] 160: Log Analysis Engine

[0058] 170: Warning Unit

[0059] 600, 700, 800: Electronic devices

[0060] 900: Network

[0061] DS: Security Description

[0062] k: Quantity reserved

[0063] KW: Emergency Keywords

[0064] LG: Raw log data

[0065] mp: Memory stress coefficient

[0066] n: scheduled time

[0067] NLG: Normalized Log Data

[0068] NM: Waiting Quantity

[0069] pr: Priority coefficient

[0070] QN: Number of people in the queue

[0071] RK: Security Level

[0072] RM: Memory Usage

[0073] rt: Flow rate coefficient

[0074] S110, S120, S1201-S1237, S130, S131-S137, S140, S150, S160: Steps

[0075] sz0: Base batch size

[0076] TM: Waiting time

[0077] tm0: Baseline latency Detailed Implementation

[0078] The technical terms used in this specification are based on common terminology in the field. Where this specification provides explanations or definitions for certain terms, the interpretation of those terms shall be based on the explanations or definitions provided in this specification. Each of the disclosed embodiments has one or more technical features. Where feasible, those skilled in the art may selectively implement some or all of the technical features in any embodiment, or selectively combine some or all of the technical features in these embodiments.

[0079] Please refer to Figure 1 The diagram illustrates a log analysis system 100 and several electronic devices 600, 700, and 800 according to an embodiment of this disclosure. The log analysis system 100 receives raw log data LG from various sources, including electronic devices 600, 700, and 800, via a network 900. The raw log data LG from different sources may be in Syslog, Zeek conn.log, AWS JSON, or Apache (Common Log Format / Combined Log Format) formats. Syslog is a common log protocol / format, typically used in Linux, Unix, network devices, firewalls, routers, etc. Zeek conn.log is a structured plain text format (tab-delimited), typically used in Zeek network monitoring and threat detection tools. AWS JSON is a JSON format, typically used in AWS CloudTrail (for API activities), AWS VPC Flow Logs (network traffic), or AWS WAF Logs (Web Application Firewall). Apache is a plain text format, typically used in Apache HTTP Server.

[0080] In this disclosure, the log analysis system 100 utilizes clustering and normalization techniques to cluster and normalize raw log data LG from multiple sources into normalized log data NLG. These normalized log data NLGs can be hierarchically searched and analyzed based on security levels, significantly improving log processing efficiency and consistency, and reducing false alarm rates while enhancing cybersecurity analysis capabilities.

[0081] Please refer to Figure 2The diagram illustrates a block diagram of a log analysis system 100 according to an embodiment of this disclosure. The log analysis system 100 includes an input unit 110, a clustering unit 120, a log parsing engine 130, a log search engine 150, a log analysis engine 160, and an alerting unit 170. The input unit 110 receives data, such as a wireless network module, a wired network module, or a connection cable. The clustering unit 120 performs data clustering. The log parsing engine 130 performs log parsing. The log search engine 150 performs log searching. The log analysis engine 140 performs log analysis.

[0082] Cluster unit 120, log parsing engine 130, log search engine 150, and / or log analysis engine 160 are, for example, a circuit, a circuit board, a storage device storing program code, or a chip. The chip is, for example, a central processing unit (CPU), or other programmable general-purpose or special-purpose microcontroller unit (MCU), microprocessor, digital signal processor (DSP), programmable controller, application-specific integrated circuit (ASIC), graphics processing unit (GPU), image signal processor (ISP), image processing unit (IPU), arithmetic logic unit (ALU), complex programmable logic device (CPLD), field-programmable gate array (FPGA), or other similar components or combinations thereof.

[0083] The alarm unit 170 is used to issue an alarm when an anomaly is detected, such as an alarm light, an alarm horn, or a message transmission module.

[0084] In this disclosure, cluster unit 120 can cluster raw log data LG from multiple sources, and log parsing engine 130 performs normalization processing to obtain normalized log data NLG. The normalized log data NLG is simultaneously parsed to obtain a corresponding security level RK and a security description DS, enabling these normalized log data NLGs to be searched and analyzed hierarchically according to security levels, significantly improving log processing efficiency and consistency, and reducing false alarm rates while enhancing cybersecurity analysis capabilities. A flowchart is provided below to explain in detail the operation of each component.

[0085] Please refer to Figure 3 The diagram illustrates a flowchart of a log processing method according to an embodiment of the present disclosure. The log processing method is executed, for example, by a computer, chip, or computer-readable recording medium loaded with program code. The log processing method includes, for example, steps S110 to S160.

[0086] exist Figure 3 In step S110, as shown in Figures 1 and 2, the input unit 110 continuously obtains several raw log data LGs. These raw log data LGs may come from several different sources (e.g., Figure 1 (Electronic devices 600, 700, 800). The format of raw log data LG from different sources may not be exactly the same.

[0087] Next, in Figure 3 In step S120, such as Figure 2 As shown, clustering unit 120 clusters these raw log data LGs. In one embodiment, multiple raw log data LGs can be clustered into a group GP. Alternatively, a single raw log data LG can be clustered into a group GP. The raw log data LGs that trigger the clustering are, for example, from the same source (e.g., the same IP address).

[0088] According to various cluster triggering conditions, after receiving one or more raw log data LGs, cluster unit 120 will trigger cluster actions once the cluster triggering conditions are met. Cluster triggering conditions include, for example, "urgent trigger", "stress trigger", "idle trigger", "time trigger", and "quantity trigger". The priority of cluster triggering conditions is, for example, "urgent trigger", "stress trigger", "idle trigger", "time trigger", and "quantity trigger"; or "urgent trigger", "stress trigger", "idle trigger", "quantity trigger", and "time trigger".

[0089] The cluster triggering conditions for "emergency trigger" include the appearance of emergency keywords such as "CRITICAL", "ALERT", "ATTACK", and "BREACH". Once these emergency keywords appear, the original log data LG is immediately clustered into a group GP.

[0090] The "stress-triggered" clustering trigger condition is, for example, a memory utilization rate (RM) exceeding a predetermined percentage. This predetermined percentage could be, for example, 70%, 80%, or 90%. Once the memory utilization rate (RM) exceeds the predetermined percentage, the raw log data (LG) is clustered into a group (GP).

[0091] The "idle trigger" clustering condition is, for example, when the queue size QN of the log parsing engine 130 falls below a predetermined number. The predetermined number is, for example, 1 or 2. Once the queue size QN of the log parsing engine 130 falls below the predetermined number, the raw log data LG is clustered into a group GP.

[0092] The triggering condition for a "time-triggered" cluster is, for example, that a waiting time TM of the raw log data LG is higher than a predetermined time n. The predetermined time n is, for example, a dynamically adjusted value. Please refer to [link / reference]. Figure 4 The example illustrates the dynamic adjustment of the scheduled time n and the scheduled quantity k. The scheduled time n is dynamically adjusted, for example, according to the following formula (1), based on a baseline waiting time tm0, a flow rate coefficient rt, and a priority coefficient pr.

[0093] (1)

[0094] The baseline waiting time tm0 is, for example, 3 seconds. The flow rate coefficient rt is used to adjust the scheduled time n according to the flow of the current raw log data LG in order to improve the processing throughput under high flow.

[0095] The priority coefficient pr is used to shorten the processing delay based on the importance of the event, allowing the system to prioritize the processing of raw log data LG that has a higher cybersecurity risk.

[0096] The cluster triggering condition for "quantity-triggered" is, for example, when the number of waits (NM) for the raw log data LG exceeds a predetermined number (k). The predetermined number (k) is, for example, a dynamically adjusted value. Figure 4 As shown, the predetermined quantity k is dynamically adjusted according to the following formula (2), based on a baseline batch size sz0, a flow rate coefficient rt, and a memory pressure coefficient mp.

[0097] (2)

[0098] The baseline batch size sz0 is, for example, 30 records. The flow rate coefficient rt is used to adjust the scheduled time n based on the flow of the current raw log data LG in order to improve the processing throughput under high flow conditions.

[0099] The memory stress factor (mp) is used to control batch size based on memory usage, preventing the system from exhausting resources during peak load periods.

[0100] like Figure 4 As shown, to cope with highly variable log traffic and system resource status, the traffic rate coefficient rt, priority coefficient pr, and memory pressure coefficient mp are adjusted in real time according to the system's input intensity, task urgency, and resource pressure, respectively. Through modular coupling design, an adaptive, elastic, and predictable log processing control logic is achieved. The combination of these three factors enables dynamic and boundary-controlled resource allocation based on different operating scenarios, effectively improving system stability and real-time response capabilities to cybersecurity incidents, while also considering high concurrency processing efficiency and false alarm rate suppression.

[0101] The flow rate factor rt can range from -0.8 to 2.0. At high flow rates (e.g., more than 30 transactions per 3 seconds), rt can be set to 1.5. At low flow rates (e.g., less than 5 transactions per 3 seconds), rt can be set to -0.5. At medium flow rates, rt can be set to 0. By designing the flow rate factor rt, batch sizes can be increased and time shortened at high flow rates, while batch sizes can be decreased and time extended at low flow rates.

[0102] The priority coefficient pr can range from 0 to 0.9. For high-priority raw log data LG, the priority coefficient pr can be set to 0.6; for general raw log data LG, the priority coefficient pr can be set to 0. The priority coefficient pr affects the scheduled time n of "time triggering", allowing high-priority raw log data LG to have a shorter waiting time.

[0103] The memory pressure factor (mp) ranges from -0.5 to 1.0. The memory pressure factor (mp) is defined using the following algorithm: total_logs represents the total number of logs from all groups, and memory_pressure_factor represents the memory pressure factor (mp).

[0104] IF total_logs > 8000 THEN

[0105] pressure = (total_logs - 8000) / (10000 - 8000)

[0106] memory_pressure_factor = min(1.0, pressure × 0.8)

[0107] ELSE

[0108] memory_pressure_factor = 0

[0109] The memory stress factor mp affects the predetermined quantity k of "quantity trigger". By designing the memory stress factor mp, the batch size can be increased when memory pressure increases, thereby speeding up processing.

[0110] Furthermore, the predetermined time n and predetermined quantity k can be bounded by the following equations (3) and (4) to prevent end cases:

[0111] The scheduled time n = max(3, min(30, scheduled time n)) (3)

[0112] That is, the scheduled time n is limited to between 3 seconds and 30 seconds.

[0113] The number of orders k = max(30, min(1000, number of orders k))(4)

[0114] That is, the number of orders k is limited to between 30 and 1000.

[0115] Please refer to the examples in scenarios one through three below.

[0116] In Scenario 1, the raw log data LG represents normal traffic and has a general priority. The traffic is 20 entries / 3 seconds (i.e., medium traffic); the priority is "general"; and the total number of log entries across all groups is 5000. Based on this data, the following information can be analyzed.

[0117] The flow rate coefficient rt is 0; the priority coefficient pr is 0; and the memory pressure coefficient mp is 0. Further analysis is performed to determine the predetermined time n and the predetermined quantity k.

[0118] The scheduled time n = 3 × (1 + 0 - 0) = 3 seconds.

[0119] The number of orders is k = 30 × (1 + 0 + 0) = 30.

[0120] In Scenario 2, the raw log data LG is of high traffic and high priority. The traffic is 50 log entries per 3 seconds (i.e., high traffic); the priority is "high"; and the total number of log entries in all groups is 5000. Based on this data, the following information can be analyzed.

[0121] The flow rate coefficient rt is 1.5; the priority coefficient pr is 0.6; and the memory pressure coefficient mp is 0. Further analysis is performed to determine the predetermined time n and the predetermined quantity k.

[0122] The scheduled time n = 3 × (1.5 - 0.6) = 5.7 seconds.

[0123] The number of orders is k = 30 × (1 + 1.5 + 0) = 75.

[0124] In scenario three, the raw log data LG is characterized by low traffic but high pressure. The traffic is 5 entries / 3 seconds (low traffic); the priority is "normal"; and the total number of log entries across all groups is 9000 (high pressure). Based on this data, the following information can be analyzed.

[0125] The flow rate coefficient rt is -0.5; the priority coefficient pr is 0; and the memory pressure coefficient mp is 0.4, i.e., (9000-8000) / (10000-800) × 0.8=0.4. Further analysis is performed to determine the predetermined time n and the predetermined quantity k.

[0126] The scheduled time n = 3 × (1 + (-0.5) - 0) = 1.5 seconds.

[0127] The number of orders is k = 30 × (1 + (-0.5) + 0.4) = 27.

[0128] In summary, the cluster unit 120 implements clustering of raw log data through technologies such as traffic detection, dynamic parameter calculation, load control, and memory protection. In the traffic detection technology, the traffic detector cluster unit 120 monitors the number of logs within a 5-second sliding window. In the dynamic parameter calculation technology, the cluster unit 120 uses algorithms to adjust parameters in real time. In the load control technology, the cluster unit 120 manages the processing queue to prevent overload. In the memory protection technology, the cluster unit 120 monitors and enforces memory limits.

[0129] Please refer to Figures 5A-5C, which illustrate a detailed flowchart of step S120 according to an embodiment of this disclosure. This process effectively manages and aggregates raw log data (LG) from multiple sources through a series of dynamic adjustment and judgment mechanisms, thereby improving the response speed and resource utilization efficiency of the log processing system. Figure 5A In step S1201, the system first initializes the parameters and data structures, including the group management structure, traffic statistics variables and various trigger thresholds, to ensure a stable initial state for subsequent operations.

[0130] Next, in Figure 5AIn step S1202, the system initiates a regular maintenance task to perform background operations such as garbage collection, statistics reset, and cleaning up expired groups in order to maintain the stability of the system's long-term operation.

[0131] Then, in Figure 5A In step S1203, the system enters a standby state, waiting for external devices or systems to continuously input raw log data LG, in order to prepare data for subsequent clustering and analysis processes.

[0132] exist Figure 5A In step S1204, the system begins to receive raw log data LG and performs preliminary receive buffering and queue management.

[0133] Next, in step S1205, the system extracts the source IP address and priority information (used to calculate the priority coefficient pr) from the received log data, so as to carry out subsequent traffic classification, group judgment and scheduling decisions.

[0134] In step S1206, the system updates traffic information, including the number of packets, data size, and time distribution for each source IP, as the basis for calculating the traffic rate coefficient rt.

[0135] Next, in step S1207, the system determines whether the cluster conditions for "emergency triggering" are met. This determination is mainly for specific high-priority events or abnormal traffic bursts. If the conditions are met, the system will process them immediately; if not, the system will execute the standard procedure.

[0136] If the "emergency trigger" condition is met, in step S1208, the system immediately clusters the original log data LG into a group GP and quickly sends it to the log parsing engine 130 to meet the real-time processing requirements.

[0137] If no emergency condition is triggered, in step S1209, the system selects a candidate group GP from the existing groups for further judgment.

[0138] In step S1210, the system determines whether a group GP corresponding to the IP address already exists. If it does exist, the process will proceed... Figure 5B Proceed to step S1211; if it does not exist, proceed to step S1212.

[0139] In step S1211, the system decides to directly use the existing group GP to simplify processing and reduce memory overhead.

[0140] In step S1212, if there is no existing group corresponding to the IP address, the system needs to determine whether the current number of groups exceeds the threshold set by the system in order to avoid excessive groups causing resource exhaustion.

[0141] In step S1213, if the number of groups exceeds the threshold, the system executes an elimination mechanism, selecting the least recently used (LRU) group GP for deletion to free up space for new groups.

[0142] If the threshold is not exceeded, in step S1214, the system directly creates a new group GP to correspond to the new source IP.

[0143] Following step S1215, the newly received raw log data LG is added to the group GP for subsequent batch analysis.

[0144] Next, in step S1216, the system analyzes the traffic rate coefficient rt and evaluates the current data transmission rate of the source IP in order to dynamically adjust subsequent clustering strategies.

[0145] In step S1217, the priority coefficient pr of the original log is analyzed to assess the importance of the data.

[0146] In step S1218, the system analyzes the memory pressure coefficient mp to assess the current system resource pressure status.

[0147] In step S1219, the system dynamically adjusts the predetermined waiting time n based on the baseline waiting time tm0, the priority coefficient rt, and the priority coefficient pr, and simultaneously dynamically adjusts the predetermined quantity k based on the baseline batch size sz0, the priority coefficient rt, and the memory pressure coefficient mp, in order to optimize cluster latency and efficiency.

[0148] Then at Figure 5C In step S1220, the system evaluates the cluster triggering conditions, including multiple conditions such as "quantity trigger", "time trigger", "idle trigger" and "pressure trigger", to determine whether to perform clustering.

[0149] If the "quantity trigger" condition is met, in step S1221, the raw log data LG with a waiting quantity NM higher than the predetermined quantity k will be clustered.

[0150] If the "time-triggered" condition is met, in step S1222, the raw log data LG cluster with a waiting time TM longer than the predetermined time n will be processed.

[0151] If the "idle trigger" condition is met, in step S1223, when the queue number QN is lower than a predetermined value, all waiting raw log data LG will be clustered.

[0152] If the "stress trigger" condition is met, in step S1224, when the memory utilization rate RM is higher than the preset ratio, all waiting raw log data LG will be clustered immediately.

[0153] In step S1225, the system determines whether the queue is full. If it is full, proceed to step S1226; if it is not full, proceed to step S1227.

[0154] In step S1226, the group GP is marked as delayed processing to reduce real-time load.

[0155] In step S1227, the group GP is sent directly to the log parsing engine 130 for parsing.

[0156] The process then proceeds synchronously to steps S1228 and S1232.

[0157] In step S1228, the group status is reset, such as clearing the counter and updating the status flag.

[0158] In step S1232, asynchronous processing of the language model is performed to support backend deep analysis or natural language understanding.

[0159] After completing step S1228, proceed to step S1229 to perform a memory protection check to prevent resource leaks.

[0160] Next, in step S1230, the system determines whether memory usage exceeds the limit. If it does, proceed to step S1231; otherwise, return to step S1203 and continue waiting for new data.

[0161] In step S1231, cluster behavior is forcibly triggered to alleviate memory pressure.

[0162] After step S1232 is completed, the process proceeds to step S1233, where the system waits for the log parsing engine 130 to complete the processing of the raw log data LG in the group GP.

[0163] In step S1234, when the original log data LG has been processed, the system updates the relevant status.

[0164] In step S1235, the queue status is updated to reflect the latest processing results.

[0165] In step S1236, the system processes the delay groups sequentially to ensure that all data can be parsed in the end.

[0166] Through the steps S1201 to S1237 described above, the cluster unit 120 can efficiently and dynamically cluster raw log data LG from different sources and with different characteristics, and achieve real-time analysis and resource optimization.

[0167] exist Figure 3 In step S130, such as Figure 2As shown, the multiple raw log data entries LG are sequentially input into the log parsing engine 130 to output several normalized log data entries NLG. Within the same group GP, these normalized log data entries NLG record the same security level RK and the same security description DS.

[0168] Please refer to Figure 6 The diagram illustrates a detailed flowchart of step S130 according to an embodiment of this disclosure. The log parsing engine 130, for example, uses large language model (LLM) and retrieval-augmented generation (RAG) techniques for inference. Step S130 includes, for example, steps S131 to S137. Figure 6 In step S131, the log parsing engine 130 performs the action of packaging prompts. The prompts include, but are not limited to, security analysis prompts and normalization prompts. The former is used to trigger the model to infer the security rank (RK) and security description (DS) of the log data; the latter is used to guide the model to convert the final output into a specific structured or standardized format.

[0169] Generally speaking, security analysis prompts include role setting (e.g., you are a security auditor / compliance expert); task description (e.g., assess risks according to established categories and policies); classification criteria / policy summary (e.g., provide actionable standards (e.g., hate speech, violence, self-harm, privacy data, malicious commands, fraud, etc.)); output format (e.g., require structured output (JSON) including risk_level, categories, rationale, actions); evidence-based (e.g., cite fragments from the input as the basis for judgment); action suggestions (e.g., approve, block, rewrite, reject, escalate to manual review); conservative principles (e.g., when uncertain, go for a higher risk level or hand it over to a human review), but are not limited to these. Regularization prompts include explicit output patterns (e.g., field name, type, allowed values); regularization rules (e.g., unit conversion, time zone, language, case sensitivity); abbreviation extensions; error handling (e.g., assigning null or unknown to uncertain values, along with notes); strict formatting (e.g., outputting only JSON, without other text); consistency (e.g., always using the same rule for the same field to avoid context drift), etc., but are not limited to these.

[0170] Continuing from Figure 6In step S132, tokenization and embedding are performed. For example, if the prompt word is "analyze log groups → cybersecurity rating → regularized output", the tokenization step will decompose the sentence into a series of tokens, such as "analysis", "log", "group", "cybersecurity", "rating", etc. Embedding converts these tokens into high-dimensional vectors, enabling the model to not only perform plain text comparison, but also capture contextual relationships and semantic features.

[0171] Next, in Figure 6 Step S133 involves task recognition and decomposition. This step comprises four sub-steps: task recognition, task classifier activation, marking as single / compound tasks, and triggering the task decomposition module. Through this multi-stage processing, the task flow can be refined and structured, facilitating subsequent execution and scheduling.

[0172] In "task cognition", the log parsing engine 130 extracts key actions from prompt words, such as "analysis", "rating" and "regularization", and then recognizes the multi-stage processing requirements contained in the prompt words.

[0173] Next, in the "task classifier startup" phase, the log parsing engine 130 uses the built-in task classifier to determine the task's attributes (such as analytical, generative, translational, or summarizing) and further determine whether the task is a single or complex task. For example, from actions such as "analysis," "rating," and "regularization," it can be identified that the task involves multiple steps and is therefore a complex task.

[0174] In the "Mark as Single / Complex Task" stage, if a task is determined to be complex, the overall objective needs to be broken down into multiple sub-tasks. For example, "Analyze Log Groups" corresponds to the step of parsing the raw log content; "Security Rating" corresponds to assessing the security score of the parsed data; and "Regularized Output" outputs the aforementioned results in a specific format.

[0175] Next, in the "Trigger Task Decomposition Module," the log parsing engine 130 initiates the task decomposition module, breaking down the overall task into smaller executable units, determining the execution order and dependencies, to ensure that the output of previous steps can serve as input for subsequent steps. This process is similar to a project manager first scheduling tasks and then starting to execute them sequentially.

[0176] At Figure 6In step S134, Retrieval-Augmented Generation (RAG) is performed. Since large language models may not be able to directly access internal log data or specific cybersecurity rating rules, this step first converts the question or prompt words into vector form, and then performs similarity searches in internal or external knowledge bases. The search results (such as log content, rating rules, format examples, etc.) are dynamically added to the prompt words to provide the model with more contextual references, thereby improving inference accuracy and reducing the risk of "illusions."

[0177] Then, in Figure 6 In step S135, correlation analysis and security rating are performed. Based on the enhanced retrieval content, the log parsing engine 130 performs semantic and structured analysis on the data in the log group, thereby deriving the corresponding security level RK and security description DS.

[0178] Next, in Figure 6 In step S136, a normalization format conversion is performed. This step fills in the analysis and rating results into the target fields according to the system-defined output template or user requirements, ultimately generating structured normalized log data (NLG).

[0179] Subsequently, Figure 6 In step S137, the generated normalized log data NLG is output to the backend system or database. Within the same group GP, this log data shares the same security level RK and security description DS. For example, as Figure 7 As shown, in the normalized log data NLG, the "security_rank" field is the inferred security level RK, and the "security_describe" field is the corresponding security description DS.

[0180] Through the steps S131 to S137 described above, the log parsing engine 130 is able to achieve high-precision and standardized security level assessment and log normalization processing when faced with raw log data from multiple sources, through prompt word packaging, task recognition and decomposition, enhanced retrieval generation, and structured output.

[0181] Next, in Figure 3 In step S140, these normalized log data NLGs are stored in the log search engine 140.

[0182] Then, in Figure 3 In step S150, the log search engine 150 performs a tiered search based on the security level NK. For example, please refer to... Figure 8 This illustrates the hierarchical structure of normalized log data (NLG). For example... Figure 8The raw log data (LG) on the left is disorganized and difficult to search because it lacks a security level (RK) and security description (DS). For example... Figure 8 The normalized log data (NLG) on the right has a security level (RK) and a security description (DS), which enables hierarchical searching and greatly improves search efficiency.

[0183] Next, in Figure 3 In step S160, the log analysis engine 160 performs layered analysis based on the security level NK.

[0184] According to the above embodiments, the log processing method and log processing system 100 disclosed herein have at least the following unexpected advantages.

[0185] The log processing method and log processing system 100 disclosed herein can improve log processing efficiency and consistency. Through a unified and extensible log structure, regardless of the source (Syslog, AWS JSON, or other formats), standardized processing can be performed, reducing the adaptation costs for subsequent analysis systems. Furthermore, dynamic encapsulation and group hierarchical design ensure that log data possesses structure and semantic relevance before entering SIEM or other security analysis systems, improving analysis efficiency.

[0186] The log processing method and log processing system 100 disclosed herein can enhance semantic understanding and information security analysis capabilities. This technology can introduce lightweight LLM models (such as LLaMA3.2 and Gemma3) for semantic parsing, which can more accurately understand log context and potential abnormal behavior compared to traditional NLP methods. In addition, by fine-tuning the model to adapt to multi-source log formats and a unified output format, the model's generalization ability and practicality are improved.

[0187] The log processing method and log processing system 100 disclosed herein can reduce false alarm rates and improve cybersecurity incident correlation analysis. This technology, through log slicing and grouping, can treat logs from different sources but with similar times as the same event, improving the accuracy of incident correlation analysis. Furthermore, by combining Retrieval Enhanced Generation (RAG) technology with a cybersecurity behavior database, it can expand model knowledge in real time, enabling dynamic analysis and description of emerging cybersecurity threats.

[0188] The log processing method and log processing system 100 disclosed herein can perform adaptive and rapid log clustering. This technology requires no manual adjustment; the system automatically adapts to various scenarios, thus possessing adaptability. Furthermore, this technology simultaneously considers performance, real-time performance, and resource constraints, thus possessing a balance. Moreover, this technology uses boundary constraints to prevent extreme parameters from affecting system stability, thus possessing stability. In addition, the algorithm of this technology can guarantee the predictability of parameter changes.

[0189] The algorithm disclosed here ensures optimal processing performance under various load conditions. For example, compared to log-by-login processing, this technology improves throughput by 3 to 5 times. Response latency for urgent events is less than 1 second; for high-priority events, the response latency is between 3 and 5 seconds; and for general logs, the response latency is between 5 and 30 seconds. Furthermore, aggregation of logs from the same source (IP address) provides a more complete attack chain analysis. Larger language models can analyze a larger log context, improving detection accuracy and enabling batch processing.

[0190] Grouped processing reduces the false alarm rate of individual log entries, thus achieving noise reduction. Dynamic parameter adjustment enables smooth processing, balancing peak traffic. Multi-level protection mechanisms prevent memory overflow, providing resource protection. Automatic degradation processing is available in case of model service anomalies, providing fault recovery capabilities.

[0191] In summary, although this disclosure has been presented above with reference to embodiments, it is not intended to limit the scope of this disclosure. Those skilled in the art to which this disclosure pertains can make various modifications and refinements without departing from the spirit and scope of this disclosure. Therefore, the scope of protection of this disclosure shall be determined by the appended claims.

Claims

1. A log processing method, characterized in that, include: Multiple raw log data entries were continuously obtained, and these raw log data entries came from multiple sources; Cluster the multiple raw log data sets; The multiple raw log data are sequentially input into a log parsing engine to output multiple normalized log data. In the same group, the multiple normalized log data record the same security level and the same security description. as well as Store these multiple normalized log data entries into a log search engine.

2. The log processing method as described in claim 1, characterized in that, In the step of clustering these multiple raw log data, the multiple raw log data that triggered the cluster came from the same source.

3. The log processing method as described in claim 1, characterized in that, In the process of clustering these multiple raw log data, the cluster triggering conditions include: An urgent keyword has appeared.

4. The log processing method as described in claim 1, characterized in that, In the process of clustering these multiple raw log data, the cluster triggering conditions include: One memory usage rate is higher than a predetermined percentage.

5. The log processing method as described in claim 1, characterized in that, In the process of clustering these multiple raw log data, the cluster triggering conditions include: The number of entries in the queue of this log parsing engine is less than a predetermined number.

6. The log processing method as described in claim 1, characterized in that, In the process of clustering these multiple raw log data, the cluster triggering conditions include: The waiting time for multiple raw log data entries is longer than a predetermined time.

7. The log processing method as described in claim 6, characterized in that, The scheduled time is dynamically adjusted.

8. The log processing method as described in claim 6, characterized in that, The scheduled time is dynamically adjusted based on a baseline waiting time, a flow rate coefficient, and a priority coefficient.

9. The log processing method as described in claim 1, characterized in that, In the process of clustering these multiple raw log data, the cluster triggering conditions include: The number of waiting records for multiple raw log data entries is higher than the predetermined number.

10. The log processing method as described in claim 9, characterized in that, The pre-ordered quantity is dynamically adjusted.

11. The log processing method as described in claim 9, characterized in that, The predetermined quantity is dynamically adjusted based on a baseline batch size, a flow rate coefficient, and a memory pressure coefficient.

12. The log processing method as described in claim 1, characterized in that, Including: The log search engine performs tiered searches based on this security level.

13. A log processing system, characterized in that, include: An input unit is used to continuously obtain multiple raw log data from multiple sources; A cluster unit is used to cluster the multiple raw log data; A log parsing engine is used to sequentially input multiple raw log data to output multiple normalized log data. In the same group, the multiple normalized log data record the same security level and the same security description. as well as A log search engine, where multiple normalized log data entries are stored.

14. The log processing system as described in claim 13, characterized in that, The multiple raw log data that triggered the cluster came from the same source.

15. The log processing system as described in claim 13, characterized in that, The cluster triggering conditions used by this cluster unit include: An urgent keyword has appeared.

16. The log processing system as described in claim 13, characterized in that, The cluster unit uses a cluster triggering condition, which includes: One memory usage rate is higher than a predetermined percentage.

17. The log processing system as described in claim 13, characterized in that, The cluster unit uses a cluster triggering condition, which includes: The number of entries in the queue of this log parsing engine is less than a predetermined number.

18. The log processing system as described in claim 13, characterized in that, The cluster unit uses a cluster triggering condition, which includes: The waiting time for multiple raw log data entries is longer than a predetermined time.

19. The log processing system as described in claim 18, characterized in that, The cluster unit dynamically adjusts the scheduled time.

20. The log processing system as described in claim 18, characterized in that, The cluster unit dynamically adjusts the predetermined time based on a baseline waiting time, a flow rate coefficient, and a priority coefficient.

21. The log processing system as described in claim 13, characterized in that, The cluster unit uses a cluster triggering condition, which includes: The number of waiting records for multiple raw log data entries is higher than the predetermined number.

22. The log processing system as described in claim 21, characterized in that, The cluster unit dynamically adjusts this predetermined number.

23. The log processing system as described in claim 21, characterized in that, The cluster unit dynamically adjusts the predetermined number based on a baseline batch size, a traffic rate coefficient, and a memory pressure coefficient.

24. The log processing system as described in claim 13, characterized in that, The log search engine performs tiered searches based on this security level.

25. A computer-readable recording medium for recording a program instruction, characterized in that, When this program instruction is executed by a processor, it causes the processor to perform the following steps: Multiple raw log data entries were continuously obtained, and these raw log data entries came from multiple sources; Cluster the multiple raw log data sets; The multiple raw log data are sequentially input into a log parsing engine to output multiple normalized log data. In the same group, the multiple normalized log data record the same security level and the same security description. as well as Store these multiple normalized log data entries into a log search engine.

Images