Log analysis method and device, computer device and storage medium
By performing preliminary analysis and block processing on the logs, and combining parallel analysis technology, the problems of latency and resource consumption in log analysis of security protection equipment were solved, achieving efficient, real-time, and reliable log analysis.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI DOUXIANG INFORMATION TECH CO LTD
- Filing Date
- 2026-04-21
- Publication Date
- 2026-07-14
AI Technical Summary
In existing technologies, log analysis methods generated by security protection devices suffer from significant delays, huge token consumption, and extended inference time when dealing with extremely long logs. In high-concurrency scenarios, they can even lead to queue backlogs and system bottlenecks.
Initial summary information is obtained by performing preliminary analysis on the target logs. The logs are then divided into blocks based on the initial summary information and preset term thresholds. Target threads are assigned to each log block to achieve parallel analysis.
It effectively solves the problems of excessive lexical consumption, significant inference latency, large computational resource consumption, and low judgment efficiency caused by directly inputting ultra-long logs into large models, improves the efficiency of log analysis and resource utilization, and ensures the accuracy and stability of log judgment.
Smart Images
Figure CN122394885A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of network security technology, and in particular to a log analysis method, apparatus, computer device, and storage medium. Background Technology
[0002] Currently, most methods for analyzing logs generated by security protection devices (such as Web (World Wide Web) application firewalls) involve directly inputting the raw logs into a large model for analysis.
[0003] However, when the logs are too long, log analysis exhibits significant latency issues, consumes a huge amount of tokens, prolongs inference time, and can even cause queue backlogs and system bottlenecks in high-concurrency scenarios. Summary of the Invention
[0004] Therefore, it is necessary to provide a log analysis method, apparatus, computer equipment, and storage medium that can improve the efficiency of log analysis in order to address the above-mentioned technical problems.
[0005] Firstly, this application provides a log analysis method. The method includes:
[0006] A preliminary analysis of the target logs is performed to obtain initial summary information of the target logs;
[0007] Based on the initial summary information and the preset term threshold, the target log is divided into blocks to obtain the log blocks associated with the target log.
[0008] Assign target threads to each log block;
[0009] Each target thread analyzes each log block to obtain the analysis results of the target log.
[0010] In one embodiment, a preliminary analysis of the target log is performed to obtain initial summary information of the target log, including:
[0011] A preliminary analysis of the target log was conducted to obtain the total number of terms in the target log.
[0012] If the total number of lexical elements exceeds the first preset threshold, entity information and keyword information are extracted from the target log.
[0013] Based on entity information and keyword information, determine the initial summary information of the target log.
[0014] In one embodiment, initial summary information of the target log is determined based on entity information and keyword information, including:
[0015] Based on entity information and keyword information, determine the key information of the target log; among which, the key information includes at least one of the following: request payload, exception response code, and potential injection point;
[0016] Based on the key information in the target log, determine the initial summary information of the target log.
[0017] In one embodiment, the target log is segmented based on initial summary information and a preset term threshold to obtain log blocks associated with the target log, including:
[0018] The risk level is determined based on the initial summary information;
[0019] Determine the target word threshold based on the risk level and the preset word threshold;
[0020] Based on the target word threshold, the target log is divided into blocks to obtain the log blocks associated with the target log.
[0021] In one embodiment, determining the target word threshold based on the risk level and a preset word threshold includes:
[0022] If the risk level is lower than the level threshold, the preset word threshold will be used as the target word threshold.
[0023] If the risk level is greater than or equal to the risk level threshold, the preset word threshold is adjusted according to the risk level to obtain the target word threshold.
[0024] In one embodiment, the target log is segmented according to a target lexical threshold to obtain log blocks associated with the target log, including:
[0025] If the total number of tokens exceeds the second preset threshold, obtain the request header summary information of the target log's request header;
[0026] The request header summary information is used to replace the request header in the target log to obtain the updated target log;
[0027] Based on the target word threshold, the updated target log is divided into blocks to obtain the log blocks associated with the target log.
[0028] In one embodiment, each target thread analyzes each log block to obtain the analysis results of the target log, including:
[0029] Each log block is transmitted to the inference model in parallel through each target thread, and the inference results of the inference model are obtained by analyzing each log block.
[0030] Based on the reasoning results, determine the analysis results of the target log.
[0031] Secondly, this application also provides a log analysis apparatus. The apparatus includes:
[0032] The first analysis module is used to perform preliminary analysis on the target logs and obtain initial summary information of the target logs;
[0033] The chunking module is used to chunk the target log based on the initial summary information and the preset word threshold to obtain the log chunks associated with the target log.
[0034] The allocation module is used to allocate target threads to each log block;
[0035] The second analysis module is used to analyze each log block through each target thread to obtain the analysis results of the target log.
[0036] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:
[0037] A preliminary analysis of the target logs is performed to obtain initial summary information of the target logs;
[0038] Based on the initial summary information and the preset term threshold, the target log is divided into blocks to obtain the log blocks associated with the target log.
[0039] Assign target threads to each log block;
[0040] Each target thread analyzes each log block to obtain the analysis results of the target log.
[0041] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:
[0042] A preliminary analysis of the target logs is performed to obtain initial summary information of the target logs;
[0043] Based on the initial summary information and the preset term threshold, the target log is divided into blocks to obtain the log blocks associated with the target log.
[0044] Assign target threads to each log block;
[0045] Each target thread analyzes each log block to obtain the analysis results of the target log.
[0046] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:
[0047] A preliminary analysis of the target logs is performed to obtain initial summary information of the target logs;
[0048] Based on the initial summary information and the preset term threshold, the target log is divided into blocks to obtain the log blocks associated with the target log.
[0049] Assign target threads to each log block;
[0050] Each target thread analyzes each log block to obtain the analysis results of the target log.
[0051] The aforementioned log analysis method, apparatus, computer equipment, and storage medium first perform preliminary analysis on the target log to obtain initial summary information, then complete log block processing based on the initial summary content and preset token thresholds, and allocate target threads to each log block to achieve parallel analysis. This effectively solves the technical problems caused by directly inputting ultra-long logs into large models, such as excessive token consumption, significant inference latency, large computational resource consumption, and low judgment efficiency. This application, while significantly reducing inference latency, improving analysis efficiency, and optimizing resource utilization, ensures the accuracy and stability of log judgment, and achieves efficient, real-time, and reliable analysis of various types of logs. Attached Figure Description
[0052] Figure 1 This is a diagram illustrating the application environment of the log analysis method provided in this embodiment.
[0053] Figure 2 This is a flowchart illustrating the first log analysis method provided in this embodiment;
[0054] Figure 3 A flowchart illustrating the process of determining the initial summary information of the target log in this embodiment;
[0055] Figure 4 This is a schematic diagram of the process of segmenting the target log into blocks provided in this embodiment;
[0056] Figure 5 This is a flowchart illustrating the second log analysis method provided in this embodiment;
[0057] Figure 6 This is a structural block diagram of a log analysis device provided in this embodiment;
[0058] Figure 7 This is an internal structural diagram of the computer device provided in this embodiment. Detailed Implementation
[0059] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0060] Currently, most methods for analyzing logs generated by security protection systems (such as Web (World Wide Web) application firewalls) involve directly inputting the raw logs into a large model for analysis.
[0061] However, when the logs are too long, log analysis exhibits significant latency issues, consumes a huge amount of tokens, prolongs inference time, and can even cause queue backlogs and system bottlenecks in high-concurrency scenarios.
[0062] To address the aforementioned technical problems, the log analysis method provided in this application can be applied to, for example... Figure 1 In the application environment shown, specifically within an AI Agent (Artificial Intelligence Agent) in a security protection system, the AI Agent performs preliminary analysis of the target logs to obtain initial summary information. Based on the initial summary information and preset lexical thresholds, the target logs are segmented into blocks, resulting in log blocks associated with the target logs. Target threads are assigned to each log block. Each target thread then analyzes each log block to obtain the analysis results of the target logs.
[0063] In this context, a security protection system refers to a system that ensures the secure operation of a user's business or network systems. For example, a security protection system can be an enterprise Security Operations Center (SOC), a Security Orchestration, Automation and Response (SOAR) system, or a cloud-native security protection platform. Security protection devices refer to security devices that maintain the security of user networks and services, such as Web (World Wide Web) application firewalls. In this application, the security protection system can receive protection logs generated by the security protection devices and analyze these logs to obtain analysis results; these results may include information such as fault types and fault injection points.
[0064] In one embodiment, such as Figure 2 As shown, a log analysis method is provided, which can be applied to... Figure 1 The security protection system in the text, specifically using an AI Agent that can be applied to the security protection system as an example, includes the following steps:
[0065] S201, Perform preliminary analysis on the target log to obtain initial summary information of the target log.
[0066] The target log refers to the logs generated by security protection devices that require alarm and fault analysis. The initial summary information refers to the summary information obtained from the preliminary analysis of the target log.
[0067] As an optional implementation of this application, a lightweight text digest model or an extractive digest model is invoked to perform semantic compression on the target log. While preserving the log semantics and attack context, redundant fields, duplicate request headers, and invalid content are removed to obtain the refined content of the target log. Based on the refined content, initial digest information is extracted.
[0068] Another optional implementation method according to this application is to pre-build a keyword library, perform full-text scanning and matching on the target logs, extract key segments (e.g., sentences and paragraphs) containing the target keywords, and aggregate the extracted key segments to obtain initial summary information. The keyword library includes, but is not limited to, keywords such as attack characteristics, request structure, and abnormal fields.
[0069] S202, based on the initial summary information and the preset term threshold, the target log is divided into blocks to obtain the log blocks associated with the target log.
[0070] As an optional implementation of this application, key paragraphs in the target log are determined based on the initial summary information. These key paragraphs are then segmented according to a preset term threshold to obtain log blocks associated with the target log.
[0071] Another optional implementation of this application involves determining a target lexical threshold based on initial summary information and a preset lexical threshold. The target log is then segmented according to the target lexical threshold to obtain log blocks associated with the target log.
[0072] Optionally, in this embodiment, after obtaining each log block of the target log, a block type is added to each log block according to the position information of each log block in the target log, such as a header block, body block, and tail block, to facilitate the subsequent inference model to perform inference for each log block.
[0073] S203, assign target threads to each log block.
[0074] The target thread refers to the thread used to process each log block.
[0075] As an optional implementation of this application, an idle thread is allocated to each log block in turn according to the order in which the log blocks are generated, using a round-robin strategy to ensure that thread resources are evenly distributed among the log blocks, avoid excessive load on a single thread, and ensure the fairness and stability of parallel processing.
[0076] Another optional implementation of this application involves marking each log block with a priority based on the initial digest information (e.g., the payload block is high priority, and the request header block is low priority). Candidate threads are sorted according to thread load, and a predetermined number of candidate threads with lower thread loads are selected as target threads. The target threads are then sorted according to thread load to obtain a sorting result. Based on the sorting result and the priority of each log block, a target thread is assigned to each log block. For example, high-priority log blocks are assigned to target threads with lower thread loads.
[0077] Another optional implementation of this application is to obtain the current load status of each candidate thread and select the candidate thread with the best current load status as the target thread.
[0078] S204: Each target thread analyzes each log block to obtain the analysis results of the target log.
[0079] As an optional implementation of this application, each target thread independently performs feature extraction, attack identification, and risk assessment on the allocated log blocks to obtain the sub-analysis results of each log block; after all threads complete the inference, the self-analysis results corresponding to each log block are aggregated and analyzed to form the analysis results of the target log.
[0080] As another optional implementation of this application, each target thread performs vectorization processing on the corresponding log block to obtain the semantic vector corresponding to the log block. The semantic vectors corresponding to each log block are weighted and fused to obtain a global feature vector. The global feature vector is input into a trained classifier, and the classifier outputs the analysis results of the target log.
[0081] Another optional implementation of this application embodiment is to transmit each log block to the inference model in parallel through each target thread, and obtain the inference result output by the inference model after analyzing each log block. Based on the inference result, the analysis result of the target log is determined. In this embodiment, an optional implementation of transmitting each log block to the inference model in parallel to obtain the inference result output by the inference model after analyzing each log block is to pre-create a backup log for each log block. Each log block is transmitted to the inference model in parallel. The inference model determines the order of each log block in the target log based on the block type (each log block has a corresponding block type, such as header, body, and tail) and the vector similarity between log blocks. For each log block, the inference model obtains the context information of the log block based on its order in the target log, and obtains the inference result of the log block based on its content and context information. A result sequence number is added to the inference result based on the order of the log block in the target log (for example, if the log block is the second in the target log, then the result sequence number is also the second). In this embodiment, an optional implementation method for determining the analysis result of the target log based on the inference result is to perform overall inference based on the inference result corresponding to each log block and the result sequence number corresponding to each inference result to obtain the overall inference result. The analysis result of the target log is determined based on the original inference length, final latency index, and overall inference result of the target log. The final latency index refers to the time elapsed from receiving the target log to generating the final analysis result. In this embodiment, during the analysis and inference process of each log block, an optional implementation method for log analysis is to obtain the inference time of each log block. Log blocks with inference times exceeding a preset threshold are designated as target blocks. The backup log corresponding to the target block is obtained. The backup log is divided into blocks to obtain log sub-blocks corresponding to the backup log, and a sub-block sequence number and sub-block identifier are added to each log sub-block. Each log sub-block is distributed to each target thread, which sends it to the inference model. The inference model performs inference on the backup log based on the sub-block sequence number and sub-block content to obtain the inference result of the backup log, i.e., the inference result of the target block.
[0082] The aforementioned log analysis method first performs preliminary analysis on the target log to obtain initial summary information, then completes log block processing based on the initial summary content and preset token thresholds, and allocates target threads to each log block to achieve parallel analysis. This effectively solves the technical problems caused by directly inputting ultra-long logs into large models, such as excessive token consumption, significant inference latency, large computational resource consumption, and low judgment efficiency. This application, while significantly reducing inference latency, improving analysis efficiency, and optimizing resource utilization, ensures the accuracy and stability of log judgment, and achieves efficient, real-time, and reliable analysis of various types of logs.
[0083] In one embodiment, to make the obtained initial summary information more accurate, such as Figure 3 As shown, one optional implementation method for performing preliminary analysis on the target log to obtain initial summary information of the target log includes:
[0084] S301, Perform a preliminary analysis of the target log to obtain the total number of terms in the target log.
[0085] The total number of tokens refers to the total number of tokens contained in the target log.
[0086] As an optional implementation of this application, the target log text is input into a word segmenter, which encodes and segments the log content to generate a word sequence corresponding to the target log. The total number of words in the word sequence is taken as the total number of words in the target log.
[0087] Another optional implementation of this application embodiment is to segment the target log and then count the number of terms in each segment. Finally, the number of terms in each log segment is summed as the total number of terms in the target log. In this embodiment, an optional implementation of segmenting the target log is to segment the target log according to its structure, such as request headers, cookies, request payloads, and response bodies, to obtain each log segment.
[0088] S302, when the total number of lexical elements is greater than the first preset threshold, extract entity information and keyword information from the target log.
[0089] Optionally, in this embodiment, if the total number of lexical units is not greater than a first preset threshold, the target log is directly input into the inference model, and the inference model analyzes and infers the target log to obtain the log analysis result.
[0090] Optionally, in this embodiment, when the total number of lexical units exceeds a first preset threshold, feature extraction is performed on the target log based on an entity feature library and a keyword feature library to extract entity information and keyword information from the target log. The entity feature library records various types of entities, and the keyword feature library records various types of keywords. Entity information and keyword information matching the entities in the entity feature library and the various types of keywords in the keyword feature library are extracted from the target log.
[0091] S303, Determine the initial summary information of the target log based on entity information and keyword information.
[0092] As an optional implementation of this application, a first content segment associated with the entity information is obtained from the target log based on entity information. A second content segment associated with the keyword information is obtained from the target log based on keyword information. The first and second content segments are then aggregated to obtain initial summary information of the target log.
[0093] Another optional implementation method of this application is to predefine a summary template, which includes fixed fields such as request information, risk characteristics, abnormal location, and key payload. The extracted entity information and keyword information are filled into the corresponding template positions according to type to obtain the initial summary information of the target log.
[0094] As another optional implementation of this application, key information of the target log is determined based on entity information and keyword information; wherein, the key information includes at least one of request payload, exception response code, and potential injection point. Initial summary information of the target log is determined based on the key information of the target log. In this embodiment, an optional implementation for determining the initial summary information of the target log based on the key information of the target log is to perform correlation analysis and content integration on the key information of the target log to obtain the initial summary information.
[0095] In this embodiment, the total number of lexical units is obtained by first performing a preliminary analysis on the target log. When the total number of lexical units exceeds a first preset threshold, entity information and keyword information are extracted from the log and initial summary information is generated accordingly. This not only reflects the necessity of generating initial summary information, but also improves the accuracy of the initial summary information obtained.
[0096] In one embodiment, to improve the rationality of target log segmentation, such as Figure 4 As shown, based on the initial summary information and a preset term threshold, the target log is segmented into blocks to obtain log blocks associated with the target log. An optional implementation method includes:
[0097] S401, Determine the risk level based on the initial summary information.
[0098] The risk level refers to the risk level of the device associated with the target log.
[0099] As an optional implementation of this application, a thesaurus of risk keywords is pre-constructed, the initial summary information is matched and statistically analyzed with the risk keywords, and a risk score is determined based on the number and type of the hit risk keywords and the risk level corresponding to the risk keywords. The risk level corresponding to the initial summary information is obtained by mapping the score range to the corresponding risk level.
[0100] Another optional implementation of this application is to convert the initial summary information into a semantic vector, input the semantic vector into a trained risk classifier, and then output the risk level from the risk classifier.
[0101] S402, determine the target word threshold based on the risk level and the preset word threshold.
[0102] The target token threshold refers to the final token threshold, which is also the block size. The target log is divided into blocks based on the block size, such as 200 tokens, 500 tokens, etc.
[0103] As an optional implementation of this application, if the risk level is less than the level threshold, a preset word threshold is used as the target word threshold. For example, if the level threshold is 3, i.e., the risk level is level 2, then the risk level is determined to be less than the level threshold, and the preset word threshold (e.g., 500 tokens) is used as the target word threshold.
[0104] Another optional implementation of this application embodiment is to adjust the preset word threshold according to the risk level when the risk level is greater than or equal to the level threshold, thereby obtaining the target word threshold. In this embodiment, an optional implementation of adjusting the preset word threshold according to the risk level to obtain the target word threshold is to determine the difference between the risk level and the level threshold. The product of the difference and a preset value is used as the adjustment amount. The difference between the preset word threshold and the adjustment amount is used as the target word threshold. For example, if the difference between the risk level and the level threshold is 2, and the preset threshold is 100, then the adjustment amount is 200. If the preset word threshold is 500 tokens, then the difference between 500 and 200, 300, is used as the target word threshold, i.e., 300 tokens. In this embodiment, when the risk of the target log is determined to be higher than the threshold, in order to improve log analysis efficiency, the target log is divided into more log blocks, and then multiple target threads perform parallel processing, improving log analysis efficiency, achieving timely risk detection and timely processing, and effectively improving the stability and security of the business system.
[0105] S403, Based on the target word threshold, the target log is divided into blocks to obtain the log blocks associated with the target log.
[0106] As an optional implementation of this application, the target log is divided into blocks using a target lexical threshold as the block size or block dimension to obtain log blocks associated with the target log. For example, if the target lexical threshold is 500 tokens and the target log contains 20,000 tokens, then the target log can be divided into 40 log blocks.
[0107] Another optional implementation of this application is to obtain the request header summary information of the target log's request header when the total number of lexical elements exceeds a second preset threshold. The request header summary information is then used to replace the request header in the target log to obtain an updated target log. Based on the target lexical threshold, the updated target log is segmented to obtain log blocks associated with the target log. The second preset threshold is greater than the first preset threshold. In this embodiment, when the total number of log lexical elements exceeds the second preset threshold, extracting the request header summary information and replacing the original lengthy request header with the summary effectively solves the technical problems of excessively long log lexical elements, excessive number of blocks, high model inference pressure, and dilution of key attack features caused by excessive redundant fields such as request headers and cookies. By significantly compressing the length of non-core fields without losing the basic identifier information of the request header, and then segmenting according to the target lexical threshold, the number of blocks can be reduced, parallel thread overhead can be lowered, and log processing speed can be improved. Simultaneously, core security information such as request payload and potential injection points is kept intact and not compressed, achieving a balance between lightweight long logs and accurate risk assessment.
[0108] In this embodiment, the risk level is first determined based on the initial summary information, and then the target word threshold is dynamically determined by combining the risk level with the preset word threshold. Finally, the log segmentation is completed according to the target word threshold. This can effectively solve the technical problems of poor flexibility of traditional fixed threshold segmentation, high latency of inference for high-risk logs, and difficulty in ensuring contextual relevance. At the same time, it can achieve adaptive matching between segmentation strategy and risk level, improving the rationality of segmentation while taking into account the accuracy and timeliness of target log analysis.
[0109] In one embodiment, such as Figure 5 As shown, one optional implementation of a log analysis method includes:
[0110] S501, Perform a preliminary analysis of the target log to obtain the total number of terms in the target log.
[0111] S502, when the total number of tokens is greater than the first preset threshold, extract entity information and keyword information from the target log.
[0112] S503, Based on entity information and keyword information, determine the key information of the target log. The key information includes at least one of the following: request payload, exception response code, and potential injection point.
[0113] S504, Determine the initial summary information of the target log based on the key information of the target log.
[0114] S505, Determine the risk level based on the initial summary information.
[0115] S506: When the risk level is less than the level threshold, the preset word threshold will be used as the target word threshold.
[0116] S507: When the risk level is greater than or equal to the level threshold, the preset word threshold is adjusted according to the risk level to obtain the target word threshold.
[0117] S508: Based on the target word threshold, the target log is divided into blocks to obtain the log blocks associated with the target log.
[0118] S509, if the total number of tokens is greater than the second preset threshold, obtain the request header summary information of the request header of the target log.
[0119] S510 uses the request header summary information to replace the request header in the target log, thus obtaining the updated target log.
[0120] S511, based on the target word threshold, the updated target log is divided into blocks to obtain the log blocks associated with the target log.
[0121] S512 transmits each log block to the inference model in parallel through the target thread, and obtains the inference results output by the inference model after analyzing each log block.
[0122] S513, Based on the reasoning results, determine the analysis results of the target log.
[0123] This application first performs preliminary analysis on the target log to obtain initial summary information, then completes log block processing based on the initial summary content and preset token thresholds, and allocates target threads to each log block to achieve parallel analysis. This effectively solves the technical problems caused by directly inputting ultra-long logs into large models, such as excessive token consumption, significant inference latency, large computational resource consumption, and low judgment efficiency. This application, while significantly reducing inference latency, improving analysis efficiency, and optimizing resource utilization, ensures the accuracy and stability of log judgment, and achieves efficient, real-time, and reliable analysis of various types of logs.
[0124] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0125] Based on the same inventive concept, this application also provides a log analysis apparatus for implementing the log analysis method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more log analysis apparatus embodiments provided below can be found in the limitations of the log analysis method described above, and will not be repeated here.
[0126] In one embodiment, such as Figure 6 As shown, a log analysis device 1 is provided, including: a first analysis module 10, a block segmentation module 20, an allocation module 30, and a second analysis module 40, wherein:
[0127] The first analysis module 10 is used to perform preliminary analysis on the target log to obtain initial summary information of the target log;
[0128] The segmentation module 20 is used to segment the target log according to the initial summary information and the preset word threshold to obtain the log blocks associated with the target log.
[0129] Allocation module 30 is used to allocate target threads to each log block;
[0130] The second analysis module 40 is used to analyze each log block through each target thread to obtain the analysis results of the target log.
[0131] In one embodiment, the first analysis module is further specifically used for:
[0132] A preliminary analysis of the target log was conducted to obtain the total number of terms in the target log.
[0133] If the total number of lexical elements exceeds the first preset threshold, entity information and keyword information are extracted from the target log.
[0134] Based on entity information and keyword information, determine the initial summary information of the target log.
[0135] In one embodiment, the first analysis module is further specifically used for:
[0136] Based on entity information and keyword information, determine the key information of the target log; among which, the key information includes at least one of the following: request payload, exception response code, and potential injection point;
[0137] Based on the key information in the target log, determine the initial summary information of the target log.
[0138] In one embodiment, the segmentation module is further specifically used for:
[0139] The risk level is determined based on the initial summary information;
[0140] Determine the target word threshold based on the risk level and the preset word threshold;
[0141] Based on the target word threshold, the target log is divided into blocks to obtain the log blocks associated with the target log.
[0142] In one embodiment, the segmentation module is further specifically used for:
[0143] If the risk level is lower than the level threshold, the preset word threshold will be used as the target word threshold.
[0144] If the risk level is greater than or equal to the risk level threshold, the preset word threshold is adjusted according to the risk level to obtain the target word threshold.
[0145] In one embodiment, the segmentation module is further specifically used for:
[0146] If the total number of tokens exceeds the second preset threshold, obtain the request header summary information of the target log's request header;
[0147] The request header summary information is used to replace the request header in the target log to obtain the updated target log;
[0148] Based on the target word threshold, the updated target log is divided into blocks to obtain the log blocks associated with the target log.
[0149] In one embodiment, the second analysis module is further specifically used for:
[0150] Each log block is transmitted to the inference model in parallel through each target thread, and the inference results of the inference model are obtained by analyzing each log block.
[0151] Based on the reasoning results, determine the analysis results of the target log.
[0152] The modules in the aforementioned log analysis device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.
[0153] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores log analysis-related information. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a log analysis method.
[0154] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0155] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:
[0156] A preliminary analysis of the target logs is performed to obtain initial summary information of the target logs;
[0157] Based on the initial summary information and the preset term threshold, the target log is divided into blocks to obtain the log blocks associated with the target log.
[0158] Assign target threads to each log block;
[0159] Each target thread analyzes each log block to obtain the analysis results of the target log.
[0160] In one embodiment, when the processor executes the computer program, it further performs the following steps: performing a preliminary analysis of the target log to obtain initial summary information of the target log, including:
[0161] A preliminary analysis of the target log was conducted to obtain the total number of terms in the target log.
[0162] If the total number of lexical elements exceeds the first preset threshold, entity information and keyword information are extracted from the target log.
[0163] Based on entity information and keyword information, determine the initial summary information of the target log.
[0164] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining initial summary information of the target log based on entity information and keyword information, including:
[0165] Based on entity information and keyword information, determine the key information of the target log; among which, the key information includes at least one of the following: request payload, exception response code, and potential injection point;
[0166] Based on the key information in the target log, determine the initial summary information of the target log.
[0167] In one embodiment, when the processor executes the computer program, it further performs the following steps: dividing the target log into blocks based on initial summary information and a preset term threshold to obtain log blocks associated with the target log, including:
[0168] The risk level is determined based on the initial summary information;
[0169] Determine the target word threshold based on the risk level and the preset word threshold;
[0170] Based on the target word threshold, the target log is divided into blocks to obtain the log blocks associated with the target log.
[0171] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining a target lexical threshold based on a risk level and a preset lexical threshold, including:
[0172] If the risk level is lower than the level threshold, the preset word threshold will be used as the target word threshold.
[0173] If the risk level is greater than or equal to the risk level threshold, the preset word threshold is adjusted according to the risk level to obtain the target word threshold.
[0174] In one embodiment, when the processor executes the computer program, it further performs the following steps: dividing the target log into blocks based on a target lexical threshold to obtain log blocks associated with the target log, including:
[0175] If the total number of tokens exceeds the second preset threshold, obtain the request header summary information of the target log's request header;
[0176] The request header summary information is used to replace the request header in the target log to obtain the updated target log;
[0177] Based on the target word threshold, the updated target log is divided into blocks to obtain the log blocks associated with the target log.
[0178] In one embodiment, when the processor executes the computer program, it further performs the following steps: analyzing each log block through each target thread to obtain the analysis results of the target log, including:
[0179] Each log block is transmitted to the inference model in parallel through each target thread, and the inference results of the inference model are obtained by analyzing each log block.
[0180] Based on the reasoning results, determine the analysis results of the target log.
[0181] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:
[0182] A preliminary analysis of the target logs is performed to obtain initial summary information of the target logs;
[0183] Based on the initial summary information and the preset term threshold, the target log is divided into blocks to obtain the log blocks associated with the target log.
[0184] Assign target threads to each log block;
[0185] Each target thread analyzes each log block to obtain the analysis results of the target log.
[0186] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: performing a preliminary analysis of the target log to obtain initial summary information of the target log, including:
[0187] A preliminary analysis of the target log was conducted to obtain the total number of terms in the target log.
[0188] If the total number of lexical elements exceeds the first preset threshold, entity information and keyword information are extracted from the target log.
[0189] Based on entity information and keyword information, determine the initial summary information of the target log.
[0190] In one embodiment, when the computer program is executed by a processor, it further performs the following steps: determining initial summary information of the target log based on entity information and keyword information, including:
[0191] Based on entity information and keyword information, determine the key information of the target log; among which, the key information includes at least one of the following: request payload, exception response code, and potential injection point;
[0192] Based on the key information in the target log, determine the initial summary information of the target log.
[0193] In one embodiment, when the computer program is executed by a processor, it further performs the following steps: dividing the target log into blocks based on initial summary information and a preset term threshold to obtain log blocks associated with the target log, including:
[0194] The risk level is determined based on the initial summary information;
[0195] Determine the target word threshold based on the risk level and the preset word threshold;
[0196] Based on the target word threshold, the target log is divided into blocks to obtain the log blocks associated with the target log.
[0197] In one embodiment, when the computer program is executed by a processor, it further performs the following steps: determining a target lexical threshold based on a risk level and a preset lexical threshold, including:
[0198] If the risk level is lower than the level threshold, the preset word threshold will be used as the target word threshold.
[0199] If the risk level is greater than or equal to the risk level threshold, the preset word threshold is adjusted according to the risk level to obtain the target word threshold.
[0200] In one embodiment, when the computer program is executed by a processor, it further performs the following steps: dividing the target log into blocks based on a target lexical threshold to obtain log blocks associated with the target log, including:
[0201] If the total number of tokens exceeds the second preset threshold, obtain the request header summary information of the target log's request header;
[0202] The request header summary information is used to replace the request header in the target log to obtain the updated target log;
[0203] Based on the target word threshold, the updated target log is divided into blocks to obtain the log blocks associated with the target log.
[0204] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: analyzing each log block through each target thread to obtain the analysis results of the target log, including:
[0205] Each log block is transmitted to the inference model in parallel through each target thread, and the inference results of the inference model are obtained by analyzing each log block.
[0206] Based on the reasoning results, determine the analysis results of the target log.
[0207] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:
[0208] A preliminary analysis of the target logs is performed to obtain initial summary information of the target logs;
[0209] Based on the initial summary information and the preset term threshold, the target log is divided into blocks to obtain the log blocks associated with the target log.
[0210] Assign target threads to each log block;
[0211] Each target thread analyzes each log block to obtain the analysis results of the target log.
[0212] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: performing a preliminary analysis of the target log to obtain initial summary information of the target log, including:
[0213] A preliminary analysis of the target log was conducted to obtain the total number of terms in the target log.
[0214] If the total number of lexical elements exceeds the first preset threshold, entity information and keyword information are extracted from the target log.
[0215] Based on entity information and keyword information, determine the initial summary information of the target log.
[0216] In one embodiment, when the computer program is executed by a processor, it further performs the following steps: determining initial summary information of the target log based on entity information and keyword information, including:
[0217] Based on entity information and keyword information, determine the key information of the target log; among which, the key information includes at least one of the following: request payload, exception response code, and potential injection point;
[0218] Based on the key information in the target log, determine the initial summary information of the target log.
[0219] In one embodiment, when the computer program is executed by a processor, it further performs the following steps: dividing the target log into blocks based on initial summary information and a preset term threshold to obtain log blocks associated with the target log, including:
[0220] The risk level is determined based on the initial summary information;
[0221] Determine the target word threshold based on the risk level and the preset word threshold;
[0222] Based on the target word threshold, the target log is divided into blocks to obtain the log blocks associated with the target log.
[0223] In one embodiment, when the computer program is executed by a processor, it further performs the following steps: determining a target lexical threshold based on a risk level and a preset lexical threshold, including:
[0224] If the risk level is lower than the level threshold, the preset word threshold will be used as the target word threshold.
[0225] If the risk level is greater than or equal to the risk level threshold, the preset word threshold is adjusted according to the risk level to obtain the target word threshold.
[0226] In one embodiment, when the computer program is executed by a processor, it further performs the following steps: dividing the target log into blocks based on a target lexical threshold to obtain log blocks associated with the target log, including:
[0227] If the total number of tokens exceeds the second preset threshold, obtain the request header summary information of the target log's request header;
[0228] The request header summary information is used to replace the request header in the target log to obtain the updated target log;
[0229] Based on the target word threshold, the updated target log is divided into blocks to obtain the log blocks associated with the target log.
[0230] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: analyzing each log block through each target thread to obtain the analysis results of the target log, including:
[0231] Each log block is transmitted to the inference model in parallel through each target thread, and the inference results of the inference model are obtained by analyzing each log block.
[0232] Based on the reasoning results, determine the analysis results of the target log.
[0233] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.
[0234] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0235] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A log analysis method, characterized in that, The method includes: A preliminary analysis of the target log is performed to obtain initial summary information of the target log; Based on the initial summary information and the preset term threshold, the target log is divided into blocks to obtain the log blocks associated with the target log; Assign target threads to each log block; Each target thread analyzes each log block to obtain the analysis results of the target log.
2. The method according to claim 1, characterized in that, The preliminary analysis of the target log to obtain initial summary information of the target log includes: A preliminary analysis of the target log is performed to obtain the total number of terms in the target log; If the total number of lexical elements is greater than a first preset threshold, entity information and keyword information are extracted from the target log. Based on the entity information and the keyword information, the initial summary information of the target log is determined.
3. The method according to claim 2, characterized in that, The step of determining the initial summary information of the target log based on the entity information and the keyword information includes: Based on the entity information and the keyword information, the key information of the target log is determined; wherein, the key information includes at least one of request payload, exception response code, and potential injection point; Based on the key information of the target log, determine the initial summary information of the target log.
4. The method according to claim 2, characterized in that, The step of segmenting the target log based on the initial summary information and a preset term threshold to obtain log blocks associated with the target log includes: The risk level is determined based on the initial summary information; The target word threshold is determined based on the risk level and the preset word threshold. Based on the target lexical threshold, the target log is divided into blocks to obtain log blocks associated with the target log.
5. The method according to claim 4, characterized in that, The step of determining the target word threshold based on the risk level and the preset word threshold includes: If the risk level is less than the level threshold, the preset word threshold will be used as the target word threshold. If the risk level is greater than or equal to the risk level threshold, the preset word threshold is adjusted according to the risk level to obtain the target word threshold.
6. The method according to claim 4, characterized in that, The step of segmenting the target log according to the target lexical threshold to obtain the log block associated with the target log includes: If the total number of tokens is greater than a second preset threshold, obtain the request header summary information of the request header of the target log; The request header summary information is used to replace the request header in the target log to obtain the updated target log; Based on the target lexical threshold, the updated target log is divided into blocks to obtain the log blocks associated with the target log.
7. The method according to any one of claims 1-6, characterized in that, The step of analyzing each log block through each target thread to obtain the analysis results of the target log includes: Each of the target threads transmits the log blocks in parallel to the inference model, and the inference results output by the inference model after analyzing the log blocks are obtained. Based on the reasoning results, the analysis results of the target log are determined.
8. A log analysis device, characterized in that, include: The first analysis module is used to perform preliminary analysis on the target log to obtain initial summary information of the target log; The segmentation module is used to segment the target log according to the initial summary information and the preset word threshold to obtain the log block associated with the target log; The allocation module is used to allocate target threads to each log block; The second analysis module is used to analyze each log block through each target thread to obtain the analysis results of the target log.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the log analysis method according to any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the log analysis method according to any one of claims 1 to 7.