A security protection system and method based on log centralized analysis

Abstract

The application belongs to the technical field of computers, and particularly relates to a security protection system and method based on log centralized analysis, which comprises: a multi-source log collection layer for collecting original logs; a log data processing layer for receiving the original logs and performing data processing and verification, and outputting log data; an intelligent analysis engine layer for performing real-time analysis on the log data through multiple built-in engines and multiple pre-trained processing models, and outputting risk identification results; each processing model is obtained through offline mode learning and training based on historical logs; a security application layer for performing multi-scene security research and judgment, hierarchical alarm, attack tracing and automatic disposal instruction issuing according to the risk identification results, and synchronously generating corresponding full-amount operation data; and a visual display layer for receiving the full-amount operation data and rendering and outputting. The application can greatly reduce the false alarm and missed alarm rates, and greatly reduce the artificial operation cost.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and specifically to a security protection system and method based on centralized log analysis. Background Technology

[0002] As digital transformation deepens across industries, enterprise network architectures encompass a large number of hardware and software assets, including network devices, security devices, servers, databases, and email gateways. These devices continuously generate massive amounts of scattered logs. Existing traditional log-based security solutions suffer from the following inherent technical shortcomings:

[0003] First, log collection channels are fragmented, with logs from multiple devices scattered and collection and parsing standards inconsistent, making it difficult to form a unified data standard.

[0004] Second, traditional security solutions rely on fixed thresholds and rule matching, resulting in loud alarm noise, high false alarm and false negative rates, and an inability to accurately identify abnormal behavior.

[0005] Third, the handling of security incidents relies on manual operation and lacks automated handling capabilities, resulting in high manual operating costs and a certain lag in risk handling.

[0006] Therefore, the aforementioned deficiencies result in insufficient security protection capabilities for enterprises, making them unable to adapt to the security operation needs under digital transformation. Summary of the Invention

[0007] In view of the technical defects mentioned in the background art, the purpose of this invention is to provide a security protection system and method based on centralized log analysis, which aims to at least solve one of the technical problems in the related art to a certain extent.

[0008] To achieve the above objectives, in a first aspect, embodiments of the present invention provide a security protection system based on centralized log analysis, the system comprising:

[0009] The multi-source log collection layer is used to collect raw logs generated by various types of devices and business systems across the entire network;

[0010] The log data processing layer is used to receive the raw logs, perform data processing and verification, and output log data in a unified and standardized format.

[0011] The intelligent analysis engine layer is used to perform real-time analysis on the log data through multiple built-in engines and multiple pre-trained processing models, and output risk identification results; wherein, each processing model is learned and trained based on historical logs in offline mode.

[0012] The security application layer is used to execute multi-scenario security assessment, hierarchical alarm, attack tracing and automatic handling instructions based on the risk identification results, and simultaneously generate corresponding full operational data;

[0013] The visualization layer is used to receive the full amount of operational data and render and output a security posture dashboard, user profiles, alarm dashboards, and compliance audit reports.

[0014] As a specific implementation of this application, the intelligent analysis engine layer includes a streaming computing engine, an SPL analysis engine, an anomaly detection engine, and a threat intelligence engine; the anomaly detection engine is configured to perform anomaly identification of logs or metrics based on at least one of hierarchical clustering, VAE, CVAE, KDE, and IsolationForest; and to perform offline word segmentation training on historical standardized logs based on Spark to generate log pattern trees, parameter trees, behavior whitelist self-learning models, and multi-class unsupervised anomaly detection models.

[0015] As one specific implementation of this application, the anomaly detection model specifically includes:

[0016] The VAE / CVAE model is adapted to periodic metrics, the IsolationForest model is adapted to discrete hardware metrics, the KDE model is adapted to batch tasks, and the hierarchical clustering model is adapted to log text.

[0017] As one specific implementation of this application, the threat intelligence engine simultaneously connects to open-source OSINT intelligence and commercial threat intelligence, and performs real-time matching of IP, domain name, URL and malicious file hash blacklist.

[0018] As a specific implementation of this application, the whitelist self-learning model adopts a time-series sliding window continuous update mechanism, periodically collects normal log behavior data within the period to iteratively correct the baseline threshold, dynamically adapts to business access volume, and adapts to the periodic changes in operation and maintenance habits over time.

[0019] As a specific implementation of this application, the security application layer includes an asset vulnerability association module and a SOAR automated response module.

[0020] The SOAR automated response module supports tiered handling strategy configuration. High-risk attacks are automatically blocked and isolated, medium-risk events are pushed to manual handling work orders, and low-risk behaviors are only recorded in audit logs without triggering automatic operations. It has a built-in handling effect write-back mechanism. After the device executes the blocking and isolation command, it sends the execution result back to the intelligent analysis engine layer for secondary verification of whether the risk has been eliminated and to update the asset risk label.

[0021] As one specific implementation of this application, the intelligent analysis engine layer performs scheduling through deployed fusion-nested scheduling logic, and the execution flow of the fusion-nested scheduling logic is as follows:

[0022] The streaming computing engine first calls the hierarchical clustering model to complete the coarse classification of log text, and then calls the corresponding sub-models according to the log data type. Periodic business indicators call the VAE / CVAE model, discrete hardware indicators call the IsolationForest model, and batch task indicators call the KDE model.

[0023] The SPL analysis engine reads the coarse classification results from the streaming computing engine, nests and loads the whitelist self-learning model for secondary verification, and sends the abnormal features obtained from the joint verification of multiple models back to the anomaly detection engine for risk weighting and scoring, forming a multi-layered nested linkage mechanism.

[0024] Secondly, embodiments of the present invention also provide a security protection method based on centralized log analysis, applied to a security protection system based on centralized log analysis as described in the first aspect, the method comprising the following steps:

[0025] Collect raw logs generated by various types of devices and business systems across the entire network through multiple protocol collection channels;

[0026] The original logs are processed and validated to output log data in a standardized format.

[0027] Based on Spark, offline word segmentation training of historical standardized logs is used to generate log pattern trees, parameter trees, behavior whitelist self-learning models, and multi-class unsupervised anomaly detection models.

[0028] By using streaming computing and the SPL engine to process standardized logs in real time, combined with offline trained models and internal and external network threat intelligence, multi-dimensional risk identification is performed, and risk identification results are output.

[0029] Based on the risk identification results, multi-scenario security assessment, tiered alerts, and attack tracing and automatic handling instructions are issued, and corresponding full operational data is generated simultaneously.

[0030] The system retains all operational data to meet regulatory storage requirements and provides visualized outputs such as security posture dashboards, user profiles, alarm dashboards, and compliance audit reports.

[0031] As one specific implementation of this application, the anomaly detection model specifically includes:

[0032] The VAE / CVAE model is adapted to periodic metrics, the IsolationForest model is adapted to discrete hardware metrics, the KDE model is adapted to batch tasks, and the hierarchical clustering model is adapted to log text.

[0033] As one specific implementation of this application, the risk identification includes a multi-model fusion nested scheduling sub-step:

[0034] Load the log pattern tree, parameter tree, sub-models of various unsupervised anomaly detection models, and whitelist self-learning models generated by Spark offline training;

[0035] The streaming computing engine first coarsely divides the real-time log text using a hierarchical clustering model, and then performs the first layer of anomaly detection based on nested matching of corresponding exclusive sub-models according to the log metric type.

[0036] The analysis engine nests and calls the whitelist self-learning model to perform secondary verification on the first-layer discrimination result and output secondary anomaly markers.

[0037] By fusing the two-layer model for discrimination labeling and weighted risk scoring, a high-confidence risk identification result is obtained after fusion.

[0038] The technical solution provided by the embodiments of the present invention has the following significant advantages:

[0039] 1. By adopting a five-layer hierarchical data flow architecture, the entire chain of log collection, processing, analysis, handling and display is connected, which solves the defects of traditional platform modules being fragmented and data not being able to communicate with each other, realizes the unified collection and standardized governance of all logs, and improves the integrity and availability of log data;

[0040] 2. It integrates multiple types of unsupervised algorithms, combined with offline log mode training and whitelist self-learning mechanism, which breaks away from the limitations of traditional solutions that use fixed thresholds, and greatly reduces the false alarm and false negative rates.

[0041] 3. Through multi-scenario security assessment, combined with asset vulnerability correlation and SOAR automated response modules, the entire process of security incident detection, tracing, and handling is automated, significantly reducing manual operating costs and shortening risk response time;

[0042] 4. It retains all complete data throughout the entire process and outputs compliance audit reports, reducing the labor costs and compliance risks for enterprise compliance audits. Attached Figure Description

[0043] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below.

[0044] Figure 1 This is a block diagram illustrating the principle of a security protection system based on centralized log analysis provided in an embodiment of the present invention.

[0045] Figure 2This is a flowchart of a security protection method based on centralized log analysis provided in an embodiment of the present invention;

[0046] Figure 3 This is a schematic diagram of a fusion nested scheduling process provided in an embodiment of the present invention. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0049] Throughout this specification, references to "an embodiment," "an example," or "an example" mean that a particular feature, structure, or characteristic described in connection with that embodiment or example is included in at least one embodiment of the invention. Therefore, the phrases "in an embodiment," "in an embodiment," "an example," or "an example" appearing in various places throughout the specification do not necessarily refer to the same embodiment or example. Furthermore, specific features, structures, or characteristics can be combined in one or more embodiments or examples in any suitable combination and / or sub-combination.

[0050] Syslog: System log transmission protocol used for standardized reporting of device logs;

[0051] Agent: A data collection agent program deployed on the host / business side to collect local logs;

[0052] API: Application Programming Interface;

[0053] Kafka: A distributed real-time message queue that enables high-throughput log caching and distribution;

[0054] Rsyslog: A high-performance log forwarding service program;

[0055] SPL: Search Processing Language;

[0056] VAE: Variational Autoencoder;

[0057] CVAE: Conditional Variational Autoencoder;

[0058] KDE: Kernel Density Estimation;

[0059] IsolationForest: An unsupervised anomaly detection algorithm for isolated forests.

[0060] Spark: A distributed offline big data computing framework used for offline training of log models;

[0061] CEP: Complex Event Processing (streaming time-series analysis logic).

[0062] OSINT: Open Source Intelligence;

[0063] SOAR: Security Orchestration, Automation and Response;

[0064] CMDB: Configuration Management Database;

[0065] EDR: Endpoint Detection and Response (interconnected devices for handling).

[0066] CVSS: Common Vulnerability Scoring System;

[0067] SQL: Structured Query Language;

[0068] XSS: Cross-Site Scripting;

[0069] VPN: Virtual Private Network;

[0070] IP: Internet Protocol address;

[0071] URL: Uniform Resource Locator.

[0072] It should be noted that, unless otherwise stated, the technical terms used in this embodiment have the common meaning as understood in the relevant technical field.

[0073] Please refer to Figure 1 This invention provides a security protection system based on centralized log analysis, the system comprising:

[0074] The multi-source log collection layer is used to collect raw logs generated by various types of devices and business systems across the entire network;

[0075] The log data processing layer is used to receive the raw logs, perform data processing and verification, and output log data in a unified and standardized format.

[0076] The intelligent analysis engine layer is used to perform real-time analysis on the log data through multiple built-in engines and multiple pre-trained processing models, and output risk identification results; wherein, each processing model is learned and trained based on historical logs in offline mode.

[0077] The security application layer is used to execute multi-scenario security assessment, hierarchical alarm, attack tracing and automatic handling instructions based on the risk identification results, and simultaneously generate corresponding full operational data;

[0078] The visualization layer is used to receive the full amount of operational data and render and output a security posture dashboard, user profiles, alarm dashboards, and compliance audit reports.

[0079] In this embodiment, the multi-source log collection layer supports collection methods such as Syslog, Agent, API, and Kafka, and the collection objects include network devices, perimeter security devices, hosts, databases, business applications, VPN devices, bastion hosts, and email gateways.

[0080] The system's raw log data, analysis records, and handling records are automatically stored for no less than 6 months to meet the mandatory log retention requirements of the Cybersecurity Classified Protection 2.0.

[0081] The processing and verification include cleaning, field extraction, desensitization, time synchronization, and integrity verification.

[0082] In application, the intelligent analysis engine layer includes a streaming computing engine, an SPL analysis engine, an anomaly detection engine, and a threat intelligence engine. The anomaly detection engine is configured to perform anomaly identification on logs or metrics based on at least one of the following algorithms: hierarchical clustering, VAE, CVAE, KDE, and IsolationForest. It also performs offline word segmentation training on historical standardized logs based on Spark to generate log pattern trees, parameter trees, behavior whitelist self-learning models, and multi-class unsupervised anomaly detection models. These models are then persistently stored in a model repository.

[0083] The log pattern tree is a hierarchical tree structure built by Spark offline word segmentation and sentence decomposition of historical logs; the tree nodes store fixed text templates for normal logs and fixed event flows, which are used to match the log text structure in real time and quickly identify new log formats that have never appeared before.

[0084] The parameter tree exists in relation to the log pattern tree and is used to record the normal value range and distribution range of variable fields (such as login IP, operation duration, access count, and return code) in each standard log; the variable parameters in the log are compared in real time, and parameters that exceed the normal range are identified as abnormal parameters.

[0085] The behavior whitelist self-learning model is a dynamic baseline model automatically trained based on long-term historical normal account, IP and device access behavior; it continuously learns users' regular login time periods, frequently used addresses, high-frequency operation commands and access asset ranges, without the need for manual whitelist entry, can automatically update the baseline with the business cycle, and is used to review the results of a single model detection and filter false alarms caused by normal business fluctuations.

[0086] Furthermore, the whitelist self-learning model adopts a time-series sliding window continuous update mechanism, periodically collects normal log behavior data within the period to iteratively correct the baseline threshold, dynamically adapts to the periodic changes in business access volume and maintenance habits over time.

[0087] The anomaly detection model specifically includes:

[0088] The model includes a VAE / CVAE model adapted to periodic metrics, an IsolationForest model adapted to discrete hardware metrics, a KDE model adapted to batch tasks, and a hierarchical clustering model adapted to log text. Among them, the hierarchical clustering is a hierarchical text clustering algorithm that aggregates log text hierarchically after word segmentation, distinguishing normal log text clusters from abnormal and unfamiliar log texts.

[0089] The threat intelligence engine connects to both open-source OSINT intelligence and commercial threat intelligence, and performs real-time matching of IP addresses, domain names, URLs, and malicious file hash blacklists.

[0090] In this embodiment, the security application layer includes a built-in VPN security analysis module, an operation and maintenance audit module, a phishing email detection module, a brute-force attack detection module, an asset vulnerability association module, and a SOAR automated response module.

[0091] The VPN security analysis module identifies logins from uncommon IPs / devices / time periods, shared accounts, brute-force attacks, and abnormal VPN device resource load, outputs VPN account risk tags, and pushes them to the asset vulnerability association module for correlation analysis.

[0092] The operation and maintenance audit module accesses the bastion host operation and maintenance logs, calls the behavior whitelist self-learning model and hierarchical clustering algorithm to identify sensitive commands, unauthorized access, sensitive table operations and zombie accounts; outputs operation and maintenance behavior risk characteristics to the preset profile fusion model to generate a 360-degree risk profile of operation and maintenance personnel.

[0093] The phishing email detection module parses the full logs of the email gateway, matches them with the built-in sensitive keyword library, trusted email whitelist, and malicious attachment characteristics; combined with SPL cross-source correlation analysis, it distinguishes between mass phishing emails and fraudulent emails, marks email risks, and synchronizes them to the alarm link.

[0094] The brute-force attack detection module performs time-series sliding window statistics on streaming logs, aggregates and identifies consecutive login failures of the same account / IP within a short period of time, high-frequency port scanning, and high-frequency interface error behavior, marks brute-force risks, and links the asset database to lock down the victimized devices.

[0095] The asset vulnerability association module uses the IP address and asset number in the log as an index to associate the CMDB asset database and third-party vulnerability scanning results in real time. It adds an asset importance level, corresponding vulnerability number and vulnerability severity score to each risk log, providing a weighting basis for comprehensive risk weighting scoring; and realizes four-dimensional information binding and tracing of attack logs, victimized assets, system vulnerabilities and operation and maintenance behavior.

[0096] The SOAR automated response module receives tiered alarms from preceding modules, supports visual script arrangement, and performs differentiated handling according to risk level, specifically:

[0097] The SOAR automated response module supports tiered handling strategy configuration. High-risk attacks are automatically blocked and isolated, medium-risk events are pushed to manual handling work orders, and low-risk behaviors are only recorded in audit logs without triggering automatic operations. It has a built-in handling effect write-back mechanism. After the device executes the blocking and isolation command, it sends the execution result back to the intelligent analysis engine layer for secondary verification of whether the risk has been eliminated and to update the asset risk label.

[0098] It should be noted that the streaming engine binds the obtained threat intelligence, asset, vulnerability, and operational behavior data to a single risk log in real time. The SPL engine links them by time, account, and IP to form a complete chain of evidence for attack tracing, which can be used for SOAR handling, visual profiling, and compliance auditing.

[0099] Furthermore, to improve the accuracy of anomaly identification and avoid data fragmentation caused by the independent operation of each algorithm, the intelligent analysis engine layer is scheduled through a deployed fusion-nested scheduling logic. The execution flow of the fusion-nested scheduling logic is as follows:

[0100] The streaming computing engine first calls the hierarchical clustering model to complete the coarse classification of log text, and then calls the corresponding sub-models according to the log data type. Periodic business indicators call the VAE / CVAE model, discrete hardware indicators call the IsolationForest model, and batch task indicators call the KDE model.

[0101] The SPL analysis engine reads the coarse classification results from the streaming computing engine, nests and loads the whitelist self-learning model for secondary verification, and sends the abnormal features obtained from the joint verification of multiple models back to the anomaly detection engine for risk weighting and scoring, forming a multi-layered nested linkage mechanism.

[0102] Specifically, the risk-weighted scoring calculation rules are as follows: four types of weight coefficients are pre-configured, namely log pattern anomaly weight W1, parameter deviation weight W2, whitelist baseline deviation weight W3, and intelligence / asset vulnerability risk weight W4; the comprehensive risk value of a single log entry is S = A1×W1 + A2×W2 + A3×W3 + A4×W4; where A1~A4 are the individual anomaly scores output by each sub-model, ranging from 0 to 10; the system presets thresholds: S≥7 is considered high risk, 3≤S<7 is considered medium risk, and S<3 is considered low risk, corresponding to triggering the SOAR graded handling strategy; the weight coefficients can be manually adjusted based on the business asset level, with high-value assets automatically amplifying the weight of vulnerabilities and operational behaviors.

[0103] Furthermore, in another embodiment, by receiving the hierarchical abnormal feature tags output by the nested scheduling, and nesting four types of risks—operation and maintenance, VPN, phishing emails, and unauthorized access—the system extracts the risk scores for various account behaviors. Based on asset and vulnerability weights, the risk scores are weighted and fused to generate a comprehensive account risk coefficient and construct a 360-degree user risk profile. The high-risk accounts and trusted accounts identified by the profile are sent back to a unified model repository to iteratively update the whitelist self-learning model baseline, completing the bidirectional nested closed-loop optimization of the detection model and the profile model.

[0104] The above solution has the following significant advantages:

[0105] 1. By adopting a five-layer hierarchical data flow architecture, the entire chain of log collection, processing, analysis, handling and display is connected, which solves the defects of traditional platform modules being fragmented and data not being able to communicate with each other, realizes the unified collection and standardized governance of all logs, and improves the integrity and availability of log data;

[0106] 2. It integrates multiple types of unsupervised algorithms, combined with offline log mode training and whitelist self-learning mechanism, which breaks away from the limitations of traditional solutions that use fixed thresholds, and greatly reduces the false alarm and false negative rates.

[0107] 3. Through multi-scenario security assessment, combined with asset vulnerability correlation and SOAR automated response modules, the entire process of security incident detection, tracing, and handling is automated, significantly reducing manual operating costs and shortening risk response time;

[0108] 4. It retains all complete data throughout the entire process and outputs compliance audit reports, reducing the labor costs and compliance risks for enterprise compliance audits.

[0109] Based on the same inventive concept, this invention also provides a security protection method based on centralized log analysis, applied to the security protection system based on centralized log analysis described in the first aspect, with reference to... Figure 2 The method includes the following steps:

[0110] S101 collects raw logs generated by various types of devices and business systems across the entire network through multiple protocol collection channels;

[0111] S102, perform data processing and verification on the original log, and output log data in a unified and standardized format;

[0112] S103 is based on Spark for offline word segmentation training of historical standardized logs, generating log pattern trees, parameter trees, behavior whitelist self-learning models and multi-class unsupervised anomaly detection models.

[0113] S104 processes standardized logs in real time through streaming computing and the SPL engine, combines offline trained models with internal and external network threat intelligence to perform multi-dimensional risk identification and output risk identification results.

[0114] S105, based on the risk identification results, execute multi-scenario security assessment, hierarchical alarm and attack tracing and automatic handling instructions, and simultaneously generate corresponding full operational data;

[0115] S106, retain the full amount of operational data to meet the regulatory storage period, and visualize and output the security situation dashboard, user profiles, alarm dashboards and compliance audit reports.

[0116] The anomaly detection model specifically includes:

[0117] The VAE / CVAE model is adapted to periodic metrics, the IsolationForest model is adapted to discrete hardware metrics, the KDE model is adapted to batch tasks, and the hierarchical clustering model is adapted to log text.

[0118] Furthermore, referring to Figure 3 The risk identification includes a multi-model fusion and nested scheduling sub-step:

[0119] Load the log pattern tree, parameter tree, sub-models of various unsupervised anomaly detection models, and whitelist self-learning models generated by Spark offline training;

[0120] The streaming computing engine first coarsely divides the real-time log text using a hierarchical clustering model, and then performs the first layer of anomaly detection based on nested matching of corresponding exclusive sub-models according to the log metric type.

[0121] The analysis engine nests and calls the whitelist self-learning model to perform secondary verification on the first-layer discrimination result and output secondary anomaly markers.

[0122] By fusing the two-layer model for discrimination labeling and weighted risk scoring, a high-confidence risk identification result is obtained after fusion.

[0123] It should be noted that for a more detailed description of the workflow of the method embodiments, please refer to the aforementioned system embodiments section, which will not be repeated here.

[0124] The specific beneficial effects of the entire plan include:

[0125] 1. By adopting a five-layer hierarchical data flow architecture, the entire chain of log collection, processing, analysis, handling and display is connected, which solves the defects of traditional platform modules being fragmented and data not being able to communicate with each other, realizes the unified collection and standardized governance of all logs, and improves the integrity and availability of log data;

[0126] 2. It integrates multiple types of unsupervised algorithms, combined with offline log mode training and whitelist self-learning mechanism, which breaks away from the limitations of traditional solutions that use fixed thresholds, and greatly reduces the false alarm and false negative rates.

[0127] 3. Through multi-scenario security assessment, combined with asset vulnerability correlation and SOAR automated response modules, the entire process of security incident detection, tracing, and handling is automated, significantly reducing manual operating costs and shortening risk response time;

[0128] 4. It retains all complete data throughout the entire process and outputs compliance audit reports, reducing the labor costs and compliance risks for enterprise compliance audits.

[0129] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can also be implemented in other ways. The embodiments described above are merely illustrative. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the figures. For example, two consecutive blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified functions or actions, or using a combination of dedicated hardware and computer instructions.

[0130] Furthermore, the functional modules in the various embodiments of this invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part. When using each module, information is collected and stored only with the full authorization of the relevant user or organization and in compliance with relevant laws and regulations, and the security and privacy of the data are protected. Unauthorized access is strictly prohibited. Data processing will be carried out within the scope stipulated by law and will not exceed the authorized purpose and scope. At the same time, the authorizing party has the right to access, correct, delete, restrict processing, refuse, etc., of its personal data, and strictly comply with applicable laws and regulations and conduct compliance reviews.

[0131] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. For those skilled in the art, this invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A security protection system based on centralized log analysis, characterized in that, The system includes: The multi-source log collection layer is used to collect raw logs generated by various types of devices and business systems across the entire network; The log data processing layer is used to receive the raw logs, perform data processing and verification, and output log data in a unified and standardized format. The intelligent analysis engine layer is used to perform real-time analysis on the log data through multiple built-in engines and multiple pre-trained processing models, and output risk identification results; wherein, each processing model is learned and trained based on historical logs in offline mode. The security application layer is used to execute multi-scenario security assessment, hierarchical alarm, attack tracing and automatic handling instructions based on the risk identification results, and simultaneously generate corresponding full operational data; The visualization layer is used to receive the full amount of operational data and render and output a security posture dashboard, user profiles, alarm dashboards, and compliance audit reports.

2. The system as described in claim 1, characterized in that, The intelligent analysis engine layer includes a streaming computing engine, an SPL analysis engine, an anomaly detection engine, and a threat intelligence engine. The anomaly detection engine is configured to perform anomaly identification on logs or metrics based on at least one of the following algorithms: hierarchical clustering, VAE, CVAE, KDE, and IsolationForest. It also performs offline word segmentation training on historical standardized logs based on Spark to generate log pattern trees, parameter trees, behavior whitelist self-learning models, and multi-class unsupervised anomaly detection models.

3. The system as described in claim 2, characterized in that, The anomaly detection model specifically includes: The VAE / CVAE model is adapted to periodic metrics, the IsolationForest model is adapted to discrete hardware metrics, the KDE model is adapted to batch tasks, and the hierarchical clustering model is adapted to log text.

4. The system as described in claim 3, characterized in that, The threat intelligence engine connects to both open-source OSINT intelligence and commercial threat intelligence, and performs real-time matching of IP addresses, domain names, URLs, and malicious file hash blacklists.

5. The system as described in claim 3, characterized in that, The whitelist self-learning model adopts a time-series sliding window continuous update mechanism, periodically collects normal log behavior data within the period to iteratively correct the baseline threshold, dynamically adapts to business access volume, and adapts to the periodic changes in operation and maintenance habits over time.

6. The system as described in claim 5, characterized in that, The security application layer includes an asset vulnerability association module and a SOAR automated response module. The SOAR automated response module supports the configuration of tiered handling strategies. High-risk attacks are automatically blocked and isolated, medium-risk events are pushed to manual handling work orders, and low-risk behaviors are only recorded in audit logs without triggering automatic operations. The built-in processing effect write-back mechanism means that after the device executes the blocking and isolation command, it sends the execution result back to the intelligent analysis engine layer for secondary verification of whether the risk has been eliminated and to update the asset risk label.

7. The system as described in claim 3, characterized in that, The intelligent analysis engine layer performs scheduling through deployed fusion-nested scheduling logic, and the execution flow of the fusion-nested scheduling logic is as follows: The streaming computing engine first calls the hierarchical clustering model to complete the coarse classification of log text, and then calls the corresponding sub-models nested according to the log data type. Periodic business indicators nested call the VAE / CVAE model, discrete hardware indicators nested call the IsolationForest model, and batch task indicators nested call the KDE model. The SPL analysis engine reads the coarse classification results from the streaming computing engine, nests and loads the whitelist self-learning model for secondary verification, and sends the abnormal features obtained from the joint verification of multiple models back to the anomaly detection engine for risk weighting and scoring, forming a multi-layered nested linkage mechanism.

8. A security protection method based on centralized log analysis, characterized in that, The method applied to the security protection system based on centralized log analysis as described in claim 1 includes the following steps: Collect raw logs generated by various types of devices and business systems across the entire network through multiple protocol collection channels; The original logs are processed and validated to output log data in a standardized format. Based on Spark, offline word segmentation training of historical standardized logs is used to generate log pattern trees, parameter trees, behavior whitelist self-learning models, and multi-class unsupervised anomaly detection models. By using streaming computing and the SPL engine to process standardized logs in real time, combined with offline trained models and internal and external network threat intelligence, multi-dimensional risk identification is performed, and risk identification results are output. Based on the risk identification results, multi-scenario security assessment, tiered alerts, and attack tracing and automatic handling instructions are issued, and corresponding full operational data is generated simultaneously. The system retains all operational data to meet regulatory storage requirements and provides visualized outputs such as security posture dashboards, user profiles, alarm dashboards, and compliance audit reports.

9. The method as described in claim 8, characterized in that, The anomaly detection model specifically includes: The VAE / CVAE model is adapted to periodic metrics, the IsolationForest model is adapted to discrete hardware metrics, the KDE model is adapted to batch tasks, and the hierarchical clustering model is adapted to log text.

10. The method as described in claim 8, characterized in that, The risk identification process includes a multi-model fusion and nested scheduling sub-step: Load the log pattern tree, parameter tree, sub-models of various unsupervised anomaly detection models, and whitelist self-learning models generated by Spark offline training; The streaming computing engine first coarsely divides the real-time log text using a hierarchical clustering model, and then performs the first layer of anomaly detection based on nested matching of corresponding exclusive sub-models according to the log metric type. The analysis engine nests and calls the whitelist self-learning model to perform secondary verification on the first-layer discrimination result and output secondary anomaly markers. By fusing the two-layer model for discrimination labeling and weighted risk scoring, a high-confidence risk identification result is obtained after fusion.

Images