Information security management and monitoring system based on big data

Abstract

The invention discloses an information security management and monitoring system based on big data, and relates to the technical field of computer information security, and the system comprises a data governance and standardization module, a multi-modal threat intelligent analysis engine, a dynamic alarm verification and association analysis module, an automatic response arrangement module and a situation awareness and risk prediction module. The invention aims to solve the problems of lagging security threat detection, low alarm effectiveness and weak situation awareness capability, and provides a multi-mode security threat detection method through data standardization, multi-mode fusion analysis, alarm verification association and automatic response arrangement. The deep detection of known and unknown threats is realized, the warning effectiveness is obviously improved, the global perception and risk prediction of the security situation are realized, and an intelligent security operation closed loop integrating perception, analysis, decision making and response is formed.

Description

Technical Field

[0001] This invention relates to the field of computer information security technology, and in particular to an information security management and monitoring system based on big data. Background Technology

[0002] In the field of information technology, information security has become a crucial cornerstone for ensuring organizational business continuity and the security of core data assets. With the widespread adoption of cloud computing, the Internet of Things, and mobile internet, cyberattacks are becoming increasingly complex, covert, and automated, posing a severe challenge to traditional security systems based on perimeter defense and static rules. Building intelligent security management platforms capable of responding to unknown threats and achieving proactive defense has become an inevitable trend in the industry.

[0003] Among these, information security monitoring and analysis based on big data technology is a core technological direction for current security operations. This technology aims to improve the detection, response, and tracing capabilities of security incidents by collecting massive amounts of multi-source network and security log data and using methods such as data mining, machine learning, and correlation analysis to identify potential attack patterns, abnormal behaviors, and internal threats from seemingly unrelated events.

[0004] In existing technologies, security information and incident management systems typically rely on predefined rule bases or static threat intelligence for matching alerts. Their analytical capabilities are limited by the breadth and depth of rule coverage, making it difficult to effectively address new and complex attacks such as zero-day attacks and advanced persistent threats. Furthermore, the log data collected by these systems is heterogeneous in format and scattered in source, lacking unified data governance and standardized processing procedures, resulting in inconsistent data quality and affecting the accuracy and efficiency of subsequent analysis. In addition, traditional monitoring systems often lack the ability to automatically verify, prioritize, and perform in-depth cross-event correlation analysis after alerts are generated, leading to alert fatigue for security operations teams. A large number of low-value alerts overwhelm genuine high-risk threats, resulting in delayed response decisions. In a dynamically changing network environment, existing systems struggle to achieve real-time, accurate perception of the security posture and risk prediction, failing to provide forward-looking data support for security management decisions. Summary of the Invention

[0005] The purpose of this invention is to provide an information security management and monitoring system based on big data, in order to solve the problems of lagging security threat detection, low alarm effectiveness, and weak situational awareness caused by existing technologies relying on static rules, lacking data governance, and having insufficient alarm processing capabilities.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: A big data-based information security management and monitoring system includes: The data governance and standardization module processes the raw security data in real time. This module incorporates a unified data model that defines standardized security data fields across all dimensions, including network traffic, endpoint logs, application logs, threat intelligence, and asset information. The processing begins with format recognition and field extraction of the raw data using a pre-built parser plugin library, which supports log formats from over 50 mainstream security devices and systems. Subsequently, the extracted fields are mapped and reconstructed according to the unified data model. For missing critical fields, an intelligent fill algorithm based on asset context and historical data patterns is used to complete them. Finally, the mapped and reconstructed data records are assigned globally unique event identifiers and millisecond-accurate timestamps, and data lineage information is injected to form a standardized security event stream output.

[0007] The multimodal threat intelligent analysis engine performs deep inspection of standardized security event flows. This engine operates in parallel with three sub-engines: a rule matching sub-engine, a behavioral baseline analysis sub-engine, and a graph association mining sub-engine. The rule matching sub-engine loads dynamically updatable threat intelligence rule sets and internal compliance policy rule sets for high-speed pattern matching. The behavioral baseline analysis sub-engine employs an unsupervised learning algorithm to build a dynamic behavioral baseline model for each monitored entity. This model uses a 7-day learning cycle to continuously calculate the mean and variance of the entity's behavioral characteristics across 12 dimensions, including access frequency, data throughput, and operation time distribution. When real-time behavioral characteristics deviate from the baseline by more than 3 standard deviations, an abnormal event is triggered. The graph association mining sub-engine abstracts entities and behaviors in the standardized event flow into nodes and edges, constructing a dynamic knowledge graph in real time. This sub-engine incorporates a graph neural network model based on random walk and community detection algorithms to identify hidden association paths and abnormal clustering patterns between nodes, thereby discovering potential lateral movement and coordinated attack clues. The output events of the above three sub-engines are uniformly submitted to the event fusion processor. Based on a preset confidence fusion algorithm, the processor performs deduplication and confidence weighting on the same threat indication from different sub-engines to generate a preliminary threat alarm event.

[0008] The dynamic alarm verification and correlation analysis module refines and deepens initial threat alarm events. This module first initiates the alarm verification process, which calls a verification probe group to automatically verify the attack indicators involved in the alarm event. The verification probe group includes port scanning probes, vulnerability existence verification probes, and file hash reputation query probes. The verification results are returned as Boolean values ​​and bound to the original alarm event. Next, the module performs cross-alarm correlation analysis, employing a correlation rule engine based on time window sliding and causal reasoning. This engine retrieves all verified alarm events within a preset 30-minute time window and, based on the attack chain model and preset causal logic rules, links alarm events with progressive stages or conditional dependencies into composite security events. Finally, the module performs risk quantification scoring on composite security events and isolated high-risk verified alarms. The scoring model comprehensively considers four factors: attack technique level, value of affected assets, credibility of verification results, and event freshness, outputting a risk score between 0 and 100.

[0009] The automated response orchestration module executes tiered response actions. This module predefines a multi-level response script library, ranging from notification and blocking to attribution. Upon receiving a high-risk event from the dynamic alarm verification and correlation analysis module, the response decision-maker automatically matches and instantiates the corresponding response script based on the event type and risk score. A response script consists of a series of atomic actions, which invoke the control interfaces of the firewall, intrusion prevention system, endpoint detection and response platform, and ticketing system through standardized application programming interfaces (APIs). During execution, the module records the execution status and result of each atomic action, forming a complete response handling chain log. For complex scripts requiring manual intervention, the module automatically generates a handling ticket containing event details, correlation analysis results, and suggested action steps, and pushes it to the security operations platform.

[0010] The Situation Awareness and Risk Prediction module visualizes the overall security situation and predicts risk trends. This module obtains a continuous stream of standardized data from the Data Governance and Standardization module and acquires quantified security events from the Dynamic Alarm Verification and Correlation Analysis module. Based on this data, the Situation Awareness component calculates and displays the overall network attack surface index, regional threat intensity index, and core asset risk index in real time. Simultaneously, the Risk Prediction component employs an ensemble learning algorithm, specifically a combination of gradient boosting decision trees and long short-term memory neural networks. This model uses a multi-dimensional situational indicator sequence from the past 72 hours and external threat intelligence activity as input features to predict the probability of the network suffering specific types of attacks and the changing trend of the overall risk level within the next 24 hours. The prediction results are overlaid on the Situation Awareness view as time-series curves, providing prediction confidence intervals.

[0011] The input of the data governance and standardization module is connected to an external, multi-source, heterogeneous security data source, and its output is connected to the input of the multimodal threat intelligent analysis engine. The output of the multimodal threat intelligent analysis engine is connected to the input of the dynamic alarm verification and correlation analysis module. The output of the dynamic alarm verification and correlation analysis module is connected to the inputs of the automated response orchestration module and the situational awareness and risk prediction module, respectively. The situational awareness and risk prediction module also receives standardized data streams from the data governance and standardization module.

[0012] As one embodiment of the present invention, the specific process of establishing a dynamic behavioral baseline model in the behavioral baseline analysis sub-engine is as follows: Using the network protocol address or host identifier of the monitored entity as an index, a dedicated feature time series queue is created in the behavioral feature repository. The feature extractor extracts the entity's numerical features in 12 preset dimensions from the standardized security event stream in real time and pushes the feature vectors into the queue in chronological order. The baseline learner uses the data from the most recent 7 days in the queue as samples and performs a model update once every morning. The update process first calculates the exponentially weighted moving average of each feature dimension as the baseline center value, and then calculates its exponentially weighted moving standard deviation as the fluctuation range threshold. During real-time detection, the feature comparator compares the entity's feature vector within the current time window with the latest baseline center value dimension by dimension, calculating the Mahalanobis distance. When the Mahalanobis distance exceeds a dynamic threshold set based on 3 times the standard deviation, it is determined to be a behavioral anomaly and an abnormal event record is generated.

[0013] As one embodiment of the present invention, the process of constructing a dynamic knowledge graph in real time in the graph association mining sub-engine is as follows: The graph builder continuously monitors the standardized security event stream, uniformly mapping the source address, destination address, user account, process hash, and accessed domain name in the event to graph nodes, and mapping the event type to directed edges between nodes. Both nodes and edges carry timestamp and event identifier attributes. The graph computing engine periodically performs time-decay-based edge weight updates on the entire graph, with earlier edge weights decaying over time. The community detection algorithm runs every 5 minutes, identifying tightly connected subsets of nodes in the graph and marking these subsets as potential attack groups or normal business clusters. The anomaly detection model focuses on whether newly emerging edges connect nodes that originally belonged to different communities, or whether a high-density star-shaped connection structure has formed in a short period of time. Such patterns are identified as potential lateral movement or command and control communication.

[0014] As one embodiment of the present invention, the specific calculation model for risk quantification scoring in the dynamic alarm verification and correlation analysis module is as follows: A base score of 50 points is set. The attack technology factor is assigned a value between 0 and 20 points based on the attack chain stage and technical complexity matched by the event; advanced persistent threat penetration stages are assigned a value of 20 points. The asset value factor is assigned a value between 0 and 15 points based on the criticality level of the affected asset in a predefined asset pool; the criticality level of the core database server corresponds to 15 points. The verification credibility factor is assigned a value of 15 points based on the return results of automated verification probes; events that are verified as true are assigned a value of 0 points, while those that cannot be verified or are false are assigned a value of 0 points. The event freshness factor is calculated using an exponential decay function based on the interval between the event's occurrence time and the current time; events within one hour receive a full score of 10 points. The values ​​of the above four factors are added together to obtain the final risk score, with a maximum total score of 100 points.

[0015] As one embodiment of the present invention, the matching and execution logic of response scripts in the automated response orchestration module is as follows: The response decision-maker has a built-in script rule matrix, where rows represent different event type codes and columns represent different risk score ranges. Each matrix cell is associated with one or more response script identifiers. When a high-risk event is input, the decision-maker first parses its event type and risk score, and then locates the corresponding cell in the script rule matrix. If a cell is associated with multiple scripts, they are sorted according to the scripts' preset execution priority. The decision-maker loads the script definition file with the highest priority. This file is in Extensible Markup Language (EXPLAIN) format, which sequentially defines the types of atomic actions, the target system's application programming interface (API) address, the calling parameters, and the success and failure branches of subsequent actions. The script execution engine parses and executes each atomic action sequentially, calling the API of the external system through a secure channel. The execution result of each action is fed back to the engine in real time. The engine determines the subsequent action branch based on the result and the script definition until the script execution is completed or enters a manual approval pending state.

[0016] In one embodiment of the present invention, the system runs on a layered data processing framework, which includes a stream processing layer, a batch processing layer, and a service layer. The stream processing layer, built on a distributed stream computing engine, is responsible for real-time reception and processing of data pipelines from the data governance and standardization module, the multimodal threat intelligent analysis engine, and the dynamic alarm verification and correlation analysis module. The batch processing layer, built on a distributed big data computing framework, is responsible for offline deep mining of historical data at fixed daily time periods to update behavioral baseline models, train risk prediction models, and generate periodic security operation reports. The service layer provides a unified application programming interface gateway and microservice cluster, encapsulating core business capabilities such as dynamic alarm verification, response orchestration, and situational awareness querying, for use by the front-end visualization platform and external systems. Data synchronization and command transmission between the three layers are achieved through a high-throughput message queue.

[0017] Compared with the prior art, the beneficial technical effects of the present invention are as follows: This invention constructs a data governance and standardization module to forcibly transform multi-source heterogeneous raw security data into a standard event stream that conforms to a unified data model. This process not only solves the integration problem caused by chaotic data formats, but also significantly improves the integrity, consistency, and traceability of data through intelligent field filling and data lineage injection. This lays a solid foundation for standardized high-quality data, ensuring the reliability and consistency of inputs for all subsequent advanced analysis modules, and fundamentally overcoming analysis biases and failures caused by data quality issues.

[0018] The multimodal threat intelligent analysis engine designed in this invention innovatively integrates three detection paradigms: rule matching, behavioral baseline analysis, and graph association mining. This design breaks through the limitations of single detection methods. Behavioral baseline analysis can discover unknown threats that deviate from normal patterns, while graph association mining can reveal hidden connections between discrete events. The three sub-engines operate in parallel and achieve full coverage detection of known rule attacks, abnormal internal behaviors, and complex association attacks through confidence fusion. This greatly improves the breadth and depth of threat discovery, especially the detection capability of advanced persistent threats and zero-day attacks.

[0019] This invention introduces a dynamic alarm verification and correlation analysis module, which adds automated verification and deep correlation steps after alarm generation. Automated verification actively confirms the authenticity of attack indicators through probes, effectively filtering false alarms. Cross-alarm correlation analysis connects fragmented events into a complete attack narrative based on an attack chain model, and prioritizes events by combining a risk quantification scoring model. This mechanism frees security operations personnel from massive amounts of low-value alarms, enabling them to focus on verified and correlated high-risk composite events, thereby significantly improving alarm response efficiency and decision-making accuracy.

[0020] This invention achieves closed-loop security management from detection and analysis to response and prediction through an automated response orchestration module and a situational awareness and risk prediction module. The response orchestration module automatically transforms analysis conclusions into executable action sequences, accelerating threat containment. The situational awareness and risk prediction module utilizes historical and real-time data to provide a globally visualized security status and future risk trends, enabling security management to shift from passive response to proactive early warning and planning. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the overall technical solution architecture of the information security management and monitoring system based on big data proposed in this invention; Figure 2 This is a schematic diagram of the core principle framework of the multimodal threat intelligent analysis engine in this invention. Detailed Implementation

[0022] The features and exemplary embodiments of various aspects of the present invention will now be described in detail. To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely intended to explain the present invention and not to limit the present invention. For those skilled in the art, the present invention can be practiced without some of these specific details. The following description of the embodiments is merely to provide a better understanding of the present invention by illustrating examples of the invention. Example

[0023] This invention provides an information security management and monitoring system based on big data, the overall technical architecture of which is shown in the attached figure. Figure 1As shown, the system consists of five core functional units: a data governance and standardization module, a multimodal threat intelligent analysis engine, a dynamic alarm verification and correlation analysis module, an automated response orchestration module, and a situational awareness and risk prediction module. These units are interconnected through high-throughput, low-latency data pipelines, forming a complete closed loop from raw data collection to security decision execution. The data governance and standardization module serves as the system's data entry point, connecting to external multi-source heterogeneous security data sources, including but not limited to network firewall logs, intrusion detection system alarms, endpoint agent logs, application server access records, cloud platform operation audit streams, third-party threat intelligence feeds, and asset configuration management databases. Its output provides the multimodal threat intelligent analysis engine with a standardized security event stream that is structurally unified and semantically consistent. The multimodal threat intelligent analysis engine performs deep threat identification on the standardized event stream and transmits its preliminary alarm results to the dynamic alarm verification and correlation analysis module. After completing alarm purification and event correlation, this module pushes high-confidence composite security events to the automated response orchestration module to trigger action, and to the situational awareness and risk prediction module to support global situational modeling. Meanwhile, the situational awareness and risk prediction module also directly receives raw standardized data streams from the data governance and standardization module to build baseline indicators required for long-term trend analysis.

[0024] The data governance and standardization module is the cornerstone of the entire system's data quality. This module integrates a complete unified data model that clearly defines comprehensive security data field standards, covering network traffic metadata, terminal process behavior logs, user authentication records, application interface call traces, external threat intelligence entries, and asset attribute descriptions. All fields adopt preset naming conventions, data types, value ranges, and unit systems to ensure semantic consistency across data sources. When raw security data enters this module, the format recognizer first matches the corresponding parser plugin based on data header features or transport protocol identifiers. The parser plugin library incorporates dedicated parsing logic for over 50 mainstream security devices and systems, covering common formats such as Cisco firewalls, Palo Alto next-generation firewalls, Microsoft Windows event logs, Linux system logs, Apache / Nginx web server logs, VMware vCenter operation logs, and AWS CloudTrail audit streams. After a successful match, the parser plugin performs field extraction operations, extracting key information from the raw text or binary data into intermediate structured objects.

[0025] Subsequently, the field mapper reconstructs the intermediate objects based on the unified data model. For key fields defined in the model but missing in the original data, including the business department to which the asset belongs, the host operating system version, and the service exposure level, the system initiates an intelligent fill subroutine. This subroutine first queries the asset context knowledge base. If the target asset is already registered in the database, its static attributes are directly completed; if not registered, it is filled based on the behavioral patterns of similar historical entities. For example, if a newly appearing IP address frequently accesses database port 3306, and its network segment is known to be an application server cluster, its role can be inferred to be a "database client," and it can be assigned the corresponding asset category label. After mapping and filling are completed, the event encapsulator generates a globally unique event identifier for each record. This identifier uses a 128-bit conflict-free string based on a combination of timestamp, source system identifier, and sequence number; it also injects a timestamp accurate to milliseconds and attaches data lineage information, including the original data source name, parser plugin version, processing node ID, and processing time. The resulting standardized security event stream is continuously output in JSON format through a message queue for downstream modules to consume.

[0026] The multimodal threat intelligent analysis engine receives the standardized security event stream described above and executes three complementary detection logics in parallel. Its internal structure is shown in the attached figure. Figure 2 As shown, the engine comprises three parallel processing channels: a rule matching sub-engine, a behavior baseline analysis sub-engine, and a graph association mining sub-engine. These three share the same input data stream but operate independently, with the results ultimately integrated by an event fusion processor. The rule matching sub-engine loads two dynamically updated rule sets: one is an externally subscribed threat intelligence rule set containing known malicious IP addresses, domain names, file hashes, and attack payload characteristics; the other is an internally defined compliance policy rule set covering scenarios such as operations violating the principle of least privilege, sensitive data leakage, and high-risk command execution outside of working hours. Rule matching employs a high-speed pattern matching algorithm based on finite state automata, achieving a throughput of millions of events per second on a single node. Successfully matched events are marked as "rule hit alerts," carrying the unique identifier of the triggered rule, the rule type, and the confidence weight.

[0027] The behavior baseline analysis sub-engine focuses on detecting abnormal activities that deviate from normal behavior patterns. This sub-engine establishes a dedicated dynamic behavior baseline model for each monitored entity.

[0028] The model construction process is as follows: First, a feature time series queue with a capacity of 7 days is created in a distributed key-value store, using the entity identifier as the key. The feature extractor parses the entity's numerical features across 12 preset dimensions in real time from the standardized event stream, including average number of sessions per hour, total daily data outflow, operation time entropy, access target diversity index, command execution frequency, file read / write rate, login failure count, privileged operation percentage, API call error rate, lateral movement attempt count, resource usage fluctuation coefficient, and external connection change rate. Each feature vector is pushed into the corresponding queue in the order of event timestamps. The baseline learner starts a model update task once a day at 02:00 AM, using the most recent 7 days of valid data in the queue as samples, calculating the exponentially weighted moving average for each feature dimension as the baseline center value, with a weight decay factor set to 0.9; simultaneously, the exponentially weighted moving standard deviation is calculated as the fluctuation range threshold. In the real-time detection phase, the feature comparator aggregates the feature vectors of entities within the current window using a 5-minute sliding window and calculates the Mahalanobis distance between them and the latest baseline center value. When the Mahalanobis distance exceeds a dynamic threshold set based on 3 times the standard deviation, it is judged as an abnormal behavior and a "behavior deviation alarm" event is generated.

[0029] The graph association mining sub-engine mines potential threats from a relational perspective. This sub-engine continuously monitors standardized security event streams and abstracts the entities involved in the events into graph nodes. Entities include source IPs, destination IPs, users, processes, domains, and files. Simultaneously, the events themselves are abstracted into directed relation edges to represent behavior types such as access, execution, writing, and connection. Each node and each relation edge carries a timestamp, an event identifier, and the corresponding original event type attribute. The graph builder injects these nodes and relationships into an in-memory graph database in real time, thus forming a dynamic knowledge graph that continuously evolves over time.

[0030] The graph computation engine performs a full graph analysis task at fixed time intervals, preferably five minutes. In each analysis task, a time decay mechanism is first applied to all relation edges in the graph to update their weights. The weight update uses an exponential decay model, calculated using the following formula:

[0031] in, This represents the edge weight at the current moment. Indicates the initial weights. This represents the time decay coefficient, with a preferred value of 0.001. Indicates the current system time. This indicates the creation time of the corresponding edge.

[0032] After updating edge weights, the graph computing engine runs the Louvain community discovery algorithm based on modularity optimization to perform cluster analysis on nodes in the dynamic knowledge graph, thereby identifying subsets of nodes with close connections. Based on the type distribution and behavioral characteristics of nodes within these subsets, the system automatically performs semantic labeling, classifying them into normal business clusters or suspicious attack groups. Subsequently, the anomaly detection model scans and analyzes newly generated relationship edges. When it detects an edge connecting two nodes that have never interacted before and belong to different communities, or when a star topology with more than twenty incoming edges is formed around the same central node within a consecutive ten-minute time window, it determines that there is potential lateral movement or command and control communication behavior and generates a graph association anomaly alarm.

[0033] Alarm events generated by the aforementioned sub-engines are uniformly submitted to the event fusion processor. The event fusion processor maintains a sliding time window with a default length of thirty minutes. Within this window, the processor clusters alarm events with the same attack indicators, including but not limited to identical malicious IP addresses or identical file hash values. For multiple alarm events within the same cluster, the event fusion processor calculates a comprehensive confidence score based on a preset confidence fusion algorithm. The confidence fusion algorithm uses a weighted average model, where rule-matching alarms have a weight of 0.4, behavior deviation alarms have a weight of 0.3, and graph association anomaly alarms have a weight of 0.3. When the calculated comprehensive confidence score exceeds a threshold of 0.6, the system generates a preliminary threat alarm event. This preliminary threat alarm event includes the original alarm list, the fused confidence score, the involved entity information, and the corresponding time range.

[0034] Upon receiving an initial threat alert event, the dynamic alert verification and correlation analysis module first initiates the alert verification process. The verification probe scheduler selects the appropriate verification probe group based on the type of alert event. When the alert event involves a suspicious IP address, a port scanning probe is scheduled to perform a SYN scan on the commonly used service ports of that IP address to confirm whether the target is in an active listening state. When the alert event involves vulnerability exploitation characteristics, a vulnerability existence verification probe is scheduled to send a specific request to the target system using a non-destructive probe script to verify whether the target system has the corresponding vulnerability. When the alert event contains a suspicious file hash value, a file hash reputation query probe is scheduled to query the local cache or cloud reputation service to obtain the known malicious rating of the file. All verification probes run in an isolated sandbox environment to ensure that the verification process does not affect the production system. The verification result is returned in Boolean form and bound to the corresponding alert event, thus forming a verified alert event.

[0035] Subsequently, the module performs cross-alarm correlation analysis. The correlation rule engine retrieves all verified alarm events within a 30-minute time window and matches them against preset attack chain models and causal logic rules. For example, if a "credential guessing successful" alarm occurs first, followed by a "remote desktop login" alarm within 5 minutes, it is determined to be a progression of the "initial access → execution" stage; if a "suspicious file download" alarm is immediately followed by a "process injection" alarm, it is determined to be a "delivery → persistence" chain. The engine links alarm events with such stage dependencies into a composite security event and marks its attack stage sequence; isolated high-risk alarms that cannot be correlated are retained as independent events.

[0036] Finally, the risk quantification scorer provides a unified score for all complex security events and isolated high-risk alerts. The scoring model employs a weighted cumulative mechanism, with a maximum total score of 100 points, of which the base score is fixed at 50 points. The remaining scores are composed of attack technology factors, asset value factors, verification credibility factors, and event novelty factors.

[0037] The attack technique factor is assigned a value based on the stage of the security incident in the attack chain and the corresponding technical complexity. Specifically, the reconnaissance stage corresponds to 5 points, the initial access stage to 10 points, the execution stage to 8 points, the persistence stage to 12 points, the privilege escalation stage to 15 points, the defense and evasion stage to 10 points, the credential access stage to 12 points, the discovery stage to 8 points, the lateral movement stage to 15 points, the collection stage to 10 points, the command and control stage to 15 points, and the data penetration stage to 20 points. When an incident involves attack techniques specific to zero-day vulnerabilities or advanced persistent threats, an additional 5 points are added to the above, and the total score for this attack technique factor cannot exceed 20 points.

[0038] The asset value factor is assigned based on the criticality level of the affected asset in the predefined asset pool. Ordinary terminal assets are assigned 3 points, development servers are assigned 6 points, test databases are assigned 9 points, production application servers are assigned 12 points, and core database servers are assigned 15 points.

[0039] The verification credibility factor is assigned a value based on the alarm verification result. When the alarm is verified to be true, it is assigned 15 points; when the verification result is partially true, such as confirming that the port is open but not confirming the existence of the vulnerability, it is assigned 8 points; when the verification cannot be completed or the verification result is negative, it is assigned 0 points.

[0040] The event freshness factor is calculated based on the exponential decay model, and its formula is as follows:

[0041] in, This indicates the event's freshness score. This represents the time difference between the event's occurrence time and the current system time, expressed in seconds. Using this model, events occurring within one hour can achieve a near-perfect event freshness score, typically exceeding 9.5.

[0042] The scores of each factor are added to the base score to obtain the final risk score for the corresponding security event.

[0043] Upon receiving a high-risk event with a risk score higher than 70, the automated response orchestration module initiates the response decision-making process. Internally, the response decision-maker maintains a script rule matrix. The row indices of this matrix correspond to event type codes, which are 128 predefined types. The column indices correspond to risk score ranges, which are sequentially divided into 0–30, 31–50, 51–70, 71–85, and 86–100. Each cell in the matrix is ​​associated with one or more response script identifiers, and the execution priority is indicated for each script.

[0044] After parsing the event type code and risk score of the input event, the response decision-maker locates the corresponding matrix cell and loads the highest-priority response script definition file according to priority order. The response script definition file is described using Extensible Markup Language (XML) and is used to define a set of atomic response action sequences. Taking a lateral movement event with a risk score of 88 as an example, the corresponding response script could include the following actions in sequence: calling the firewall's application programming interface (API) to add an access denial rule between the source IP address and the target network segment; calling the endpoint detection and response platform's API to perform memory forensics on the source host and freeze suspicious processes; and calling the ticket system's API to create a high-priority handling ticket, which includes detailed event information, a list of associated alarms, a list of affected assets, and suggested handling steps.

[0045] The script execution engine sequentially calls the application programming interfaces (APIs) of various external systems through a two-way transport layer secure encrypted channel, and waits for a synchronous response after each atomic action is completed. When an action is executed successfully, it continues to the next action; when an action fails, it jumps to a preset rollback process or a manual approval branch. The execution status and results of all response actions are recorded in the response handling chain log for post-event audit analysis and continuous optimization of the response script.

[0046] The Situation Awareness and Risk Prediction module receives input data from two data sources. First, it receives a continuous standardized data stream from the Data Governance and Standardization module to calculate basic situation indicators. Second, it receives risk quantification events output from the Dynamic Alarm Verification and Correlation Analysis module to build a threat heatmap model. The Situation Awareness component performs real-time aggregation and analysis of the exposure surface data of all assets within the network. This data includes the number of open ports, the number of running services, and the number of known vulnerabilities, and calculates the overall network attack surface index accordingly. Simultaneously, the system aggregates and analyzes alarm density by geographical region or business unit, generating corresponding regional threat heatmap indices, and continuously tracks changes in the risk scores of core assets to form a core asset risk index. These indices are displayed in a visualization interface in the form of heatmaps, dashboards, and trend curves.

[0047] The risk prediction component employs an ensemble learning model to predict future risk trends. This model combines a gradient boosting decision tree model with a long short-term memory neural network model. Input features include 12-dimensional situational indicators sampled hourly over the past 72 hours, and a concurrent external threat intelligence activity index calculated based on the frequency of attack payloads captured by global honeypots. The model training process uses a sliding window mechanism and is incrementally updated every 24 hours. The model's prediction objective is the probability of a specific attack type occurring within the next 24 hours, along with the corresponding overall risk level, categorized as low, medium, high, and extremely high. The prediction results are displayed as time-series curves overlaid on the situational awareness view, accompanied by a 95% confidence interval, to assist security managers in proactively allocating resources and strengthening defense strategies.

[0048] The entire system runs on a three-layer data processing framework. The stream processing layer is built on the Apache Flink distributed stream computing engine, deploying real-time pipelines for data governance, threat analysis, and alarm verification, ensuring end-to-end processing latency of less than 500 milliseconds. The batch processing layer is built on Apache Spark, executing offline tasks daily at midnight, including retraining the behavioral baseline model, fine-tuning the risk prediction model, updating asset profiles, and generating weekly reports. The service layer consists of a Spring Cloud microservice cluster, providing a unified application programming interface gateway, encapsulating capabilities such as alarm querying, script triggering, and situational data retrieval, supporting integration with the front-end platform and third-party systems. The three layers are decoupled through a Kafka message queue, ensuring system stability and scalability under high concurrency.

[0049] Example 2 Building upon the aforementioned embodiments, this embodiment further optimizes the community detection algorithm in the graph association mining sub-engine to improve the accuracy of identifying covert attack groups. The traditional Louvain community detection algorithm primarily divides communities based on the weights of relation edges. In high-interaction scenarios, it can easily misjudge high-frequency connection clusters formed by normal business activities as attack groups, such as frequent calls between different service instances in a microservice architecture.

[0050] To address the aforementioned issues, this embodiment introduces a multi-dimensional node attribute weighting mechanism during community discovery. When constructing the dynamic knowledge graph, in addition to assigning weights to relation edges, a set of node attribute vectors is configured for each node. These node attribute vectors include node type information, node role label, node behavior entropy value, and asset criticality level. Specifically, the node type characterizes the entity category corresponding to the node, the node role label describes the node's functional role in the system, the node behavior entropy value measures the randomness of node interaction behavior, and the asset criticality level reflects the importance of the asset corresponding to the node.

[0051] When executing the community detection algorithm, the modularity calculation process considers not only the connection strength between nodes but also introduces node attribute similarity as a modifier to adjust the effective connection strength between nodes. Specifically, the effective connection strength between node i and node j... Calculate using the following formula:

[0052] in, This represents the original edge weight between node i and node j. Let α represent the cosine distance between the attribute vectors of node i and node j, and let α represent the attribute similarity adjustment coefficient, which is preferably 0.3.

[0053] Through the aforementioned correction mechanism, when two nodes differ significantly in attribute dimensions, even if there is a strong connection between them, their contribution to community formation will still be significantly weakened. Conversely, nodes with highly similar attributes and close connections are more likely to be assigned to the same community. This effectively distinguishes between high-interaction clusters formed in normal business scenarios and potential attack groups with mixed node attributes, thereby improving the semantic accuracy of community partitioning results.

[0054] Experimental results show that after introducing a multi-dimensional node attribute weighting mechanism, the system's detection accuracy for lateral movement behavior is improved by about 18%, while the false alarm rate is reduced by about 22%, verifying the significant improvement effect of this optimization strategy on the identification of covert attack groups in complex network environments.

[0055] Furthermore, this embodiment enhances the script execution mechanism of the automated response orchestration module. The original scripts were linear execution sequences, which struggled to handle conditional branches in complex attack scenarios. This embodiment introduces a state machine-driven script engine. Each script is modeled as a finite state machine, containing an initial state, intermediate states, a termination state, and state transition conditions. Transition conditions include not only the execution results (success / failure) of atomic actions but also references external real-time data, such as "if the target host's current CPU utilization is below 20%, perform memory forensics; otherwise, only record the process list." The state machine definition uses YAML format and supports nested sub-state machines to handle multi-stage responses. The execution engine dynamically parses the state transition diagram at runtime, determining the next action in real time based on the current system state and external feedback. This mechanism significantly improves the adaptability of response scripts to dynamic environments, making it particularly suitable for elastically scalable asset scenarios in cloud-native environments.

[0056] Example 3 This embodiment focuses on optimizing the prediction model for the situational awareness and risk prediction module. The original embodiment used a simple concatenation of a gradient boosting decision tree and a long short-term memory neural network, failing to fully exploit their complementarity. This embodiment designs an attention fusion mechanism. First, the gradient boosting decision tree processes structured static features, including the number of assets, the total number of vulnerabilities, and the policy compliance rate; it outputs a high-order feature representation. The long short-term memory neural network processes temporal dynamic features, including alarm rate, traffic mutations, and login anomaly frequency, outputting a temporal context vector. Subsequently, an attention layer is introduced to calculate the importance weights of dynamic features to each dimension of static features. The final prediction result is output by a weighted fusion joint feature vector through a fully connected layer. In historical data backtesting, this model achieved a 24-hour prediction AUC of 0.92 for ransomware attacks, an improvement of 0.07 compared to the original model.

[0057] Meanwhile, this embodiment enhances the intelligent fill capability of the data governance and standardization modules. The original fill logic relied on an asset database and simple pattern matching, which had limited effectiveness for unknown assets. This embodiment introduces a graph neural network to assist in fill. Known assets and their attributes are constructed into an attribute graph, where nodes represent assets and edges represent network proximity or business affiliation relationships. For unknown assets, their observable behavioral characteristics are used as input, and a graph convolutional network performs message passing on the attribute graph, aggregating the attribute information of neighboring nodes to ultimately predict their missing fields. For example, if an unknown IP frequently communicates with multiple known database servers, its "business role" field is predicted as "database client." This method improves the field fill accuracy to 93%, providing a more complete context for downstream analysis.

[0058] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape, and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A big data based information security management and monitoring system, characterized in that, include: The data governance and standardization module receives raw security data from multiple heterogeneous security data sources, processes the raw security data based on a unified data model, and outputs a standardized security event stream. The multimodal threat intelligent analysis engine performs threat detection on the standardized security event stream to generate preliminary threat alert events. The multimodal threat intelligent analysis engine includes a rule matching sub-engine, a behavior baseline analysis sub-engine, and a graph association mining sub-engine in parallel. The dynamic alarm verification and correlation analysis module automatically verifies, correlates, and quantifies the risks of the preliminary threat alarm events, and outputs the security events after risk quantification. An automated response orchestration module performs tiered response operations based on the quantified risk security events. The automated response orchestration module matches and executes predefined response scripts through a response decision-maker. The situational awareness and risk prediction module has its input terminals connected to the data governance and standardization module and the dynamic alarm verification and correlation analysis module, respectively, to realize the visualization of the global security situation and the prediction of risk trends.

2. The big data based information security management and monitoring system as claimed in claim 1, wherein, The process of establishing a dynamic behavioral baseline model in the behavioral baseline analysis sub-engine is as follows: using the network protocol address or host identifier of the monitored entity as an index, a dedicated feature time series queue is created in the behavioral feature warehouse; the feature extractor extracts the numerical features of the entity in 12 preset dimensions from the standardized security event stream in real time, and pushes the feature vectors into the queue in chronological order; the baseline learner uses the data of the most recent 7 days in the queue as samples and performs a model update once every morning. The update process first calculates the exponentially weighted moving average of each feature dimension as the baseline center value, and then calculates its exponentially weighted moving standard deviation as the fluctuation range threshold; during real-time detection, the feature comparator compares the feature vector of the entity in the current time window with the latest baseline center value dimension by dimension and calculates the Mahalanobis distance. When the Mahalanobis distance exceeds the dynamic threshold set based on 3 times the standard deviation, it is determined to be a behavioral anomaly and an abnormal event record is generated.

3. The big data based information security management and monitoring system as claimed in claim 1, wherein, The process of constructing a dynamic knowledge graph in real time in the graph association mining sub-engine is as follows: The graph builder continuously listens to the standardized security event stream, and uniformly maps the source address, destination address, user account, process hash, and accessed domain name in the event to graph nodes, and maps the event type to directed edges between nodes. Both nodes and edges carry timestamp and event identifier attributes; the graph computing engine periodically performs time decay-based edge weight updates on the entire graph, and the weights of earlier edges decay over time; The community detection algorithm runs every 5 minutes to identify tightly connected subsets of nodes in the graph and mark these subsets as potential attack groups or normal business clusters. The anomaly detection model focuses on whether newly emerging edges connect nodes that originally belonged to different communities, or whether a high-density star topology has been formed in a short period of time. Such patterns are identified as potential lateral movement or command and control communication.

4. The information security management and monitoring system based on big data according to claim 1, characterized in that, The specific calculation model for risk quantification scoring in the dynamic alarm verification and correlation analysis module is as follows: A base score of 50 points is set; the attack technology factor is assigned a value between 0 and 20 points based on the attack chain stage and technical complexity matched by the event, with advanced persistent threat penetration stages receiving a value of 20 points; the asset value factor is assigned a value between 0 and 15 points based on the criticality level of the affected asset in a predefined asset pool, with the criticality level of the core database server corresponding to 15 points; the verification credibility factor is assigned a value of 15 points based on the return results of automated verification probes, with unverifiable or false events receiving 0 points; the event freshness factor is calculated using an exponential decay function based on the interval between the event's occurrence time and the current time, with events within one hour receiving a full score of 10 points; the values ​​of the above four factors are added together to obtain the final risk score, with a maximum total score of 100 points.

5. The information security management and monitoring system based on big data according to claim 1, characterized in that, The matching and execution logic of response scripts in the automated response orchestration module is as follows: The response decision maker has a built-in script rule matrix. The rows of the matrix represent different event type codes, the columns represent different risk score ranges, and each matrix cell is associated with one or more response script identifiers. When a high-risk event is input, the response decision-maker first parses its event type and risk score, and then locates the corresponding cell in the script rule matrix. If a cell is associated with multiple scripts, they are sorted according to the scripts' preset execution priority. The response decision-maker loads the script definition file with the highest priority. The script definition file is in Extensible Markup Language (XML) format, which sequentially defines the type of atomic action, the target system's application programming interface (API) address, the calling parameters, and the success and failure branches of subsequent actions. The script execution engine parses and executes each atomic action in sequence, calls the API of the external system through a secure channel, and determines the subsequent action branch based on the execution result and the script definition, until the script is completed or enters a manual approval pending state.

6. The information security management and monitoring system based on big data according to claim 1, characterized in that: The big data-based information security management and monitoring system operates on a layered data processing framework, which includes a stream processing layer, a batch processing layer, and a service layer. The stream processing layer, built on a distributed stream computing engine, is responsible for receiving and processing data pipelines from the data governance and standardization module, the multimodal threat intelligent analysis engine, and the dynamic alarm verification and correlation analysis module in real time. The batch processing layer, built on a distributed big data computing framework, is responsible for offline deep mining of historical data at fixed daily time periods to update the dynamic behavioral baseline model of the behavioral baseline analysis sub-engine, train the combined model of the risk prediction component, and generate periodic security operation reports. The service layer provides a unified application programming interface gateway and microservice cluster, encapsulating core business capabilities such as dynamic alarm verification, response orchestration, and situational query for use by the front-end visualization platform and external systems. The stream processing layer, batch processing layer, and service layer synchronize data and transmit instructions through a high-throughput message queue.

7. The information security management and monitoring system based on big data according to claim 3, characterized in that: The community discovery algorithm introduces node attribute similarity as an adjustment factor when calculating modularity; the effective connection strength between two nodes is corrected to the original connection strength multiplied by 1 minus the product of the adjustment coefficient and the cosine distance of the node attribute vector; the node attribute vector includes node type, role label, behavior entropy value and asset criticality level; through this correction, even if nodes with large attribute differences are connected, their contribution to community formation is weakened.

8. The information security management and monitoring system based on big data according to claim 5, characterized in that: The script definition file models the response script as a finite state machine, including an initial state, intermediate states, a termination state, and state transition conditions. The state transition conditions include not only the execution results of atomic actions but also reference external real-time data. The script execution engine dynamically parses the state transition diagram during runtime and determines the next action in real time based on the current system state and external feedback.

9. The information security management and monitoring system based on big data according to claim 6, characterized in that: The risk prediction component employs an attention fusion mechanism, introducing an attention layer. The risk prediction component uses an ensemble learning algorithm, specifically a combination of a gradient boosting decision tree and a long short-term memory neural network. The gradient boosting decision tree processes structured static features and outputs high-order feature representations, while the long short-term memory neural network processes temporal dynamic features and outputs a temporal context vector. The attention layer calculates the importance weights of the temporal dynamic features to each dimension of the structured static features. The final prediction result is output from the weighted fusion joint feature vector through a fully connected layer.

10. The information security management and monitoring system based on big data according to claim 1, characterized in that: The intelligent filling algorithm of the data governance and standardization module introduces graph neural network assistance; it constructs an attribute graph by taking known assets and their attributes, with nodes representing assets and edges representing network proximity or business affiliation; for unknown assets, it takes their observable behavioral characteristics as input, performs message passing on the attribute graph through graph convolutional network, aggregates the attribute information of neighboring nodes, and predicts their missing fields.

Images