Multi-dimensional network security situation awareness system and method for big data

Abstract

This invention discloses a multi-dimensional network security situation awareness system and method for big data, specifically relating to the field of network security technology for big data. It includes a multi-source data acquisition module for collecting raw datasets; a data preprocessing and fusion module for preprocessing and fusing the raw data to form a structured dataset; a feature construction and correlation analysis module for extracting relevant features from the structured dataset and outputting multi-dimensional feature vectors; a dynamic pattern recognition and state assessment module for identifying data patterns based on the multi-dimensional feature vectors using a collaborative fusion mechanism of adaptive clustering and pattern recognition algorithms, calculating a comprehensive state index, and outputting data pattern assessment results; and an execution feedback module for generating data application instructions and business instructions. This invention achieves deep integration of multi-source data and accurate situation assessment, effectively improving the comprehensiveness, accuracy, and real-time performance of network security situation awareness in a big data environment.

Description

Technical Field

[0001] This invention relates to the field of network security technology for big data, and more specifically, to a multi-dimensional network security situation awareness system and method for big data. Background Technology

[0002] With the rapid development of big data technology, the network environment is becoming increasingly complex, and network security threats are showing characteristics of diversification, concealment, and scale. Network security situation awareness has become a key technology for ensuring network security. Existing network security situation awareness technologies often focus on analyzing single types of data, such as collecting only network traffic data or device log data. They fail to integrate multi-source information such as business system logs, device operating status, and time-series data, resulting in a one-sided perception perspective and an inability to comprehensively reflect the network security situation. In the data processing stage, traditional technologies often only perform simple missing value deletion and duplicate value removal, lacking standardized data format unification and precise noise removal methods. Furthermore, they fail to effectively align the temporal differences of multi-source data, resulting in low-quality foundational data for subsequent feature extraction and analysis. Feature extraction dimensions are limited, often confined to statistical features, ignoring temporal features and inter-entity correlation features, making it impossible to construct a multi-dimensional feature system to characterize complex security patterns. In terms of pattern recognition, they either rely solely on pre-set known pattern libraries, making it difficult to identify unknown security threats, or they use only a single clustering algorithm, whose clustering effect is poorly affected by fixed parameters, leading to insufficient pattern recognition accuracy. In the execution feedback stage, they often only implement simple alarm functions, lacking a risk-level-based hierarchical feedback mechanism, and are unable to proactively adjust process parameters and schedule computing resources for high-risk patterns, resulting in delayed response and weak intervention capabilities. These shortcomings make it difficult for existing technologies to achieve comprehensive and accurate perception of network security situation and rapid and effective risk response in a big data environment, thus failing to meet the security protection needs of complex network environments. Summary of the Invention

[0003] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a multi-dimensional network security situation awareness system and method for big data, which solves the problem that the prior art, as mentioned in the background, cannot comprehensively and accurately perceive network security situation and respond quickly to risks through the following solutions.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a multi-dimensional network security situation awareness system for big data, characterized in that it includes:

[0005] Multi-source data acquisition module: used to acquire raw datasets, including business system log data, device operating status data and time series data, and to perform preliminary processing and time stamping on the raw data;

[0006] Data preprocessing and fusion module: connected to the multi-source data acquisition module, used to perform data cleaning on the original dataset, remove missing and duplicate values, unify data structure and field definitions through format standardization, and perform noise removal; further, time-series data alignment is performed through time window slicing to form a structured dataset;

[0007] Feature construction and association analysis module: connected to the data preprocessing and fusion module, used to extract statistical features, time series features and association features from the structured dataset, construct a multi-entity association graph based on the graph model, obtain and realize the interaction relationship model between different data entities, and output multi-dimensional feature vectors;

[0008] Dynamic pattern recognition and state assessment module: connected to the feature construction and association analysis module, used to identify patterns in the data based on the multi-dimensional feature vector, using a collaborative fusion mechanism of adaptive clustering method and pattern recognition algorithm, calculate comprehensive state index, and output data pattern assessment results;

[0009] Execution feedback module: connected to the dynamic pattern recognition and state assessment module, used to generate data application instructions and business instructions based on the pattern assessment results, the instructions including data alarms, process parameter adjustments and computing resource scheduling.

[0010] Preferred methods for multi-dimensional network security situation awareness based on big data include:

[0011] S1: Collect the raw dataset, which includes business system log data, device operating status data, and time series data, and perform preliminary processing and time stamping on the raw data;

[0012] S2: Perform data cleaning on the original dataset, remove missing and duplicate values, unify the data structure and field definitions through format standardization, and perform noise removal; further, perform time-series data alignment through time window slicing to form a structured dataset;

[0013] S3: Extract statistical features, time-series features, and correlation features from the structured dataset, construct a multi-entity relationship graph based on the graph model, obtain the interaction relationship model between different data entities, and output multi-dimensional feature vectors;

[0014] S4: Based on the multi-dimensional feature vector, an adaptive clustering method and a pattern recognition algorithm are used to identify patterns in the data, calculate the comprehensive state index, and output the data pattern evaluation results.

[0015] S5: Generate data application instructions and business instructions based on the model evaluation results. The instructions include data alarms, process parameter adjustments, and computing resource scheduling.

[0016] The technical effects and advantages of this invention are as follows:

[0017] 1. This invention uses a multi-source data acquisition module to cover three core data types: business system logs, device operating status, and time-series data. It adopts a distributed architecture, a multi-level monitoring network, and a real-time capture engine, combined with a clock synchronization mechanism, to achieve comprehensive acquisition of multi-source data and ensure time-series consistency. This solves the problems of single data acquisition and one-sided perspective in traditional technologies, and provides a comprehensive data foundation for situational awareness.

[0018] 2. The data preprocessing stage of this invention significantly improves data quality through hierarchical filling, three-layer deduplication, precise noise reduction, and time-series alignment; feature extraction covers three types of features: statistical, time-series, and correlation features, and a multi-entity interaction model is constructed by combining correlation graphs to output standardized multi-dimensional feature vectors, overcoming the shortcomings of traditional technologies such as single feature dimensions and poor data quality, and providing high-quality feature support for accurate pattern recognition;

[0019] 3. This invention adopts a collaborative mechanism of adaptive clustering and gradient boosting tree model, which can accurately match known patterns and effectively discover unknown threats, and comprehensively quantify situational risks through state index. The execution feedback link realizes graded alarms, targeted parameter adjustments and dynamic resource scheduling according to risk level, forming a "assessment-execution-feedback" closed loop, solving the problems of low pattern recognition accuracy and delayed response of traditional technologies, and greatly improving the accuracy of network security situational awareness and risk response efficiency. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the system structure of the present invention;

[0021] Figure 2 This is a schematic diagram of the feature construction and analysis process of the present invention;

[0022] Figure 3 This is a flowchart of the overall method of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] As attached Figure 1 and Figure 2The multi-dimensional network security situation awareness system for big data shown includes: a multi-source data acquisition module: used to collect raw datasets, including business system log data, device operating status data and time series data, and to organize the raw data and transmit it to the data preprocessing and fusion module;

[0025] Data preprocessing and fusion module: connected to the multi-source data acquisition module, used to perform data cleaning on the original dataset, remove missing and duplicate values, unify data structure and field definitions through format standardization, and perform noise removal; further, time-series data alignment is performed through time window slicing to form a structured dataset;

[0026] Feature construction and association analysis module: connected to the data preprocessing and fusion module, used to extract statistical features, time series features and association features from the structured dataset, construct a multi-entity association graph based on the graph model, obtain and realize the interaction relationship model between different data entities, and output multi-dimensional feature vectors;

[0027] Dynamic pattern recognition and state assessment module: connected to the feature construction and association analysis module, used to identify patterns in the data based on the multi-dimensional feature vector, using a collaborative fusion mechanism of adaptive clustering method and pattern recognition algorithm, calculate comprehensive state index, and output data pattern assessment results;

[0028] Execution feedback module: connected to the dynamic pattern recognition and state assessment module, used to generate data application instructions and business instructions based on the pattern assessment results, the instructions including data alarms, process parameter adjustments and computing resource scheduling.

[0029] It should be specifically noted that the function of the multi-source data acquisition module is to collect business system log data, equipment operation status data, and time series data, and to perform preliminary processing and time stamping on the raw data.

[0030] It should be further explained that the reasons for collecting the business system log data include: this type of data records user operation trajectories, system call logs, and security event logs, such as user login failure records and abnormal database access, which can directly reflect the security vulnerabilities of the business system; the reason for collecting the device operating status data is that it includes indicators such as server CPU utilization and network device throughput. When the firewall CPU utilization consistently exceeds 80%, it may indicate a denial-of-service attack; the purpose of collecting the time-series stream data is to correlate the causal relationship of security events through timestamps, such as multiple port scans initiated by the same network address within a short period of time.

[0031] It should be further explained that when collecting log data from the aforementioned business systems, a distributed log collection architecture is adopted: log collection agents based on distributed log collection tools are embedded in web servers and database servers, and the formats of network server access logs and database audit logs are parsed using regular expressions. The parsed data is then transmitted to the log aggregation node via a message queue. At the same time, log processing tools are used to standardize the time fields in the logs to ensure that the timestamp accuracy of cross-system logs reaches the millisecond level.

[0032] It should be further explained that when collecting the device operating status data, a multi-level monitoring network is constructed: For network devices such as routers and switches, object identifier values ​​such as interface traffic and CPU temperature are periodically obtained from the management information database through Simple Network Management Protocol version 3, and a trap alarm mechanism is used to capture abnormal device interruption events in real time; For server hardware, the intelligent platform management interface is used to read motherboard sensor data, and when the "reallocation sector count" in the hard drive self-monitoring analysis and reporting technical parameters exceeds the threshold, an early warning collection process is triggered; For virtualization devices, virtualization indicators such as virtual machine CPU preemption rate and memory balloon expansion rate are obtained through the virtualization platform application programming interface, and the collection frequency is dynamically adjusted according to the device load, once every five minutes under normal operating conditions, and increased to once every ten seconds during abnormal conditions.

[0033] It should be further explained that when collecting the time-series stream data, a real-time stream data capture engine is constructed: for session data in network traffic, a packet ring buffer acceleration technology is used to achieve line-speed acquisition, and a hash algorithm is used to map packets of the same session to the same processing thread to ensure time-series consistency; for alarm data generated by sensors, a real-time data channel is established using an open platform communication unified architecture protocol. When the intrusion detection system detects a malicious network address, it immediately encapsulates the alarm event in the format of "device identifier-timestamp-alarm level" and pushes it to the stream processing cluster; all the collected time-series data is clock-synchronized through a network time protocol server to ensure that the time deviation of cross-device data does not exceed fifty milliseconds.

[0034] It should be specifically noted that the data preprocessing and fusion module performs data cleaning on the original dataset, removes missing and duplicate values, standardizes the data structure and field definitions through format standardization, performs noise removal, and aligns time-series data through time window slicing to form a structured dataset.

[0035] It should be further explained that the specific methods for performing data cleaning on the raw data include: for missing fields in the business system log data, a hierarchical filling strategy is adopted—if it is a key missing field (such as user identifier, operation time) in user operation logs (such as login records), it is marked as invalid data; for non-key fields (such as terminal type), the forward filling method is used, using the value of the previous valid record in the same session; for missing values ​​in device operation status data, the sliding window interpolation method is used, with a 5-minute window, and cubic spline interpolation is used to fill in the missing points of sensor sampling; sudden missing segments (more than 10 seconds) in time series data are marked as abnormal intervals, and subsequently filled by cross-validation with data from other devices of the same source; for duplicate value removal, a three-layer deduplication mechanism is constructed: first, preliminary deduplication is performed based on the data collection timestamp and unique device identifier; second, the business system logs are deduplicated using the triple hash of "user identifier + operation type + timestamp"; finally, the device operation status data is filtered using the sliding window median filter to retain the unique valid value within the window.

[0036] Further explanation is needed regarding the specific methods for standardizing and unifying data structures and field definitions: A cross-data source metadata mapping dictionary is established, uniformly mapping "access status code" in business system logs to the "status code" field; "CPU utilization" in device operating status data is uniformly standardized to "central processing unit utilization," with the unit standardized as a percentage; timestamps in time-series data are uniformly converted to Coordinated Universal Time (UTC) format, accurate to the millisecond level; for unstructured log data (such as network device alarm logs), a regular expression template library is used for parsing, for example, parsing "Device [R1] experienced a port scan attack on 2025-10-13 10:30:00" into structured fields of "Device ID = R1, Timestamp = 2025-10-13 10:30:00, Event Type = Port Scan"; a unified encoding table is established for enumerated values ​​(such as log levels) in multi-source data, mapping "ERROR," "Error," and "High Risk" to the code "3."

[0037] It should be further explained that the specific methods for noise removal include: filtering abnormally high-frequency operation records in the business system log data (such as the same user logging in more than 20 times per minute) by using a sliding window to calculate anomaly thresholds (mean + 3 times the standard deviation); sensor noise removal from equipment operating status data uses wavelet transform denoising, and for vibration sensor data, a db4 wavelet basis is selected for 5-level decomposition, and soft thresholds are set for high-frequency coefficients, calculated based on "heuristic thresholds". σ is the standard deviation of the high-frequency coefficients after wavelet decomposition, and N is the number of samples of the high-frequency coefficients. The effective vibration characteristics are retained after reconstruction. The burst impulse noise in the time-series stream data (such as instantaneous spikes in network traffic) is processed by median filtering. The window size is dynamically adjusted according to the data sampling frequency. A 5-point window is used for second-level sampling data and a 21-point window is used for millisecond-level sampling data to ensure that the event mutation characteristics are retained while filtering out noise.

[0038] Further explanation is needed regarding the specific method of aligning time-series data through time window slicing: First, a globally unified time benchmark is established, and clock synchronization is performed on all data sources through a network time protocol server to ensure that the time deviation across devices does not exceed 50 milliseconds; then, using 10-second as the basic time window unit, the frequency of operations is aggregated according to the window for business system log data, and the mean, variance, and other statistics within the window are calculated for device operating status data, while event count features are extracted for time-series stream data according to the window; for time-series alignment of multi-source data, a dynamic time warping algorithm (DTW) optimization strategy is adopted: using the sampling time sequence of device operating status data as the benchmark sequence, the time warp path between business system logs and time-series stream data is calculated, and the similarity of time series is measured by Euclidean distance; for long-cycle time-series data (such as daily system logs), a hierarchical alignment strategy is adopted, first performing coarse alignment at the daily level, and then fine alignment at the second level within the day to ensure the time-series consistency of data at different time granularities.

[0039] It should be specifically noted that the function of the feature construction and association analysis module is to extract statistical features, time series features and association features from the structured dataset, construct a multi-entity association graph based on the graph model, obtain and realize the interaction relationship model between different data entities, and output multi-dimensional feature vectors.

[0040] Further explanation is needed regarding the specific methods for extracting general statistical features, including: For the business system log data in the structured dataset, calculating the mean of user operation counts by time window (using the time window of the data preprocessing and fusion module) – that is, summing all user operation counts within a certain time window and dividing by the total number of operations, reflecting operational activity; the variance of login duration – first calculating the difference between each login duration and the mean login duration, then squaring these differences and averaging them, reflecting operational stability; the 90th percentile of the occurrence of abnormal status codes – sorting the occurrence of abnormal status codes within a certain window from smallest to largest, and taking the value at the 90th percentile after sorting, reflecting the concentration of abnormal events; and for device operating status data, extracting the mean of CPU occupancy (the average number of CPU occupancy values ​​within a certain window). The following metrics are calculated: (1) the sum of processor utilization values ​​divided by the total number of values, reflecting average load; (2) the variance of memory utilization (the average of the squares of the differences between each memory utilization value and the mean, reflecting load fluctuation); (3) the 75th percentile of network throughput (the values ​​at the 75th percentile after sorting network throughput values ​​within the window, reflecting high-frequency load levels); (4) the mean of attack event frequency (the total number of attack events within the window divided by the number of windows, reflecting attack activity); (5) the variance of event interval duration (the average of the squares of the differences between each event interval and the mean interval, reflecting the stability of attack rhythm); and (6) the 95th percentile of traffic peaks (the values ​​at the 95th percentile after sorting traffic peaks within the window, reflecting extreme traffic pressure). All statistical characteristics are rounded to two decimal places to ensure consistent numerical precision.

[0041] Further explanation is needed regarding the specific methods for extracting time-series features: For the trend change rate, using continuous time windows as units, calculate the trend change rate of user operation frequency in the business system logs—the calculation method is to subtract the user operation frequency in the previous time window from the user operation frequency in the current time window, and then divide the difference by the user operation frequency in the previous time window; a positive result indicates an upward trend in operation frequency. For the device operation status data, calculate the trend change rate of disk I / O (subtract the disk I / O value in the previous window from the current window value, and divide the difference by the previous window value, reflecting changes in storage access pressure); for the time-series stream data, calculate the trend change rate of the number of abnormal events (subtract the number of abnormal events in the previous window from the number of events in the current window, and divide the difference by the number of events in the previous window, reflecting the speed of evolution of security risks); for periodic features… Autocorrelation analysis is used to calculate the 24-hour periodic autocorrelation coefficient for the scheduled task operation sequence of the business system logs—that is, to measure the correlation between the operation sequence and its sequence after a 24-hour delay. If the absolute value of the correlation coefficient is greater than or equal to 0.6, it is considered that there is a significant 24-hour period. The 1-hour periodic autocorrelation coefficient is calculated for the CPU load sequence of the device operation status data (to measure the correlation between the load sequence and the sequence after a 1-hour delay, and to identify short-period fluctuation patterns). The 12-hour periodic autocorrelation coefficient is calculated for the attack event sequence of the time-series data (to measure the correlation between the attack sequence and the sequence after a 12-hour delay, and to capture the time pattern of attack behavior). The periodic feature is represented by a tuple of "whether there is a period (0 indicates no period, 1 indicates period) + period length (in minutes)".

[0042] Further explanation is needed regarding the specific methods for extracting association features, including: For entity co-occurrence frequency, defining core entities as users, devices, and security events, calculating the co-occurrence frequency of "user-device" within the same time window—that is, the number of times a user and a device appear together within the same time window, then dividing by the total number of all time windows, and normalizing the result to between 0 and 1 (the larger the value, the more frequent the co-occurrence); the co-occurrence frequency of "device-security event" (the number of times a device and a security event appear together within the same window divided by the total number of windows, also normalized to between 0 and 1); and for interaction intensity, quantifying the closeness of interaction between entities: User-device interaction strength is represented by the product of data transfer volume (in MB) and interaction duration (in seconds) per unit time. This product is then divided by the maximum product value of this type of interaction across all time windows to obtain a normalized result, which is then rounded to three decimal places. Device-device interaction strength is represented by the product of business dependency weight (pre-set based on system topology, ranging from 0 to 1) and actual communication frequency. For example, if the dependency weight of the application server on the database server is 0.9, and the two communicate 15 times within a certain window, and the maximum communication frequency between this type of device is 20 times, then the interaction strength is 0.9 multiplied by 15 and then divided by 20 to obtain the final result.

[0043] It should be further explained that the specific method for constructing a multi-entity relationship graph based on the general association analysis method of graph models is as follows: First, define the node set of the graph, which uniformly includes three types of entities: user nodes (named with unique user identifiers), device nodes (named with unique device identifiers), and security event nodes (named with event types). Node attributes include entity type and historical activity (the number of interactions in the last 24 hours divided by the maximum possible number of interactions in the last 24 hours to obtain a normalized value). The maximum possible number of interactions = the total number of time windows in the last 24 hours × the maximum number of interactions per window. Wherein, the total number of time windows in the last 24 hours = 24 × 3600 ÷ 10 = 8640 (based on a time granularity of 10 seconds / window). Secondly, the edge set and general weight calculation rules are defined. All edges between entities represent "associations". The edge weights are calculated using standardized frequency: the actual number of times the entities co-occur is divided by the total number of time windows to ensure that the weight range is between 0 and 1 (the larger the value, the stronger the association). Finally, the graph structure is stored through an adjacency matrix. The matrix dimension is the total number of nodes multiplied by the total number of nodes. The element in the i-th row and j-th column of the matrix represents the edge weight between the i-th node and the j-th node. The diagonal elements (i.e., the weight of the node and itself) are set as the degree centrality of the node. The calculation method is the number of connected nodes of the node divided by the total number of all nodes, reflecting the importance of the node in the graph, thus forming a generalized association graph.

[0044] It needs further explanation that the specific method for achieving multi-dimensional modeling of the interaction relationships between different data entities is as follows: Based on the constructed relationship graph, the interaction relationships are modeled from three dimensions: First, the breadth of interaction, which calculates the number of connected nodes for each entity (i.e., node degree) and then divides it by the total number of all nodes. The resulting ratio reflects the scope of the entity's association (the larger the ratio, the wider the scope of association); Second, the depth of interaction, which quantifies the tightness of the association between entities through the mean and maximum value of edge weights. For example, if the average edge weight of a user node is 0.3, it means that the average tightness between the user and all associated entities is 0.3, and the maximum value is the value with the largest weight among all associated edges of the user; Third, the interaction path, which uses a breadth-first search algorithm to mine typical interaction paths of "user-device-security event", counts the frequency of each path, and then divides it by the total frequency of all paths to obtain the proportion, such as the proportion of the "user A-server B-malicious code event" path in all paths. The path length is limited to 3 to 5 nodes to focus on key associations.

[0045] It should be further explained that the specific organization method of the output multi-dimensional feature vector is as follows: the extracted general statistical features, time series features, and correlation features are integrated according to the rule of "feature type - entity type - specific indicator". Among them, the general statistical features include 12 dimensions (4 for business system logs, 4 for device operating status, and 4 for time series flow), the time series features include 9 dimensions (3 for trend change rate and 6 for periodic features), and the correlation features include 15 dimensions (6 for co-occurrence frequency, 6 for interaction intensity, and 3 for graph interaction dimension). All feature values ​​are processed by min-max standardization - that is, each feature value is subtracted from the minimum value of the feature, and then divided by the difference between the maximum and minimum values ​​of the feature, and mapped to 0 to 1 to eliminate the difference in units. The final output multi-dimensional feature vector is 36-dimensional and stored in the form of a structured array. The array element naming format is as follows: "statistics - device - central processing unit mean", "time series - log - operation frequency trend rate", "correlation - graph - interaction breadth ratio".

[0046] It should be specifically noted that the function of the dynamic pattern recognition and state assessment module is to identify patterns in the data based on the multi-dimensional feature vector, using a collaborative fusion mechanism of adaptive clustering method and pattern recognition algorithm, calculate the comprehensive state index, and output the data pattern assessment results.

[0047] It should be further explained that the specific implementation of the adaptive clustering method is as follows: For the 36-dimensional multi-dimensional feature vector, a density-based adaptive clustering algorithm is adopted, and pattern discovery is achieved by dynamically adjusting the cluster radius r and the minimum number of samples minPts; wherein: the initial value of the cluster radius r is set to 1 / 3 of the mean Euclidean distance between all samples; if a single cluster contains more than 30% of the total number of samples, the radius is reduced by r = r × 0.8 (to avoid the clusters being too large); the formula for calculating the minimum number of samples minPts is: minPts = d × 2, where d = 36, corresponding to a minimum number of samples of 72, to ensure that each cluster contains enough samples to reflect a stable pattern;

[0048] During clustering, business system logs, device operating status, and time-series stream features are assigned weights of 30%, 40%, and 30%, respectively, and then weighted by weighted Euclidean distance D. w (x i ,x j Measuring sample similarity: , where x i,k Let w be the k-th dimension feature value of the i samples; k The weights are the weights of the k-th dimension features.

[0049] It should be further noted that the pattern recognition algorithm uses a gradient boosting tree model, and the input is a multi-dimensional feature vector x=[x1,x2,...,x...]. 36 (Each dimension is a dimensionless standardized value), the output is the probability distribution P(c|x) of 12 known patterns, where: Let c be the pattern category; P(c|x) is the probability that a sample belongs to category c; the model optimizes parameters through 5-fold cross-validation: tree depth (5≤T≤8), learning rate η=0.05; feature importance is evaluated based on the reduction of Gini impurity, key features such as attack event frequency feature x5, CPU utilization trend feature x 12 Its importance weight is higher than that of other features.

[0050] It should be further explained that the collaborative fusion mechanism of adaptive clustering and pattern recognition algorithms adopts a two-stage processing flow of "first recognition, then clustering": Known pattern matching: calculate the maximum probability (max) of a sample belonging to each known pattern. c P(c|x), if max c If P(c|x≥0.6), then it is determined to be the known pattern; Unknown pattern mining: for max c For samples with P(c|x<0.6), calculate μ relative to the cluster center. m Weighted distance: , where μ m =[μ m,1 ,μ m,2 ,...,μ m,d] is the center vector of the m-th cluster, r m Let D be the clustering radius of cluster m; if D w (x,μ m ) <r m If it is, it is determined to be the corresponding unknown mode; otherwise, it is marked as an isolated point.

[0051] It should be further explained that the calculation of the comprehensive status index adopts a three-level index system (all dimensionless, ranging from 0 to 100), and the formula is as follows, including: Sub-index calculation: Attack risk sub-index S a (Reflecting the degree of security attack risk): , where f att The frequency of attack events in the current window (times / window, window is 10 seconds); f att,max Maximum historical attack frequency (times / window); For the attack type distribution entropy (p) i (where i is the percentage of attacks of type i); n known The number of known attack pattern samples (n) total The total number of samples (number of samples);

[0052] Equipment Health Sub-index S dev (Reflecting equipment operational stability): , where q cpu,95% 95th percentile of CPU utilization (%); q cpu,95%,max The highest 95th percentile in history (%); σ mem σ represents the variance of memory utilization (%²). mem,max The historical maximum variance (%²) is given; the ratio is dimensionless. Disk I / O trend change rate (io) curr The current window's I / O value, io prev (The previous window's I / O value); The normal trend threshold is set based on three times the standard deviation of historical normal data.

[0053] Business Anomaly Sub-index S b (Reflecting the degree of abnormality in business operations): , where r abn The percentage of abnormal logins (%); r abn,max This represents the highest percentage in history (%). The rate of change of trend in operation frequency (ops) curr ops represents the number of operations performed on the current window. prev (Number of operations in the previous window); The normal trend threshold (set based on 3 times the standard deviation of historical normal data) is used, and the ratio of the absolute value to the threshold is dimensionless; f coUser-device co-occurrence frequency (times / window); f co,max This represents the highest historical frequency (times per window).

[0054] Comprehensive State Index SS=S a ×0.5+S dev ×0.3+S b ×0.2, where weights of 0.5, 0.3, and 0.2 correspond to the importance of attack risk, device health, and business anomaly, respectively. The result is rounded to the nearest integer, ranging from 0 to 100 (the higher the value, the more severe the security situation).

[0055] It should be further noted that the data pattern evaluation results are output in structured JSON format, including: Pattern recognition results: known patterns (name, frequency of occurrence, percentage), unknown patterns (cluster number, core features, number of samples), number of outliers; State index: comprehensive state index and sub-index scores for attack risk, device health, and business anomaly; Risk level: divided into 5 levels according to S (0-20 safe, 21-40 low risk, 41-60 medium risk, 61-80 high risk, 81-100 extremely high risk); Key features: the 3 features that contribute the most to the current index and their contribution, such as "attack event frequency (35%)".

[0056] It should be specifically noted that the function of the execution feedback module is to generate data application instructions and business instructions based on the data pattern evaluation results output by the dynamic pattern recognition and state evaluation module. The instructions include data alarms, process parameter adjustments, and computing resource scheduling.

[0057] It should be further explained that the data pattern evaluation results include four core pieces of information: pattern recognition results (distribution of known patterns, unknown patterns and outliers), comprehensive status index and sub-index scores (attack risk, device health, business anomaly), risk level (5-level classification), and key impact features (features that contribute the most to the current index). This information provides a quantitative basis and target orientation for instruction generation.

[0058] It should be further explained that the generation method of the data alerts is based on a graded alert mechanism formulated according to risk level and key pattern characteristics:

[0059] Risk Level 1 (Safe, Overall Status Index 0-20): No alarms are triggered; only the normal system operation log is recorded, including the current overall index and each sub-index.

[0060] Risk Level 2 (Low Risk, 21-40): Triggers "Alert", displays a blue alert message on the system console, including "The current network is in a low-risk state, the main impact feature is [Key Feature 1], and continuous monitoring is recommended";

[0061] Risk Level 3 (Medium Risk, 41-60): Triggers "Warning and Alarm". In addition to console notifications, an alarm message is pushed to the mobile application of the maintenance personnel, along with risk details (such as "Medium Risk: Attack Risk Sub-Index 58, mainly due to the increase in the proportion of abnormal logins") and 3 related key characteristic evidences.

[0062] Risk Level 4 (High Risk, 61-80): Triggers an "Emergency Alarm", activates the system's audible and visual alarm devices, and sends SMS messages and application push notifications to the operations and maintenance manager and all members of the security team. The alarm information includes the real-time comprehensive index, the name of the high-risk mode (e.g., "DDoS attack mode accounts for 35%)", and the expected scope of impact (e.g., "may cause Web server response delay").

[0063] Risk Level 5 (Extremely High Risk, 81-100): Triggers a "Disaster Alarm". In addition to all emergency alarm measures, it automatically dials the emergency contact number of the operations and maintenance manager and attaches an emergency response plan summary (such as "Immediately start the traffic scrubbing equipment and isolate the attack source IP") to the alarm message.

[0064] All alarm information includes a timestamp, risk level, key characteristic value, and associated device ID, and is archived and stored in the format of "risk level - timestamp" for easy traceability.

[0065] It should be further explained that the specific methods for adjusting the process parameters are based on targeted strategies formulated according to key impact characteristics and high-risk pattern types:

[0066] When the attack risk sub-index is ≥70 and the pattern recognition result contains "DDoS attack", adjust the network firewall parameters: add the network segment where the attack source IP is located to the temporary blacklist (valid for 1 hour), enable the traffic shaping function to limit the maximum bandwidth of the network segment (not exceeding 10% of the total bandwidth), and adjust the detection sensitivity of the intrusion detection system (from "standard" to "strict").

[0067] When the device health sub-index is ≥60 and the core characteristic is "CPU utilization rate 95th percentile is too high", adjust the server running parameters: set the CPU usage limit for non-core processes (not exceeding 30% of a single core), start the process priority scheduling mechanism (set the priority of core business processes to the highest), and if the memory utilization rate exceeds 80% synchronously, automatically trigger the memory page replacement policy (replace the memory of inactive processes to the disk swap area).

[0068] When the business anomaly sub-index is ≥50 and the key feature is "the proportion of abnormal logins is increasing", adjust the business system parameters: temporarily add login verification steps (such as two verification codes), limit the number of login attempts from the same IP (no more than 5 times every 10 minutes), and trigger temporary locking for accounts that fail to log in 3 times in a row (lock duration 15 minutes).

[0069] The parameter adjustment follows the "minimum intervention principle," with the adjustment range not exceeding 30% of the baseline parameter. The adjustment effect is automatically evaluated every 30 minutes. If the corresponding sub-index decreases by ≥20% compared to the value before adjustment, the adjustment is maintained; otherwise, the default parameter is restored and an alternative scheme is activated.

[0070] It should be further explained that the implementation mechanism for computing resource scheduling dynamically allocates resources based on risk level and resource load status:

[0071] When the risk level is 3 or above, resource expansion scheduling will be initiated: For servers carrying core business, virtual machine instances will be automatically added (number = current number of instances × risk level coefficient, where the coefficient is 0.5 for level 3, 1.0 for level 4, and 2.0 for level 5). The number of expansion instances will be rounded up to ensure sufficient resources. If the rounding result exceeds the resource pool limit, the maximum number of available instances in the resource pool will be used. Expansion instances will be selected from the backup nodes in the resource pool first to ensure deployment is completed within 3 minutes.

[0072] When the device health sub-index is ≥70 and the load of a server exceeds the threshold (CPU ≥85% or memory ≥90%), load migration scheduling is initiated: non-core business (such as log analysis and backup processes) on the server are migrated to a server with a lower load (CPU ≤50% and memory ≤60%). The migration process adopts the "copy first, switch later" mode to ensure that the business interruption time is ≤10 seconds.

[0073] If the risk level drops to level 2 or below and remains so for 30 minutes, resource reclamation scheduling will be initiated: redundant resources will be released in the order of "expansion first, reclamation later" and "non-core first, core later", standby virtual machine instances will be shut down, migrated services will be moved back to the original servers, and the reclaimed resources will be put back into the resource pool for standby.

[0074] Resource scheduling is automated through cloud platform APIs. The scheduling process records indicators such as resource usage and migration time in real time, which serve as the basis for subsequent optimization of scheduling strategies.

[0075] It should be further explained that the collaboration mechanism between the execution feedback module and other modules is as follows: the module receives the evaluation results pushed by the dynamic pattern recognition and status assessment module in real time (the push frequency is consistent with the time window, i.e., 10 seconds / time). When a change in risk level or the emergence of a new high-risk pattern is detected, the instruction generation process is immediately triggered. After the instruction is executed, the execution results (such as alarm confirmation status, comparison of indicators before and after parameter adjustment, and resource scheduling completion status) are fed back to the dynamic pattern recognition and status assessment module for feature weight correction in the next round of evaluation (e.g., if the attack risk decreases after a parameter is adjusted, the importance weight of the corresponding feature of that parameter is increased), forming a closed-loop control of "evaluation-execution-feedback".

[0076] refer to Figure 3 Multi-dimensional network security situation awareness methods for big data include:

[0077] S1: Collect the raw dataset, which includes business system log data, device operating status data, and time series data, and perform preliminary processing and time stamping on the raw data;

[0078] S2: Perform data cleaning on the original dataset, remove missing and duplicate values, unify the data structure and field definitions through format standardization, and perform noise removal; further, perform time-series data alignment through time window slicing to form a structured dataset;

[0079] S3: Extract statistical features, time-series features, and correlation features from the structured dataset, construct a multi-entity relationship graph based on the graph model, obtain the interaction relationship model between different data entities, and output multi-dimensional feature vectors;

[0080] S4: Based on the multi-dimensional feature vector, an adaptive clustering method and a pattern recognition algorithm are used to identify patterns in the data, calculate the comprehensive state index, and output the data pattern evaluation results.

[0081] S5: Generate data application instructions and business instructions based on the model evaluation results. The instructions include data alarms, process parameter adjustments, and computing resource scheduling.

[0082] Secondly: The accompanying drawings of the embodiments disclosed in this invention only involve the structures involved in the embodiments disclosed in this invention. Other structures can refer to the general design. In the absence of conflict, the same embodiment and different embodiments of this invention can be combined with each other.

[0083] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-dimensional network security situation awareness system for big data, characterized in that, include: Multi-source data acquisition module: used to acquire raw datasets, including business system log data, device operating status data and time series data, and to perform preliminary processing and time stamping on the raw data; Data preprocessing and fusion module: connected to the multi-source data acquisition module, used to perform data cleaning on the original dataset, remove missing and duplicate values, unify data structure and field definitions through format standardization, and perform noise removal processing; Further time-series data alignment is performed through time window slicing to form a structured dataset; Feature construction and association analysis module: connected to the data preprocessing and fusion module, used to extract statistical features, time series features and association features from the structured dataset, construct a multi-entity association graph based on the graph model, obtain and realize the interaction relationship model between different data entities, and output multi-dimensional feature vectors; Dynamic pattern recognition and state assessment module: connected to the feature construction and association analysis module, used to identify patterns in the data based on the multi-dimensional feature vector, using a collaborative fusion mechanism of adaptive clustering method and pattern recognition algorithm, calculate comprehensive state index, and output data pattern assessment results; Execution feedback module: connected to the dynamic pattern recognition and state assessment module, used to generate data application instructions and business instructions based on the pattern assessment results, the instructions including data alarms, process parameter adjustments and computing resource scheduling.

2. The multi-dimensional network security situation awareness system for big data as described in claim 1, characterized in that: When collecting log data from business systems, the multi-source data acquisition module adopts a distributed log acquisition architecture, deploying log acquisition agents in web servers and database servers. It parses the formats of network server access logs and database audit logs in appropriate ways, transmits the parsed data to the log aggregation node via a message queue, and standardizes the time field in the logs.

3. The multi-dimensional network security situation awareness system for big data as described in claim 1, characterized in that: When standardizing the format, the data preprocessing and fusion module establishes metadata mapping rules across data sources, and standardizes the naming and unit specifications for similar fields in different data sources; for unstructured log data, it uses a template-based parsing method to convert it into structured fields; and it establishes unified encoding rules for enumeration information in the data to achieve consistent representation of similar enumeration information.

4. The multi-dimensional network security situation awareness system for big data as described in claim 1, characterized in that: When integrating features and outputting feature vectors, the feature construction and association analysis module classifies and integrates the extracted statistical features, time-series features and association features according to the rule of "feature type - entity type - specific indicator", standardizes all feature values, and finally stores and outputs multi-dimensional feature vectors in a structured form.

5. The multi-dimensional network security situation awareness system for big data as described in claim 1, characterized in that: The collaborative fusion mechanism of the adaptive clustering method and the pattern recognition algorithm in the dynamic pattern recognition and state assessment module has the following specific operation logic: First, the pattern recognition algorithm is used to match the multi-dimensional feature vectors with known security patterns; then, for the multi-dimensional feature vectors that have not matched the known security patterns, the adaptive clustering method is used to mine the unknown security patterns; at the same time, the known pattern matching results output by the pattern recognition algorithm are used to help adjust the core parameters of the adaptive clustering method. The unknown pattern features discovered by the adaptive clustering method are added to the sample library of the pattern recognition algorithm, forming a collaborative processing flow of dynamic interaction between the two.

6. The multi-dimensional network security situation awareness system for big data as described in claim 5, characterized in that: The matching process specifically includes: the pattern recognition algorithm first calls a preset known security pattern feature library, which contains feature templates corresponding to security patterns; then, the input standardized multi-dimensional feature vector is compared with the feature templates of each known security pattern in the feature library to calculate the similarity; finally, based on a preset similarity threshold, the known security pattern matched by the feature vector is determined. If the similarity reaches the threshold, it is marked as the corresponding known pattern; otherwise, it is determined as an unmatched known pattern.

7. The multi-dimensional network security situation awareness system for big data as described in claim 1, characterized in that: When the execution feedback module generates a data alarm, the alarm information includes a timestamp, risk level, key feature value, and associated device identifier. It sets a tiered triggering method from recording system operation logs to initiating emergency alarm measures according to the risk level, and archives the data according to the "risk level - timestamp" rule. When adjusting process parameters, it follows the principle of minimum intervention to control the adjustment range, formulates corresponding strategies for different types of high-risk modes, and periodically evaluates and decides whether to maintain the current adjustment status or restore the default parameter values.

8. A multi-dimensional network security situation awareness method for big data, used to implement the multi-dimensional network security situation awareness system for big data as described in any one of claims 1-7, characterized in that, include: S1: Collect the raw dataset, which includes business system log data, device operating status data, and time series data, and perform preliminary processing and time stamping on the raw data; S2: Perform data cleaning on the original dataset, remove missing and duplicate values, unify the data structure and field definitions through format standardization, and perform noise removal. Further time-series data alignment is performed through time window slicing to form a structured dataset; S3: Extract statistical features, time-series features, and correlation features from the structured dataset, construct a multi-entity relationship graph based on the graph model, obtain the interaction relationship model between different data entities, and output multi-dimensional feature vectors; S4: Based on the multi-dimensional feature vector, an adaptive clustering method and a pattern recognition algorithm are used to identify patterns in the data, calculate the comprehensive state index, and output the data pattern evaluation results. S5: Generate data application instructions and business instructions based on the model evaluation results. The instructions include data alarms, process parameter adjustments, and computing resource scheduling.

Images