Big data-based network anomaly perception detection method and system

Abstract

The application discloses a network anomaly perception detection method and system based on big data, and relates to the technical field of network security.The steps of the method include: collecting network parameters, task parameters and behavior parameters in real time, normalizing and constructing a plurality of parameter sequences, dividing the plurality of parameter sequences into a plurality of candidate groups through preset first and second constraints, analyzing the abnormal correlation degree of the parameter sequences in each candidate group, calculating the abnormal correlation scores of the candidate groups, and dividing the candidate groups into a first response pool and a second response pool based on the abnormal correlation scores; for the candidate groups in the first response pool, if the detection resources of each candidate group in the second response pool are determined, the remaining detection resources are selected to trigger a perception evaluation task in combination with the abnormal correlation scores; and for the candidate groups in the second response pool, an abnormal response strategy is executed.The application improves the detection rate of network anomaly perception and the utilization rate of detection resources.

Description

Technical Field

[0001] This invention relates to the field of network security technology, specifically to a method and system for network anomaly detection based on big data. Background Technology

[0002] With the expansion of network scale and the popularization of emerging technologies such as big data and cloud computing, the complexity of architecture is growing exponentially. The large number of users makes network security increasingly important. However, communication networks are controlled by physical devices, and once attacked, they may cause irreparable damage, accompanied by abnormal network traffic.

[0003] Existing network anomaly detection systems have the following shortcomings:

[0004] On the one hand, the massive logs and parameters generated by network devices, servers, and applications are usually analyzed independently, and network anomalies are becoming increasingly concealed and often correlated across multiple dimensions. In cases such as DDoS attacks, privilege escalation, and abnormal traffic, relying on single-dimensional parameters is no longer sufficient to meet the needs of accurate perception. For example, when a system is attacked, the slight traces left by the attack in different dimensions such as network and behavior cannot be linked together, resulting in many low-reliability alarms and a high false alarm rate.

[0005] On the other hand, during the network anomaly detection process, high-priority anomalies such as DDoS attacks and core data leaks are often overtaken by less important anomalies, resulting in unreasonable resource allocation for detection. This forces important anomalies to be handled with inefficient resources, paralyzing response capabilities and leading to low detection and response efficiency. At the same time, the lack of predictive ability for future anomalies means that response strategies cannot be adapted to potential anomalies in advance, and can only passively deal with anomalies that have already occurred. After detecting network anomalies, pre-configured template anomaly response strategies are usually adopted, such as initiating fixed-bandwidth isolation upon detecting an anomaly, which further reduces the timeliness and resource adaptability of anomaly handling. Summary of the Invention

[0006] (a) Technical problems to be solved

[0007] To address the shortcomings of existing technologies, this invention provides a network anomaly perception and detection method and system based on big data. By real-time collection and normalization of multi-dimensional parameters such as network, task, and behavior, and then applying first and second constraints to intelligently divide candidate groups, it can fuse and correlate weak anomaly signals scattered across different dimensions. By calculating anomaly correlation scores and accordingly assigning candidate groups to primary and secondary response pools, the accuracy of network anomaly perception and detection is improved. For the secondary response pool, anomaly response strategies are executed; for the primary response pool, the remaining resources allocated from the secondary response pool are intelligently utilized to trigger perception evaluation tasks, improving resource utilization and solving the problems mentioned in the background technology.

[0008] (II) Technical Solution

[0009] To achieve the above objectives, the present invention provides the following technical solution:

[0010] Firstly, this application provides a network anomaly detection method based on big data, the method comprising:

[0011] Real-time collection of network parameters, task parameters, and behavior parameters, and normalization to construct network parameter sequences, task parameter sequences, and behavior parameter sequences. Through preset first and second constraints, several parameter sequences are divided into several candidate groups.

[0012] Based on each candidate group, the degree of abnormal correlation of parameter sequences within the group is analyzed, the abnormal correlation score of each candidate group is calculated, and the candidate groups are divided into a first-level response pool and a second-level response pool based on the abnormal correlation score.

[0013] For candidate groups in the primary response pool, if the detection resources for each candidate group in the secondary response pool are determined, the remaining detection resources are selected to trigger the perception evaluation task based on the anomaly association score.

[0014] For candidate groups in the secondary response pool, execute the abnormal response strategy.

[0015] Furthermore, the network parameters include data traffic, transmission protocol, source IP, and destination IP; the task parameters include node computing load, port scan frequency, and network connection frequency; and the behavioral parameters include data tracking operations and log statistics.

[0016] Furthermore, through preset first and second constraints, several parameter sequences are divided into several candidate groups, including:

[0017] Calculate the pairwise similarity between any two parameter sequences to form a similarity matrix; use a clustering algorithm to analyze the similarity matrix, automatically group the sequences, and obtain several preliminary groups, including network parameter sequence groups, task parameter sequence groups, and behavior parameter sequence groups;

[0018] The initial grouping is reorganized based on the first and second constraints to form several candidate groups;

[0019] In this process, the first constraint uses a Cartesian product to perform a verification and recombination on each preliminary group to obtain a first candidate group. Under the condition of satisfying the first constraint, the second constraint presets a time window and identifies the timestamps of abnormal occurrences in the first candidate group. It sets a time deviation for the timestamps and bundles the timestamps with a time deviation less than the time deviation threshold to obtain the corresponding second candidate combination. At the same time, a pair of parameter sequences is randomly selected from the second candidate group as a candidate causal pair, including the first sequence and the second sequence. The second candidate group is then subjected to a second verification and recombination to obtain the final candidate group.

[0020] Furthermore, the secondary verification and reassembly process also includes:

[0021] Statistical matching: After each abnormal time point in the first sequence occurs, the number of successful matches is recorded when an abnormal time point also occurs in the second sequence within a preset maximum lag time window;

[0022] Generate reference: Perform multiple random permutations on the abnormal time point sequence of the second sequence, and repeat the statistical matching step after each permutation to obtain a reference distribution of the number of random matches;

[0023] Test judgment: If the statistical significance of the number of successful matches in the reference distribution is higher than the preset confidence level, then the candidate causal pair is judged to satisfy the second constraint.

[0024] Furthermore, the degree of abnormal correlation among parameters within the group is analyzed, and the abnormal correlation score for each candidate group is calculated, including:

[0025] Evaluation metrics were extracted based on candidate groups, including: number of successful matches and time deviation;

[0026] The abnormal match rate is obtained based on the number of successful matches; the timing deviation coefficient is obtained based on the time deviation.

[0027] By performing a transformation analysis on the time series deviation coefficient, the positive value of the time series deviation is obtained. Combined with the abnormal matching rate, the abnormal association score is obtained by weighted summation.

[0028] Furthermore, based on the anomaly correlation score, the remaining detection resources are selected to trigger a perception assessment task, including:

[0029] After the detection resource allocation for all candidate groups in the secondary response pool is determined, for all candidate groups in the primary response pool, the remaining detection resources are used to extract the abnormal patterns of the parameter sequences within the groups, and the abnormal patterns are characterized as at least one of network type, attack type, or behavior type.

[0030] The first triplet is formed based on the anomaly pattern, anomaly association score, and anomaly duration, and is labeled as the range analysis group; the second triplet is formed based on the anomaly pattern, anomaly association score, and the number of anomaly parameter sequences, and is labeled as the intensity analysis group; the range analysis space and intensity analysis space are formed by the range analysis group and intensity analysis group that the anomaly pattern appears in the current calculation period, respectively.

[0031] The standard score of the current measurement cycle range analysis group in the range analysis space is marked as the first evaluation score;

[0032] The standard score of the current measurement cycle intensity analysis group in the intensity analysis space is marked as the second evaluation score;

[0033] The first and second evaluation scores are weighted and summed to obtain a comprehensive evaluation index. The network scheduling level is determined by batch sorting according to the comprehensive evaluation index. The network scheduling level is matched with a preset resource allocation rule, and the resource allocation rule includes at least the following: if the network scheduling level is determined to be level three, the corresponding candidate group is aggregated into the level two response pool.

[0034] Furthermore, determining the degree of network scheduling includes comparing the comprehensive evaluation index with preset first threshold intervals, second threshold intervals, and third threshold intervals:

[0035] If the comprehensive evaluation index is within the first threshold range, it is determined to be a level 1 network scheduling level.

[0036] If the comprehensive evaluation index is in the second threshold range, it is determined to be a level 1 network scheduling level.

[0037] If the comprehensive evaluation index is in the third threshold range, it is judged as a level three network scheduling level.

[0038] Furthermore, the exception response strategy allocation is executed, including:

[0039] The detection resources in the secondary response pool are classified and quantized to obtain the resource vector R: R=[r1, r2, ..., r m ], where m is the number of resource types to be detected, including the number of traffic filtering threads, port status management computing power, and IP routing isolation bandwidth, r i Let i represent the quantization value of the i-th type of detection resource, where i ranges from [1, m].

[0040] The sample set is aggregating all candidate groups in the secondary response pool and using a pre-set perception detection model to train and output the abnormal patterns at the next N time points, which are then used as sample labels. During training, the minimum resource requirement of the quantized resource vector is used as the objective function to obtain the initial policy code. The policy code is iteratively optimized using a particle swarm optimization algorithm, and the policy code with the highest fitness is output as the abnormal response policy. The minimum resource requirement is to minimize the number of traffic filtering threads, the computing power for port status management, and the bandwidth for IP routing isolation.

[0041] Furthermore, the anomaly response strategy is represented by traffic blocking rules, port blocking duration, and IP isolation policies for the next N time points.

[0042] Secondly, this application provides a network anomaly detection system based on big data, the system comprising:

[0043] The sequence construction module is used to collect network parameters, task parameters, and behavior parameters in real time, and normalize them to construct network parameter sequences, task parameter sequences, and behavior parameter sequences. Through preset first and second constraints, several parameter sequences are divided into several candidate groups.

[0044] The anomaly correlation module is used to analyze the degree of anomaly correlation of parameter sequences within each candidate group, calculate the anomaly correlation score of each candidate group, and divide the candidate groups into a primary response pool and a secondary response pool based on the anomaly correlation score.

[0045] The strategy execution module, for candidate groups in the primary response pool, if the detection resources for each candidate group in the secondary response pool are determined, combines the anomaly correlation score to select the remaining detection resources to trigger the perception evaluation task; for candidate groups in the secondary response pool, it executes the anomaly response strategy.

[0046] (III) Beneficial Effects

[0047] This invention provides a network anomaly detection method and system based on big data, which has the following beneficial effects:

[0048] 1. This invention divides candidate groups by first and second constraints. The first constraint verifies and reorganizes the candidate groups by Cartesian product verification, forcing them to be constructed as triplets of network parameter sequences, task parameter sequences, and behavior parameter sequences. The second constraint uses a preset time window and specific timestamps to accurately filter out abnormal parameter sequences with strong temporal correlation, significantly reducing invalid combinations of coincidental times, providing a precise data foundation for subsequent model training, and enhancing the interpretability of network security events.

[0049] 2. This invention targets the primary response pool and utilizes the remaining detection resources of the secondary response pool to construct a range analysis group and an intensity analysis group. It performs multimodal quantitative evaluation of anomaly perception in the primary response pool from two directions: anomaly duration and anomaly parameter sequence. It also calculates a comprehensive evaluation index to accurately determine the network scheduling degree in each perception process and automatically triggers resource allocation rules. This significantly shortens the response time from threat discovery to in-depth analysis, ensures that the computing resources of the entire system are maximized, and avoids resource idleness.

[0050] 3. This invention introduces a perception and detection model for the secondary response pool, using the minimum resource requirement of the quantized resource vector as the objective function to obtain the initial strategy code; it uses a particle swarm optimization algorithm to iteratively optimize the strategy code, and outputs the strategy code with the highest fitness as the anomaly response strategy; thus improving the timeliness of anomaly handling. Attached Figure Description

[0051] Figure 1 This is a flowchart illustrating a network anomaly detection method according to an exemplary embodiment;

[0052] Figure 2 This is a schematic diagram of a network anomaly perception and detection system according to an exemplary embodiment. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] Example 1:

[0055] This invention provides a network anomaly detection method based on big data; Figure 1 This is a flowchart illustrating a network anomaly detection method according to an exemplary embodiment; please refer to [link / reference]. Figure 1 The method includes the following steps:

[0056] The system collects network parameters, task parameters, and behavioral parameters in real time, and normalizes them to construct network parameter sequences, task parameter sequences, and behavioral parameter sequences. Based on preset first and second constraints, these parameter sequences are divided into several candidate groups. Network parameters include data traffic, transmission protocols, source IPs, and destination IPs, reflecting the network's communication and attack status. Task parameters include node computational load, port scan frequency, and network connection frequency. Behavioral parameters include embedded operations and log statistics.

[0057] In this embodiment, the above data can be collected in real time using an agent, log collection tool, network traffic analysis tool, data tracking tool, or monitoring scanning tool. Specifically, for network parameters, Suricata is used to collect data traffic, and its collection dimensions include, but are not limited to, packets per second (PPS), bytes per second (BPS), traffic fluctuation amplitude, and traffic percentage of a specific port; that is, the data output is time-series numerical. Suricata is used to capture packets and collect protocols, and its collection dimensions include, but are not limited to, protocol type, such as TCP / UDP / ICMP / HTTP, and protocol exception flags; that is, the data output is time-series categorized. The source IP is obtained using a WHOIS library or IP location interface. The source addresses are highly dispersed and irregular. The collection dimensions include, but are not limited to, the number of source IPs, their geographical distribution, and their access frequency; the data output is time-series statistical. Network traffic analysis tools, such as NetFlow+nfdump, are used to collect target IPs. The target addresses are highly concentrated, and the collection dimensions include, but are not limited to, the number of target IPs and their access percentage; the data output is also time-series statistical. For task parameters, Prometheus is used to collect node computational load, and the collection dimensions include, but are not limited to, CPU utilization, memory utilization, and the number of processes; the data output is a time-series numerical vector. Monitoring and scanning tools are used to collect port scan frequency and network connection frequency. The data collection dimensions for frequency scanning include, but are not limited to, the number of port scans performed by a single IP on the same node, and the number of scans performed by multiple IPs on the same port; the data output is a time-series numerical type. Namp is used to collect network connection frequency, and its collection dimensions include, but are not limited to, the number of TCP connections to a node and the connection success rate; the data output is a time-series numerical vector type. For behavioral parameters, tracking tools are used to collect the number of tracking operations, and its collection dimensions include, but are not limited to, the frequency of a certain operation by the same user, such as: a user attempting to log in 10 times within 1 minute, all of which failed; the trigger ratio of key operations; the time distribution of operations; and the tracking type, such as: download, login, browsing, payment; the data output is a time-series statistical type. Log collection tools are used to collect... The system collects log statistics, including but not limited to log duplication rate and the number of abnormal commands, meaning the data output is time-series statistical. Additionally, a fixed time window and sliding step can be set. Within each time window, collected data parameters of the same type are aggregated to form a value at a specific point in time, ensuring that parameter data at the same moment can be correlated. Simultaneously, invalid data is removed because the dimensions of the aforementioned parameters differ significantly. For example, data traffic might be in GB, while CPU load is between 0 and 1, so normalization is necessary to scale them to a uniform range, avoiding the impact of magnitude differences on subsequent model calculations. For example, for numerical parameters such as data traffic and port scan frequency, Min-Max normalization is used.For categorical parameters, such as transmission protocols, normalization is performed using tag encoding or one-hot encoding: Tag encoding: suitable for ordered classification, mapping classification values ​​to 0, 1, 2..., for example: transmission protocol: TCP=0, UDP=1, ICMP=2, HTTP=3; One-hot encoding: suitable for unordered classification, avoiding numerical bias, creating a binary vector for each classification, for example: tracking operations: login = [1, 0, 0, 0], browsing = [0, 1, 0, 0], download = [0, 0, 1, 0], payment = [0, 0, 0, 1]); based on a preset detection period and real-time sampling frequency, each parameter is sampled into a fixed-length sequence to obtain the network parameter sequence, task parameter sequence, and behavior parameter sequence;

[0058] Specifically, by using preset first and second constraints, several parameter sequences are divided into several candidate groups, including:

[0059] Calculate the pairwise similarity between any two parameter sequences to form a similarity matrix; use a clustering algorithm to analyze the similarity matrix, automatically group the sequences, and obtain several preliminary groups, including network parameter sequence groups, task parameter sequence groups, and behavior parameter sequence groups;

[0060] For network parameter sequences, the similarity between corresponding sequences is calculated based on the DTW algorithm. All pairwise similarity values ​​form a corresponding similarity matrix, and density clustering algorithm is used for analysis. Based on the similarity values ​​in the matrix, network parameter sequences with a density that reaches a preset threshold are aggregated into a group, indicating that the similarity between the two is high, and a preliminary grouping of network parameter sequences is formed.

[0061] For task parameter sequences, the similarity between corresponding sequences is calculated based on the cosine similarity algorithm. Similarity matrices are formed by calculating the similarity between load sequences, port scan frequency sequences, and network connection frequency sequences using computational nodes. Hierarchical clustering is then employed, aggregating task parameter sequences that meet the clustering conditions based on trend consistency by calculating the distance between sequences, thus forming preliminary groupings of task parameter sequences. For behavior parameter sequences, the similarity between corresponding sequences is calculated based on the edit distance algorithm and the cosine similarity algorithm. Similarity matrices are formed by calculating the similarity between the tracking operation sequences and the log repetition rate sequences. Threshold similarity clustering is then used, comparing the similarity with a preset benchmark similarity (e.g., setting the benchmark to 0.6). Sequences with similarities greater than the preset benchmark are grouped together. Through this process, each type of parameter sequence first forms a structured similarity matrix through pairwise similarity calculation, and then automatically groups them using an adapted clustering algorithm. This results in preliminary groupings for network, task, and behavior parameter sequences, providing a foundation for subsequent integration of candidate groups.

[0062] The initial grouping is verified and reorganized based on the preset first and second constraints to form several candidate groups;

[0063] The first constraint includes: performing a Cartesian product on each preliminary group to verify and reorganize it, obtaining the first candidate group, and each combination is in the form of (a, b, c); where a is taken from at least one network parameter sequence group, b is taken from at least one task parameter sequence group, and c is taken from at least one behavior parameter sequence group; through the Cartesian product verification and reorganization of the first constraint, the candidate group is required to be constructed in the form of a triple of network parameter sequence a - task parameter sequence b - behavior parameter sequence c. In actual operation, this covers multi-dimensional chain anomaly scenarios such as network anomalies triggering task failures and task failures causing behavior anomalies, ensuring that all potential multi-dimensional anomaly association patterns are included in the candidate group, providing a more comprehensive data analysis foundation for subsequent models;

[0064] The second constraint includes: under the condition of satisfying the first constraint, a preset time window is used to identify the timestamps of anomalies in the first candidate group, a time deviation is set for the timestamps, and timestamps with time deviations less than the time deviation threshold are bundled to obtain the corresponding second candidate combinations; for example: identifying the abnormal timestamps of the first candidate group, and then bundling the timestamps with time deviations less than the time deviation threshold; for example: only classifying network anomaly t1, task anomaly t1+3s, and behavior anomaly t1+4s into the same second candidate group, while excluding irrelevant task anomalies of t1+30s; this process can accurately filter abnormal parameter sequences with strong temporal correlation, significantly reduce invalid combinations of time coincidences, and improve the candidate group The purity of anomaly correlations lays a high-quality data foundation for subsequent causal testing. Identifying anomalies can be aided by network models, such as random forest models, which is standard knowledge and will not be elaborated upon here. Furthermore, the parameters within a candidate group themselves provide context for the cause of the anomaly. In practice, this context allows for determining whether the anomaly is caused by a network attack or a resource failure, enhancing the interpretability of network security incidents and facilitating rapid root cause identification by operations personnel. Additionally, a parameter sequence pair is randomly selected from the second candidate group as a candidate causal pair, including the first and second sequences. This second candidate group undergoes secondary verification and recombination to obtain the final candidate group, which is then used for training the subsequent model.

[0065] The secondary verification and recombination process includes: Statistical matching: After each anomalous time point in the first sequence occurs, the number of successful matches is counted within a preset maximum lag time window when the second sequence also has an anomalous time point. Assuming the first sequence has 10 anomalous points and the maximum lag time window is set to 2 minutes, the second sequence also has an anomalous point within two minutes after 8 anomalous points, so the number of successful matches is 8; Generate reference: The anomalous time point sequence of the second sequence is randomly permuted multiple times. After each permutation, the above statistical matching steps are repeated to obtain a reference distribution of the number of random matches; Verification and judgment: If the statistical significance of the number of successful matches in the reference distribution is higher than the preset confidence level, then the candidate causal pair is judged to satisfy the second constraint, that is, the anomalous change in the first sequence significantly precedes the anomalous change in the second sequence in time.

[0066] Based on each candidate group, the degree of abnormal correlation of parameter sequences within the group is analyzed, the abnormal correlation score of each candidate group is calculated, and the candidate groups are divided into a first-level response pool and a second-level response pool based on the abnormal correlation score.

[0067] This includes analyzing the degree of abnormal correlation among parameters within the analysis group and calculating the abnormal correlation score for each candidate group, including:

[0068] Evaluation metrics were extracted based on candidate groups, including: number of successful matches and time deviation;

[0069] The abnormal match rate is obtained based on the number of successful matches, which is represented as the proportion of the number of successful matches of all parameter sequences at abnormal time points within the group to the total number of abnormal points. It should be noted that the higher this proportion is, the stronger the correlation between the abnormal time points of the two types of sequences, and it is the core indicator for measuring abnormal correlation.

[0070] The time deviation coefficient is obtained based on the time deviation, which is represented as the ratio of the average deviation value of the abnormal time point to the standard deviation threshold. It should be noted that the smaller the coefficient, the better the synchronization of the abnormal time points of the two types of parameter sequences. To convert it into a positive index that the larger the value, the better the synchronization, it is necessary to calculate 1 minus the time deviation coefficient, which is used to help verify the time authenticity of the abnormal association.

[0071] By performing a transformation analysis on the time series deviation coefficient, a positive value of the time series deviation is obtained. Combined with the anomaly matching rate, a weighted summation is used to obtain the anomaly association score, which comprehensively reflects the degree of anomaly association between the parameter sequences within the candidate group. A higher score indicates a more significant anomaly association between the two types of sequences, reflecting both high matching of anomalies and high temporal synchronicity; conversely, a lower anomaly association score indicates a lower degree of anomaly association between the two types of sequences. Specifically, the positive value of the time series deviation = 1 - the time series deviation coefficient; therefore, the anomaly association score = anomaly matching rate × first weight + (1 - time series deviation coefficient) × second weight. It should be noted that... It is clear that all parameters involved have been dimensionless before the formula calculation to ensure that the formula has physical meaning. The first and second weights are dynamic weights, automatically assigned by the system. The acquisition process is as follows: Historical statistics: Collect raw data on abnormal associations at historical anomaly time points, including the number of successful matches for each candidate group, the total number of anomalies, the average deviation value at anomaly time points, and the standard deviation threshold. Calculate the abnormal match rate and positive time-series deviation value for each candidate group in the historical dataset. Label sorting: Based on expert experience rules, assign a true association validity label to each historical candidate group, setting it to 0-1. The interval is defined as follows: 1 represents a completely true association, and 0 represents a random association. All historical candidate groups are sorted from low to high according to the true association validity label to form a historical association dataset. Every three digits, the historical association dataset is evaluated, and the resulting dataset is used as the test set. Model training: A lightweight fully connected neural network is preset as the weight learning model. The model input layer has two feature nodes, corresponding to the abnormal matching rate and the positive value of temporal deviation, respectively. There are two hidden layers, each with 16 neurons, both using the ReLU activation function. The output layer has two weight nodes, corresponding to the dynamic weights of the two indicators, denoted as the first weight and the second weight, and the sum of the first weight and the second weight is 1. At the same time, by learning the mapping relationship between the abnormal matching rate, the positive value of temporal deviation, and the true association validity label, the first weight and the second weight are adjusted to minimize the error between the calculated result of abnormal matching rate × first weight + positive value of temporal deviation × second weight and the true association validity label.

[0072] Based on the anomaly correlation score, candidate groups are divided into a primary response pool and a secondary response pool, including:

[0073] Set a minimum activation threshold to filter out candidate groups whose abnormal association scores are lower than the minimum activation threshold; for those scores below the minimum activation threshold, continue network perception.

[0074] Calculate the first quantile of the abnormal association scores for the remaining candidate groups;

[0075] Candidate groups with anomaly correlation scores less than or equal to the first quantile are assigned to the first-level response pool;

[0076] Candidate groups with anomaly correlation scores greater than the first quantile are assigned to the secondary response pool; the secondary anomaly response pool has a higher priority than the primary anomaly response pool.

[0077] In this embodiment, the system resource status is monitored in real time, and the required detection resources are allocated to the candidate groups in the secondary response pool first. If the system detection resources are insufficient, the resources occupied by the primary anomaly response pool tasks are suspended or reduced and reallocated to the secondary response pool. Specifically, this includes: setting checkpoints for the tasks in the primary response pool and saving their current state, then interrupting their execution to release resources; after the secondary response pool tasks have finished executing and released resources, the released resources are automatically used to resume the execution of the interrupted primary response pool tasks from the checkpoint; triggering anomaly response strategies for the candidate groups in the secondary anomaly response pool, and allocating remaining resources to the candidate groups in the primary anomaly response pool to execute interruptible deep perception evaluation tasks.

[0078] For candidate groups in the primary response pool, if the detection resources for each candidate group in the secondary response pool are determined, the remaining detection resources are selected to trigger the perception evaluation task based on the anomaly association score.

[0079] Based on the anomaly correlation score, select the remaining detection resources to trigger the perception assessment task, including:

[0080] After the detection resource allocation for all candidate groups in the secondary response pool is determined, for all candidate groups in the primary response pool, the remaining detection resources are used to extract the abnormal patterns of the group's parameter sequences, including network parameter sequences, task parameter sequences, and behavioral parameter sequences, and these are classified and labeled as network-related, attack-related, and behavioral-related. Network-related anomalies manifest as traffic fluctuations, protocol anomalies, and IP scanning anomalies; attack-related anomalies typically manifest as port intrusions, brute-force attacks, and malicious code injection; and behavioral-related anomalies typically manifest as sudden increases in CPU load, memory overflows, and service interruptions, ensuring that each candidate group corresponds to at least one abnormal pattern. It should be noted that after the resource allocation for high-priority tasks is determined, the remaining detection resources are often idle. In the environment of remaining detection resources, a perception and evaluation task is triggered. Furthermore, when the remaining detection resources are exhausted, the detection resources of the primary response pool are used to further trigger a perception and evaluation task. In this embodiment, the remaining detection resources are represented by the number of traffic filtering threads, idle port management computing power, and unused routing isolation bandwidth.

[0081] The first triplet is formed based on the anomaly pattern, anomaly association score, and anomaly duration, and is labeled as the range analysis group; the second triplet is formed based on the anomaly pattern, anomaly association score, and the number of anomaly parameter sequences, and is labeled as the intensity analysis group; where the anomaly duration is the absolute value of the difference between the start time of the anomaly and the end time of the anomaly, for example: if the anomaly lasts from 10:00:00 to 10:05:30, then the duration is 330 seconds;

[0082] The range analysis space and intensity analysis space are constructed by the range analysis group and intensity analysis group that show abnormal patterns in the current measurement period, respectively; the standard score of the range analysis group in the range analysis space in the current measurement period is marked as the first evaluation score; the standard score of the intensity analysis group in the intensity analysis space in the current measurement period is marked as the second evaluation score;

[0083] For example: In the range analysis space of anomaly patterns, extract the mean and standard deviation of the anomaly duration of historical range analysis groups, and label them as μ1 and σ1, respectively; in the intensity analysis space of anomaly patterns, extract the mean and standard deviation of the number of anomaly parameter sequences of historical intensity analysis groups, and label them as μ2 and σ2, respectively; then the first evaluation score = (current candidate group anomaly duration - μ1) / σ1, which is equivalent to the range standard score; the second evaluation score = (current candidate group anomaly parameter sequence number - μ2) / σ2, the second evaluation score, is equivalent to the intensity standard score; the first and second evaluation scores are weighted and summed to obtain the comprehensive evaluation index; the comprehensive evaluation index is sorted in batches according to the comprehensive evaluation index, and compared with the preset first threshold interval, second threshold interval, and third threshold interval, and matched with the preset resource allocation rules: if the comprehensive evaluation index is within the first threshold interval, the comprehensive evaluation index at this time is marked as n1, and assigned the character, and n1 is combined with the first-level character to generate the first-level scheduling level, judging that the abnormal risk level is low, and only basic detection resources are allocated, such as: regular traffic scanning; if the comprehensive evaluation index is within the second threshold interval, the comprehensive evaluation index at this time is marked as n2, and assigned the character, and n2 is combined with the second-level character to generate the second-level scheduling level, judging that the abnormal The risk level is medium, and an enhanced detection strategy is assigned. If the comprehensive evaluation index is within the third threshold interval, the comprehensive evaluation index at this time is marked as n3 and assigned the character. The n3 is combined with the third-level character to generate a three-level scheduling degree. If the abnormal risk level is high, the corresponding candidate group is aggregated into the second-level response pool. The first threshold interval is less than the second threshold interval, which is less than the third threshold interval. It should be noted that the first threshold interval, the second threshold interval, and the third threshold interval are based on the statistical range of the average value pj and the standard deviation bz of the comprehensive evaluation index. Usually, the first threshold interval is set to [pj-2bz, pj-bz), the second threshold interval is set to [pj-bz, pj+3bz), and the third threshold interval is set to [pj+3bz, +∞). The specific values ​​are defined according to the actual situation.

[0084] By classifying anomaly patterns and constructing dual triples, the anomaly perception of the primary pool candidate groups is decomposed into anomaly pattern types, namely network type, attack type, and behavior type. Range dimension analysis is performed through anomaly association score and anomaly duration, and intensity dimension analysis is performed through anomaly association score and the number of anomaly parameter sequences, forming a complete anomaly perception profile. Furthermore, scheduling indicators are evaluated through range analysis space and intensity analysis space to achieve cross-pattern and cross-cycle comparability. Batch sorting by comprehensive index and batch matching of resource rules improve batch processing efficiency and adapt to large-scale primary pool scenarios.

[0085] For candidate groups in the secondary response pool, execute the exception response strategy;

[0086] Implement exception response strategies, including:

[0087] The detection resources in the secondary response pool are classified and quantized to obtain the resource vector R: R=[r1, r2, ..., r m ], where m is the number of resource types to be detected, including the number of traffic filtering threads, port status management computing power, and IP routing isolation bandwidth, r i Let i represent the quantization value of the i-th type of detection resource, where i ranges from [1, m].

[0088] The sample set is aggregated into the secondary response pool and all candidate groups in the secondary response pool. An anomaly pattern for the next N time steps is trained using a pre-defined perception detection model and used as sample annotation. During training, the minimum resource requirement of the quantized resource vector is used as the objective function to obtain the initial policy code. The policy code is iteratively optimized using a particle swarm optimization algorithm, and the policy code with the highest fitness is output as the anomaly response policy. The minimum resource requirement is defined as minimizing the number of traffic filtering threads, minimizing the computational power for port status management, and minimizing the bandwidth for IP routing isolation. The anomaly response policy is represented by the traffic interception rules, port blocking duration, and IP isolation policy for the next N time steps.

[0089] The particle swarm optimization algorithm is used for iterative policy encoding, including: initializing the policy encoding, which is a real-number vector whose dimension is determined by the number of candidate groups in the secondary pool and the resource type, representing the abnormal response schemes for each candidate group; the position of each particle in the particle swarm is then a policy encoding; the quality of the policy encoding is calculated, i.e., the fitness of the particle swarm is calculated, and the velocity and position of the particles are continuously updated based on their positions and fitness, simulating the process of the swarm finding the optimal solution; in each iteration, each particle is extrapolated through a perception detection model to predict its future effect after execution and obtain a fitness; after multiple iterations, the particle swarm will converge to the optimal solution. The algorithm terminates at the position with the highest response, when the maximum number of iterations is reached or the quality of the solution no longer significantly improves. During training, the number of future time points N = future time period / future time interval, where the future time period, future time interval, preset time period, preset time interval, and the number of individuals in the initial population are all user-defined parameters. For example, if the preset time period and future time period are both set to 3 hours, and the future time interval and preset time interval are both set to 5 minutes, then the number of future time points N = 18, and the number of individuals in the initial population is set to 50. It should be noted that the specific numbers involved are only illustrative examples, and the specific parameters should be set according to the actual situation.

[0090] Example 2:

[0091] This invention provides a network anomaly detection system based on big data; Figure 2This is a schematic diagram of a network anomaly detection system according to an exemplary embodiment; please refer to [link / reference]. Figure 2 The system includes: a sequence construction module, an exception association module, and a policy execution module, and the sequence construction module, the exception association module, and the policy execution module are connected to each other.

[0092] The sequence construction module is used to collect network parameters, task parameters, and behavior parameters in real time, and normalize them to construct network parameter sequences, task parameter sequences, and behavior parameter sequences. Through preset first and second constraints, several parameter sequences are divided into several candidate groups.

[0093] The anomaly correlation module is used to analyze the degree of anomaly correlation of parameter sequences within each candidate group, calculate the anomaly correlation score of each candidate group, and divide the candidate groups into a primary response pool and a secondary response pool based on the anomaly correlation score.

[0094] The strategy execution module, for candidate groups in the primary response pool, if the detection resources for each candidate group in the secondary response pool are determined, combines the anomaly correlation score to select the remaining detection resources to trigger the perception evaluation task; for candidate groups in the secondary response pool, it executes the anomaly response strategy.

[0095] The setting of the interval or threshold size is for the purpose of facilitating comparison. The size of the threshold depends on the amount of sample data and the number of bases set by those skilled in the art for each set of sample data, as long as it does not affect the ratio between the parameter and the quantized value.

[0096] In the application, the various formulas mentioned are all calculated by removing dimensions and taking their numerical values. The formulas are derived from the most recent real-world situation by collecting a large amount of data and simulating it with software. The formulas are set by those skilled in the art according to the actual situation.

[0097] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0098] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0099] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A network anomaly perception detection method based on big data, characterized in that, The method comprises: Real-time acquisition of network parameters, task parameters and behavior parameters, and normalization to construct network parameter sequences, task parameter sequences and behavior parameter sequences, and division of a plurality of parameter sequences into a plurality of candidate groups through preset first and second constraints; Based on each candidate group, the abnormal association degree of the parameter sequence in the group is analyzed, the abnormal association score of each candidate group is calculated, and the candidate groups are divided into a first response pool and a second response pool based on the abnormal association score; Wherein, analyzing the abnormal association degree of the parameters in the group and calculating the abnormal association score of each candidate group comprises: Based on the candidate group, evaluation indexes are extracted, including the number of successful matches and time deviation; Based on the number of successful matches, an abnormal matching rate is obtained; based on the time deviation, a time sequence deviation coefficient is obtained; Through conversion analysis of the time sequence deviation coefficient, a time sequence deviation positive value is obtained, and combined with the abnormal matching rate, the abnormal association score is obtained through weighted summation; For the candidate groups in the first response pool, when the detection resources of each candidate group in the second response pool are determined, the remaining detection resources are selected to trigger the perception evaluation task based on the abnormal association score; for the candidate groups in the second response pool, an abnormal response strategy is executed; Wherein, selecting the remaining detection resources to trigger the perception evaluation task based on the abnormal association score comprises: After the detection resource allocation of all candidate groups in the second response pool is determined, for all candidate groups in the first response pool, the abnormal mode of the parameter sequence in the group is extracted using the remaining detection resources, and the abnormal mode is at least one of network type, attack type or behavior type; Based on the abnormal mode, the abnormal association score and the abnormal duration, a first triple is formed and marked as a range analysis group; based on the abnormal mode, the abnormal association score and the number of abnormal parameter sequences, a second triple is formed and marked as an intensity analysis group; the range analysis group and the intensity analysis group that appear in the current calculation period form a range analysis space and an intensity analysis space, respectively; The standard score of the range analysis group in the current calculation period in the range analysis space is marked as the first evaluation score; The standard score of the intensity analysis group in the current calculation period in the intensity analysis space is marked as the second evaluation score; The first evaluation score and the second evaluation score are weighted and summed to obtain a comprehensive evaluation index; the comprehensive evaluation index is batch sorted to determine the network scheduling degree, and the network scheduling degree is matched with the preset resource allocation rule, and the resource allocation rule at least includes: if it is determined to be a third scheduling degree, the corresponding candidate groups are aggregated into the second response pool; Wherein, the abnormal response strategy comprises: The detection resource classification quantization of the secondary response pool obtains a resource vector R: R=[r1, r2, …, rm], wherein m is the type number of the detection resource, including the number of traffic filtering threads, the port state control algorithm and the IP routing isolation bandwidth, r m i represents the quantization value of the ith type of detection resource, and i ranges from 1 to m. i ​ All candidate groups aggregated into the second response pool and the second response pool are taken as a sample set, an abnormal mode of future N time points is trained and output as a sample label through a preset perception detection model; in the training process, the minimum resource demand of the quantitative resource vector is taken as the objective function to obtain the initialized strategy code; the particle swarm optimization algorithm is used to iteratively optimize the strategy code, and the strategy code with the highest fitness is output as the abnormal response strategy; wherein, the minimum resource demand is the minimization of the number of traffic filtering threads, the minimization of the algorithm power for port state control, and the minimization of the bandwidth for IP routing isolation.

2. The big data based network anomaly perception detection method according to claim 1, characterized in that, The network parameters include data traffic, transmission protocol, source IP, and target IP; the task parameters include node computing load, port scanning frequency, and network connection frequency; and the behavior parameters include a burying point operation and log statistics. 3.The big data based network anomaly perception detection method of claim 1, wherein, The several parameter sequences are divided into several candidate groups through the preset first constraint and second constraint, including: The similarity between any two parameter sequences is calculated to form a similarity matrix; a clustering algorithm is used to analyze the similarity matrix to automatically group the sequences to obtain several preliminary groups, including a network parameter sequence group, a task parameter sequence group, and a behavior parameter sequence group; The preliminary groups are checked and reorganized based on the first constraint and the second constraint to form several candidate groups; The first constraint is used to perform one check and reorganization on each preliminary group to obtain a first candidate group, and the second constraint is used to set a time window under the condition of meeting the first constraint, identify time stamps at which the first candidate group appears abnormally, set a time deviation for the time stamps, bundle the time stamps with a time deviation less than a time deviation threshold, and obtain a corresponding second candidate group; meanwhile, any pair of parameter sequences in the second candidate group is selected as a candidate causal pair, including a first sequence and a second sequence, the second candidate group is checked and reorganized twice to obtain a final candidate group.

4. The big data based network anomaly perception detection method of claim 3, wherein, The process of the twice check and reorganization further includes: Statistical matching: After each abnormal time point of the first sequence appears, the number of successful matches of abnormal time points of the second sequence in a preset maximum lag time window is counted; Reference generation: The abnormal time point sequence of the second sequence is randomly permuted multiple times, the statistical matching step is repeated after each permutation to obtain a reference distribution of the number of random matches; Test determination: If the number of successful matches is higher than a preset confidence level in the reference distribution, it is determined that the candidate causal pair meets the second constraint.

5. The big data based network anomaly perception detection method of claim 1, wherein, The determination of the network scheduling degree includes: comparing the comprehensive evaluation index with a preset first threshold interval, a second threshold interval, and a third threshold interval: If the comprehensive evaluation index is in the first threshold interval, it is determined as a first-level network scheduling degree; If the comprehensive evaluation index is in the second threshold interval, it is determined as a first-level network scheduling degree; If the comprehensive evaluation index is in the third threshold interval, it is determined as a third-level network scheduling degree.

6. The big data based network anomaly perception detection method of claim 1, wherein, The abnormal response strategy is represented by a traffic interception rule, a port ban duration, and an IP isolation strategy at N future time points. 7.A big data based network anomaly perception detection system, characterized in that, The system includes: A sequence construction module, which is used to collect network parameters, task parameters, and behavior parameters in real time, normalize and construct network parameter sequences, task parameter sequences, and behavior parameter sequences, divide several parameter sequences into several candidate groups through preset first and second constraints; An abnormal association module, which is used to analyze the abnormal association degree of parameter sequences in each candidate group, calculate the abnormal association score of each candidate group, and divide the candidate groups into a first-level response pool and a second-level response pool based on the abnormal association score; The analysis of the abnormal association degree of the parameters in the group and the calculation of the abnormal association score of each candidate group include: The evaluation indexes based on the candidate groups include the number of successful matches and time deviation; The abnormal matching rate is obtained based on the number of successful matches, and the time deviation coefficient is obtained based on the time deviation; The time deviation coefficient is converted and analyzed to obtain a positive value of the time deviation, and the abnormal correlation score is obtained by combining the abnormal matching rate and weighted summation; The strategy execution module selects the remaining detection resources to trigger the perception evaluation task in combination with the abnormal correlation score for the candidate groups in the first response pool when the detection resources of each candidate group in the second response pool are determined, and executes the abnormal response strategy for the candidate groups in the second response pool. The selection of the remaining detection resources to trigger the perception evaluation task in combination with the abnormal correlation score includes: After the detection resource allocation of all candidate groups in the second response pool is determined, the abnormal mode of the parameter sequence in each group is extracted using the remaining detection resources for all candidate groups in the first response pool, and the abnormal mode represents at least one of network type, attack type or behavior type. The first triplet is formed based on the abnormal mode, the abnormal correlation score and the abnormal duration, and is marked as a range analysis group; the second triplet is formed based on the abnormal mode, the abnormal correlation score and the number of abnormal parameter sequences, and is marked as an intensity analysis group; the range analysis space and the intensity analysis space are formed by the range analysis group and the intensity analysis group in which the abnormal mode appears in the current measurement period, respectively. The standard score of the range analysis group in the current measurement period in the range analysis space is marked as the first evaluation score. The standard score of the intensity analysis group in the current measurement period in the intensity analysis space is marked as the second evaluation score. The first evaluation score and the second evaluation score are weighted and summed to obtain a comprehensive evaluation index; the network scheduling degree is determined according to the batch ordering of the comprehensive evaluation index, and the resource allocation rule is matched based on the network scheduling degree, and the resource allocation rule at least includes: if the third scheduling degree is determined, the corresponding candidate groups are aggregated into the second response pool. The execution of the abnormal response strategy includes: The detection resources in the secondary response pool are classified and quantized to obtain the resource vector R: R=[r1, r2, ..., r m ], where m is the number of resource types to be detected, including the number of traffic filtering threads, port status management computing power, and IP routing isolation bandwidth, r i Let i represent the quantization value of the i-th type of detection resource, where i ranges from [1, m]. The candidate groups aggregated into the second response pool and all candidate groups in the second response pool are taken as a sample set, and the abnormal mode of the future N time points is trained and output as a sample label through a preset perception detection model; in the training process, the minimum resource requirement of the quantitative resource vector is taken as the objective function to obtain the initialized strategy code; the particle swarm optimization algorithm is used to iteratively optimize the strategy code, and the strategy code with the highest fitness is output as the abnormal response strategy; wherein, the minimum resource requirement is to minimize the number of traffic filtering threads, minimize the algorithm power of port state control, and minimize the bandwidth of IP routing isolation.

Images