Cybersecurity-oriented monitoring data cloud processing method and system

Abstract

The present application relates to the technical field of network security monitoring, in particular, the present application relates to a network security-oriented monitoring data cloud processing method and system, aiming at solving the problem of waste of computing power and storage and response lag caused by full analysis and storage of massive alarms in traditional cloud processing, the present application completes complete redundant deduplication on alarms with completely consistent alarm type, terminal IP and timestamp through a hash algorithm, generates a primary feature fingerprint with terminal IP and alarm type, obtains a linear correlation coefficient as a microscopic fluctuation feature by fitting parameter fluctuation and timestamp difference, generates a secondary microscopic feature fingerprint by combination, determines the similar redundant alarm cluster by fingerprint similarity comparison, adopts hierarchical processing, selects representative core data for full threat analysis, and only extracts key features for lightweight storage of the rest of the alarms, reduces cloud resource consumption, and improves network security alarm processing efficiency.

Description

Technical Field

[0001] This invention relates to the field of network security monitoring technology, and more specifically, to a cloud processing method and system for monitoring data for network security. Background Technology

[0002] Network security monitoring technology is an important technology, specifically applied to the real-time cloud processing of network security monitoring alarm streams. Its core is to improve cloud computing and storage efficiency through redundancy filtering and hierarchical processing, meeting the real-time and cost-effective requirements of large-scale network security monitoring. During operation, network security monitoring equipment continuously generates massive amounts of similar and nearly identical alarm data. This data contains a large amount of completely duplicated and highly similar redundant information. Because traditional cloud processing methods perform full-volume in-depth analysis and complete storage of all alarm data, cloud computing resources and storage space are inefficiently occupied, leading to increased processing latency, decreased response speed, and increased maintenance costs, making it difficult to support efficient and stable processing of network security monitoring alarms. To solve this technical problem, we provide a cloud processing method and system for network security monitoring data. Summary of the Invention

[0003] The purpose of this invention is to provide a cloud processing method and system for monitoring data for network security, so as to solve the problems mentioned in the background art.

[0004] To achieve the above objectives, one objective of this invention is to provide a cloud processing method for monitoring data for network security, comprising the following steps: S1. Receive the original network security monitoring alarm stream and use a hash algorithm to perform full redundancy filtering and deduplication on alarm data with completely identical alarm type, terminal IP and timestamp, and output the alarm dataset after preliminary filtering. S2. Perform redundancy determination on the alarm data in the pre-filtered alarm dataset, combine the terminal IP and alarm type in the current alarm data to generate a first-level feature fingerprint for identifying the macro source and category, extract multiple alarm data under the same first-level feature fingerprint, obtain the alarm parameters and timestamps containing specific values ​​in the multiple alarm data respectively, calculate the absolute difference between the corresponding alarm parameters between any two alarm data as the parameter fluctuation, and calculate the absolute difference between the timestamps between the two alarm data as the timestamp difference. Finally, use the least squares method to fit the parameter fluctuation and the timestamp difference, calculate the linear correlation coefficient, use the linear correlation coefficient as the micro fluctuation feature value, and combine the first-level feature fingerprint with the micro fluctuation feature value to generate a second-level micro feature fingerprint that characterizes the macro consistency and micro fluctuation pattern of the alarm data. S3. Perform pairwise similarity comparison on all secondary micro-feature fingerprints. Specifically, compare the closeness of the micro-fluctuation feature values ​​and the similarity of the primary feature fingerprints. When the overall similarity of the secondary micro-feature fingerprints of two alarm data is greater than or equal to a preset threshold, it is determined that the two alarm data belong to the same redundant alarm data cluster. S4. Perform hierarchical processing on each identified redundant alarm data cluster. From the same redundant alarm data cluster, arbitrarily select one alarm data as representative core data and input it into the backend full threat analysis module for complete in-depth analysis and response decision-making. At the same time, for each redundant alarm data in the redundant alarm data cluster other than the representative core data, bypass the full threat analysis module and extract the alarm type, terminal IP, key alarm parameters and occurrence timestamp through a lightweight feature extraction routine. Store the set in a dedicated cache database to avoid repeated full analysis calculations.

[0005] A second objective of this invention is to provide a system for implementing the cloud processing method for network security monitoring data, including any one of the above-described methods, comprising: The data receiving and fully redundant filtering unit is used to receive the original network security monitoring alarm stream and use a hash algorithm to filter and deduplicate alarm data that are completely consistent in alarm type, terminal IP and timestamp, and output the alarm dataset after preliminary filtering. The feature extraction and fingerprint generation unit is used to concatenate the terminal IP and alarm type in the alarm data of the pre-filtered alarm dataset to generate a first-level feature fingerprint, group the alarm data according to the first-level feature fingerprint, calculate the parameter fluctuation and timestamp difference for the alarm data in the same group, apply the least squares method to fit the parameter fluctuation and timestamp difference to obtain the linear correlation coefficient as the micro fluctuation feature value, and finally combine the first-level feature fingerprint and the micro fluctuation feature value to generate a second-level micro feature fingerprint. The redundancy determination unit is used to compare the similarity of the generated secondary micro-feature fingerprints. By comparing the similarity of the primary feature fingerprints and the closeness of the micro-fluctuation feature values, it determines whether the alarm data belongs to the same redundancy alarm data cluster based on a preset threshold. The hierarchical processing unit is used to select one alarm data from the cluster as a representative sub-core data according to preset rules and send it to the full threat analysis module for in-depth analysis. At the same time, for each other alarm data in the cluster, the feature extraction routine is started to extract the set of key feature fields and store the set of key feature fields in a dedicated cache database that is independent of the storage system of the full threat analysis module.

[0006] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention uses a hash algorithm to quickly deduplicat fully redundant alarms with identical alarm types, terminal IPs, and timestamps, reducing invalid data at the source. It aggregates alarms of the same source and type based on primary feature fingerprints and constructs secondary micro-feature fingerprints by combining the linear correlation coefficient between parameter fluctuations and timestamp differences. This accurately identifies clusters of highly similar redundant alarms, enabling refined judgment of redundant alarms. In cloud processing, a hierarchical processing mechanism is employed, selecting the earliest representative alarm within the cluster for full-scale in-depth threat analysis. Other similar alarms only have their key features extracted and are lightweighted and stored in a dedicated cache, bypassing repetitive full-scale analysis calculations. This significantly reduces cloud computing power consumption and storage resource usage, improving alarm processing speed and data processing efficiency. Attached Figure Description

[0007] Figure 1 This is a flowchart illustrating the overall workflow of the present invention; Figure 2 This is a schematic diagram of the overall structure of the present invention; Figure 3 Flowchart for complete redundancy removal of network security monitoring data; Figure 4 This is a comparison chart showing the effects of traditional cloud processing and the monitoring data processing of this invention.

[0008] The meanings of the labels in the diagram are as follows: 1. Data receiving and full redundancy filtering unit; 2. Feature extraction and fingerprint generation unit; 3. Class redundancy determination unit; 4. Hierarchical processing unit. Detailed Implementation

[0009] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0010] Please see Figure 1 As shown, one of the objectives of this embodiment is to provide a cloud processing method for monitoring data for network security, including the following steps: S1. Receive the original network security monitoring alarm stream and use a hash algorithm to perform full redundancy filtering and deduplication on alarm data with completely identical alarm type, terminal IP and timestamp, and output the alarm dataset after preliminary filtering. S2. Perform redundancy judgment on the alarm data in the initially filtered alarm dataset. Combine the terminal IP and alarm type in the current alarm data to generate a first-level feature fingerprint for identifying the macro source and category. Extract multiple alarm data under the same first-level feature fingerprint. Obtain the alarm parameters and timestamps containing specific values ​​in the multiple alarm data. Calculate the absolute difference between the corresponding alarm parameters between any two alarm data as the parameter fluctuation. Calculate the absolute difference between the timestamps between two alarm data as the timestamp difference. Finally, use the least squares method to fit the parameter fluctuation and timestamp difference to calculate the linear correlation coefficient. Use the linear correlation coefficient as the micro fluctuation feature value. Combine the first-level feature fingerprint with the micro fluctuation feature value to generate a second-level micro feature fingerprint that characterizes the macro consistency and micro fluctuation pattern of the alarm data. S3. Perform pairwise similarity comparison on all secondary micro-feature fingerprints. Specifically, compare the closeness of micro-fluctuation feature values ​​and the similarity of primary feature fingerprints. When the overall similarity of the secondary micro-feature fingerprints of two alarm data is greater than or equal to the preset threshold, it is determined that the two alarm data belong to the same redundant alarm data cluster. S4. Perform hierarchical processing on each identified redundant alarm data cluster. From the same redundant alarm data cluster, arbitrarily select one alarm data as representative core data and input it into the backend full threat analysis module for complete in-depth analysis and response decision-making. At the same time, for each redundant alarm data in the redundant alarm data cluster other than the representative core data, bypass the full threat analysis module and extract the alarm type, terminal IP, key alarm parameters and occurrence timestamp through a lightweight feature extraction routine. Store the set in a dedicated cache database to avoid repeated full analysis calculations.

[0011] Using a hash algorithm, complete redundancy filtering and deduplication are performed on alarm data with identical alarm types, terminal IPs, and timestamps. Specifically, this includes: Based on the alarm type string, terminal IP address string, and timestamp string contained in each received original alarm data, a first hash key is generated by concatenating the strings according to a preset first string concatenation order. The first hash key is then input into a preset hash table for querying. If the same first hash key already exists in the hash table, the current alarm data is determined to be completely redundant and is discarded directly. If the same first hash key does not exist in the hash table, the current alarm data is added to the pre-filtered alarm dataset, and the first hash key is inserted into the hash table, thus completing the filtering and deduplication of alarm data with completely identical alarm type, terminal IP, and timestamp.

[0012] After initiating the cloud processing flow for network security monitoring data, the system first continuously receives raw network security monitoring alarm streams reported by various network security monitoring devices. To quickly eliminate completely duplicate and invalid alarm data and reduce the subsequent computing and storage load on the cloud, a hash algorithm is then used to perform complete redundancy filtering and deduplication. The hash algorithm is an efficient algorithm that maps an input of arbitrary length to a fixed-length hash value through a hash function. It features fast query speed, low collision probability, and small storage footprint, and is the core technology for achieving rapid alarm deduplication. The specific implementation method is as follows: Please see Figure 3 As shown, each received raw alarm data is parsed one by one, and three core unique identifier fields are precisely extracted: an alarm type string representing the threat category, a terminal IP address string identifying the terminal that triggered the alarm, and a timestamp string recording the precise time the alarm occurred. Then, according to a predefined first string concatenation order, these three strings are concatenated in a fixed order to form a unique first hash key corresponding to this alarm. In the fully redundant deduplication process of cloud-based monitoring data processing for network security, the first string concatenation order is a pre-fixed, globally unified, and irreversible standardized string concatenation rule. Its core design goal is to ensure that when the alarm type, terminal IP, and timestamp fields are exactly the same, the concatenated strings are completely identical; if any field is different, the concatenated strings will inevitably be different. At the same time, a dedicated separator is used to avoid hash key conflicts caused by character concatenation between fields. This fixed concatenation order is strictly defined as follows: Alarm type string → Dedicated unambiguous separator → Terminal IP address string → Dedicated unambiguous separator → Timestamp string. The dedicated separator must be a special character that will never appear in the alarm data to prevent abnormal concatenation results caused by concatenated characters in the field content. The specific concatenation process is as follows: First, remove leading and trailing whitespace, newline characters, and tabs from the alarm type string, terminal IP address string, and timestamp string to ensure absolute consistency in field format. Then, arrange the three preprocessed core fields sequentially, inserting dedicated delimiters between adjacent fields, and concatenate the three independent strings into a single, unambiguous concatenated string. The fourth step is hash conversion: input the concatenated string into a preset hash algorithm to generate the first hash key used for deduplication verification. To more intuitively understand the concatenation rules and process, an example using actual network security alarm data is provided. Assume the core fields of a certain original alarm data are: The alarm type string is "Brute Force Attack", the terminal IP address string is "10.0.0.128", and the timestamp string is "2024-03-25 16:45:22.789". Following the concatenation order and rules of the first string, after preprocessing, the concatenation operation is performed directly, resulting in the complete concatenated string "Brute Force Attack|10.0.0.128|2024-03-25 16:45:22.789". This string is then hashed to obtain a unique first hash key. If the three items of another alarm data... If the heart field is completely identical to this entry, the concatenated string and hash key will be exactly the same, and the system will determine it as completely redundant data and discard it. If any field is different, for example, if the terminal IP changes to "10.0.0.129", the concatenated string will be "Brute-force attack|10.0.0.129|2024-03-2516:45:22.789", and the first hash key generated will definitely be different. The system will determine it as a valid alarm and add it to the preliminary filtering dataset. This fixed connection order and standardized concatenation process ensures the determination of completely redundant alarms from the source. The accuracy of the hash key is the core foundation for the reliable operation of the hash deduplication process. The hash key can accurately mark completely duplicate alarms that are "the same terminal, the same type, and the same time". The system maintains a hash table in advance to record processed alarms. The hash table is an efficient data structure that stores data in key-value pairs and supports fast query with O(1) time complexity. It is specifically used to store the first hash key that has passed the deduplication verification. After the first hash key is generated, the system uses it as the query key to input the hash table to perform matching retrieval. If the query result shows that there is already a completely identical first hash key in the hash table, the current alarm is directly determined to be completely redundant data with the same alarm type, terminal IP and timestamp. The alarm is immediately discarded and does not enter the subsequent processing stage. If the query result shows that there is no matching first hash key in the hash table, the current alarm is determined to be valid data that is not completely redundant. It is added to the alarm dataset after preliminary filtering. At the same time, the first hash key corresponding to the alarm is inserted into the hash table to complete the retention mark for subsequent alarm duplication verification. This completes the filtering and deduplication of completely redundant data in the original alarm stream and outputs a pre-filtered alarm dataset without completely duplicate data.

[0013] In the initially filtered alarm dataset, the alarm type field and the terminal IP address field are read. The string of the alarm type field and the string of the terminal IP address field are concatenated according to a preset second string concatenation order, which is different from the first string concatenation order, to generate a composite string, which serves as a first-level feature fingerprint. This is used to filter out a subset of alarm data with the same alarm source and type.

[0014] Further explanation is needed regarding the process of generating a first-level feature fingerprint and filtering an alarm subset after obtaining the initially filtered alarm dataset. The specific implementation method is as follows: Each valid alarm record in the initially filtered alarm dataset is read one by one. The field parsing engine extracts the alarm type string (representing the threat category) and the terminal IP address string (uniquely identifying the terminal that triggered the alarm). Then, these two fields are concatenated according to a predefined second string concatenation order, which is completely different from the first string concatenation order in the full redundancy deduplication stage. The field parsing engine is a pre-compiled, standardized parsing module adapted to mainstream network security alarm formats. Its specific extraction process is as follows: The pre-filtered single alarm data is loaded into the parsing engine's memory buffer in structured message format. The alarm data is stored in a standardized key-value pair format. Based on the built-in global field mapping rule library, the parsing engine traverses all key names in the alarm message. First, it matches the key name corresponding to the alarm type that represents the threat category. After locating the target key, it directly extracts the corresponding value as the original alarm type text. Then, it matches the key name corresponding to the terminal IP that identified the terminal where the alarm occurred. After locating the target key, it extracts the corresponding value as the original terminal IP text. Immediately after extraction, a text cleaning operation is performed, automatically removing leading and trailing whitespace characters, newline characters, tab characters, and escape characters from both original text segments. Invalid format characters, such as symbols, are uniformly converted into UTF-8 encoded plain text strings. Simultaneously, the extraction results are validated to ensure that the alarm type string is not empty and the terminal IP string conforms to the IPv4 / IPv6 format specification. Abnormal records that fail validation are directly discarded. Finally, clean and valid alarm type and terminal IP address field strings are obtained. After completing the core field extraction and cleaning, a concatenation operation is performed according to a pre-fixed second string concatenation order that is completely different from the first string concatenation order in the complete redundancy deduplication stage (alarm type string + | + terminal IP string + | + timestamp string). The fixed rule for the sequence is that the terminal IP address string comes first, followed by the alarm type field string. Adjacent fields are separated by a dedicated, unambiguous separator `#`, completely different from the first-stage separator `|`. This character will not appear in the standard values ​​of alarm type and terminal IP, completely avoiding concatenation conflicts. During concatenation, the preprocessed terminal IP address string, the dedicated separator `#`, and the alarm type field string are combined sequentially to form a continuous, unambiguous composite string. This composite string is the first-level feature fingerprint used for macro-clustering. For example, the extracted and cleaned terminal IP address string is `10.1.25.36`, and the alarm type field string... The string "Brute-force attack" is concatenated according to the second string concatenation order to obtain the first-level feature fingerprint "10.1.25.36#Brute-force attack". The first-level feature fingerprint generated by this fixed concatenation rule can accurately mark alarm groups with the same terminal and the same alarm type, providing a unique macro-identifier for the subsequent filtering and aggregation of alarm data subsets. This fingerprint can uniquely mark alarm groups with "the same terminal IP and the same alarm type" at the macro level. For example, if the terminal IP of an alarm is 172.16.0.5 and the alarm type is SQL injection attack, the first-level feature fingerprint obtained after concatenating the second string according to the second string concatenation order is 172.16.0.5# SQL injection attack: After generating the primary feature fingerprint of a single alarm, this fingerprint is used as the clustering basis to traverse the entire initially filtered alarm dataset. All alarm data with identical primary feature fingerprints are automatically grouped into the same analysis group. Finally, several alarm datasets containing only alarms of the same source and type are selected.

[0015] All alarm data with the same primary feature fingerprint are grouped into the same analysis group. For each alarm data in the same analysis group, its load information is parsed to extract alarm parameters with numerical features. Load information refers to the core business data segment in the alarm data packet that carries the specific quantitative characteristics of the alarm, the terminal operating status, and network traffic details, in addition to basic identifier fields such as alarm type, terminal IP, and timestamp. The system uses the built-in deep packet parsing module to disassemble the load information field by field, perform format verification and feature recognition, and accurately extract numerical alarm parameters with clear numerical attributes and quantifiable comparison. Alarm parameters include packet length, request frequency, traffic size, CPU utilization percentage and memory usage. At the same time, the timestamp in each alarm data is extracted. For any two different alarm data in the analysis group, the values ​​of the same alarm parameter are aligned and the absolute value of the difference between the two values ​​is calculated. The absolute value result is defined as the parameter fluctuation. Further explanation is needed. After completing the standardized extraction of parameters and timestamps, for any two distinct alarm data sets within the current analysis group, the values ​​of the same type of numerical alarm parameters corresponding to the two alarm data sets are aligned, using parameter type as the sole matching benchmark. This strictly adheres to the principle of comparison within the same category, achieving alignment of packet length with packet length, request frequency with request frequency, traffic size with traffic size, CPU utilization percentage with CPU utilization percentage, and memory usage value with memory usage value. This mechanism fundamentally prevents erroneous comparisons across parameter types. After parameter value alignment is completed, the system calculates the absolute value of the difference between the corresponding parameter values ​​of the two alarm data sets in real time, and formally defines this absolute value calculation result as the parameter fluctuation. Specifically, after completing the alignment of the same type of numerical alarm parameters for any two different alarm data sets within the same analysis group, the variables of the two alarm data sets involved in the calculation (defined as alarm data X and alarm data Y) and the aligned single target numerical parameter are standardized. Let the standardized value of the currently aligned parameter in alarm data X be... The standardized value of the same parameter in the alarm data Y is All parameters have been pre-converted to units and pure digital form, in integer or floating-point format, and can be directly used in arithmetic operations. The system then first calculates the difference between the original values ​​of this parameter for the two alarm data points. To eliminate the positive and negative differences caused by the order of numerical values ​​and retain only the fluctuation range characteristics of the parameters, the system performs an absolute value operation on the original differences. The final absolute value result is the parameter fluctuation amount corresponding to that parameter, and its standard calculation formula is as follows: In actual execution, the system will repeat the above calculation process for each of the five numerical alarm parameters in a fixed order: packet length, request frequency, traffic volume, CPU utilization percentage, and memory usage. It will never compare parameters across different types, ensuring that the fluctuation of each parameter accurately corresponds to the difference in a single characteristic. To ensure traceability of the calculation process, the system synchronously binds the parameter type name, the unique identifier of the two alarm data points, the calculation timestamp, and the final fluctuation value. For a more intuitive explanation of the calculation process, consider the following example using actual alarm data: the packet length parameter of two alarms within the same analysis group. Alarm X has a value of 1500 bytes, and Alarm Y has a value of 1496 bytes. Substituting these values ​​into the formula, we get the parameter fluctuation = |1500-1496| = 4. For the CPU utilization percentage parameter, Alarm X has a value of 45.2%, and Alarm Y has a value of 42.8%, so the parameter fluctuation = |45.2-42.8| = 2.4. For the memory usage parameter, Alarm X has a value of 8192MB, and Alarm Y has a value of 8192MB, so the parameter fluctuation = |8192-8192| = 0. Through this calculation process, we can obtain the full-dimensional parameter fluctuation of any two alarms within the same analysis group.

[0016] After calculating the parameter fluctuation and extracting the timestamp difference for any two alarm data points within the same analysis group, the system automatically aggregates multiple valid alarm data pairs within the same analysis group (completely redundant and single invalid alarm pairs have been removed). Simultaneously, it collects two core quantitative data points corresponding to each alarm data pair. The absolute value obtained by subtracting the standardized timestamps of the two alarms in the same alarm pair is defined as the timestamp difference, which serves as the independent variable for the fitting analysis. This value is uniformly converted to a second-level standard value to ensure dimensional consistency. The parameter fluctuation calculated for the same type of parameters in the same alarm pair is used as the dependent variable for the fitting analysis. The system strictly adheres to the principle of independent fitting for a single parameter, collecting corresponding independent and dependent variable sample sets for five types of numerical parameters: packet length, request frequency, traffic volume, CPU utilization percentage, and memory usage, ensuring that the fitting results for different parameters do not interfere with each other. Subsequently, the system uses the least squares method to perform linear fitting. The least squares method is a classic mathematical optimization method that finds the optimal linear match by minimizing the sum of squared errors. It can accurately fit the univariate linear relationship between the timestamp difference and the parameter fluctuation. The target linear equation for the fitting is... （ ,in （ For timestamp difference, （ For parameter fluctuation, （ To fit the slope of the straight line, （ To define the intercept, in the specific calculation, the effective sample size is first defined as... （ The sample set is （ First, calculate the basic statistic: the mean of the independent variable. （ The mean of the dependent variable （ Then calculate the fitting slope. （ ,intercept （ After completing the linear fitting, the system further calculates the linear correlation coefficient and the Pearson correlation coefficient. This coefficient is a core indicator that measures the linear correlation between timestamp differences and parameter fluctuations, and its value range is [value range missing]. （ The closer the absolute value is to 1, the stronger the linear correlation; the closer it is to 0, the stronger the linear correlation. The specific calculation formula is as follows: The system performs the aforementioned least squares fitting and linear correlation coefficient calculation process on the sample data of each type of parameter, ultimately obtaining the fitting line parameters and linear correlation coefficients for each of the five types of parameters. The linear correlation coefficient is treated as a scalar micro-fluctuation characteristic value. The string representation of the primary feature fingerprint and the string representation of the micro-fluctuation characteristic value are concatenated according to a preset third string concatenation order to generate a secondary micro-feature fingerprint. Further explanation is needed: after completing the least squares linear fitting and linear correlation coefficient calculation for each numerical alarm parameter within the same analysis group, this linear correlation coefficient is used as the core quantitative indicator characterizing the micro-fluctuation pattern of the alarm, further completing the generation of the secondary micro-feature fingerprint. The specific implementation method is as follows: The linear correlation coefficient obtained from the fitting is defined as a scalar micro-fluctuation feature value. This micro-fluctuation feature value is a single numerical scalar with a fixed value range between [-1, 1]. It directly reflects the linear correlation between the parameter fluctuation and the timestamp difference. To ensure the consistency of accuracy in subsequent string concatenation and similarity comparison, the system performs a standardized string conversion on this micro-fluctuation feature value. The value is rounded according to a preset fixed precision (retaining 4 decimal places), and then the pure numerical value is converted into a standard numeric string without extra symbols or spaces, eliminating the possibility of errors due to precision differences. This leads to fingerprint matching anomalies. Subsequently, the system calls the preset third string concatenation order to perform the concatenation operation. This concatenation order is a globally fixed rule of the system and is completely different from the first and second string concatenation orders in terms of field composition, arrangement order, and delimiters. This avoids fingerprint confusion at the source. Its fixed rule is that the primary feature fingerprint string comes first, followed by the micro-fluctuation feature value string. Adjacent strings are separated by a special unambiguous delimiter (such as $) that does not repeat the delimiters (|, #) in the preceding order. This delimiter is a special character that will never appear in alarm text, IP address, or numerical strings, completely avoiding concatenation conflicts and field ambiguities. Before concatenation, the system performs format validation on the primary feature fingerprint string and the micro-fluctuation feature value string again, removing leading and trailing whitespace characters, escape characters, and other invalid content to ensure that both strings are in a regular and clean text format. During concatenation, the preprocessed primary feature fingerprint string, the special delimiter $, and the micro-fluctuation feature value string are combined in sequence to form a continuous composite string that simultaneously carries the macro-source category characteristics and micro-fluctuation pattern characteristics of the alarm. This composite string is the fingerprint fingerprint string. The final secondary micro-feature fingerprint; to intuitively illustrate the splicing process, an example is given: if the primary feature fingerprint of an alarm is "192.168.1.100#port scan", the calculated linear correlation coefficient is 0.9642, which is converted into the micro-fluctuation feature value string "0.9642". After splicing according to the third string concatenation order, the final generated secondary micro-feature fingerprint is "192.168.1.100#port scan$0.9642". This secondary micro-feature fingerprint can uniquely identify the macro-attributes and micro-fluctuation patterns of this alarm.

[0017] The secondary micro-feature fingerprints generated for each alarm data are read sequentially. The primary feature fingerprint and micro-fluctuation feature value are parsed from each secondary micro-feature fingerprint. When comparing pairs, it is determined whether the primary feature fingerprints in two secondary micro-feature fingerprints are completely identical. If they are different, they are directly determined to be dissimilar. If the primary feature fingerprints are identical, the micro-fluctuation feature values ​​in the two secondary micro-feature fingerprints are further extracted and compared. The degree of similarity between the two micro-fluctuation feature values ​​is calculated. The degree of similarity is determined by calculating the absolute value of the difference between the two micro-fluctuation feature values ​​and comparing the absolute value with a preset fluctuation threshold. If the absolute value is less than or equal to the fluctuation threshold, the degree of similarity of the micro-fluctuation feature values ​​is considered to meet the condition, and the overall similarity of the secondary micro-feature fingerprints of the two alarm data reaches the preset threshold. The system first reads the secondary micro-feature fingerprints generated for each alarm data point sequentially, according to the storage order of the alarm data in the initially filtered dataset, and loads them into the memory comparison buffer. Since the secondary micro-feature fingerprints are constructed using a fixed format of "primary feature fingerprint + micro-fluctuation feature value with a dedicated delimiter," the system then activates a string parsing module based on delimiter matching. Using the dedicated unambiguous delimiter used during concatenation as the sole segmentation criterion, each secondary micro-feature fingerprint is precisely split into two independent parts: the left string represents the primary feature fingerprint, and the right string represents the micro-fluctuation feature value. After the split, the right-hand micro-fluctuation feature value string is automatically converted to standard floating-point format for subsequent numerical values. In preparation for the operation, the left-side primary feature fingerprint portion is stored in its original string format for text equality matching. Subsequently, the system employs a full, non-repeating pairwise comparison strategy, traversing all alarm data pairs and sequentially performing hierarchical similarity judgment on the secondary micro-feature fingerprints of any two different alarm data pairs. After splitting the secondary micro-feature fingerprints by the delimiter '$' to obtain the primary feature fingerprint strings corresponding to the two alarms to be compared, the system immediately performs primary feature fingerprint equality matching verification. This is a macro-level rigid pre-verification step for redundant alarm judgment, employing strict matching rules of no ambiguity, no wildcards, and complete bit-by-bit consistency. Technically, this ensures that only macro-level homogeneous alarms with the same terminal IP and alarm type can proceed to subsequent micro-level fluctuation feature comparison. The two primary feature fingerprints obtained from the splitting are labeled as FP1_A` (the primary feature fingerprint of the first alarm) and FP1_B` (the primary feature fingerprint of the second alarm), respectively. First, standardized format preprocessing is performed on both fingerprints. This is a key technical step to avoid misjudgment due to format errors. This includes uniformly converting to UTF-8 encoding, removing all leading and trailing whitespace characters, tabs, newlines, escape characters, and other invalid format characters, restoring the original form of special characters, and unifying character case. This ensures that only core valid characters are retained in the two fingerprints, without any format differences interfering. After preprocessing, the system performs length pre-verification optimization, a core technical design to improve comparison efficiency. The total character length of FP1_A` and FP1_B` is obtained first. If the two lengths are not equal, there is no need to perform character-by-character comparison; the two primary feature fingerprints are directly determined to be inconsistent, and the corresponding secondary micro-feature fingerprints are dissimilar, indicating that the two alarms do not belong to the same class of redundant alarms. Data cluster; if the lengths are completely equal, the system initiates a precise character-by-character matching. Starting from the first character on the left of the string, it performs a rigid one-to-one comparison of each character of 'FP1_A' and 'FP1_B' from left to right. The comparison range covers the terminal IP's numeric and dot separators, the first-level feature fingerprint's dedicated separator '#', and all characters of the alarm type, including Chinese, English, and numbers. It is required that the characters at each position must be completely identical, without any deviation. If any difference is found in any position during the comparison process, the system immediately terminates the character-by-character comparison, determines that the first-level feature fingerprints are inconsistent, and that the two alarms are not macroscopically from the same source and do not belong to the same type of redundant alarm data cluster. If all characters at each position are completely matched, the system formally determines that the two first-level feature fingerprints are completely identical, confirming that the two alarms come from the same terminal IP and the same alarm type, belonging to macroscopically from the same source data, and can proceed to the next step of verifying the proximity of micro-fluctuation feature values.To illustrate the matching process more clearly, let's look at some actual alarm data: Example 1: Two primary feature fingerprints to be compared are `FP1_A=192.168.1.100#port scan` and `FP1_B=192.168.1.101#port scan`. After preprocessing, they are the same length. However, when comparing character by character until the 12th character, `FP1_A` is `0` and `FP1_B` is `1`. Since the characters are inconsistent, it is directly determined that they are not macroscopically homologous. Example 2: `FP1_A=10.0.0.5#brute-force attack` and `FP1_B=10.0.0`. .5#Abnormal Remote Login”, compare it character by character to the alarm type field after the separator “#”, the characters are completely different, and it is determined that it is not from the same macro source; Example 3, FP1_A=172.16.0.8#SQL Injection Attack` and FP1_B=172.16.0.8#SQL Injection Attack”, after preprocessing, the length is the same and the characters match completely, and it is determined to be from the same macro source, and can continue to the subsequent micro comparison. The system will automatically enter the second step of micro fluctuation feature value proximity verification, extract the micro fluctuation feature value in the form of standardized floating point number corresponding to the two alarm data, and mark them respectively. and According to the formula The absolute value of the difference between two micro-fluctuation characteristic values ​​is calculated. This absolute value is used to quantify the degree of similarity between their micro-fluctuation patterns. The system is pre-configured with a preset fluctuation threshold, which is calibrated through network security business testing. This threshold is the critical standard for determining whether micro-fluctuation characteristics can be considered to originate from the same source. The calculated absolute value is then used to... Compare the magnitude with the preset fluctuation threshold. If the values ​​are less than or equal to a preset fluctuation threshold, then the similarity between the two micro-fluctuation feature values ​​is determined to meet the requirement of homologous fluctuation. Combined with the verification result of identical primary feature fingerprints from the first step, the overall similarity of the secondary micro-feature fingerprints of the two alarm data sets is finally determined to reach a preset threshold, indicating they belong to the same redundant alarm data cluster. If the fluctuation exceeds the preset threshold, it is determined that the micro-fluctuation characteristics are too different, and the two alarm data do not belong to the same redundant alarm data cluster.

[0018] For each redundant alarm data cluster identified based on the similarity of secondary micro-feature fingerprints, a temporary processing list is created for the redundant alarm data cluster. According to the preset filtering rules, one alarm data is selected from all the alarm data contained in the redundant alarm data cluster. The filtering rule is to select the one with the earliest timestamp. The selected alarm data is marked as representative sub-core data and removed from the temporary processing list. It is then transmitted to the full threat analysis module. The full threat analysis module performs a deep analysis process on the representative sub-core data and generates a security response decision based on the analysis results.

[0019] It should be further explained that, for each identified redundant alarm data cluster, a tiered processing and analysis process is initiated, and the specific implementation method is as follows: For each independent redundant alarm data cluster, a dedicated isolated cache space is allocated in memory, and a temporary processing list is created. This list is an ordered, structured linear storage queue, strictly sorted according to the original occurrence time of the alarm data. It completely includes all alarm data in the current cluster that meets the similarity threshold, and each alarm retains all original fields, secondary micro-feature fingerprints, numerical parameters, timestamps, and other complete information. A unique cluster ID is bound to the temporary processing list, completely isolating it from the lists of other redundant clusters to prevent data cross-contamination. Representative data is selected according to a preset fixed filtering rule: the alarm data with the earliest timestamp in the cluster is selected as the core representative, because the alarm with the earliest timestamp is the most redundant. The original alarm triggered first in the cluster can completely preserve the initial characteristics of the threat occurrence and is the only core data with the most in-depth analysis value. During the selection process, the system first extracts the standardized timestamps of all alarms in the temporary processing list (which have been uniformly converted to Unix second-level numerical format and can be directly and accurately compared in size). By traversing and comparing the values, the alarm data with the smallest timestamp value is found, which is the target data. After finding it, the system adds a representative sub-core data exclusive identifier tag to the alarm data and removes it securely from the temporary processing list. Only the other redundant alarm data in the cluster are kept in the list to await lightweight processing. Through the cloud encrypted secure transmission channel, this representative sub-core data is transmitted to the backend full threat analysis module in the full original message format.

[0020] After the representative sub-core data that has been marked is sent to the full threat analysis module through an encrypted transmission channel to start the deep analysis process, the pre-built lightweight feature extraction routine is automatically started to efficiently process all other alarm data that are still in the corresponding temporary processing list and belong to the same type of redundant alarm data cluster. This lightweight feature extraction routine is a low-computing-power, high-throughput lightweight processing program designed specifically for redundant alarms. It does not need to parse the full alarm load or perform complex threat analysis, but only focuses on the rapid extraction of core features. During runtime, the routine reads the remaining alarm data one by one according to the storage order of the alarms in the temporary processing list. It quickly locates the interface through lightweight fields and accurately extracts a set of key feature fields defined by a pre-unified standard from a single alarm. This set contains at least four types of core information: alarm type, terminal IP, core alarm parameter values ​​(data packet length, request frequency, traffic size, CPU utilization percentage, memory usage value), and complete timestamp. During the extraction process, the field data is simultaneously standardized, and invalid characters such as whitespace and escape characters are removed, and the text encoding and numerical values ​​are unified. The system ensures that the feature data is well-organized and searchable. After extracting the key features of a single alarm, the system encapsulates all extracted fields into key-value pairs as structured records. Each structured record is bound to a unique identifier ID of its respective redundant alarm data cluster, achieving a correlation with representative sub-core data. Finally, the system batch writes the structured records of all remaining alarms into a dedicated, independently deployed cache database. This cache database is a lightweight storage unit in the network security monitoring data cloud processing system, specifically designed for storing key features of redundant alarms. It is completely independent of the deep analysis results storage system of the full threat analysis module in terms of both physical storage and logical architecture. It does not occupy core analysis storage resources, retaining only key feature data for alarm tracing and statistics. This effectively retains redundant alarm data while avoiding the cloud computing power and storage resource consumption caused by full storage and repeated deep analysis, thus improving the efficiency of monitoring data cloud processing. To quantify the technical advantages of this invention in cloud resource optimization and processing efficiency, a comparison chart of computing power, storage, and efficiency between traditional processing methods and this invention is presented, as shown below. Figure 4 As shown.

[0021] Please see Figure 2 As shown, a second objective of this invention is to provide a system for implementing a network security-oriented monitoring data cloud processing method, including any one of the above-mentioned features, comprising: The data receiving and fully redundant filtering unit 1 is used to receive the original network security monitoring alarm stream and use a hash algorithm to filter and deduplicate alarm data that are completely consistent in alarm type, terminal IP and timestamp, and output the alarm dataset after preliminary filtering. The feature extraction and fingerprint generation unit 2 is used to concatenate the terminal IP and alarm type in the alarm data of the initially filtered alarm dataset to generate a first-level feature fingerprint. The alarm data is grouped according to the first-level feature fingerprint. The parameter fluctuation and timestamp difference are calculated for the alarm data in the same group. The least squares method is applied to fit the parameter fluctuation and timestamp difference to obtain a linear correlation coefficient as a micro-fluctuation feature value. Finally, the first-level feature fingerprint and the micro-fluctuation feature value are combined to generate a second-level micro-feature fingerprint. The redundancy determination unit 3 is used to compare the similarity of the generated secondary micro-feature fingerprints. By comparing the similarity of the primary feature fingerprints and the closeness of the micro-fluctuation feature values, it determines whether the alarm data belongs to the same redundancy alarm data cluster based on the preset threshold. The hierarchical processing unit 4 is used to select one alarm data from the cluster as a representative sub-core data according to preset rules and send it to the full threat analysis module for in-depth analysis. At the same time, for each other alarm data in the cluster, the feature extraction routine is started to extract the set of key feature fields and store the set of key feature fields in a dedicated cache database that is independent of the storage system of the full threat analysis module.

[0022] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A cloud processing method for monitoring data for network security, characterized in that: Includes the following steps: S1. Receive the original network security monitoring alarm stream and use a hash algorithm to perform full redundancy filtering and deduplication on alarm data with completely identical alarm type, terminal IP and timestamp, and output the alarm dataset after preliminary filtering. S2. Perform redundancy determination on the alarm data in the pre-filtered alarm dataset, combine the terminal IP and alarm type in the current alarm data to generate a first-level feature fingerprint for identifying the macro source and category, extract multiple alarm data under the same first-level feature fingerprint, obtain the alarm parameters and timestamps containing specific values ​​in the multiple alarm data respectively, calculate the absolute difference between the corresponding alarm parameters between any two alarm data as the parameter fluctuation, and calculate the absolute difference between the timestamps between the two alarm data as the timestamp difference. Finally, use the least squares method to fit the parameter fluctuation and the timestamp difference, calculate the linear correlation coefficient, use the linear correlation coefficient as the micro fluctuation feature value, and combine the first-level feature fingerprint with the micro fluctuation feature value to generate a second-level micro feature fingerprint that characterizes the macro consistency and micro fluctuation pattern of the alarm data. S3. Perform pairwise similarity comparison on all secondary micro-feature fingerprints. Specifically, compare the closeness of the micro-fluctuation feature values ​​and the similarity of the primary feature fingerprints. When the overall similarity of the secondary micro-feature fingerprints of two alarm data is greater than or equal to a preset threshold, it is determined that the two alarm data belong to the same redundant alarm data cluster. S4. Perform hierarchical processing on each identified redundant alarm data cluster. From the same redundant alarm data cluster, arbitrarily select one alarm data as representative core data and input it into the backend full threat analysis module for complete in-depth analysis and response decision-making. At the same time, for each redundant alarm data in the redundant alarm data cluster other than the representative core data, bypass the full threat analysis module and extract the alarm type, terminal IP, key alarm parameters and occurrence timestamp through a lightweight feature extraction routine. Store the set in a dedicated cache database to avoid repeated full analysis calculations.

2. The cloud processing method for monitoring data for network security according to claim 1, characterized in that: The process of using a hash algorithm to perform complete redundancy filtering and deduplication on alarm data with identical alarm types, terminal IPs, and timestamps specifically includes: Based on the alarm type string, terminal IP address string, and timestamp string contained in each received original alarm data, a first hash key is generated by concatenating them according to a preset first string concatenation order. The first hash key is then input into a preset hash table for querying. If the same first hash key already exists in the hash table, the current alarm data is determined to be completely redundant and is discarded directly. If the same first hash key does not exist in the hash table, the current alarm data is added to the alarm dataset after preliminary filtering, and the first hash key is inserted into the hash table, thus completing the filtering and deduplication of alarm data with completely identical alarm type, terminal IP, and timestamp.

3. The cloud processing method for monitoring data for network security according to claim 2, characterized in that: From the pre-filtered alarm dataset, the alarm type field and the terminal IP address field are read. The string of the alarm type field and the string of the terminal IP address field are concatenated according to a preset second string concatenation order that is different from the first string concatenation order to generate a composite string, which serves as the first-level feature fingerprint. This is used to filter out a subset of alarm data with the same alarm source and type.

4. The cloud processing method for monitoring data for network security according to claim 3, characterized in that: Based on the primary feature fingerprint, all alarm data with the same primary feature fingerprint in the initially filtered alarm dataset are grouped into the same analysis group. For each alarm data in the same analysis group, its load information is parsed, and alarm parameters with numerical features are extracted. The alarm parameters include data packet length, request frequency, traffic size, CPU utilization percentage, and memory usage. At the same time, the timestamp in each alarm data is extracted. For any two different alarm data in the analysis group, the values ​​of the same alarm parameter corresponding to them are aligned, and the absolute value of the difference between the two values ​​is calculated. The obtained absolute value result is defined as the parameter fluctuation.

5. The cloud processing method for monitoring data for network security according to claim 4, characterized in that: Collect the parameter fluctuation and timestamp difference corresponding to multiple pairs of alarm data within the same analysis group, use the timestamp difference as the independent variable and the parameter fluctuation as the dependent variable, apply the least squares method to perform linear fitting on the timestamp difference and the parameter fluctuation, and calculate the linear correlation coefficient.

6. The cloud processing method for monitoring data for network security according to claim 5, characterized in that: The linear correlation coefficient is treated as a scalar form of micro-fluctuation feature value. The string of the first-level feature fingerprint and the string of the micro-fluctuation feature value are concatenated according to the preset third string concatenation order to finally generate a second-level micro-feature fingerprint.

7. The cloud processing method for monitoring data for network security according to claim 6, characterized in that: The secondary micro-feature fingerprints generated for each alarm data are read sequentially. The primary feature fingerprint and micro-fluctuation feature value are parsed from each secondary micro-feature fingerprint. When comparing pairs, it is determined whether the primary feature fingerprint parts of two secondary micro-feature fingerprints are completely identical. If they are different, they are directly determined to be dissimilar. If the primary feature fingerprint parts are identical, the micro-fluctuation feature values ​​in the two secondary micro-feature fingerprints are further extracted and compared. The degree of similarity between the two micro-fluctuation feature values ​​is calculated. The degree of similarity is determined by calculating the absolute value of the difference between the two micro-fluctuation feature values ​​and comparing the absolute value with a preset fluctuation threshold. If the absolute value is less than or equal to the fluctuation threshold, the degree of similarity of the micro-fluctuation feature values ​​is considered to meet the condition, and the overall similarity of the secondary micro-feature fingerprints of the two alarm data reaches the preset threshold.

8. The cloud processing method for monitoring data for network security according to claim 7, characterized in that: For each redundant alarm data cluster identified based on the similarity of secondary micro-feature fingerprints, a temporary processing list is created for the redundant alarm data cluster. According to a preset filtering rule, one alarm data is selected from all alarm data contained in the redundant alarm data cluster. The filtering rule is to select the one with the earliest timestamp. The selected alarm data is marked as representative sub-core data and removed from the temporary processing list. It is then transmitted to the full threat analysis module. The full threat analysis module performs a deep analysis process on the representative sub-core data and generates a security response decision based on the analysis results.

9. The cloud processing method for monitoring data for network security according to claim 8, characterized in that: After sending representative sub-core data into the full threat analysis module, for each other alarm data that remains in the temporary processing list and belongs to the same type of redundant alarm data cluster, a lightweight feature extraction routine is started for processing. The feature extraction routine reads the remaining alarm data one by one and extracts a predefined set of key feature fields from each alarm data. The set of key feature fields includes at least the alarm type, terminal IP, core alarm parameter values, and a complete timestamp. The extracted set of key feature fields is stored in a dedicated cache database in a structured record format. The cache database is independent of the storage system of the full threat analysis module.

10. A system for implementing the cloud processing method for monitoring data for network security as described in any one of claims 1-9, characterized in that, include: The data receiving and fully redundant filtering unit (1) is used to receive the original network security monitoring alarm stream and use a hash algorithm to filter and deduplicate alarm data with completely consistent alarm type, terminal IP and timestamp, and output the alarm dataset after preliminary filtering. The feature extraction and fingerprint generation unit (2) is used to concatenate the terminal IP and alarm type in the alarm data in the alarm dataset after preliminary filtering to generate a first-level feature fingerprint, group the alarm data according to the first-level feature fingerprint, calculate the parameter fluctuation and timestamp difference for the alarm data in the same group, apply the least squares method to fit the parameter fluctuation and timestamp difference to obtain the linear correlation coefficient as the micro fluctuation feature value, and finally combine the first-level feature fingerprint and the micro fluctuation feature value to generate a second-level micro feature fingerprint; The redundancy determination unit (3) is used to compare the similarity of the generated secondary micro-feature fingerprints. By comparing the similarity of the primary feature fingerprints and the closeness of the micro-fluctuation feature values, it determines whether the alarm data belongs to the same redundancy alarm data cluster based on the preset threshold. The hierarchical processing unit (4) is used to select an alarm data from the cluster as a representative sub-core data according to preset rules and send it to the full threat analysis module for in-depth analysis. At the same time, for each other alarm data in the cluster, the feature extraction routine is started to extract the set of key feature fields and store the set of key feature fields in a dedicated cache database that is independent of the storage system of the full threat analysis module.

Images