Network security data processing methods, devices, and storage media based on big data
By performing time-series segmentation and feature extraction on network security data, constructing mutation and non-mutation labels, calculating density coefficients and feature points, and filtering abnormal logs, the accuracy problem of multi-source network security data analysis in existing technologies is solved, and more efficient anomaly detection is achieved.
Patent Information
- Application Number
- CN202611409580.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-11
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies only identify and analyze abnormal data in collected network data by setting security thresholds, which fails to effectively analyze the fluctuation of security logs of multi-source network security data, resulting in missed and false alarms of abnormal data.
By dividing security log data into time series, extracting log mutation features and channel parsing difference features, constructing mutation and non-mutation labels, calculating density coefficients and feature points, performing stable feature point analysis, and filtering abnormal mutation logs.
It improves the accuracy and robustness of network security data processing, reduces false alarm rates, and ensures the integrity and efficiency of detection.
Smart Images

Figure CN123122958A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data management, and in particular to a network security data processing method, apparatus, and storage medium based on big data. Background Technology
[0002] Cybersecurity data is the foundation for security monitoring, threat detection, and incident response. However, during the collection, transmission, storage, and processing of multi-source cybersecurity data, the problem of malicious data modification is widespread. Plaintext log data can be intercepted and tampered with by man-in-the-middle attacks. During log parsing, inconsistent encoding, field truncation, or non-standard formats can also cause unexpected changes to log content. These problems directly affect the correlation analysis of security incidents, attack attribution, and the credibility of electronic evidence, and may even lead to missed or false alarms of security threats. In cybersecurity data analysis, security logs are core data that is focused on. Security logs are automatically generated by firewalls, intrusion detection systems, web application firewalls, operating systems, databases, and application systems during operation. They can record in detail user operations, system state changes, security policy triggering, and abnormal events. Therefore, analyzing security logs is a critical link in cybersecurity data processing, and ensuring the integrity of logs during transmission and processing is an important prerequisite for ensuring the accuracy and reliability of security analysis results.
[0003] For example, Chinese Patent Publication No. CN119254520A discloses a network security alarm system and method based on big data. A network data acquisition module collects network traffic and security data from network devices and system logs. The data obtained by the network data acquisition module is stored in a database. Then, the data in the database is called and security analysis is performed according to preset alarm rules to calculate a security alarm index. Next, the security alarm index is compared with a preset security threshold range. If the security alarm index is within the security threshold range, the current network security is determined; otherwise, the current network is determined to have a security risk. Then, based on the security risk determination by the network security analysis module, alarm information is generated and sent to the system backend. Finally, the alarm information is sorted in descending order according to the security alarm index, and the sorted order is used as the response priority.
[0004] However, the following problems still exist in the existing technology:
[0005] Simply setting security thresholds to identify and analyze abnormal data in collected network data fails to identify abnormal data in the aggregated network security data by performing fluctuation analysis of security logs from multiple sources. Summary of the Invention
[0006] To address this, the present invention provides a network security data processing method, apparatus, and storage medium based on big data, which overcomes the problem in the prior art that only identifies and analyzes abnormal data in collected network data by setting security thresholds, and fails to identify abnormal data in the summarized network security data by performing fluctuation analysis of security logs on network security data collected from multiple sources.
[0007] To achieve the above objectives, in a first aspect, the present invention provides a network security data processing method based on big data, comprising:
[0008] The acquired security log data is divided into several log entries in a time sequence according to a preset time window. Several log entries in each time window constitute an analysis set. Several log mutation features that affect log state mutations are extracted from each analysis set.
[0009] Based on the log mutation characteristics and channel parsing difference characteristics, the channel parsing difference degree is determined to determine the mutation label of each analysis set;
[0010] The predetermined number of analysis sets identified as mutation tags are combined in chronological order to construct mutation observation sequences. Based on the continuous state and number of non-mutation tags, several non-mutation comparison sequences corresponding to each analysis set are constructed.
[0011] Log entries in the non-mutation comparison sequences are extracted and the corresponding key log fields are analyzed to determine feature points. The density coefficient of each feature point is calculated to determine the stable feature points for each mutation observation sequence.
[0012] Based on the key fields of the logs, log entries of the same type belonging to different analysis sets in the mutation observation sequence are extracted. Based on the stable feature points, the detection features of each log entry of the same type are extracted to obtain a stable observation group for mutation analysis.
[0013] Extract log entries containing a single stable feature point for feature point response analysis;
[0014] The mutation analysis involves extracting mutation features from stable observation groups to filter abnormal mutation logs, while the feature point response analysis involves determining the feature point response degradation degree based on the field completeness of a single stable feature point relative to each single stable feature point in the previous classification set to filter abnormal mutation logs.
[0015] Furthermore, the process of extracting several log mutation features that affect log state mutations from each of the aforementioned analysis sets includes,
[0016] Extract the parsed data corresponding to the security log data, and determine the log length, key log fields, and time interval between adjacent security logs in the analysis set;
[0017] The log length change rate is determined based on the log length, the keyword frequency change rate is determined based on the log key fields, and the timestamp interval variation coefficient is determined based on the time interval between adjacent security logs. These are then normalized and weighted to serve as log mutation features.
[0018] Furthermore, the process of determining the channel resolution difference degree based on the log mutation characteristics and channel resolution difference characteristics to determine the mutation label of each set includes,
[0019] In response to the log mutation feature being greater than or equal to a preset log mutation feature threshold, the channel parsing difference feature is determined and obtained;
[0020] Calculate the channel resolution difference degree based on the channel resolution difference characteristics;
[0021] If the channel resolution difference is greater than or equal to the preset resolution difference threshold, then a mutation label is set;
[0022] If the channel resolution difference is less than a preset resolution difference threshold, then a non-mutation label is set.
[0023] Furthermore, the process of constructing several non-mutation contrast sequences corresponding to each analysis set based on the continuous states and continuous quantities of non-mutation tags includes,
[0024] Select the analysis set that is determined to be a non-mutation tag, and determine the adjacent non-mutation tag analysis sets as non-mutation tag groups;
[0025] If the non-mutation tag group meets the preset quantity condition, then the non-mutation tag group is determined to be a non-mutation comparison sequence;
[0026] The preset quantity condition is that the number of consecutive analysis sets included in the non-mutation tag group is greater than or equal to a preset consecutive quantity threshold.
[0027] Further, the process of calculating the density coefficient of each feature point to determine the stable feature points for each of the said mutation observation sequences includes,
[0028] Log entries are extracted from the non-mutation comparison sequence, and the key log fields corresponding to each log entry are parsed and identified.
[0029] Identify key log fields related to abnormal behavior as feature points;
[0030] Feature points that meet the density screening criteria are identified as stable feature points.
[0031] The density screening condition is that the density coefficient is greater than a preset density coefficient threshold. The density coefficient is determined by the total number of occurrences of each feature point in the non-mutation comparison sequence and the total number of classification sets contained in the non-mutation comparison sequence.
[0032] Furthermore, the process of extracting detection features from each similar log entry based on stable feature points to obtain a stable observation group and performing mutation analysis includes:
[0033] Log entries with the same key field in the mutation observation sequence are grouped into the same type of log entries;
[0034] The detection features are determined based on the frequency of occurrence of each stable feature point in each of the aforementioned log entries of the same type to classify the log entries contained in the same type of log entries as abnormal mutation logs or stable observation groups;
[0035] A log key field contained in a single log entry in the stable observation group that does not exist in other log entries in the stable observation group is identified as a mutation feature;
[0036] Log entries exhibiting the aforementioned mutation characteristics are identified as anomalous mutation logs.
[0037] Furthermore, detection features are determined based on the frequency of occurrence of each stable feature point in each of the aforementioned similar log entries to classify the log entries contained in the same type of log entries into either abnormal mutation logs or stable observation groups, wherein,
[0038] Select similar log entries containing more than one stable feature point to determine the detection features;
[0039] If the detected feature is greater than the preset detected feature threshold, then the log entries included in the same type of log entries are classified as abnormal mutation logs.
[0040] If the detection feature is less than or equal to the preset detection feature threshold, then the log entries included in the same type of log entries are classified as a stable observation group.
[0041] Furthermore, the process of extracting log entries containing single stable feature points for feature point response analysis includes,
[0042] Sequentially obtain the number of fields contained in the single stable feature point in the previous category set of the category set to which the log entry containing the single stable feature point belongs;
[0043] Based on the number of fields, the field completeness of the single stable feature point relative to each single stable feature point in the previous classification set is calculated.
[0044] A field completeness curve is constructed with time as the horizontal axis and the field completeness as the vertical axis. The slope of the field completeness curve is determined to be the feature point response degradation degree.
[0045] If the feature point response degradation is negative, then the log entry containing a single stable feature point is determined to be an abnormal mutation log.
[0046] Secondly, the present invention provides a network security data processing device based on big data, comprising:
[0047] The data acquisition module is used to divide the acquired security log data into several log entries in a time sequence according to a preset time window. Several log entries in each time window constitute an analysis set, and several log mutation features that affect log status mutations are extracted from each analysis set.
[0048] A data labeling module, which is connected to the data acquisition module, is used to determine the channel resolution difference degree based on the log mutation characteristics and channel resolution difference characteristics, so as to determine the mutation label of each analysis set;
[0049] The data classification module is connected to the data acquisition module and the data labeling module respectively. It is used to combine a preset number of analysis sets that are determined as mutation tags in chronological order to construct a mutation observation sequence, and to construct several non-mutation comparison sequences corresponding to each analysis set based on the continuous state and continuous number of non-mutation tags.
[0050] The feature extraction module, which is connected to the data classification module, is used to extract log entries from the non-mutation comparison sequence, analyze the corresponding log key fields to determine feature points, and calculate the density coefficient of each feature point to determine the stable feature points for each mutation observation sequence.
[0051] The data analysis module, connected to the feature extraction module, is used to extract similar log entries belonging to different analysis sets in the mutation observation sequence based on the log key fields, extract the detection features of each similar log entry based on stable feature points to obtain stable observation groups, perform mutation analysis, and extract log entries containing a single stable feature point for feature point response analysis.
[0052] Thirdly, the present invention also provides a storage medium for storing computer-executable instructions, which, when executed, implement the steps of the big data-based network security data processing method described in any one of the first aspects.
[0053] Compared with the prior art, the beneficial effect of the present invention is that it fully considers the impact of different log sources and different parsing channels on the expression of log fields, making the determination of mutation tags more consistent with the actual data characteristics and improving the accuracy of data processing.
[0054] In particular, by extracting change features from three dimensions—log length, frequency of key fields, and time interval between adjacent logs—this invention can reflect the abrupt changes in the log stream from different perspectives, thereby improving the robustness of anomaly detection.
[0055] In particular, the density coefficient of this invention is jointly determined by the total number of feature points appearing in non-mutation comparison sequences and the total number of classification sets contained in the sequence, reflecting the frequency and coverage of feature points in normal behavior, filtering out occasional and sparse fields, and reducing the computational load and false alarm rate of subsequent analysis.
[0056] In particular, this invention extracts log entries of individual stable feature points for independent analysis, ensuring that all suspicious logs are included in the detection process, thereby improving the completeness of the detection. Attached Figure Description
[0057] Figure 1 This is a flowchart illustrating the network security data processing method based on big data according to the present invention.
[0058] Figure 2 This is a schematic diagram illustrating the process of determining mutation tags for each set in an embodiment of the present invention;
[0059] Figure 3 This is a flowchart illustrating the process of determining stable feature points for each mutation observation sequence in an embodiment of the present invention.
[0060] Figure 4 This is a schematic diagram of the network security data processing device based on big data according to an embodiment of the present invention. Detailed Implementation
[0061] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.
[0062] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.
[0063] It should be noted that, in the description of this invention, unless otherwise explicitly specified and limited, the term "connection" should be interpreted broadly. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.
[0064] Please see Figure 1As shown, this is a flowchart illustrating the network security data processing method based on big data according to the present invention. An embodiment of the present invention provides a network security data processing method based on big data, comprising:
[0065] Step S1: Divide the acquired security log data into several log entries in a time sequence according to a preset time window. Several log entries in each time window constitute an analysis set. Extract several log mutation features that affect log state mutations from each analysis set.
[0066] Specifically, the preset time window is used to divide several log entries of the acquired security log data into time sequences. Several log entries within a time window constitute an analysis set. The preset time window can be set according to the average log generation rate and detection sensitivity requirements. The higher the average log generation rate, the smaller the time window can be set; the higher the detection sensitivity, the smaller the time window can be set. Using the timestamp of the first log entry as a reference, the window number for each log entry is calculated based on its timestamp. For empty windows, if a window contains no log entries, it is marked as an empty region, and subsequent feature extraction is skipped or it is merged with the previous non-empty window.
[0067] Step S2: Determine the channel resolution difference degree based on log mutation characteristics and channel resolution difference characteristics to determine the mutation label of each analysis set;
[0068] Step S3: Combine the predetermined number of analysis sets that are determined as mutation tags in chronological order to construct mutation observation sequences, and construct several non-mutation comparison sequences corresponding to each analysis set based on the continuous state and continuous number of non-mutation tags.
[0069] Step S4: Extract log entries from the non-mutation comparison sequences, analyze the corresponding log key fields to determine feature points, and calculate the density coefficient of each feature point to determine the stable feature points for each mutation observation sequence.
[0070] Understandably, the key features are the critical fields in the logs.
[0071] Step S5: Extract similar log entries from different analysis sets in the mutation observation sequence based on key log fields, extract detection features of each similar log entry based on stable feature points to obtain stable observation groups, and perform mutation analysis.
[0072] Step S6: Extract log entries containing a single stable feature point for feature point response analysis;
[0073] Among them, mutation analysis is used to extract mutation features from stable observation groups to filter abnormal mutation logs, and feature point response analysis is used to determine the feature point response degradation degree based on the field completeness of a single stable feature point relative to each single stable feature point in the previous classification set to filter abnormal mutation logs.
[0074] Specifically, by periodically acquiring network security data and analyzing security logs, and combining log mutation characteristics with channel parsing difference characteristics to calculate channel parsing difference, the impact of different log sources and parsing channels on log field expression is fully considered, making mutation label determination more consistent with actual data characteristics. Since network security data has a time sequence, the continuous state and continuous quantity of non-mutation labels are analyzed to construct a non-mutation comparison sequence. The log key fields of the non-mutation comparison sequence are analyzed to determine the log key fields that appear in the corresponding system during daily operation as feature points. Then, the density of feature points appearing in the mutation observation series with mutation labels is analyzed to determine the stable feature points of the mutation observation series. Log entries of the same type belonging to different analysis sets in the mutation observation sequence are aggregated, and detection features are extracted using stable feature points. The detection of the same type improves detection efficiency and accuracy of network security data processing based on big data.
[0075] Specifically, in step S1, the process of extracting several log mutation features that affect log state mutations for each analysis set includes,
[0076] Extract the parsed data corresponding to the security log data, and determine the log length, key log fields, and time interval between adjacent security logs in the analysis set;
[0077] Specifically, the log length is the length of the log entry in bytes, the log key fields are security-related key fields, and the time interval between adjacent security log entries is the time interval between each adjacent security log entry.
[0078] The log length change rate is determined based on the log length, the keyword frequency change rate is determined based on the log key fields, and the timestamp interval variation coefficient is determined based on the time interval between adjacent security logs. These are then normalized and weighted to serve as log mutation features.
[0079] Specifically, the log length change rate is the abrupt change in the average byte length of log entries, used to reflect changes in log format, injection of attack payloads, or output of abnormal debugging information. The process of determining the log length change rate includes: obtaining the average byte length of all log entries in the current analysis set, calculating the average byte length of the previous analysis set in the time sequence, calculating the byte length difference between the average byte length of all log entries in the current analysis set and the average byte length of the previous analysis set in the time sequence, and determining the ratio of the byte length difference to the average byte length of all log entries in the current analysis set as the log length change rate.
[0080] Specifically, the keyword frequency change rate is a sudden change in the frequency of security-related keywords, used to reflect the switching of security event states. The process of determining the keyword frequency change rate includes: obtaining the total number of occurrences of keywords in the current analysis set, obtaining the total number of occurrences of keywords in the previous analysis set in chronological order, calculating the difference between the total number of occurrences of keywords in the current analysis set and the total number of occurrences of keywords in the previous analysis set in chronological order, and determining the ratio of the difference in the total number of occurrences of keywords to the total number of occurrences of keywords in the current analysis set as the keyword frequency change rate.
[0081] Understandably, security-related keywords refer to key fields that can directly or indirectly reflect security incidents, attack behaviors, abnormal states, policy violations, or risk indicators. These keywords typically originate from specific fields in logs, such as event type, source / destination IP, port, user, operation, status code, error message, etc. These keywords need to be preset according to the log type; for example, SSH logs can select "Failed password" and "Accepted," while web logs can select "403," "404," "500," "SQL," etc.
[0082] Specifically, the timestamp interval variation coefficient is the abrupt change in the dispersion of the time interval between adjacent log entries, used to reflect the regular changes in the log generation rate. The process of determining the timestamp interval variation coefficient includes: obtaining the mean of the timestamp intervals of adjacent logs in the current analysis set, obtaining the standard deviation of the timestamp intervals of adjacent logs in the previous analysis set in the time sequence, and determining the ratio of the standard deviation of the timestamp intervals of adjacent logs in the previous analysis set to the mean of the timestamp intervals of adjacent logs in the current analysis set as the timestamp interval variation coefficient.
[0083] Specifically, for ease of calculation, during normalization, the log length change rate, keyword frequency change rate, and timestamp interval variation coefficient are all normalized to the range of [0,1]. There is no limitation on the normalization method. For example, those skilled in the art can use the common extreme value normalization method for processing. Of course, other methods can also be used, which will not be elaborated here.
[0084] Specifically, when normalized weighted processing is used as a log mutation feature, the correlation between the keyword frequency change rate and the safe time mutation is the strongest, followed by the log length change rate and the timestamp interval coefficient of variation. For example, the weights of the log length change rate, the keyword frequency change rate, and the timestamp interval coefficient of variation are set to 0.3, 0.4, and 0.3, respectively.
[0085] Please see Figure 2As shown, this is a schematic diagram of the process for determining the mutation labels of each set according to an embodiment of the present invention. In step S2, the process of determining the channel resolution difference degree based on log mutation characteristics and channel resolution difference characteristics to determine the mutation labels of each set includes:
[0086] Step S201: In response to the log mutation feature being greater than or equal to a preset log mutation feature threshold, determine the acquisition channel parsing difference feature;
[0087] If the log mutation feature is less than the preset log mutation feature threshold, then a non-mutation label is set.
[0088] Specifically, a preset log mutation feature threshold is used to quantify the degree of abnormal fluctuation in security logs. Then, by comparing the log mutation features with the log mutation feature threshold, it is determined whether to test the parsing capability of the security log transmission channel. In this embodiment, the higher the user's requirement for the accuracy of judging the degree of abnormal fluctuation in security logs, the smaller the log mutation feature threshold. Preferably, the log mutation feature threshold is the product of the log mutation feature threshold reference value and the log mutation feature deviation factor. The log mutation feature threshold reference value is the average value of the log mutation feature thresholds corresponding to the same log collection system in historical data. The log mutation feature deviation factor can be set by those skilled in the art based on their requirements for the accuracy of judging the degree of abnormal fluctuation in security logs. The higher the accuracy requirement, the smaller the value should be. The value range can be [1.0, 1.1], and preferably, it can be 1.05.
[0089] Step S202: Calculate the channel resolution difference degree based on the channel resolution difference characteristics;
[0090] In this embodiment, the channel refers to different log data sources or log transmission protocol channels. The channel parsing difference is the degree of abrupt change in the field structure and parsing format of the logs from each channel, used to reflect the abnormality of the merged log format. The process of determining the channel parsing difference includes:
[0091] For each channel, extract the channel resolution difference feature vector. The dimensions of the channel resolution difference feature vector include, but are not limited to, the number of fields, the distribution of field types, the timestamp format encoding, and the distribution of severity level fields. Calculate the cosine distance between the channel resolution difference feature vectors of each channel in the current analysis set and the channel resolution difference feature vectors of the previous analysis set in the time sequence. Take the maximum cosine distance among all channels as the channel resolution difference degree. This will not be elaborated further here.
[0092] If the channel resolution difference is greater than or equal to the preset resolution difference threshold, then a mutation label is set;
[0093] If the channel resolution difference is less than the preset resolution difference threshold, then a non-mutation label is set.
[0094] Specifically, a preset resolution difference threshold is used to quantify the confidence level of a mutation occurring in the security logs within the analysis set. Then, by comparing the resolution difference with the resolution difference threshold, it is determined whether to set a mutation label for the analysis set. In this embodiment, the higher the user's requirement for the accuracy of the confidence level that no mutation has occurred in the security logs within the analysis set, the smaller the resolution difference threshold. Preferably, the resolution difference threshold can be set by those skilled in the art based on their requirement for the accuracy of the confidence level that no mutation has occurred in the security logs within the analysis set. The higher the accuracy requirement, the smaller the threshold should be. The value range can be [0.1, 0.3], and preferably, it can be 0.2.
[0095] Specifically, this invention extracts variation features from three dimensions: log length, key field frequency, and time interval between adjacent logs. This allows it to reflect abrupt changes in the log stream from different perspectives. The log length change rate reflects abrupt changes in log content complexity; the keyword frequency change rate reflects the increase or decrease of keywords related to specific security events, associating with abnormal behavior; and the timestamp interval variation coefficient reflects fluctuations in the log generation rhythm, such as changes in time intervals caused by sudden traffic or intermittent attacks. Through weighted fusion, the contribution of each dimension can be adjusted according to the actual scenario, further obtaining channel parsing difference features and enabling secondary discrimination of suspicious sets. This effectively improves the accuracy of detection, enhances the robustness of anomaly detection, and further improves the accuracy of big data-based network security data processing.
[0096] Specifically, in step S3, the process of constructing several non-mutation contrast sequences corresponding to each analysis set based on the continuous state and number of non-mutation tags includes,
[0097] Select the analysis set that is determined to be a non-mutation tag, and determine the adjacent non-mutation tag analysis sets as non-mutation tag groups;
[0098] If the number of non-mutated tag groups meets the preset quantity condition, then the non-mutated tag group is determined to be a non-mutated comparison sequence;
[0099] The preset quantity condition is that the number of consecutive analysis sets included in the non-mutation tag group is greater than or equal to the preset consecutive quantity threshold.
[0100] Understandably, a preset consecutive number threshold is set to ensure that the non-mutated comparison sequence contains control regions at least three adjacent time points in order to avoid randomness. For example, the preset consecutive number threshold is set to 5.
[0101] Please see Figure 3The diagram illustrates the process of determining stable feature points for each mutation observation sequence according to an embodiment of the present invention. In step S4, the density coefficient of each feature point is calculated to determine the stable feature points for each mutation observation sequence.
[0102] Step S401: Extract log entries from the non-mutation comparison sequence, parse and identify the log key fields corresponding to each log entry;
[0103] Step S402: Identify key log fields related to abnormal behavior as feature points;
[0104] Understandably, in implementation, the method for determining security-related keywords in the logs related to abnormal behavior can be referenced in the above text.
[0105] Step S403: Feature points that meet the density screening conditions are identified as stable feature points;
[0106] The density screening condition is that the density coefficient is greater than the preset density coefficient threshold. The density coefficient is determined by the total number of each feature point appearing in the non-mutation comparison sequence and the total number of classification sets contained in the non-mutation comparison sequence.
[0107] Specifically, a density coefficient is set to quantify the frequency of feature points in non-mutation contrast sequences. In practice, the density coefficient is the ratio of the total number of each feature point in the non-mutation contrast sequence to the total number of classification sets contained in the non-mutation contrast sequence.
[0108] Specifically, a preset density coefficient threshold is set to quantify the frequency of feature points in non-mutation comparison sequences. Then, by comparing the density coefficient with the density coefficient threshold, it is determined whether a feature point is a stable feature point. In this embodiment, the higher the user's requirement for accuracy in determining stable feature points, the larger the density coefficient threshold. Preferably, the density coefficient threshold can be set by those skilled in the art based on their requirements for accuracy in determining stable feature points; the higher the accuracy requirement, the larger the threshold. Preferably, the density coefficient threshold is the product of a density coefficient threshold reference value and a density coefficient deviation factor. The density coefficient threshold reference value is the average density coefficient threshold corresponding to the same log collection system in historical data. The density coefficient deviation factor can be set by those skilled in the art based on their requirements for accuracy in determining stable feature points; the higher the accuracy requirement, the larger the deviation factor. The value range can be [1.05, 1.2], preferably 1.1.
[0109] Specifically, in step S5, the process of extracting detection features from similar log entries based on stable feature points to obtain stable observation groups and performing mutation analysis includes:
[0110] Log entries with the same key field in the mutation observation sequence are grouped into the same type of log entries;
[0111] The detection features are determined based on the frequency of occurrence of each stable feature point in each similar log entry to classify log entries contained in the same type of log entry into abnormal mutation logs or stable observation groups;
[0112] Identify a log key field in a single log entry within a stable observation group that is not present in other log entries within the stable observation group as a mutation feature;
[0113] Log entries exhibiting mutation characteristics are identified as anomalous mutation logs.
[0114] Specifically, in step S5, detection features are determined based on the frequency of occurrence of each stable feature point in each similar log entry to classify the log entries contained in the same type of log entry into either abnormal mutation logs or stable observation groups.
[0115] Select similar log entries containing more than one stable feature point to determine the detection features;
[0116] It is understandable that the detection feature is the standard deviation of the occurrence frequency of each stable feature point in each log entry of the same type. Therefore, log entries of the same type that contain one or no stable feature points do not have a detection feature.
[0117] If the detected feature is greater than the preset detected feature threshold, then log entries included in the same type of log entries are classified as abnormal mutation logs.
[0118] If the detected feature is less than or equal to the preset detected feature threshold, then log entries included in the same category are classified as a stable observation group.
[0119] Specifically, a preset detection feature threshold is set to quantify the fluctuation of the detection frequency of stable feature point data contained in similar log entries. Then, by comparing the detection features with the detection feature threshold, the log entries contained in similar log entries are determined to be a stable observation group. In this embodiment, the higher the user's requirement for the accuracy of the determination of the stable observation group, the smaller the detection feature threshold. Preferably, the detection feature threshold can be set by those skilled in the art based on their requirement for the accuracy of the determination of the stable observation group. The higher the accuracy requirement, the smaller the threshold should be. Preferably, the detection feature threshold is the product of the detection feature threshold reference value and the detection feature deviation factor. The detection feature threshold reference value is the average value of the detection feature thresholds corresponding to the same log collection system in historical data. The detection feature deviation factor can be set by those skilled in the art based on their requirement for the accuracy of the determination of stable feature points. The higher the accuracy requirement, the smaller the threshold should be. The value range can be [1.0, 1.1], preferably 1.05.
[0120] Specifically, this invention constructs a comparison sequence by selecting adjacent non-mutation label analysis sets that meet a preset continuous quantity threshold. This reduces interference from scattered and isolated normal time periods. The density coefficient is determined by the total number of feature points appearing in the non-mutation comparison sequence and the total number of classification sets contained in the sequence, reflecting the frequency and coverage of feature points in normal behavior. It filters out occasional and sparse fields, avoids false feature points caused by individual noisy logs, and reduces the computational load and false alarm rate of subsequent analysis. Log entries with the same key fields in the mutation observation sequence are grouped into the same category, which facilitates comparison within homogeneous data and eliminates differences caused by different log types. Based on the occurrence frequency of stable feature points, detection features are calculated. First, abnormal mutation logs that deviate significantly from the group are divided. The remaining logs that meet the threshold constitute a stable observation group. Further analysis within the stable observation group identifies log key fields that are unique to a single log entry and not found in other entries as mutation features. Logs with this feature are judged as abnormal mutation logs. This can discover hidden attacks that appear normal in overall pattern but have abnormal individual fields, further improving the accuracy of network security data processing based on big data.
[0121] Specifically, in step S6, the process of extracting log entries containing a single stable feature point for feature point response analysis includes,
[0122] Sequentially obtain the number of fields contained in the single stable feature point in the previous category set of the category set to which the log entry containing the single stable feature point belongs;
[0123] Calculate the field completeness of a single stable feature point relative to each single stable feature point in the previous classification set based on the number of fields.
[0124] Specifically, in implementation, field completeness is set to, for example, the ratio of the number of fields contained in a single stable feature point in the current set to the number of fields contained in a single stable feature point in the previous classification set.
[0125] Construct a field completeness curve with time on the x-axis and field completeness on the y-axis, and determine the slope of the field completeness curve as the feature point response degradation degree;
[0126] It is understandable that the field completeness curve is generated by removing the coupling of points with a large degree of dispersion. This is an existing technology and will not be elaborated here.
[0127] If the feature point response degradation is negative, then the log entry containing a single stable feature point is identified as an anomalous mutation log.
[0128] Specifically, the aforementioned mutation analysis of this invention mainly targets similar log entries containing more than one stable feature point. For log entries containing only a single stable feature point, since they cannot participate in the calculation of group detection features, they are prone to becoming detection blind spots. Extracting such log entries for independent analysis ensures that all suspicious logs are included in the detection process, avoiding the omission of logs that contain only a single key feature but are actually abnormal, and significantly improving the completeness of detection. If the completeness of a certain key field continues to decline in the time series, it means that the amount of information contained in that field is decreasing, and it is very likely that it has suffered an attack or data collection failure. Quickly judging each log containing a single stable feature point not only improves detection efficiency but also further improves the accuracy of network security data processing based on big data.
[0129] Please see Figure 4 As shown, this is a structural schematic diagram of a network security data processing device based on big data according to an embodiment of the present invention. As an implementation of the methods shown in the above figures, the present invention provides an embodiment of a network security data processing device based on big data. This device embodiment is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices, including:
[0130] The data acquisition module is used to divide the acquired security log data into several log entries in a time sequence according to a preset time window. Several log entries in each time window constitute an analysis set, and several log mutation features that affect log status mutations are extracted from each analysis set.
[0131] The data labeling module, which is connected to the data acquisition module, is used to determine the channel resolution difference degree based on log mutation characteristics and channel resolution difference characteristics, so as to determine the mutation label of each analysis set;
[0132] The data classification module, which is connected to the data acquisition module and the data labeling module respectively, is used to combine a preset number of analysis sets that are identified as mutation tags in chronological order to construct a mutation observation sequence, and to construct several non-mutation comparison sequences corresponding to each analysis set based on the continuous state and continuous number of non-mutation tags.
[0133] The feature extraction module, which is connected to the data classification module, is used to extract log entries from non-mutation comparison sequences, analyze the corresponding log key fields to determine feature points, and calculate the density coefficient of each feature point to determine stable feature points for each mutation observation sequence.
[0134] The data analysis module, connected to the feature extraction module, is used to extract similar log entries belonging to different analysis sets in the mutation observation sequence based on key log fields, extract detection features of each similar log entry based on stable feature points to obtain stable observation groups, perform mutation analysis, and extract log entries containing a single stable feature point for feature point response analysis.
[0135] This application provides a storage medium for storing computer-executable instructions, which, when executed, implement the steps of any of the above embodiments of the network security data processing method based on big data.
[0136] This application provides a storage medium for storing computer-executable instructions, which, when executed, implement the steps of any of the above embodiments of the network security data processing method based on big data.
[0137] It should be noted that the embodiments concerning storage media in this specification and the embodiments of the network security data processing method based on big data in this specification are based on the same inventive concept. Therefore, the specific implementation of this embodiment can be referred to the implementation of the corresponding network security data processing method based on big data mentioned above, and the repeated parts will not be described again.
[0138] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.
[0139] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, when implementing the embodiments of this specification, the functions of each unit can be implemented in one or more software and / or hardware.
[0140] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0141] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0142] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0143] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0144] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0145] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.
[0146] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.
[0147] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.
Claims
1. A network security data processing method based on big data, characterized in that, include: The acquired security log data is divided into several log entries in a time sequence according to a preset time window. Several log entries in each time window constitute an analysis set. Several log mutation features that affect log state mutations are extracted from each analysis set. Based on the log mutation characteristics and channel parsing difference characteristics, the channel parsing difference degree is determined to determine the mutation label of each analysis set; The predetermined number of analysis sets identified as mutation tags are combined in chronological order to construct mutation observation sequences. Based on the continuous state and number of non-mutation tags, several non-mutation comparison sequences corresponding to each analysis set are constructed. Log entries in the non-mutation comparison sequences are extracted and the corresponding key log fields are analyzed to determine feature points. The density coefficient of each feature point is calculated to determine the stable feature points for each mutation observation sequence. Based on the key fields of the logs, log entries of the same type belonging to different analysis sets in the mutation observation sequence are extracted. Based on the stable feature points, the detection features of each log entry of the same type are extracted to obtain a stable observation group for mutation analysis. Extract log entries containing a single stable feature point for feature point response analysis; The mutation analysis involves extracting mutation features from stable observation groups to filter abnormal mutation logs, while the feature point response analysis involves determining the feature point response degradation degree based on the field completeness of a single stable feature point relative to each single stable feature point in the previous classification set to filter abnormal mutation logs.
2. The network security data processing method based on big data according to claim 1, characterized in that, The process of extracting several log mutation features that affect log state mutations from each of the aforementioned analysis sets includes, Extract the parsed data corresponding to the security log data, and determine the log length, key log fields, and time interval between adjacent security logs in the analysis set; The log length change rate is determined based on the log length, the keyword frequency change rate is determined based on the log key fields, and the timestamp interval variation coefficient is determined based on the time interval between adjacent security logs. These are then normalized and weighted to serve as log mutation features.
3. The network security data processing method based on big data according to claim 2, characterized in that, The process of determining the channel resolution difference degree based on the log mutation characteristics and channel resolution difference characteristics, in order to determine the mutation label of each set, includes the following: In response to the log mutation feature being greater than or equal to a preset log mutation feature threshold, the channel parsing difference feature is determined and obtained; Calculate the channel resolution difference degree based on the channel resolution difference characteristics; If the channel resolution difference is greater than or equal to the preset resolution difference threshold, then a mutation label is set; If the channel resolution difference is less than a preset resolution difference threshold, then a non-mutation label is set.
4. The network security data processing method based on big data according to claim 3, characterized in that, The process of constructing several non-mutation contrast sequences corresponding to each analysis set based on the continuous states and continuous quantities of non-mutation tags includes: Select the analysis set that is determined to be a non-mutation tag, and determine the adjacent non-mutation tag analysis sets as non-mutation tag groups; If the non-mutation tag group meets the preset quantity condition, then the non-mutation tag group is determined to be a non-mutation comparison sequence; The preset quantity condition is that the number of consecutive analysis sets included in the non-mutation tag group is greater than or equal to a preset consecutive quantity threshold.
5. The network security data processing method based on big data according to claim 4, characterized in that, The process of calculating the density coefficient of each feature point to determine stable feature points for each of the said mutation observation sequences includes, Log entries are extracted from the non-mutation comparison sequence, and the key log fields corresponding to each log entry are parsed and identified. Identify key log fields related to abnormal behavior as feature points; Feature points that meet the density screening criteria are identified as stable feature points. The density screening condition is that the density coefficient is greater than a preset density coefficient threshold. The density coefficient is determined by the total number of occurrences of each feature point in the non-mutation comparison sequence and the total number of classification sets contained in the non-mutation comparison sequence.
6. The network security data processing method based on big data according to claim 5, characterized in that, The process of extracting detection features from stable feature points of each similar log entry to obtain a stable observation group and performing mutation analysis includes: Log entries with the same key field in the mutation observation sequence are grouped into the same type of log entries; The detection features are determined based on the frequency of occurrence of each stable feature point in each of the aforementioned log entries of the same type to classify the log entries contained in the same type of log entries as abnormal mutation logs or stable observation groups; A log key field contained in a single log entry in the stable observation group that does not exist in other log entries in the stable observation group is identified as a mutation feature; Log entries exhibiting the aforementioned mutation characteristics are identified as anomalous mutation logs.
7. The network security data processing method based on big data according to claim 6, characterized in that, The detection features are determined based on the frequency of occurrence of each stable feature point in each of the aforementioned log entries of the same type, thereby classifying the log entries contained in the same type of log entries into either abnormal mutation logs or stable observation groups, wherein... Select similar log entries containing more than one stable feature point to determine the detection features; If the detected feature is greater than the preset detected feature threshold, then the log entries included in the same type of log entries are classified as abnormal mutation logs. If the detection feature is less than or equal to the preset detection feature threshold, then the log entries included in the same type of log entries are classified as a stable observation group.
8. The network security data processing method based on big data according to claim 7, characterized in that, The process of extracting log entries containing single stable feature points for feature point response analysis includes, Sequentially obtain the number of fields contained in the single stable feature point in the previous category set of the category set to which the log entry containing the single stable feature point belongs; Based on the number of fields, the field completeness of the single stable feature point relative to each single stable feature point in the previous classification set is calculated. A field completeness curve is constructed with time as the horizontal axis and the field completeness as the vertical axis. The slope of the field completeness curve is determined to be the feature point response degradation degree. If the feature point response degradation is negative, then the log entry containing a single stable feature point is determined to be an abnormal mutation log.
9. A big data-based network security data processing apparatus applied to the big data-based network security data processing method according to any one of claims 1-8, characterized in that, include: The data acquisition module is used to divide the acquired security log data into several log entries in a time sequence according to a preset time window. Several log entries in each time window constitute an analysis set, and several log mutation features that affect log status mutations are extracted from each analysis set. A data labeling module, which is connected to the data acquisition module, is used to determine the channel resolution difference degree based on the log mutation characteristics and channel resolution difference characteristics, so as to determine the mutation label of each analysis set; The data classification module is connected to the data acquisition module and the data labeling module respectively. It is used to combine a preset number of analysis sets that are determined as mutation tags in chronological order to construct a mutation observation sequence, and to construct several non-mutation comparison sequences corresponding to each analysis set based on the continuous state and continuous number of non-mutation tags. The feature extraction module, which is connected to the data classification module, is used to extract log entries from the non-mutation comparison sequence, analyze the corresponding log key fields to determine feature points, and calculate the density coefficient of each feature point to determine the stable feature points for each mutation observation sequence. The data analysis module, connected to the feature extraction module, is used to extract similar log entries belonging to different analysis sets in the mutation observation sequence based on the log key fields, extract the detection features of each similar log entry based on stable feature points to obtain stable observation groups, perform mutation analysis, and extract log entries containing a single stable feature point for feature point response analysis.
10. A storage medium for storing computer-executable instructions, characterized in that, When the stored computer-executable instructions are executed, they implement the steps of the network security data processing method based on big data as described in any one of claims 1-8.
Citation Information
Patent Citations
Network security alarm system and method based on big data
CN119254520A
