A heterogeneous network security big data governance method and system
By building a pre-classification model and multi-computing level node allocation, combining semantic labels and big data analysis, the problems of low data processing efficiency and classification difficulties in heterogeneous networks are solved, and efficient and accurate data governance and decision support are achieved.
Patent Information
- Application Number
- CN202411304034.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-19
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-09-19
AI Technical Summary
Traditional data processing methods are inefficient when facing massive and diverse heterogeneous network data, and classification methods are difficult to accurately identify unknown or unlabeled data categories and attributes, and cannot adapt to changes in cyber attack methods.
Build a pre-classification model, dynamic allocation and data pre-processing of processing nodes at multiple computing levels, combined with semantic label annotation and big data analysis technology, features are extracted and visualized to realize rapid data classification and efficient governance.
It significantly improves data processing efficiency and classification accuracy, ensures efficient utilization of resources, shortens classification time, improves decision-making efficiency, and provides timely and effective support for network security.
Smart Images

Figure CN118820910B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of network data processing technology, and in particular to a heterogeneous network security big data management method and system. Background Art
[0002] With the rapid development of information technology, the network environment is becoming increasingly complex, and the surge in heterogeneous data sources has brought unprecedented challenges to the governance of network security big data. Traditional data processing and classification methods are unable to cope with massive and diverse heterogeneous network data. For example:
[0003] 1. Traditional data processing methods often use serial processing mode, which cannot fully utilize the parallel processing capabilities of modern computing resources, resulting in low data processing efficiency and difficulty in meeting real-time requirements. In addition, the lack of unified standards and specifications in the data preprocessing stage makes data cleaning, deduplication, normalization and other steps time-consuming, further reducing processing efficiency.
[0004] 2. The diversity and complexity of heterogeneous network data make it difficult to directly apply traditional classification methods. Existing classification algorithms often cannot accurately identify the categories or attributes of large amounts of unknown or unlabeled data. At the same time, the accuracy and reliability of classification results are also affected due to the weak correlation between data or inaccurate feature extraction. In addition, with the continuous evolution and changes in network attack methods, traditional classification models are difficult to adapt to new threat forms, resulting in a significant reduction in classification results.
[0005] Therefore, it is necessary to provide a heterogeneous network security big data governance method and system to solve the above technical problems. Summary of the invention
[0006] In order to solve the above technical problems, the present invention provides a heterogeneous network security big data governance method and system for solving the problems of low processing efficiency, difficult classification and low accuracy in processing heterogeneous network data.
[0007] The present invention provides a heterogeneous network security big data governance method, the governance method comprising the following steps:
[0008] Build a pre-classification model based on historical heterogeneous network data;
[0009] Obtain current heterogeneous network data and pre-classify homogeneous data to obtain multiple initial classification data sets;
[0010] Preset multiple processing nodes of different computing levels, and assign processing nodes of corresponding computing levels to each initial classification data set;
[0011] By allocating corresponding processing nodes to the initial classification data set, the initial classification data set is cleaned, and all data in the processed initial classification data set are marked with semantic labels;
[0012] Based on the relevance of semantic tags, all processed initial classification data sets are extracted and classified through the processing center, and the final classification data is obtained after unified format conversion and storage;
[0013] Big data analysis technology is used to extract features from the final classified data, and visualization technology is used to display the characteristic regularity results.
[0014] Preferably, the steps of constructing a pre-classification model based on historical heterogeneous network data are as follows:
[0015] Collect data samples from historical heterogeneous network data sources, including but not limited to network traffic, logs, and events;
[0016] Preprocess the data samples, extract features from the preprocessed data samples, and identify the mapping relationship between the features and the types of the data samples;
[0017] The mapping relationship between features and data sample types is used as a training set, and the deep learning model is used as an architecture for training to obtain a pre-classification model.
[0018] Preferably, the steps of acquiring the current heterogeneous network data and pre-classifying the homogeneous data to obtain multiple initial classification data sets are as follows:
[0019] Acquire current heterogeneous network data through real-time data stream collection system;
[0020] The current heterogeneous network data is input into the constructed pre-classification model, the homologous features in the current heterogeneous network data are identified, the current heterogeneous network data is preliminarily classified, and multiple initial classification data sets are obtained.
[0021] Preferably, the steps of presetting a plurality of processing nodes of different computing levels and assigning a processing node of a corresponding computing level to each initial classification data set are specifically as follows:
[0022] Evaluate the computing level of each processing node based on its computing power, memory size, and network bandwidth;
[0023] According to the size of each initial classification data set and the required computing resources, the processing requirement level of each initial classification data set is evaluated, and according to the evaluated processing requirement level result, a processing node of the corresponding computing level is allocated to each initial classification data set.
[0024] Preferably, the steps of allocating corresponding processing nodes to the initial classification data set, cleaning the initial classification data set, and marking semantic labels on all data in the processed initial classification data set are specifically as follows:
[0025] On each processing node, the allocated initial classification data set is cleaned, including removing invalid data, correcting erroneous data, and filling missing values;
[0026] Add semantic labels to the data in the processed initial classification dataset.
[0027] Preferably, based on the association of semantic tags, all processed initial classification data sets are extracted and classified by the processing center, and the final classification data is obtained after the format is uniformly converted and stored, and the specific steps are as follows:
[0028] The processing center collects the processed data from each processing node, identifies the relevance of semantic tags based on natural language processing technology, extracts the data from the processed initial classification data set, and performs secondary classification;
[0029] Perform unified format conversion on secondary classified data, including standardization of data format and unification of data types;
[0030] Store the data in a unified format in a suitable data warehouse or database to obtain the final classified data.
[0031] Preferably, the steps of extracting features from the final classified data using big data analysis technology and displaying the feature regularity results using visualization technology are as follows:
[0032] Use data mining technology to extract features that reflect the inherent laws and trends of the data from the final classified data;
[0033] Analyze the extracted features through statistical analysis and association analysis algorithms to obtain the results of the analyzed data regularity;
[0034] Use visualization tools to output the analyzed data pattern results as display graphs.
[0035] A heterogeneous network security big data governance system, the governance system comprising:
[0036] Model building module, used to build pre-classification models based on historical heterogeneous network data;
[0037] The pre-classification module is used to obtain the current heterogeneous network data and pre-classify the homogeneous data to obtain multiple initial classification data sets;
[0038] A node allocation module is used to preset a plurality of processing nodes of different computing levels and allocate a processing node of a corresponding computing level to each initial classification data set;
[0039] A node processing module is used to clean the initial classification data set by assigning corresponding processing nodes to the initial classification data set, and mark semantic labels on all data in the processed initial classification data set;
[0040] The central processing module is used to extract and classify all processed initial classification data sets based on the relevance of semantic tags through the processing center, and store them after unified format conversion to obtain the final classification data;
[0041] The visualization display module is used to extract features from the final classified data using big data analysis technology, and to display the feature regularity results through visualization technology.
[0042] Compared with related technologies, the heterogeneous network security big data governance method and system provided by the present invention has the following beneficial effects:
[0043] The present invention significantly improves data processing efficiency and ensures efficient resource utilization and reduced error rates by combining dynamic allocation of multiple preset computing level processing nodes, efficient execution of data preprocessing and standardization, accurate implementation of feature extraction and optimization, and application of pre-classification models and clarity of semantic labeling. At the same time, by using efficient classification algorithms and visual decision-making aids, rapid classification of heterogeneous network big data is achieved, which not only shortens the classification time, but also improves the accuracy of classification and decision-making efficiency, thereby providing users with more timely and effective support for network security big data governance. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 A schematic diagram of a process for heterogeneous network security big data governance method provided by the present invention;
[0045] Figure 2 A system block diagram of a heterogeneous network security big data governance system provided by the present invention. DETAILED DESCRIPTION
[0046] The present invention will be further described below in conjunction with the accompanying drawings and implementation modes.
[0047] Embodiment 1
[0048] like Figure 1 As shown, a heterogeneous network security big data governance method includes the following steps:
[0049] 100. Build a pre-classification model based on historical heterogeneous network data;
[0050] In the specific implementation process, step 100 specifically includes the following steps:
[0051] 101. Collect data samples from historical heterogeneous network data sources, including but not limited to network traffic, logs, and events;
[0052] Specifically, data samples are collected from historical heterogeneous network data sources by using data collection software or API interfaces, including collecting data from internal enterprise networks, cloud service providers, third-party security services, open source projects, etc., such as obtaining network traffic data from firewalls, intrusion detection systems (IDS), and network traffic analysis tools; extracting log data from log files of servers, applications, and devices; and collecting security event data from security event management (SIEM) systems.
[0053] 102. Preprocess the data sample, extract features from the preprocessed data sample, and identify the mapping relationship between the features and the type of the data sample;
[0054] Specifically, first remove noise, outliers, missing values, etc. in the data samples; secondly, reduce the redundancy of the data samples through deduplication operations; finally, normalize or standardize the numerical data to eliminate the impact of different dimensions and distributions on data analysis, and extract features from the preprocessed data samples, including but not limited to statistical characteristics of network traffic (such as packet size, transmission rate, traffic pattern, etc.), keywords or event sequences in logs, and specific attributes in security events. Then, through automatic labeling, use existing classification models or rules to preliminarily classify the data samples, and establish a mapping relationship between features and types based on the classification results.
[0055] 103. The mapping relationship between features and data sample types is used as a training set, and the deep learning model is used as an architecture for training to obtain a pre-classification model.
[0056] 200. Acquire current heterogeneous network data, and pre-classify homogeneous data to obtain multiple initial classification data sets;
[0057] In the specific implementation process, step 200 specifically includes the following steps:
[0058] 201. Obtain current heterogeneous network data through a real-time data stream collection system;
[0059] Specifically, a real-time data stream collection system is deployed at key nodes of the network. It should be noted that these nodes include firewalls at the network boundary, core switches, and important servers or terminal devices. The collection system will then begin to capture data flows that meet the rules. The data flows include network packets, log entries, event notifications, and other forms.
[0060] 202. Input the current heterogeneous network data into the constructed pre-classification model, identify homologous features in the current heterogeneous network data, perform preliminary classification on the current heterogeneous network data, and obtain multiple initial classification data sets;
[0061] The current heterogeneous network data is preprocessed, specifically including first removing noise, outliers and missing values in the data samples; secondly, reducing the redundancy of the data samples through deduplication operations, and inputting the preprocessed current heterogeneous network data into the constructed pre-classification model, identifying homologous features in the current heterogeneous network data through the constructed pre-classification model, and dividing the current heterogeneous network data into multiple initial classification data sets, each of which contains data items with similar features.
[0062] 300. Preset a plurality of processing nodes of different computing levels, and assign a processing node of a corresponding computing level to each initial classification data set;
[0063] In the specific implementation process, step 300 specifically includes the following steps:
[0064] 301. Evaluate the computing level of each processing node according to the computing power, memory size, and network bandwidth of the processing node;
[0065] Specifically, collect key performance indicator data of each processing node, including but not limited to CPU model and core number, memory capacity, disk I / O speed, network bandwidth, etc., and conduct quantitative evaluation of the computing power, memory size and network bandwidth of each processing node based on the collected key performance indicator data. Specifically, convert the key performance indicator data into comparable metrics (such as floating-point operations per second, memory bandwidth, network throughput, etc.) and then conduct quantitative evaluation. According to the evaluation results, divide the processing nodes into different computing levels;
[0066] For example, based on simple threshold judgments, such as classifying nodes with more than 16 CPU cores and more than 64GB of memory as high-level nodes; or using more complex clustering algorithms (such as K-means) to automatically divide the levels. The level division should ensure that the nodes in the same level are similar in performance to facilitate subsequent resource allocation and management.
[0067] 302. Evaluate the processing requirement level of each initial classification data set according to the size of each initial classification data set and the required computing resources, and allocate processing nodes of corresponding computing levels to each initial classification data set according to the evaluated processing requirement level results;
[0068] Specifically, the size, complexity and required computing resources (such as CPU time, memory usage, network bandwidth, etc.) of each initial classified data set are evaluated, and based on the evaluation results, the initial classified data sets are divided into different processing requirement levels. In this embodiment, the levels are automatically divided based on simple rules (such as data with a volume greater than 10GB and requiring complex calculations are divided into high levels) or more complex evaluation models (such as prediction models based on machine learning); then, corresponding processing nodes are allocated to each initial classified data set according to the processing requirement level and the processing node calculation level divided in step 301.
[0069] 400. Clean the initial classification data set by assigning corresponding processing nodes to the initial classification data set, and mark all data in the processed initial classification data set with semantic labels;
[0070] In the specific implementation process, step 400 specifically includes the following steps:
[0071] 401. On each processing node, the allocated initial classification data set is cleaned, specifically including removing invalid data, correcting erroneous data, and filling missing values;
[0072] Specifically, by using Python's Pandas library to perform duplicate deletion and conditional screening to identify and delete invalid data in the initial classification data set, and check and correct erroneous data in the data set, finally fill in statistical values (such as mean, median, mode) and retain missing values as separate categories.
[0073] 402. Adding semantic labels to the data in the processed initial classification data set;
[0074] Specifically, NLP technology is used to identify keywords and topics in text data, and corresponding semantic tags are added to them accordingly.
[0075] 500. Based on the relevance of semantic tags, all processed initial classification data sets are extracted and classified through the processing center, and the final classification data is obtained after the format is uniformly converted and stored;
[0076] In the specific implementation process, step 500 specifically includes the following steps:
[0077] 501. Collect the data processed by each processing node through the processing center, identify the relevance of semantic tags according to natural language processing technology, extract the data in the processed initial classification data set and perform secondary classification;
[0078] Specifically, the processing center establishes a connection with each processing node through a network communication protocol (such as HTTP, FTP, etc.), and collects the data processed by each processing node in real time. After collecting the data, the processing center uses NLP technology to analyze the correlation between semantic tags, further extracts data subsets with similar or related semantics, and performs secondary classification;
[0079] Exemplarily, the association network between tags is identified by calculating the co-occurrence frequency or semantic similarity between tags or using a graph algorithm (such as PageRank or community detection algorithm).
[0080] 502. Perform unified format conversion on secondary classified data, including standardization of data format and unification of data types;
[0081] Specifically, the processing center converts the secondary classified data into a unified format according to preset data format standards (such as JSON, XML, CSV, etc.). It should be noted that during the processing, the field types in the data also need to be unified. For example, the character strings representing dates are uniformly converted into date types, and the character strings representing amounts are uniformly converted into floating-point types, etc. This helps to ensure the consistency and accuracy of the data and avoid type mismatches or precision loss in subsequent data processing.
[0082] 503. Store the data in a unified format into a suitable data warehouse or database to obtain the final classified data;
[0083] Specifically, the processing center uses data loading tools (such as ETL tools, SQL statements, etc.) to load the data in a unified format into the target data warehouse or database.
[0084] 600. Use big data analysis technology to extract features from the final classified data, and use visualization technology to display the feature regularity results;
[0085] In the specific implementation process, step 600 specifically includes the following steps:
[0086] 601. Use data mining technology to extract features that reflect the inherent laws and trends of the data from the final classified data;
[0087] Specifically, RapidMiner and KNIME are used to extract features that reflect the inherent laws and trends of the data from the final classified data. In this example, a clustering algorithm is used to group the data into clusters with similar characteristics; a classification algorithm is used to predict the category of data points; association rule mining is used to discover frequent item sets and association rules; and time series analysis is used to identify periodicity, trend, and seasonal patterns in the data.
[0088] 602. Analyze the extracted features through statistical analysis and association analysis algorithms to obtain the results of the analyzed data regularity;
[0089] Specifically, the mean, median, and standard deviation methods in descriptive statistics and hypothesis testing and variance analysis in inferential statistics are used to summarize the basic situation and differences of data features, and the Apriori algorithm is used to identify the correlation between features, and finally the results of statistical analysis and correlation analysis are obtained.
[0090] 603. Use visualization tools to output the results of the analyzed data rules as display graphs;
[0091] Specifically, the analyzed data pattern results are output as display graphs through visualization tools such as Tableau, Power BI, Matplotlib, etc., where the display graphs include chart types, such as bar graphs, line graphs, scatter graphs, and heat maps.
[0092] Embodiment 2
[0093] like Figure 2 As shown, a heterogeneous network security big data governance system applied to a heterogeneous network security big data governance method, the governance system specifically includes:
[0094] Model building module, used to build pre-classification models based on historical heterogeneous network data;
[0095] The pre-classification module is used to obtain the current heterogeneous network data and pre-classify the homogeneous data to obtain multiple initial classification data sets;
[0096] A node allocation module is used to preset a plurality of processing nodes of different computing levels and allocate a processing node of a corresponding computing level to each initial classification data set;
[0097] A node processing module is used to clean the initial classification data set by assigning corresponding processing nodes to the initial classification data set, and mark semantic labels on all data in the processed initial classification data set;
[0098] The central processing module is used to extract and classify all processed initial classification data sets based on the relevance of semantic tags through the processing center, and store them after unified format conversion to obtain the final classification data;
[0099] The visualization display module is used to extract features from the final classified data using big data analysis technology, and to display the feature regularity results through visualization technology.
[0100] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0101] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.
[0102] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
Claims
1. A heterogeneous network security big data governance method, characterized in that: The governance method comprises the following steps: Build a pre-classification model based on historical heterogeneous network data; Obtain current heterogeneous network data and pre-classify homogeneous data to obtain multiple initial classification data sets; Preset multiple processing nodes of different computing levels, and assign processing nodes of corresponding computing levels to each initial classification data set. The specific steps are as follows: Evaluate the computing level of each processing node based on its computing power, memory size, and network bandwidth; According to the size of each initial classification data set and the required computing resources, the processing requirement level of each initial classification data set is evaluated, and according to the evaluated processing requirement level result, a processing node of a corresponding computing level is allocated to each initial classification data set; By allocating corresponding processing nodes to the initial classification data set, the initial classification data set is cleaned, and all data in the processed initial classification data set are marked with semantic labels; Based on the relevance of semantic tags, all processed initial classification data sets are extracted and classified through the processing center, and the final classification data is obtained after unified format conversion and storage. The specific steps are as follows: The processing center collects the processed data from each processing node, identifies the relevance of semantic tags based on natural language processing technology, extracts the data from the processed initial classification data set, and performs secondary classification; Perform unified format conversion on secondary classified data, including standardization of data format and unification of data types; Store the data in a unified format into a suitable data warehouse or database to obtain the final classified data; Big data analysis technology is used to extract features from the final classified data, and visualization technology is used to display the characteristic regularity results.
2. A heterogeneous network security big data management method according to claim 1, characterized in that: The steps of constructing a pre-classification model based on historical heterogeneous network data are as follows: Collect data samples from historical heterogeneous network data sources, including but not limited to network traffic, logs, and events; Preprocess the data samples, extract features from the preprocessed data samples, and identify the mapping relationship between the features and the types of the data samples; The mapping relationship between features and data sample types is used as a training set, and the deep learning model is used as an architecture for training to obtain a pre-classification model.
3. A heterogeneous network security big data governance method according to claim 1, characterized in that: The steps of obtaining the current heterogeneous network data and pre-classifying the homogeneous data to obtain multiple initial classification data sets are as follows: Acquire current heterogeneous network data through real-time data stream collection system; The current heterogeneous network data is input into the constructed pre-classification model, the homologous features in the current heterogeneous network data are identified, the current heterogeneous network data is preliminarily classified, and multiple initial classification data sets are obtained.
4. A heterogeneous network security big data governance method according to claim 1, characterized in that: The initial classification data set is cleaned by allocating corresponding processing nodes to the initial classification data set, and all data in the processed initial classification data set are marked with semantic labels, and the specific steps are as follows: On each processing node, the allocated initial classification data set is cleaned, including removing invalid data, correcting erroneous data, and filling missing values; Add semantic labels to the data in the processed initial classification dataset.
5. According to claim 1, a heterogeneous network security big data governance method is characterized in that: The steps of extracting features from the final classified data using big data analysis technology and displaying the feature regularity results using visualization technology are as follows: Use data mining technology to extract features that reflect the inherent laws and trends of the data from the final classified data; Analyze the extracted features through statistical analysis and association analysis algorithms to obtain the results of the analyzed data regularity; Use visualization tools to output the analyzed data pattern results as display graphs.
6. A heterogeneous network security big data management system, applied to a heterogeneous network security big data management method according to any one of claims 1 to 5, characterized in that: The governance system includes: Model building module, used to build pre-classification models based on historical heterogeneous network data; The pre-classification module is used to obtain the current heterogeneous network data and pre-classify the homogeneous data to obtain multiple initial classification data sets; The node allocation module is used to preset multiple processing nodes of different computing levels and allocate processing nodes of corresponding computing levels to each initial classification data set. The specific steps are as follows: Evaluate the computing level of each processing node based on its computing power, memory size, and network bandwidth; According to the size of each initial classification data set and the required computing resources, the processing requirement level of each initial classification data set is evaluated, and according to the evaluated processing requirement level result, a processing node of a corresponding computing level is allocated to each initial classification data set; A node processing module is used to clean the initial classification data set by allocating corresponding processing nodes to the initial classification data set, and mark semantic labels on all data in the processed initial classification data set; The central processing module is used to extract and classify all processed initial classification data sets based on the relevance of semantic tags through the processing center, and store them after unified format conversion to obtain the final classification data. The specific steps are as follows: The processing center collects the processed data from each processing node, identifies the relevance of semantic tags based on natural language processing technology, extracts the data from the processed initial classification data set, and performs secondary classification; Perform unified format conversion on secondary classified data, including standardization of data format and unification of data types; Store the data in a unified format into a suitable data warehouse or database to obtain the final classified data; The visualization display module is used to extract features from the final classified data using big data analysis technology, and to display the feature regularity results through visualization technology.
Citation Information
Cited By
Data management method and system for coal mine, and storage medium
CN120408725A
