ETL-based intelligent measurement network data fusion and modeling method and system

Abstract

The invention relates to an ETL-based intelligent measurement network data fusion and modeling method and system, and the method comprises the steps: correspondingly building a plurality of data collection modules in a preset data warehouse based on preset modular data features; the data acquisition module is used for acquiring network security data of the intelligent measurement network, preprocessing the network security data, and correspondingly storing the network security data in the data acquisition module according to the type of the network security data and modular data characteristics; analyzing the network security data in each data acquisition module to obtain current model application features; according to the current model application features and the modular data features, screening a data acquisition module adapted to the current intelligent measurement network, extracting network security data in the data acquisition module adapted to the current intelligent measurement network, and obtaining current model application feature data; and modeling by using the current model application feature data to obtain a network security data model. The method has the effect of improving the practicability of the data model by using the ETL technology.

Description

Technical Field

[0001] The present invention relates to the technical field of network data fusion and modeling, and in particular to an ETL-based intelligent measurement network data fusion and modeling method and system. Background Art

[0002] In the field of secure data processing of intelligent measurement networks, there are generally problems such as high processing complexity, diverse data sources, and inconsistent data formats. The high dimensionality of data and the wide range of sources make data fusion and synchronization particularly difficult. Although the industry has adopted some data fusion strategies, there are still obvious shortcomings in processing efficiency and the wide application of data models.

[0003] In response to the above problems, the industry currently usually relies on ETL (Extract, Transform, Load) technology and data warehouse technology to store and manage network security related data. However, when faced with massive intelligent measurement network data with complex and changing characteristics, although ETL technology has significant advantages in processing large-scale data and optimizing data management processes, the characteristics of this technology have not been effectively utilized in data integration and model building methods, which leads to confusion in the storage and management of intelligent measurement network security data, thereby reducing the practicality of the constructed data model. Summary of the invention

[0004] In order to improve the practicality of data models by using ETL technology, the present application provides an ETL-based intelligent measurement network data fusion and modeling method, system, computer device and storage medium.

[0005] In the first aspect, the above-mentioned invention objective of the present application is achieved through the following technical solutions: An ETL-based intelligent measurement network data fusion and modeling method, the ETL-based intelligent measurement network data fusion and modeling method comprising: Based on the preset modular data features, several data collection modules are correspondingly established in the preset data warehouse; Acquire network security data of the intelligent measurement network, and after preprocessing the network security data, store the data in the data acquisition module accordingly according to the type of the network security data and the modular data characteristics; Analyze the network security data in each of the data acquisition modules to obtain the current model application characteristics; According to the current model application characteristics and the modular data characteristics, a data acquisition module adapted to the current intelligent measurement network is selected, network security data in the data acquisition module adapted to the current intelligent measurement network is extracted, and current model application characteristic data is obtained; The current model is used to apply the characteristic data for modeling to obtain a network security data model.

[0006] By adopting the above technical solution, based on the preset modular data features, a number of data collection modules are correspondingly established in the preset data warehouse, and the network security data is stored according to the type and modular data features of the network security data. This modular design improves the flexibility of data management and the flexibility of the data collection and storage process, so that the data collection module can be easily adjusted or expanded according to actual needs, and different types of data are stored in the corresponding modules, which helps to reduce data redundancy and improve the efficiency of data retrieval and processing; the acquired network security data is preprocessed, including cleaning, conversion and other operations, which can remove noise data, correct erroneous data, and ensure the data quality of subsequent analysis, and then stored according to the data type and modular data features. The preprocessed and classified stored data is more standardized and consistent, which is conducive to subsequent feature extraction and model construction; by analyzing each data collection The network security data in the module can reveal the correlation and potential rules between the current network security data and the current application scenario, so that the obtained model application characteristics can be used to screen the data acquisition module that is suitable for the current intelligent measurement network, and then select the appropriate model for training; in addition to the current model application characteristics, combined with the preset modular data characteristics, the data acquisition module that is suitable for the current intelligent measurement network is screened, and the network security data is extracted for modeling, which can significantly improve the accuracy and applicability of the model; based on this, through modular data acquisition, data preprocessing, feature extraction, data screening and modeling, the effective integration and efficient utilization of network security data in the intelligent measurement network are realized, which not only improves the flexibility and efficiency of data management, but also significantly improves the accuracy and applicability of the model, providing strong support for the operation and management of the intelligent measurement network.

[0007] In a preferred example, the present application can be further configured as follows: the modular data feature is obtained by: Acquire network security data of different types of measured networks, wherein the network security data of different types of measured networks include historical network security data of different types of measured networks and data referenced by network security data models corresponding to different types of measured networks; Classifying the network security data of the different types of measured networks according to data sources to obtain classified network security data; Based on the preset data requirement concept characteristics, each type of classified network security data is analyzed respectively to obtain modular data characteristics corresponding to each type of classified network security data.

[0008] By adopting the above technical solution, network security data of different types of measurement networks are obtained, including historical data and model reference data, covering data of various types of measurement networks, ensuring the breadth and diversity of data, helping to discover the commonalities between different networks, and improving the universality of modular data features. Among them, historical data provides the behavior pattern of the measurement network in the past period of time, while model reference data reflects the performance of the network in a specific scenario. The combination of the two makes the data more representative and can more accurately reflect the actual status of the measurement network; in addition, the network security data of different types of measurement networks are classified according to the data source, such as network communication data, system logs, security alarm information, etc. The classified data is more orderly, which is convenient for subsequent data processing and analysis. Moreover, for data from different sources, its unique modular data features can be extracted more specifically, which facilitates subsequent data fusion and modeling; the preset data requirement concept features refer to the network security related data. According to the concept of requirements, for example, the timeliness of network security related data is required, each classified network security data is analyzed, and the compliance conditions for each classified network security data to meet the timeliness requirements are determined. For example, the requirement is within 24 hours, so the corresponding modular data features are used to collect network security data within 24 hours. Therefore, the preset data requirement concept features provide clear standards and directions for data analysis, ensuring the practicality and accuracy of the extracted modular data features, and when a new type of measurement network appears, its modular data features can be easily obtained in the same way without large-scale modifications to the overall process; based on this, by comprehensively collecting, clearly classifying and accurately analyzing network security data of different types of measurement networks, it provides strong support for subsequent data processing, analysis and modeling processes, improves the comprehensiveness and representativeness of the data, ensures the accuracy and consistency of the modular data features, and has good flexibility and scalability.

[0009] In a preferred example, the present application may be further configured as follows: the data requirement concept features include data novelty concept features, data emphasis concept features, and data stability concept features; The method of analyzing each of the classified network security data based on the preset data requirement concept features to obtain the modular data features corresponding to each of the classified network security data includes: Based on the novel concept features of the data, the key concept features of the data, and the stable concept features of the data, each of the classified network security data is analyzed to determine the data requirement concept features that are suitable for each of the classified network security data, and obtain the adaptive data requirement concept features that are suitable for each of the classified network security data; According to the adaptability data requirement concept feature of each of the classified network security data, determine the data collection index corresponding to each adaptability data requirement concept feature of each of the classified network security data, and the feature usage condition; The data collection index and the feature usage condition are associated to obtain modular data features corresponding to each type of classified network security data.

[0010] By adopting the above technical solutions, different data requirement concept features focus on different aspects of the data, so that the value of the data can be more comprehensively explored, ensuring that these values ​​can be fully utilized in the subsequent data processing, analysis and modeling processes, so that the modular data features obtained based on in-depth analysis can provide more accurate and powerful support for network security decisions, improve the scientificity and effectiveness of decisions, and the adaptive data requirement concept refers to the data requirement concept features that are adapted to each classified network security data. According to the adaptive data requirement concept features, the data collection index and feature usage conditions are judged, and when facing network security data of different types and characteristics, the data collection and processing strategies can be flexibly adjusted to adapt to different Data requirement concept characteristics; through clear data collection indexes and feature usage conditions, the data processing process can be optimized, unnecessary calculations and resource consumption can be reduced, and the efficiency of data processing can be improved. At the same time, clear data collection indexes and feature usage conditions provide a basis for the standardized processing of data, which helps to reduce data ambiguity and errors and improve the reliability and consistency of data; based on this, by introducing data requirement concept characteristics and conducting in-depth analysis of classified network security data based on these characteristics, it not only enhances the depth and breadth of data value mining, but also improves the flexibility and efficiency of data processing. At the same time, it promotes the standardization and normalization of data processing, and supports dynamic adjustment and optimization to adapt to the ever-changing network security environment.

[0011] In a preferred example, the present application may be further configured as follows: analyzing the network security data in each of the data acquisition modules to obtain the current model application features includes: Acquire current application domain characteristics, where the current application domain characteristics refer to standard characteristics of the application domain of the current intelligent measurement network for modeling data; Based on the current application field characteristics, the network security data in each of the data acquisition modules is analyzed to obtain the current model application characteristics corresponding to each of the data acquisition modules.

[0012] By adopting the above technical solution, the current application field characteristics are obtained. These characteristics reflect the specific needs and specifications of the intelligent measurement network in the current application field. Data analysis based on the current application field characteristics can ensure that the extracted current model application characteristics are more in line with the actual needs of the field, thereby enhancing the pertinence and effectiveness of the model application in this field, which helps to reduce the deviation and error of the model in a specific application environment and improve the prediction accuracy and stability of the model; based on the current application field characteristics, the network security data in each data acquisition module is analyzed; the current application field characteristics provide clear guidance for data filtering and preprocessing, which helps to accurately screen out data that is valuable for model application, exclude irrelevant or noise data, and improve the accuracy of data processing. By reducing unnecessary data processing steps and calculations, the application of the current application field characteristics can significantly improve the efficiency of data processing and speed up model training and application; the network security data in each data acquisition module is analyzed to obtain the current model application characteristics corresponding to each module. The current application field characteristics provide unified standards and specifications for data in different data acquisition modules, which helps to promote the integration and coordination of data between different modules and improve the overall consistency and availability of data.

[0013] In a preferred example, the present application may be further configured as follows: the data acquisition module adapted to the current intelligent measurement network is screened according to the current model application characteristics and the modular data characteristics, the network security data in the data acquisition module adapted to the current intelligent measurement network is extracted, and the current model application characteristic data is obtained, specifically including: Based on the modular data features, extracting the feature usage conditions; According to the characteristic usage conditions, the network security data in each of the data acquisition modules is analyzed, and according to the current model application characteristics, the data acquisition modules that are suitable for the current intelligent measurement network are screened, and the network security data in the data acquisition modules that meet the characteristic usage conditions and the network security data in the data acquisition modules that are suitable for the current intelligent measurement network are extracted to obtain the current model application characteristic data.

[0014] By adopting the above technical solution, through filtering of feature usage conditions, data acquisition modules that are highly relevant to the current model application and meet specific conditions can be accurately screened out. The network security data in these modules are more likely to provide valuable input for the model, thereby improving the accuracy and reliability of model prediction and decision-making, excluding data acquisition modules that are irrelevant or less relevant to the current model application, reducing redundant work in data processing and model training, and improving overall efficiency; by extracting network security data from data acquisition modules that meet the feature usage conditions and adapt to the current intelligent measurement network, it is ensured that the current model application feature data finally obtained has high quality and consistency. These data can better reflect the actual status and characteristics of the intelligent measurement network, provide more reliable input for the model, and reduce possible noise and abnormal data, reduce their adverse effects on model performance, and improve the stability and robustness of the model.

[0015] In the second aspect, the above invention objective of the present application is achieved through the following technical solutions: An ETL-based intelligent measurement network data fusion and modeling system, the ETL-based intelligent measurement network data fusion and modeling system comprising: A collection module establishment module is used to establish a number of data collection modules in a preset data warehouse based on preset modular data features; A data storage module, used for obtaining network security data of the intelligent measurement network, and storing the network security data in the data acquisition module according to the type of the network security data and the modular data characteristics after preprocessing the network security data; A current feature analysis module, used to analyze the network security data in each of the data acquisition modules to obtain the current model application features; A feature data extraction module, used to screen a data acquisition module adapted to the current intelligent measurement network according to the current model application feature and the modular data feature, extract network security data from the data acquisition module adapted to the current intelligent measurement network, and obtain current model application feature data; The model building module is used to use the current model application feature data to perform modeling and obtain a network security data model.

[0016] Optionally, the ETL-based intelligent measurement network data fusion and modeling system further includes: An analysis data acquisition module is used to acquire network security data of different types of measured networks, wherein the network security data of different types of measured networks include historical network security data of different types of measured networks and data referenced by network security data models corresponding to different types of measured networks; A data classification module, used to classify the network security data of the different types of measurement networks according to data sources to obtain classified network security data; The feature analysis module is used to analyze each of the classified network security data based on preset data requirement concept features, and obtain modular data features corresponding to each of the classified network security data.

[0017] Optionally, the data requirement concept features include data novelty concept features, data key concept features and data stability concept features; The feature analysis module comprises: A multi-feature analysis submodule, for analyzing each of the classified network security data based on the novel concept feature of the data, the key concept feature of the data, and the stable concept feature of the data, to determine the data requirement concept feature that is suitable for each of the classified network security data, and obtain the adaptability data requirement concept feature that is suitable for each of the classified network security data; An index and condition determination submodule, for determining a data collection index and a feature usage condition corresponding to each adaptability data requirement concept feature of each of the classified network security data according to the adaptability data requirement concept feature of each of the classified network security data; The index and condition association submodule is used to associate the data collection index with the feature usage condition to obtain modular data features corresponding to each type of classified network security data.

[0018] In the third aspect, the above invention objective of the present application is achieved through the following technical solutions: A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned ETL-based intelligent measurement network data fusion and modeling method are implemented.

[0019] In a fourth aspect, the above-mentioned invention objective of the present application is achieved through the following technical solutions: A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned ETL-based intelligent measurement network data fusion and modeling method.

[0020] In summary, the present application includes at least one of the following beneficial technical effects: 1. Based on the preset modular data features, several data collection modules are correspondingly established in the preset data warehouse, and the network security data is stored according to the type and modular data features. This modular design improves the flexibility of data management, as well as the flexibility of data collection and storage processes, so that the data collection modules can be easily adjusted or expanded according to actual needs, and different types of data are stored in corresponding modules, which helps to reduce data redundancy and improve the efficiency of data retrieval and processing; the acquired network security data is preprocessed, including cleaning, conversion and other operations, which can remove noise data, correct erroneous data, and ensure the data quality of subsequent analysis, and then stored according to the data type and modular data features. The preprocessed and classified stored data is more standardized and consistent, which is conducive to subsequent feature extraction and model construction; by analyzing the network security data in each data collection module The network security data can reveal the correlation and potential rules between the current network security data and the current application scenario, so that the obtained model application characteristics can be used to screen the data acquisition module that is suitable for the current intelligent measurement network, and then select the appropriate model for training; in addition to the current model application characteristics, combined with the preset modular data characteristics, the data acquisition module that is suitable for the current intelligent measurement network is screened, and the network security data therein is extracted for modeling, which can significantly improve the accuracy and applicability of the model; based on this, through the steps of modular data acquisition, data preprocessing, feature extraction, data screening and modeling, the effective integration and efficient utilization of network security data in the intelligent measurement network are realized, which not only improves the flexibility and efficiency of data management, but also significantly improves the accuracy and applicability of the model, providing strong support for the operation and management of the intelligent measurement network; 2. Obtain network security data of different types of measurement networks, including historical data and model reference data, covering data of various types of measurement networks, ensuring the breadth and diversity of data, helping to discover the commonalities between different networks, and improving the universality of modular data features. Among them, historical data provides the behavior pattern of the measurement network in the past period of time, while model reference data reflects the performance of the network in a specific scenario. The combination of the two makes the data more representative and can more accurately reflect the actual status of the measurement network; in addition, the network security data of different types of measurement networks are classified according to the data source, such as network communication data, system logs, security alarm information, etc. The classified data is more orderly, which is convenient for subsequent data processing and analysis. For data from different sources, its unique modular data features can be extracted more specifically, which facilitates subsequent data fusion and modeling; the preset data requirement concept features refer to the requirements for network security related data For example, the timeliness of network security related data is required, and each classified network security data is analyzed to determine the compliance conditions for each classified network security data when it meets the timeliness requirements. For example, it is required to be within 24 hours. In this way, the corresponding modular data features are used to collect network security data within 24 hours. Therefore, the preset data requirement concept features provide clear standards and directions for data analysis, ensuring the practicality and accuracy of the extracted modular data features. Moreover, when a new type of measurement network appears, its modular data features can be easily obtained in the same way without large-scale modifications to the overall process. Based on this, by comprehensively collecting, clearly classifying and accurately analyzing network security data of different types of measurement networks, it provides strong support for subsequent data processing, analysis and modeling processes, improves the comprehensiveness and representativeness of the data, ensures the accuracy and consistency of the modular data features, and has good flexibility and scalability. 3. Different data requirement concept features focus on different aspects of the data, so as to more comprehensively explore the value of the data and ensure that these values ​​can be fully utilized in the subsequent data processing, analysis and modeling process. The modular data features based on in-depth analysis can provide more accurate and powerful support for network security decision-making, improve the scientificity and effectiveness of decision-making. The adaptive data requirement concept refers to the data requirement concept features that adapt to each classification of network security data. According to the adaptive data requirement concept features, the data collection index and feature usage conditions are judged. When facing network security data of different types and characteristics, the data collection and processing strategies can be flexibly adjusted to adapt to different data requirement concepts. Concept features; through clear data collection indexes and feature usage conditions, the data processing process can be optimized, unnecessary calculations and resource consumption can be reduced, and the efficiency of data processing can be improved. At the same time, clear data collection indexes and feature usage conditions provide a basis for the standardized processing of data, which helps to reduce data ambiguity and errors and improve the reliability and consistency of data; based on this, by introducing data requirement conceptual features and conducting in-depth analysis of classified network security data based on these features, it not only enhances the depth and breadth of data value mining, but also improves the flexibility and efficiency of data processing. At the same time, it promotes the standardization and normalization of data processing, and supports dynamic adjustment and optimization to adapt to the ever-changing network security environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a flowchart of an implementation of the ETL-based intelligent measurement network data fusion and modeling method in an embodiment of the present application; Figure 2 It is a flowchart of the method for obtaining modular data features in the embodiment of the present application; Figure 3 It is a flowchart for implementing S03 of the ETL-based intelligent measurement network data fusion and modeling method in the embodiment of the present application; Figure 4 It is a flowchart for implementing S30 of the ETL-based intelligent measurement network data fusion and modeling method in the embodiment of the present application; Figure 5 It is a flowchart for implementing S40 of the ETL-based intelligent measurement network data fusion and modeling method in the embodiment of the present application; Figure 6 It is a principle block diagram of an ETL-based intelligent measurement network data fusion and modeling system in an embodiment of the present application; Figure 7 It is a diagram of the internal structure of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following is combined with Figure 1-7 This application is described in further detail.

[0023] In one embodiment, if Figure 1 As shown, the present application discloses an ETL-based intelligent measurement network data fusion and modeling method, which specifically includes the following steps: S10: Based on the preset modular data features, a number of data collection modules are correspondingly established in the preset data warehouse.

[0024] In this embodiment, the intelligent measurement network may be a measurement network of an intelligent industrial automation control system, for example, a smart grid system, and the measurement network may generate raw data related to network security, including but not limited to network communication data, system logs, security alarm information, etc. Modular data features refer to features used for classification according to network security data characteristics.

[0025] Specifically, network security data in the intelligent measurement network will be collected and stored in the data warehouse. The data warehouse is a digital storage system used to connect and coordinate large amounts of data from different data sources. Its purpose is to provide data support for business intelligence, reporting and analysis, and help enterprises meet regulatory requirements. It supports enterprises to transform data into insights and make smart decisions driven by data. The data warehouse will centrally store historical data and the latest data to provide enterprises with a unified real data source. Before collecting network security data in the intelligent measurement network, the network security data characteristics in the intelligent measurement network will be analyzed first, and the network security data type in the intelligent measurement network will be determined, so as to obtain modular data characteristics, that is, characteristics used for classification according to network security data characteristics, wherein the modular data characteristics can be obtained according to the network security data source of the intelligent measurement network, for example, network security data obtained from different devices have different modular data characteristics, or the network security data of the intelligent measurement network can be obtained by analyzing the network security data of the intelligent measurement network in a targeted manner according to the general possible model building needs, for example, when the current demand is security warning and risk monitoring, the corresponding modular data characteristics include power grid operation detection, surge safety warning and opening and closing risk monitoring, etc. Each modular data characteristic corresponds to a data extraction tool for extracting different data, which is used to take data out of various storage methods, perform necessary conversion and sorting, and then store it in the data warehouse; Therefore, based on the preset modular data features, several data collection modules are established in the preset data warehouse corresponding to each modular data feature, and each data collection module has a corresponding data extraction tool.

[0026] S20: Acquire network security data of the intelligent measurement network, pre-process the network security data, and store the data in a data acquisition module according to the type of network security data and modular data characteristics.

[0027] Specifically, after a data warehouse is established and several data acquisition modules are correspondingly established in the data warehouse, network security data of the intelligent measurement network, that is, original data related to network security, is obtained based on the data extraction tools in each data acquisition module and predetermined extraction conditions, such as data extraction frequency. After pre-processing such as conversion and sorting of the network security data, the network security data is correspondingly stored in the data acquisition module of the data warehouse according to the type of network security data and modular data characteristics.

[0028] S30: Analyze the network security data in each data acquisition module to obtain the current model application characteristics.

[0029] Specifically, according to the current modeling data usage requirements, the network security data in each data acquisition module is analyzed, that is, the network security data in each data acquisition module is analyzed separately, rather than analyzing the network security data in each data acquisition module centrally, because each data acquisition module corresponds to different data usage requirements. Therefore, by analyzing the network security data in each data acquisition module, it is first verified whether the network security data in each data acquisition module is extracted correctly, and then it is determined whether the network security data in each data acquisition module can meet the corresponding data usage requirements, so as to obtain the current model application characteristics corresponding to each data acquisition module, that is, the practicality of the network security data in the data acquisition module when applied to modeling.

[0030] S40: According to the current model application characteristics and modular data characteristics, a data acquisition module adapted to the current intelligent measurement network is selected, network security data in the data acquisition module adapted to the current intelligent measurement network is extracted, and current model application characteristic data is obtained.

[0031] Specifically, the current model application characteristics represent the practicality of the network security data in the data acquisition module when applied to the current modeling, while the modular data characteristics represent the practicality of the network security data when applied to modeling, which is not targeted at the current intelligent measurement network. Therefore, it is necessary to comprehensively consider the current model application characteristics and the modular data characteristics to determine the data acquisition module suitable for the current modeling, thereby screening the data acquisition module suitable for the current intelligent measurement network, and extracting the network security data in the data acquisition module suitable for the current intelligent measurement network, thereby obtaining the current model application characteristic data, that is, practical network security data suitable for current modeling needs.

[0032] S50: Use the current model to apply characteristic data to perform modeling and obtain a network security data model.

[0033] Specifically, the current model is used to apply characteristic data for modeling, thereby obtaining a network security data model that can meet current modeling requirements.

[0034] In one embodiment, if Figure 2 As shown, modular data features are obtained in the following way: S01: Acquire network security data of different types of measured networks, wherein the network security data of different types of measured networks include historical network security data of different types of measured networks and data referenced by network security data models corresponding to different types of measured networks.

[0035] Specifically, network security data of different types of measurement networks are obtained, wherein the network security data of different types of measurement networks include historical network security data of different types of measurement networks, that is, historical network security data of different data types of different types of intelligent measurement networks, and data referenced when different types of intelligent measurement networks finally establish network security data models.

[0036] S02: Classify the network security data of different types of measurement networks according to data sources to obtain classified network security data.

[0037] Specifically, historical network security data of different data types of different types of intelligent measurement networks are analyzed, and historical network security data of different data types of different types of intelligent measurement networks are classified according to data sources. Similarly, for data referenced by network security data models corresponding to different types of measurement networks, network security data of different data types referenced by network security data models corresponding to different types of measurement networks are classified according to data sources, because network security data of the same data type of each intelligent measurement network of the same type may have multiple different data sources. For example, the types of data sources can be classified according to the equipment or system that generates the data, classified according to the means of data acquisition, classified according to structured data and unstructured data, etc. Compared with network security data of different data types, the differentiation between network security data from different data sources is more obvious, the data classification in the data warehouse is more distinguished, and more different training samples can be generated for model construction. Therefore, classified network security data is obtained by classification according to data source.

[0038] S03: Based on the preset data requirement concept features, each type of classified network security data is analyzed respectively to obtain modular data features corresponding to each type of classified network security data.

[0039] Specifically, after completing the data classification according to the data source, it is also necessary to determine what kind of model training can be used for each classified network security data based on some general data usage requirements to meet what kind of data usage requirements. For example, the preset data requirement concept features include data timeliness requirements and novelty requirements. Then, the historical network security data and model reference data in each classified network security data are analyzed separately. First, based on the historical network security data in each classified network security data, several levels of corresponding timeliness requirements and novelty requirements are determined respectively. For example, according to the acquisition frequency of the classified network security data, several levels are determined in order of frequency. According to the data content of the classified network security data, the proportion of new data obtained each time is determined. According to the order of the proportion of new data, several levels are determined. Then, according to the proportion of each level, the basic requirements are divided. For example, in order of frequency, the classified network security data with the top 80% acquisition frequency are taken as the basic requirements for corresponding timeliness; in order of new data proportion, the classified network security data with the top 80% new data proportion are taken as the basic requirements for corresponding timeliness; then, based on the model reference data in each classified network security data that meets the basic requirements, the corresponding timeliness requirements and novelty requirements are judged to meet the final requirements for meeting the timeliness requirements and novelty requirements; after obtaining the final requirements, the classified network security data are returned to judge the data requirement concept features met by each classified network security data; for example, a certain type of classified network security data meets the timeliness requirements, and another type of classified network security data meets the novelty requirements; each type of classified network security data, the data requirement concept features met by each classified network security data, and the requirements of the corresponding data requirement concept features are associated to obtain modular data features.

[0040] In one embodiment, if Figure 3 As shown, the data requirement concept features include data novelty concept features, data focus concept features and data stability concept features.

[0041] In step S03, based on the preset data requirement concept features, each type of classified network security data is analyzed respectively to obtain modular data features corresponding to each type of classified network security data, including: S031: Analyze each category of network security data based on the data novelty concept characteristics, data key concept characteristics and data stability concept characteristics, determine the data requirement concept characteristics that are suitable for each category of network security data, and obtain the adaptive data requirement concept characteristics that are suitable for each category of network security data.

[0042] Specifically, the data requirement concept features include data novelty concept features, data emphasis concept features and data stability concept features, among which the novelty concept features include new data proportion requirement concept features, new source requirement concept features (i.e. the data comes from the same model of equipment but has been replaced), and new positioning requirement concept features (i.e. the object of the data source remains unchanged, but the object of the data source has changed), the data emphasis concept features include monitoring emphasis concept features and early warning emphasis concept features, and the data stability concept features include data source stability concept features, data structure stability concept features and data capability stability concept features (i.e. the data can stably represent the meaning it represents, for example, stably represent data trends); Based on each of the concept features of data novelty concept features, data focus concept features and data stability concept features, each classified network security data is analyzed to determine the data requirement concept features that each classified network security data is adapted to in the data novelty concept features, data focus concept features and data stability concept features. For example, for various data from monitoring equipment (the classified network security data at this time is classified according to the data source), in the data novelty concept features, the adapted data novelty concept features are determined to be the new source requirement concept features, because the monitoring equipment in various industrial systems basically needs to be replaced after a period of continuous use. For example, the power monitoring equipment of the smart power network has a greater possibility of replacement. Therefore, by establishing a data acquisition module corresponding to the new source requirement concept features, when there is a need to determine the risk situation after the equipment is replaced during modeling, the risk situation after the equipment is replaced can be determined by analyzing the data in the data acquisition module corresponding to the new source requirement concept features. In addition, relatively speaking, the new data ratio requirement concept features and the new positioning requirement concept features are not adapted to the data from the monitoring equipment, because the monitoring equipment is usually installed in the original position after replacement, and the position change is less likely to occur, and the monitoring equipment is usually replaced with the same model when it is replaced; Therefore, the data requirement concept characteristics that each classified network security data is adapted to in several data novel concept characteristics, several data key concept characteristics and several data stable concept characteristics are taken as the adaptive data requirement concept characteristics of each classified network security data. The adaptive data requirement concept characteristics indicate that the classified network security data has generally met data requirement concept characteristics.

[0043] S032: According to the adaptability data requirement concept feature of each classified network security data, determine the data collection index corresponding to each adaptability data requirement concept feature of each classified network security data, and the feature usage condition.

[0044] Specifically, according to the adaptability data requirement concept feature of each classified network security data, the data type corresponding to the adaptability data requirement concept feature in each classified network security data is determined. For example, when the adaptability data novelty concept feature is determined to be the new source requirement concept feature, the corresponding data type is the data from the same model but replaced equipment before and after replacement. In this way, according to the data type corresponding to the adaptability data requirement concept feature in each classified network security data, a data collection index is generated, that is, an index used in the data collection module to accurately extract the corresponding data; In addition, according to the adaptive data requirement concept characteristics of each classified network security data, the standard threshold of the data corresponding to the adaptive data requirement concept characteristics in each classified network security data is determined when the corresponding data requirements are met. For example, when the adaptive data novelty concept characteristics are determined to be the new source requirement concept characteristics, the corresponding data is the data from the same model but after the replacement of the replaced equipment. In order to avoid the extraction of non-corresponding data due to occasional data errors, the proportion of the data before and after the replacement in the extracted data can be determined based on all historical data from the same model but after the replacement of the replaced equipment, and the lowest proportion among them is used as the standard threshold. That is, when extracting data, only when the proportion of the data before and after the replacement in the extracted data is greater than the standard threshold, the extracted data will be stored in the corresponding data acquisition module. Therefore, the standard threshold of the data corresponding to the adaptive data requirement concept characteristics in each classified network security data when the corresponding data requirements are met is used as the feature usage condition.

[0045] S033: Associate data collection indexes and feature usage conditions to obtain modular data features corresponding to each category of network security data.

[0046] Specifically, the data collection index and feature usage conditions corresponding to each type of classified network security data are associated to obtain modular data features corresponding to each type of classified network security data.

[0047] In one embodiment, if Figure 4 As shown, in step S30, the network security data in each data acquisition module is analyzed to obtain the current model application characteristics, including: S31: Acquire current application domain characteristics, where the current application domain characteristics refer to standard characteristics of the application domain of the current intelligent measurement network for modeling data.

[0048] Specifically, the current application field characteristics are obtained, and the current application field characteristics refer to the standard characteristics of the application field of the current intelligent measurement network for modeling data. For example, if the application field of the current intelligent measurement network is the intelligent power field, the corresponding current application field characteristics may include requirements for a specific data format, a certain amount of data, timeliness and novelty, etc.

[0049] S32: Based on the current application field characteristics, the network security data in each data acquisition module is analyzed to obtain the current model application characteristics corresponding to each data acquisition module.

[0050] Specifically, based on the current application field characteristics, the network security data collected in each data acquisition module is analyzed to determine the performance characteristics of the network security data in each data acquisition module in terms of the requirements represented by the current application field characteristics. For example, if the current application field characteristics include novelty requirements, then the network security data in each data acquisition module is determined in terms of novelty, that is, whether the network security data in each data acquisition module contains new data, thereby obtaining the current model application characteristics corresponding to each data acquisition module.

[0051] In one embodiment, if Figure 5 As shown, in step S40, according to the current model application characteristics and modular data characteristics, the data acquisition module adapted to the current intelligent measurement network is screened, the network security data in the data acquisition module adapted to the current intelligent measurement network is extracted, and the current model application characteristic data is obtained, specifically including: S41: Extract feature usage conditions based on modular data features.

[0052] Specifically, based on the modular data features, feature usage conditions corresponding to each modular data feature are extracted.

[0053] S42: Analyze the network security data in each data acquisition module according to the characteristic usage conditions, select the data acquisition modules that are suitable for the current intelligent measurement network according to the current model application characteristics, extract the network security data in the data acquisition modules that meet the characteristic usage conditions and the network security data in the data acquisition modules that are suitable for the current intelligent measurement network, and obtain the current model application characteristic data.

[0054] Specifically, first, according to the feature usage conditions, the network security data in each data acquisition module is analyzed to determine whether the network security data in each data acquisition module meets the corresponding feature usage conditions, and then according to the current model application characteristics, the data acquisition modules that perform well in terms of the requirements of the feature representation of the current application field are screened out, thereby extracting the network security data in the data acquisition modules that meet the feature usage conditions and the network security data in the data acquisition modules that are suitable for the current intelligent measurement network, and obtaining the current model application feature data.

[0055] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0056] In one embodiment, a smart measurement network data fusion and modeling system based on ETL is provided, and the smart measurement network data fusion and modeling system based on ETL corresponds to the smart measurement network data fusion and modeling method based on ETL in the above embodiment. Figure 6 As shown, the ETL-based intelligent measurement network data fusion and modeling system includes an acquisition module establishment module, a data storage module, a current feature analysis module, a feature data extraction module and a model building module. The detailed description of each functional module is as follows: A collection module establishment module is used to establish a number of data collection modules in a preset data warehouse based on preset modular data features; A data storage module is used to obtain network security data of the intelligent measurement network, and after pre-processing the network security data, store it in the data acquisition module according to the type of network security data and modular data characteristics; The current feature analysis module is used to analyze the network security data in each data acquisition module to obtain the current model application features; A feature data extraction module is used to screen the data acquisition module adapted to the current intelligent measurement network according to the current model application features and modular data features, extract the network security data in the data acquisition module adapted to the current intelligent measurement network, and obtain the current model application feature data; The model building module is used to use the current model application feature data to build a model and obtain a network security data model.

[0057] Optionally, the ETL-based intelligent measurement network data fusion and modeling system also includes: An analysis data acquisition module is used to acquire network security data of different types of measurement networks, wherein the network security data of different types of measurement networks include historical network security data of different types of measurement networks and data referenced by network security data models corresponding to different types of measurement networks; A data classification module is used to classify network security data of different types of measurement networks according to data sources to obtain classified network security data; The feature analysis module is used to analyze each type of classified network security data based on preset data requirement concept features, and obtain modular data features corresponding to each type of classified network security data.

[0058] Optionally, the data requirement concept features include data novelty concept features, data focus concept features, and data stability concept features; The feature analysis module includes: A multi-feature analysis submodule is used to analyze each type of classified network security data based on data novelty concept features, data key concept features and data stability concept features, determine the data requirement concept features that are suitable for each type of classified network security data, and obtain the adaptive data requirement concept features that are suitable for each type of classified network security data; An index and condition determination submodule, for determining a data collection index corresponding to each adaptability data requirement concept feature of each classified network security data, and a feature use condition according to the adaptability data requirement concept feature of each classified network security data; The index and condition association submodule is used to associate data collection indexes and feature usage conditions to obtain modular data features corresponding to each type of classified network security data.

[0059] Optionally, the current feature analysis modules include: The current feature acquisition submodule is used to acquire the current application field feature. The current application field feature refers to the standard feature of the modeling data in the application field of the current intelligent measurement network. The current feature analysis submodule is used to analyze the network security data in each data acquisition module based on the current application field characteristics, and obtain the current model application characteristics corresponding to each data acquisition module.

[0060] Optionally, the feature data extraction module includes: The conditional extraction submodule is used to extract feature usage conditions based on modular data features; The screening and extraction submodule is used to analyze the network security data in each data acquisition module according to the characteristic usage conditions, screen the data acquisition modules that are suitable for the current intelligent measurement network according to the current model application characteristics, extract the network security data in the data acquisition modules that meet the characteristic usage conditions, and the network security data in the data acquisition modules that are suitable for the current intelligent measurement network, and obtain the current model application characteristic data.

[0061] For the specific definition of the ETL-based intelligent measurement network data fusion and modeling system, please refer to the definition of the ETL-based intelligent measurement network data fusion and modeling method above, which will not be repeated here. Each module in the above-mentioned ETL-based intelligent measurement network data fusion and modeling system can be implemented in whole or in part through software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0062] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data acquisition modules, current model application features, current model application feature data and network security data models, etc. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an intelligent measurement network data fusion and modeling method based on ETL is implemented.

[0063] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: Based on the preset modular data features, several data collection modules are correspondingly established in the preset data warehouse; Obtain network security data of the intelligent measurement network, pre-process the network security data, and store it in the data acquisition module according to the type and modular data characteristics of the network security data; Analyze the network security data in each data acquisition module to obtain the application characteristics of the current model; According to the current model application characteristics and modular data characteristics, the data acquisition module suitable for the current intelligent measurement network is selected, the network security data in the data acquisition module suitable for the current intelligent measurement network is extracted, and the current model application characteristic data is obtained; The current model is used to apply characteristic data for modeling to obtain a network security data model.

[0064] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: Based on the preset modular data features, several data collection modules are correspondingly established in the preset data warehouse; Obtain network security data of the intelligent measurement network, pre-process the network security data, and store it in the data acquisition module according to the type and modular data characteristics of the network security data; Analyze the network security data in each data acquisition module to obtain the application characteristics of the current model; According to the current model application characteristics and modular data characteristics, the data acquisition module suitable for the current intelligent measurement network is selected, the network security data in the data acquisition module suitable for the current intelligent measurement network is extracted, and the current model application characteristic data is obtained; The current model is used to apply characteristic data for modeling to obtain a network security data model.

[0065] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0066] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above.

[0067] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. An ETL-based intelligent measurement network data fusion and modeling method, characterized in that: The ETL-based intelligent measurement network data fusion and modeling method includes: Based on the preset modular data features, several data collection modules are correspondingly established in the preset data warehouse; Acquire network security data of the intelligent measurement network, and after preprocessing the network security data, store the data in the data acquisition module accordingly according to the type of the network security data and the modular data characteristics; Analyze the network security data in each of the data acquisition modules to obtain the current model application characteristics; According to the current model application characteristics and the modular data characteristics, a data acquisition module adapted to the current intelligent measurement network is selected, network security data in the data acquisition module adapted to the current intelligent measurement network is extracted, and current model application characteristic data is obtained; The current model is used to apply the characteristic data for modeling to obtain a network security data model.

2. The ETL-based intelligent measurement network data fusion and modeling method according to claim 1 is characterized in that: The modular data feature is obtained in the following way: Acquire network security data of different types of measured networks, wherein the network security data of different types of measured networks include historical network security data of different types of measured networks and data referenced by network security data models corresponding to different types of measured networks; Classifying the network security data of the different types of measured networks according to data sources to obtain classified network security data; Based on the preset data requirement concept characteristics, each type of classified network security data is analyzed respectively to obtain modular data characteristics corresponding to each type of classified network security data.

3. The ETL-based intelligent measurement network data fusion and modeling method according to claim 2 is characterized in that: The data requirement concept features include data novelty concept features, data focus concept features and data stability concept features; The method of analyzing each of the classified network security data based on the preset data requirement concept features to obtain the modular data features corresponding to each of the classified network security data includes: Based on the novel concept features of the data, the key concept features of the data, and the stable concept features of the data, each of the classified network security data is analyzed to determine the data requirement concept features that are suitable for each of the classified network security data, and obtain the adaptive data requirement concept features that are suitable for each of the classified network security data; According to the adaptability data requirement concept feature of each of the classified network security data, determine the data collection index corresponding to each adaptability data requirement concept feature of each of the classified network security data, and the feature usage condition; The data collection index and the feature usage condition are associated to obtain modular data features corresponding to each type of classified network security data.

4. The ETL-based intelligent measurement network data fusion and modeling method according to claim 1 is characterized in that: Analyzing the network security data in each of the data acquisition modules to obtain the current model application characteristics includes: Acquire current application domain characteristics, where the current application domain characteristics refer to standard characteristics of the application domain of the current intelligent measurement network for modeling data; Based on the current application field characteristics, the network security data in each of the data acquisition modules is analyzed to obtain the current model application characteristics corresponding to each of the data acquisition modules.

5. The ETL-based intelligent measurement network data fusion and modeling method according to claim 3 is characterized in that: The method further comprises: selecting a data acquisition module adapted to the current intelligent measurement network according to the current model application characteristics and the modular data characteristics, extracting network security data from the data acquisition module adapted to the current intelligent measurement network, and obtaining the current model application characteristic data. Based on the modular data features, extracting the feature usage conditions; According to the characteristic usage conditions, the network security data in each of the data acquisition modules is analyzed, and according to the current model application characteristics, the data acquisition modules that are suitable for the current intelligent measurement network are screened, and the network security data in the data acquisition modules that meet the characteristic usage conditions and the network security data in the data acquisition modules that are suitable for the current intelligent measurement network are extracted to obtain the current model application characteristic data.

6. An ETL-based intelligent measurement network data fusion and modeling system, characterized in that: The ETL-based intelligent measurement network data fusion and modeling system includes: A collection module establishment module is used to establish a number of data collection modules in a preset data warehouse based on preset modular data features; A data storage module, used for obtaining network security data of the intelligent measurement network, and storing the network security data in the data acquisition module according to the type of the network security data and the modular data characteristics after preprocessing the network security data; A current feature analysis module, used to analyze the network security data in each of the data acquisition modules to obtain the current model application features; A feature data extraction module, used to screen a data acquisition module adapted to the current intelligent measurement network according to the current model application feature and the modular data feature, extract network security data from the data acquisition module adapted to the current intelligent measurement network, and obtain current model application feature data; The model building module is used to use the current model application feature data to perform modeling and obtain a network security data model.

7. The ETL-based intelligent measurement network data fusion and modeling system according to claim 6 is characterized in that: The ETL-based intelligent measurement network data fusion and modeling system also includes: An analysis data acquisition module is used to acquire network security data of different types of measured networks, wherein the network security data of different types of measured networks include historical network security data of different types of measured networks and data referenced by network security data models corresponding to different types of measured networks; A data classification module, used to classify the network security data of the different types of measurement networks according to data sources to obtain classified network security data; The feature analysis module is used to analyze each of the classified network security data based on preset data requirement concept features, and obtain modular data features corresponding to each of the classified network security data.

8. The ETL-based intelligent measurement network data fusion and modeling system according to claim 7 is characterized in that: The data requirement concept features include data novelty concept features, data focus concept features and data stability concept features; The feature analysis module comprises: A multi-feature analysis submodule, for analyzing each of the classified network security data based on the novel concept feature of the data, the key concept feature of the data, and the stable concept feature of the data, to determine the data requirement concept feature that is suitable for each of the classified network security data, and obtain the adaptability data requirement concept feature that is suitable for each of the classified network security data; An index and condition determination submodule, for determining a data collection index corresponding to each adaptability data requirement concept feature of each of the classified network security data, and a feature usage condition according to the adaptability data requirement concept feature of each of the classified network security data; The index and condition association submodule is used to associate the data collection index with the feature usage condition to obtain modular data features corresponding to each type of classified network security data.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the ETL-based intelligent measurement network data fusion and modeling method as claimed in any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the ETL-based intelligent measurement network data fusion and modeling method according to any one of claims 1 to 5 are implemented.

Images