A data segmentation management method, device and data management platform

By classifying and subdividing secure data and establishing a data hierarchical architecture and field model, the problem that existing technology is difficult to meet the needs of multi-source and complex risk analysis is solved, and the efficiency and flexibility of data management and storage are improved.

CN113971500BActive Publication Date: 2024-12-03CHINA MOBILE GRP GUANGDONG CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010717176.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-07-23
Publication Date
2024-12-03
Estimated Expiration
2040-07-23

AI Technical Summary

Technical Problem

Existing data management methods are difficult to meet the multi-source and complex upper-level risk analysis needs, especially in enterprise-level data management and storage, and it is difficult to deal with the challenges of diversified and complex security data analysis.

Method used

A data segmentation management method is proposed. By classifying security data, setting up a data hierarchy architecture, and establishing a field model for each hierarchy node to form a data segmentation model. This model is used to subdivide the collected security data Processing and storage, supporting distributed retrieval systems, relational databases and distributed file systems and other storage modes.

Benefits of technology

It improves the efficiency of upper-level security risk analysis and work capabilities at the data management level, can better respond to the diverse and complex upper-level risk analysis needs, realizes unified and complete data storage, and supports flexible storage strategy configuration.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113971500B_ABST
    Figure CN113971500B_ABST
Patent Text Reader

Abstract

The present application discloses a data segmentation management method, device and data management platform, the method comprising: dividing the data categories of security data and setting the data hierarchical structure under each data category; the data categories include parsed data, basic data, data dictionary and analytical data; establishing a field model for each parent node in each data hierarchical structure to form a data segmentation model; the field model includes a core field for constraining the core structure of the same or similar data under the parent node; using the data segmentation model to segment the collected security data; storing the segmented security data. The present application provides overall functional data segmentation management constraints for data analysis by establishing a field model at the parent node, thereby improving the data segmentation management capabilities at the data management level, and thereby improving the work efficiency at the upper data analysis level, so as to better cope with the diverse and complex upper risk analysis needs.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of information security technology, and in particular to a data segmentation management method, device and data management platform. Background Art

[0002] With the rapid development of the Internet, enterprises will face constantly changing security threats and growing security risks. Traditional security protection systems and single security data analysis platforms face more severe challenges. Faced with the increasingly complex enterprise risk security management situation, multi-source and complex security data are the basis of risk analysis. The first thing to do is to collect and summarize relevant data, process on demand, and manage them in segments to support the upper layer's ability to perform risk analysis and risk situation display based on data.

[0003] At present, enterprise-level data management and storage mainly rely on big data management platforms. However, big data management platforms have numerous data sources, complex data types, large data volumes, and are limited by the work capabilities at the data management level, making it difficult to cope with diverse and complex upper-level risk analysis needs. Summary of the invention

[0004] The present application provides a data segmentation management method, device and data management platform to solve the problem that existing data management methods are difficult to meet the needs of multi-source and complex upper-level risk analysis.

[0005] In a first aspect, the present application provides a segmentation management method, comprising:

[0006] Divide the data categories of security data and set the data hierarchy architecture corresponding to each data category; wherein the data categories include parsed data, basic data, data dictionary and analytical data; the parsed data is the parsed security log data, including the flow data and file data of network equipment, security equipment, middleware, server, application system and business system, which is the master data used for security risk analysis; the basic data is the basic information data of various businesses, including asset data, personnel data, configuration data and work order data, which is used to provide basic data basis, completion and label data basis for security risk analysis; the data dictionary is various built-in analysis support data, including label data, classification data, built-in dictionary and system dictionary, which is the built-in database of the data management platform; the analytical data is the intermediate process data and analysis result data generated when performing security risk analysis, which is used to provide data support for subsequent security risk analysis;

[0007] A field model is established for each parent node in each data hierarchy to form a data segmentation model; wherein the field model includes a core field, which is set according to the necessary data attributes shared by the same or similar data under the parent node and is used to constrain the core structure of the same or similar data under the parent node;

[0008] Using the data segmentation model to perform segmentation processing on the collected security data;

[0009] The segmented security data is stored.

[0010] In one embodiment, the method further comprises:

[0011] Determine whether to generate a new data category based on the data collected from different data sources; when a new data category is generated, add a data hierarchy structure corresponding to the new data type in the data segmentation model, and establish a field model for each parent node in the new data hierarchy structure;

[0012] and / or,

[0013] Based on the data collected from different data sources, determine whether there are changes in the subcategories of each level under the data category; when there are changes in the subcategories of each level under the data category, update the data hierarchy structure corresponding to the data category, and adjust the field model of each parent node accordingly according to the updated data hierarchy structure.

[0014] In one embodiment, the storing of the segmented security data includes: storing the segmented parsed data in a distributed retrieval system, so as to query the parsed data matching the retrieval instruction from the distributed retrieval system when a retrieval instruction is received.

[0015] In one embodiment, the storing of the segmented security data includes: storing the segmented basic data and data dictionary in a relational database to support data completion and data association analysis.

[0016] In one embodiment, storing the subdivided security data includes:

[0017] The other types of data except the real-time alarm data in the segmented analysis data are stored in the distributed file system, and the real-time alarm data in the segmented analysis data are stored in the distributed message bus, so that the distributed message bus analyzes the real-time alarm data;

[0018] After the distributed message bus completes the analysis of the real-time alarm data, the real-time alarm data is stored in a relational database to support data display.

[0019] In one embodiment, storing the subdivided security data includes: storing all the subdivided security data of all data categories in a distributed file system to store the security data uniformly and completely.

[0020] In one embodiment, the method further comprises:

[0021] Setting a first preset time and a first threshold for the distributed retrieval system;

[0022] Setting a second preset time and a second threshold for the distributed message bus;

[0023] clearing the expired data in the distributed retrieval system at intervals of a first preset time; or clearing the expired data in the distributed retrieval system when the currently used storage capacity of the distributed retrieval system exceeds a first threshold;

[0024] The expired data in the distributed message bus is cleared at intervals of a second preset time; or, when the currently used storage capacity of the distributed message bus exceeds a second threshold, the expired data in the distributed message bus is cleared.

[0025] In one embodiment, the field model further includes optional fields and extended fields, wherein the optional fields are fields set to define non-essential data attributes included in the data structure; and the extended fields are reserved fields used to extend the fields of the data structure.

[0026] In one embodiment, the method further comprises:

[0027] When storing the subdivided data of each category, the basic information corresponding to the parent node under each data category is recorded. The basic information includes the storage device name, device description, department, operation and maintenance person in charge, contact information of the person in charge, device IP, user name and password.

[0028] In a second aspect, the present application provides a data storage device, comprising:

[0029] The data segmentation model construction module is used to divide the data categories of security data and set the data hierarchy architecture corresponding to each data category; wherein the data categories include parsed data, basic data, data dictionary and analytical data; the parsed data is the parsed security log data, including the flow data and file data of network equipment, security equipment, middleware, server, application system and business system, which is the master data used for security risk analysis; the basic data is the basic information data of various businesses, including asset data, personnel data, configuration data and work order data, which is used to provide basic data basis, completion and label data basis for security risk analysis; the data dictionary is various built-in analysis support data, including label data, classification data, built-in dictionary and system dictionary, which is the built-in database of the data management platform; the analytical data is the intermediate process data and analysis result data generated when performing security risk analysis, which is used to provide data support for subsequent security risk analysis;

[0030] The data segmentation model building module is further used to establish a field model for each parent node in each data hierarchy structure to form a data segmentation model; the field model includes a core field, which is set according to the necessary data attributes shared by the same or similar data under the parent node and is used to constrain the core structure of the same or similar data under the parent node;

[0031] A segmentation processing module, used to perform segmentation processing on the collected security data using the data segmentation model;

[0032] The storage module is used to store the segmented security data.

[0033] In a third aspect, the present application provides a data management platform, including:

[0034] Data collection module, used to collect data from different data sources;

[0035] A data analysis module, used for analyzing the data stored in the data management module;

[0036] The data management module is used to execute:

[0037] Divide the data categories of security data and set the data hierarchy architecture corresponding to each data category; wherein the data categories include parsed data, basic data, data dictionary and analytical data; the parsed data is the parsed security log data, including the flow data and file data of network equipment, security equipment, middleware, server, application system and business system, which is the master data used for security risk analysis; the basic data is the basic information data of various businesses, including asset data, personnel data, configuration data and work order data, which is used to provide basic data basis, completion and label data basis for security risk analysis; the data dictionary is various built-in analysis support data, including label data, classification data, built-in dictionary and system dictionary, which is the built-in database of the data management platform; the analytical data is the intermediate process data and analysis result data generated when performing security risk analysis, which is used to provide data support for subsequent security risk analysis;

[0038] A field model is established for each parent node in each data hierarchy to form a data segmentation model; wherein the field model includes a core field, which is set according to the necessary data attributes shared by the same or similar data under the parent node and is used to constrain the core structure of the same or similar data under the parent node;

[0039] Using the data segmentation model to perform segmentation processing on the collected security data;

[0040] The segmented security data is stored.

[0041] In this application, a data segmentation model is established. According to the characteristics of the enterprise itself, the data usage, analysis requirements and other factors, the data categories of security data can be divided, and the data categories can be divided into parsed data, basic data, data dictionary and analysis data. Then, for each data category, a corresponding data hierarchical architecture is set. The data hierarchical architecture can divide each large category of data into data of different levels, thereby forming a parent-child node tree structure. Each child node is the data included under its parent node. A field model is established for each parent node. The field model is used to constrain the core structure of the same or similar data under the parent node. For example, for security event data under a parent node, the field model can specify that its data structure includes core fields such as attack time, attack source IP, event type and event level. In this way, when performing data analysis, data processing and analysis can be performed through the same analysis method, which improves the efficiency of upper-level security risk analysis. In this application, the security data collected from each data source is used as input and input into the data segmentation model, so that the security data is segmented and processed, and then the segmented security data is stored, which is convenient for the upper layer to retrieve data and process and analyze data. This application establishes a field model at the parent node to provide overall functional data segmentation management constraints for data analysis, thereby improving the data segmentation management capabilities at the data management level, thereby improving the work efficiency at the upper-level data analysis level, thereby better responding to diverse and complex upper-level risk analysis needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 An exemplary data management platform architecture is shown;

[0044] Figure 2 A flowchart of a data segmentation management method is exemplarily shown;

[0045] Figure 3 The following is an exemplary structural diagram of a data segmentation management device;

[0046] Figure 4 A structural block diagram of a data management platform is shown by way of example. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] Figure 1 The present invention shows the architecture of a data management platform, which includes a data collection layer, a data management layer and a data analysis layer. The data collection layer is the bottom layer of the platform, which is used to collect data from different data sources of the enterprise (data source 1 to data source n). The data here can be various types of parsed security log data, business basic information data and other data required to support security risk analysis; the data management layer is located in the middle layer of the platform, which is used to perform segmented management and storage of the collected data, and provide good data support for subsequent data analysis; the data analysis layer is located in the upper layer of the data management layer, and provides functions such as security risk analysis of stored data and risk situation display. Since the capabilities of the data management layer directly affect the work efficiency and reliability of the upper data analysis layer, the technical solution provided in this application is mainly aimed at the data management layer, and how the data collection layer and the data analysis layer operate does not affect the essence of this solution.

[0049] Figure 2 A data segmentation management method is shown, which can be configured in Figure 1 At the data management level in the method, the method comprises:

[0050] Step S10, classifying the data categories of the security data.

[0051] Due to the different business situations of different enterprises, each enterprise will formulate specific data management systems, data standardization management methods, processes and responsible persons according to its own characteristics. According to the enterprise's data management system, data categories can be divided according to factors such as the purpose and analysis needs of different data. In this embodiment, the data is divided into 4 categories, namely, analytical data, basic data, data dictionary and analytical data. It should be noted that the division of data categories can be set according to the actual situation of the enterprise and is not limited to the description of this embodiment. In addition, with the continuous access to new data sources and the dynamic changes of data in the data collection layer, new data categories may be added on the basis of the initial 4 data categories.

[0052] Among them, parsed data mainly refers to the various types of security log data collected and parsed, including relevant data of network equipment, security equipment, middleware, servers, application systems and business systems, as well as traffic data, file data, etc. The collected data is processed into standardized data through data processing programs, and then data management and storage are performed. Parsed data is the main data source for subsequent security risk analysis.

[0053] Basic data mainly collects basic information data of various businesses, including assets, personnel, configuration, and work orders, to provide basic data basis, completion / labeling data basis, etc. for subsequent security risk analysis.

[0054] The data dictionary is mainly for various types of built-in analysis support data, including label data, classification data, built-in dictionary and system dictionary, etc., as the factory built-in database of the data management platform.

[0055] Analyze data. Security risk analysis has a relatively complex data analysis logic and often requires multiple, iterative data analysis processes. Therefore, corresponding data management capabilities are required for analyzing intermediate process data and analysis result data to provide data support for subsequent analysis.

[0056] Step S20, setting a data hierarchy structure corresponding to each data category.

[0057] In this embodiment, a corresponding data hierarchy architecture is set for each data category. The data hierarchy architecture can divide each major category of data into data of different levels, thereby forming a parent-child node tree structure. Each child node is the data included in its parent node. It can be divided into several levels according to the data volume of the enterprise, such as a four-level or five-level data hierarchy architecture, and supports configuration functions such as custom node structure and node management.

[0058] In the data hierarchy architecture, the first-level data is the major data corresponding to each data category, such as network equipment, security equipment, middleware, business system, assets and tags, etc. Each data category (i.e. each first-level data) includes multiple levels of subcategories. The first-level data is subordinate to the second-level data, which may include classifications such as data fields or business types under each major data category. For example, the second-level data under the first-level data security equipment includes intrusion detection, advanced threat, data security, etc.; the second-level data is subordinate to the third-level data, which may include specific equipment types and business system types of the second-level data, such as the second-level data intrusion detection can be subdivided into IDS (Intrusion Detection System), IPS (Intrusion Prevention System) and WAF (Web Application Firewall), etc.; the third-level data is subordinate to the fourth-level data, which supports manufacturer classification and version classification; if it is a five-level architecture, it supports classification of other items such as version. Through the data hierarchy architecture, each data category is classified and subdivided layer by layer, and the parent-child relationship between data at all levels under the same category is defined, so as to express the association and logic of the data more clearly and intuitively.

[0059] Step S30, establishing a field model for each parent node in each data hierarchy structure to form a data segmentation model.

[0060] Step S40: using the data segmentation model to perform segmentation processing on the collected security data.

[0061] For the above-mentioned tree-like data hierarchy architecture, a field model is established for each parent node therein, and the field model includes core fields. The core fields are set according to the necessary data attributes shared by the same or similar data under the parent node, and are used to constrain the core structure of the same or similar data under the parent node, thereby providing overall functional data segmentation management constraints for data analysis. For example, under intrusion detection data, security devices such as network intrusion detection and defense are managed. For security event data generated by such security devices, the field model established at the intrusion detection parent node stipulates that the data structure includes core fields such as attack time, attack source IP, attack destination IP, destination port, event type and event level. In this way, when performing data analysis, for the intrusion detection data, data processing, data analysis and result visualization and other processes can be performed through consistent analysis methods, thereby improving the efficiency of upper-level security risk analysis. The field model may include required fields, optional fields and extended fields. The required fields may define the core structure of the data, corresponding to some necessary core fields in the data structure. Optional fields are fields set to define data attributes that are included in the data structure but are not necessary. Extended fields may be reserved fields used to extend the fields of the data structure, and may support corresponding extended personalized fields for different data types. The field model may perform operations such as field modification, field redefinition, field addition and field deletion according to actual needs.

[0062] When the corresponding field models are established for the parent nodes in each data hierarchy, a data segmentation model is formed. The collected data is used as input into the data segmentation model to achieve unified management constraints on data segmentation planning and data structure, avoid missing core basic information of data, and improve the value of data in the analysis stage, thereby improving the efficiency of upper-level security risk analysis. When an enterprise operates for a long time, the data volume, data category, and data hierarchy architecture corresponding to each data category may change during the entire data life cycle. For example, the data source may change and increase. Therefore, in the process of continuous data collection, the initially constructed data segmentation model can be updated and adaptively adjusted according to the actual data changes of the enterprise.

[0063] In one implementation, based on data collected from different data sources, it is determined whether to generate a new data category; when a new data category is generated, a data hierarchy structure corresponding to the new data type is added to the data segmentation model, and a field model is established for each parent node in the new data hierarchy structure; and / or, based on data collected from different data sources, it is determined whether there are changes in subcategories of each level under the data category; when there are changes in subcategories of each level under the data category, the data hierarchy structure corresponding to the data category is updated, and the field model of each parent node is adjusted accordingly according to the updated data hierarchy structure.

[0064] When a new data category is generated, that is, when there is a major category change in security data, it is necessary to set the data hierarchy structure corresponding to the new data category and establish a field model at each parent node in the new data hierarchy structure to complete the update of the data segmentation model; when there are changes in the data category, that is, the subcategories of each level under the major category, such as the change / replacement of a subcategory in the secondary data under the analytical data, or the generation of a new subcategory in the secondary data, it is necessary to update the data hierarchy structure corresponding to the data category and adaptively adjust the field model of each parent node according to the updated data hierarchy structure, that is, the data hierarchy structure corresponding to each data category will change with the changes in the data in the operation of the enterprise, so the data segmentation model must also be changed and updated accordingly.

[0065] After data segmentation, the model supports the definition and management of basic data information. Since data is generally transmitted in the form of strings, fields, etc., each string / field in the data stream is defined to refer to basic information. In this way, after data segmentation, the data is parsed and the strings / fields of the data are converted into basic information. In this way, the data can be stored and managed in the form of Chinese or English. Then, the next step is to store the segmented data.

[0066] Step S50, storing the subdivided security data.

[0067] Applicants have found in practice that data storage is more about constraining the corresponding storage location after the needs are confirmed in the data planning stage, but the data access process is a long-term and continuous process, which will continuously access more data sources and will also cause changes in the planned default storage logic. In addition, as the project continues to advance, more data analysis scenarios need to be expanded, and the storage requirements for data analysis will also change accordingly, but there is currently a lack of more automated and flexible data storage logic support. Therefore, in order to better respond to the diverse and complex needs of upper-level risk analysis, it is necessary not only to support the segmented planning and management constraints of data, but also strategies such as storage methods and storage logic are also important means to support data analysis.

[0068] This embodiment mainly supports the data storage of unstructured data, structured data, and semi-structured data. To support different application requirements, data storage strategies are planned based on the enterprise data volume, data category, data requirements, etc. For example, a data storage model for the entire life cycle of data can be established and used as the initial value input of the data storage model. Later, as the data types are gradually enriched and the data volume continues to expand throughout the data management life cycle, the initial value is updated and adjusted. This embodiment supports simultaneous storage in multiple storage structures for different categories of segmented data. Overall, it can include four storage modes, namely relational data storage, distributed retrieval storage, distributed file storage, and distributed message bus storage, to meet different data analysis and retrieval requirements.

[0069] Among them, relational data storage, that is, data storage location in a relational database, is suitable for storing structured data with small data volume and small change cycle, such as basic data and data dictionary.

[0070] Distributed retrieval storage, that is, the data storage location is in a distributed retrieval system, which is suitable for storing data that needs to provide full-text retrieval to the outside world and supports data query and retrieval needs.

[0071] Distributed file storage, that is, the data storage location is in the distributed file system, which stores the collected original data and the standardized data after ETL (Extract Transform Load, which is used to describe the process of extracting, transforming, and loading data from the source to the destination), etc. As an important data storage and computing body, it stores a large amount of historical data.

[0072] Distributed message bus storage, that is, data storage location is in the distributed message bus, providing a distributed message processing mechanism, with high throughput, high concurrency message publishing and message subscription, for real-time data processing, supporting the output of real-time data alarms and other needs.

[0073] In this application, in order to ensure the successful storage ratio of data, storage data performance, etc., a built-in storage strategy for the data segmentation model can be established to support multi-dimensional storage strategy configuration. Different logics can be configured in the storage strategy, such as storage capacity logic, storage time logic, expired data removal logic, and data storage logic, etc., as a flexible strategy configuration method to enrich the storage strategy. The core elements of the storage strategy include: data category, data usage, data magnitude, data storage location, data storage time, storage capacity, expired data removal, etc. The storage logic involved in the storage strategy mainly includes data storage and expired data removal. The above four storage modes can establish their own corresponding storage strategies, and different element factors can be set for each storage method.

[0074] The data storage logic can be set according to the data category, including, for example:

[0075] (1) All the subdivided data of each category is stored in the distributed file system. That is, all the collected data needs to be stored in the distributed file system after being subdivided, so as to store the security data in a unified and complete manner to ensure the integrity of data storage and facilitate the query of all historical data;

[0076] (2) The segmented parsed data is stored in a distributed retrieval system to support visual analysis of the parsed data and query retrieval of the parsed data; upon receiving a retrieval instruction, the parsed data matching the retrieval instruction can be queried from the distributed retrieval system, and the retrieved parsed data can be displayed to the user in a visual form. The retrieval instruction can be, for example, a user inputting information such as "keywords" through the retrieval system of the data management platform and clicking on a retrieval control. The implementation form of the retrieval can be set according to the actual application;

[0077] (3) Store the subdivided basic data and data dictionary in a relational database to support data completion and data association analysis;

[0078] (4) For data whose category is analytical data, the real-time alarm data in the segmented analytical data is first stored in the distributed message bus, and other types of data except the real-time alarm data in the segmented analytical data are stored in the distributed file system; the distributed message bus will analyze the real-time alarm data, and after the analysis is completed, the real-time alarm data will be stored in the relational database to support data display and subsequent closed-loop processing processes.

[0079] It should be noted that the data storage logic can be flexibly configured according to the actual application, and can be stored based on the data category according to the corresponding storage logic mentioned above. For example, under each data category, a separate storage mode can be set for sub-category data of different levels.

[0080] When storing the subdivided data of each category, the basic information corresponding to the parent node under each data category is recorded. The basic information includes storage device name, device description, department, operation and maintenance person in charge, contact information of the person in charge, and other basic information, as well as storage connection information such as storage device name, device IP, user name and password. By setting the basic information for storage, data management and maintenance can be facilitated. For example, if there are abnormalities in the intrusion detection data under the security device in the parsed data, the operation and maintenance person in charge of the department described in the intrusion detection data can be contacted.

[0081] The expired data clearing logic is mainly applicable to distributed retrieval systems and distributed message buses; expired data in distributed file systems and relational databases does not need to be cleared and remains fully stored.

[0082] The data clearing logic of the distributed retrieval system is as follows: a first preset time and a first threshold are set for the distributed retrieval system, and expired data in the distributed retrieval system is cleared at intervals of the first preset time; or, when the currently used storage capacity of the distributed retrieval system exceeds the first threshold, expired data in the distributed retrieval system is cleared. When accumulating and storing data, data of the same type will be continuously updated, and the old data previously stored will become expired data. In this case, the expired data of the distributed retrieval system can be cleared regularly by setting the first preset time; or, as data is continuously accumulated and stored, the current storage capacity of the distributed retrieval system continues to grow. When it grows to the first threshold, expired data must be cleared to release the storage space occupied by expired data. Of course, whether expired data needs to be cleared can also be determined by the remaining capacity of the distributed retrieval system, which will not be described in detail in this embodiment.

[0083] The data clearing logic of the distributed message bus is: set a second preset time and a second threshold for the distributed message bus, and clear the expired data in the distributed message bus every second preset time; or, when the current used storage capacity of the distributed message bus exceeds the second threshold, clear the expired data in the distributed message bus. By setting the second preset time, the expired data of the distributed message bus is cleared regularly; or, as the data is continuously accumulated and stored, the current storage capacity of the distributed message bus continues to grow. When it grows to the second threshold, the expired data must be cleared to release the storage space occupied by the expired data. Of course, the remaining capacity of the distributed message bus can also be used to determine whether the expired data needs to be cleared, which will not be described in detail in this embodiment.

[0084] The above embodiments of the present application are exemplary. In actual applications, data categories and the hierarchical architecture under each category can be updated and adjusted according to the data of different enterprises. In addition, the storage mode is not limited to the four types listed in this embodiment. On the basis of the aforementioned technical solutions, technical personnel in this field can flexibly configure and update storage strategies to adapt to the ever-changing enterprise data and updated data segmentation models.

[0085] Figure 3 A data segmentation management device is shown, comprising a data segmentation model building module 110, a segmentation processing module 120 and a storage module 130;

[0086] The data segmentation model building module 110 is used to divide the data categories of security data and set the data hierarchy architecture corresponding to each data category; wherein the data categories include parsed data, basic data, data dictionary and analytical data;

[0087] The data segmentation model building module 110 is also used to establish a field model for each parent node in each data hierarchy to form a data segmentation model; the field model is used to constrain the core structure of the same or similar data under the parent node;

[0088] A segmentation processing module 120, used to perform segmentation processing on the collected security data using the data segmentation model;

[0089] The storage module 130 is used to store the subdivided security data.

[0090] Specifically, the storage module 130 is used to store the segmented parsed data in the distributed retrieval system.

[0091] The storage module 130 is further used to store the subdivided basic data and data dictionary in a relational database.

[0092] The storage module 130 is further used to store all the subdivided security data of all data categories in the distributed file system.

[0093] The storage module 130 is also used to first store the real-time alarm data in the segmented analysis data in a distributed message bus, and store other types of data except the real-time alarm data in the segmented analysis data in a distributed file system; after the distributed message bus completes the analysis of the real-time alarm data, the real-time alarm data is stored in a relational database.

[0094] The storage module 130 is further used to record the storage basic information corresponding to the parent node under each data category when storing the segmented data.

[0095] In some implementations, the apparatus further includes a storage policy setting module 140, the storage policy setting module 140 being configured to set a first preset time and a first threshold for the distributed retrieval system;

[0096] The storage module 130 is further configured to clear expired data in the distributed retrieval system at intervals of a first preset time; or, when the currently used storage capacity of the distributed retrieval system exceeds a first threshold, clear expired data in the distributed retrieval system.

[0097] The storage policy setting module 140 is further used to set a second preset time and a second threshold for the distributed message bus;

[0098] The storage module 130 is further configured to clear expired data in the distributed message bus at intervals of a second preset time; or, when the currently used storage capacity of the distributed message bus exceeds a second threshold, clear expired data in the distributed message bus.

[0099] The storage policy setting module 140 is also used to set or modify other policy logics related to data storage. The specific functional description of each module in this embodiment can refer to the previous embodiment, and will not be repeated in this embodiment.

[0100] Figure 4 A data management platform is shown, comprising:

[0101] A data collection module 210 is used to collect data from different data sources;

[0102] The data analysis module 230 is used to analyze the data stored in the data management module 220;

[0103] The data management module 220 is used to execute:

[0104] Divide the data categories of security data and set the data hierarchy architecture corresponding to each data category; wherein the data categories include parsed data, basic data, data dictionary and analytical data;

[0105] Establish a field model for each parent node in each data hierarchy to form a data segmentation model; wherein the field model is used to constrain the core structure of the same or similar data under the parent node;

[0106] Using the data segmentation model to perform segmentation processing on the collected security data;

[0107] The segmented security data is stored.

[0108] For other functional configurations and descriptions of the data management module 220, please refer to the first embodiment, which will not be described in detail in this embodiment.

[0109] In the above embodiments, the data categories of security data can be divided according to the characteristics of the enterprise itself, based on factors such as the purpose of the data and analysis requirements, and the data categories can be divided into parsed data, basic data, data dictionary and analytical data. Then, a corresponding data hierarchy structure is set for each data category. The data hierarchy structure can divide each large category of data into data of different levels, thereby forming a parent-child node tree structure. Each child node is the data included under its parent node. A field model is established for each parent node. The field model is used to constrain the core structure of the same or similar data under the parent node. For example, for security event data under a parent node, the field model can specify that its data structure includes core fields such as attack time, attack source IP, event type and event level. In this way, when performing data analysis, data processing and analysis can be performed through identical analysis methods, thereby improving the efficiency of upper-level security risk analysis. In this application, the security data collected from various data sources is used as input and input into the data segmentation model, so as to perform segmentation processing on the security data, and then store the segmented security data, formulate a storage strategy that is suitable for each data category in the data segmentation module, and implement the classification and storage of data of different categories, which is convenient for the upper layer to retrieve data and process and analyze data. This application provides overall functional data segmentation management constraints for data analysis, and provides more automatic and flexible data storage strategy logic, improves the data segmentation management and storage capabilities at the data management level, and then improves the work efficiency of the upper data analysis level, so as to better cope with the diverse and complex upper risk analysis needs.

[0110] The same or similar contents between the various embodiments in this application can be referenced and verified with each other.

[0111] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present invention are indicated by the appended claims.

[0112] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A data segmentation management method, characterized in that: include: Divide the data categories of security data and set the data hierarchical architecture corresponding to each data category; wherein the data categories include parsed data, basic data, data dictionary and analytical data; the parsed data is the parsed security log data, including the flow data and file data of network equipment, security equipment, middleware, server, application system and business system, which is the master data used for security risk analysis; the basic data is the basic information data of various businesses, including asset data, personnel data, configuration data and work order data, which is used to provide basic data basis, completion and label data basis for security risk analysis; the data dictionary is various built-in analysis support data, including label data, classification data, built-in dictionary and system dictionary, which is the built-in database of the data management platform; the analytical data is the intermediate process data and analysis result data generated when performing security risk analysis, which is used to provide data support for subsequent security risk analysis; the data hierarchical architecture is to divide each data category into several levels of small category data, forming a parent-child node tree structure, and each child node is the data included under its parent node; A field model is established for each parent node in each data hierarchy to form a data segmentation model; wherein the field model includes core fields, which are set according to necessary data attributes shared by the same or similar data under the parent node, and are used to constrain the core structure of the same or similar data under the parent node, so as to perform data processing and analysis through consistent analysis methods; Using the data segmentation model to perform segmentation processing on the collected security data; The segmented security data is stored.

2. The method according to claim 1, characterized in that The method further comprises: Determine whether to generate a new data category based on the data collected from different data sources; when a new data category is generated, add a data hierarchy structure corresponding to the new data type in the data segmentation model, and establish a field model for each parent node in the new data hierarchy structure; and / or, Based on the data collected from different data sources, determine whether there are changes in the subcategories of each level under the data category; when there are changes in the subcategories of each level under the data category, update the data hierarchy structure corresponding to the data category, and adjust the field model of each parent node accordingly according to the updated data hierarchy structure.

3. The method according to claim 1, characterized in that: The segmented security data is stored, including: The segmented parsed data is stored in a distributed retrieval system, so that when a retrieval instruction is received, the parsed data matching the retrieval instruction is searched from the distributed retrieval system.

4. The method according to claim 1, characterized in that The segmented security data is stored, including: The segmented basic data and data dictionary are stored in a relational database to support data completion and data association analysis.

5. The method according to claim 1, characterized in that The segmented security data is stored, including: The other types of data except the real-time alarm data in the segmented analysis data are stored in the distributed file system, and the real-time alarm data in the segmented analysis data are stored in the distributed message bus, so that the distributed message bus analyzes the real-time alarm data; After the distributed message bus completes the analysis of the real-time alarm data, the real-time alarm data is stored in a relational database to support data display.

6. The method according to claim 1, characterized in that The segmented security data is stored, including: All security data of all subdivided data categories are stored in a distributed file system to store the security data in a unified and complete manner.

7. The method according to claim 1, characterized in that The field model also includes optional fields and extended fields; the optional fields are fields set to define non-essential data attributes included in the data structure; the extended fields are reserved fields used to extend the fields of the data structure.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: When storing the subdivided data of each category, the basic information corresponding to the parent node under each data category is recorded. The basic information includes the storage device name, device description, department, operation and maintenance person in charge, contact information of the person in charge, device IP, user name and password.

9. A data segmentation management device, characterized in that: include: The data segmentation model construction module is used to divide the data categories of security data and set the data hierarchical architecture corresponding to each data category; wherein the data categories include parsed data, basic data, data dictionary and analytical data; the parsed data is the parsed security log data, including the flow data and file data of network equipment, security equipment, middleware, server, application system and business system, which is the master data used for security risk analysis; the basic data is the basic information data of various businesses, including asset data, personnel data, configuration data and work order data, which is used to provide basic data basis, completion and label data basis for security risk analysis; the data dictionary is various built-in analysis support data, including label data, classification data, built-in dictionary and system dictionary, which is the built-in database of the data management platform; the analytical data is the intermediate process data and analysis result data generated when performing security risk analysis, which is used to provide data support for subsequent security risk analysis; the data hierarchical architecture is to divide each data category into several levels of small category data, forming a parent-child node tree structure, and each child node is the data included under its parent node; The data segmentation model building module is further used to establish a field model for each parent node in each data hierarchy structure to form a data segmentation model; the field model includes a core field, which is set according to the necessary data attributes shared by the same or similar data under the parent node, and is used to constrain the core structure of the same or similar data under the parent node, so as to perform data processing and analysis through consistent analysis means; A segmentation processing module, used to perform segmentation processing on the collected security data using the data segmentation model; The storage module is used to store the segmented security data.

10. A data management platform, characterized in that: include: Data collection module, used to collect data from different data sources; A data analysis module, used for analyzing the data stored in the data management module; The data management module is used to execute: Divide the data categories of security data and set the data hierarchical architecture corresponding to each data category; wherein the data categories include parsed data, basic data, data dictionary and analytical data; the parsed data is the parsed security log data, including the flow data and file data of network equipment, security equipment, middleware, server, application system and business system, which is the master data used for security risk analysis; the basic data is the basic information data of various businesses, including asset data, personnel data, configuration data and work order data, which is used to provide basic data basis, completion and label data basis for security risk analysis; the data dictionary is various built-in analysis support data, including label data, classification data, built-in dictionary and system dictionary, which is the built-in database of the data management platform; the analytical data is the intermediate process data and analysis result data generated when performing security risk analysis, which is used to provide data support for subsequent security risk analysis; the data hierarchical architecture is to divide each data category into several levels of small category data, forming a parent-child node tree structure, and each child node is the data included under its parent node; A field model is established for each parent node in each data hierarchy to form a data segmentation model; wherein the field model includes core fields, which are set according to necessary data attributes shared by the same or similar data under the parent node, and are used to constrain the core structure of the same or similar data under the parent node, so as to perform data processing and analysis through consistent analysis methods; Using the data segmentation model to perform segmentation processing on the collected security data; The segmented security data is stored.