Network security data normalization processing method based on multi-element network security device

Abstract

The application relates to a network security data normalization processing method based on a multi-element network security device and belongs to the technical field of network security. The method comprises the following steps: acquiring network security data of each network security device; wherein the network security data comprises a data name, a device type, a data format, a coding format and data content; each kind of network security device uniquely corresponds to a device type; determining a data analysis method corresponding to the network security data of each kind of network security device; for the network security data of each kind of network security device, analyzing the data content based on the data analysis method and the data field format to generate a data content analysis result; establishing a mapping relationship between the data format of the network security data of each kind of network security device and a normalized data format; and based on the mapping relationship, mapping the generated data content analysis result into the normalized data format to obtain a normalized data set.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network security, and in particular to a network security data normalization processing method based on multiple network security devices. BACKGROUND

[0002] Network security data processing can be generally divided into three parts. The first part is data access, which is to access data of various software and hardware parts for unified processing. The second part is data fusion, which is to combine the processed data to form data with the same characteristics or attributes. The third part is data association, which is to associate data sources, time sequence relationships, etc. to form a data graph.

[0003] In the prior art, the data quality is often measured by three factors of integrity, consistency and accuracy. If the data meets the application requirements in these three aspects, it is of high quality. However, due to various machine or human reasons, the data in the real world may be missing, inconsistent and incorrect, and in addition, the timeliness and credibility of the data will also affect the understanding and processing of the data. Therefore, how to normalize the network security data, overcome the problems of low data quality, data processing delay and the like, is an urgent problem to be solved. SUMMARY

[0004] In view of the above analysis, the embodiments of the present application aim to provide a network security data normalization processing method based on multiple network security devices, to solve the problems of low data quality, data processing delay and the like caused by inconsistent network security data, data missing and the like of multiple network security devices in the prior art.

[0005] The present application discloses a network security data normalization processing method based on multiple network security devices, comprising the following steps:

[0006] Obtaining network security data of each network security device; wherein the network security data comprises data name, device type, data format, encoding format and data content; each kind of network security device corresponds to one device type uniquely;

[0007] Determining a data parsing method corresponding to the network security data of each kind of network security device;

[0008] For the network security data of each kind of network security device, the data content is parsed based on the data parsing method and the data field format, to generate a data content parsing result;

[0009] Establishing a mapping relationship between the data format of the network security data of each kind of network security device and the normalized data format; based on the mapping relationship, the generated data content parsing result is mapped into the normalized data format to obtain a normalized data set.

[0010] Based on the above method, the application further makes the following improvements:

[0011] Further, the data name is used to describe the data source information of the network security data;

[0012] The device type is used to describe the device type of the network security device;

[0013] The data format is used to describe the format of a plurality of data fields in the log file matched with the device type;

[0014] The data content is used to describe a piece of log file information of the network security data;

[0015] The encoding format is used to describe the encoding format of the data content.

[0016] Further, the data parsing method is divided into:

[0017] The encoding parsing method, including: CEF parsing method, XML parsing method and JSON parsing method;

[0018] The regular parsing method, including: regular expression parsing method and Grok regular parsing method;

[0019] The symbol parsing method, including: delimiter parsing method and key-value pair parsing method;

[0020] And,

[0021] The custom parsing method.

[0022] Further, the data parsing method corresponding to the network security data of each network security device is determined, including:

[0023] For each network security device, the historical network security data of the network security device is obtained;

[0024] The encoding format of the historical network security data is read,

[0025] If there is an encoding parsing method corresponding to the encoding format, the data content in the historical network security data is parsed by using the encoding parsing method to obtain the historical data content parsing result corresponding to the encoding parsing method;

[0026] The data field format of the historical network security data is read,

[0027] If the data field format contains an expression, the data content in the historical network security data is also parsed by using the regular parsing method to obtain the historical data content parsing result corresponding to the regular parsing method;

[0028] If only symbols are included in the data field format, the data content in the historical network security data is also parsed by using the symbol parsing method to obtain a historical data content parsing result corresponding to the symbol parsing method;

[0029] The parsing accuracy of the historical data content parsing results obtained by the above parsing methods is compared, and the parsing method with the highest parsing accuracy is selected as the data parsing method corresponding to the network security data of the network security device of this type.

[0030] Further, if the highest parsing accuracy is lower than the parsing accuracy threshold, a custom parsing method is generated according to the encoding mode and data format of the network security data of the network security device of this type; and the generated custom parsing method is determined as the data parsing method corresponding to the network security data of the network security device of this type.

[0031] Further, the mapping relationship is established by performing the following:

[0032] If the data format of the network security data is inconsistent with the data field format in the normalized data format, the inconsistent data field format in the network security data is processed by adding fields, deleting fields, cropping fields, merging fields, or renaming fields, thereby establishing a one-to-one mapping relationship between the data format of the network security data and the normalized data format.

[0033] Further, the generated data content parsing result is mapped to the normalized data format, including:

[0034] According to the processing mode of the data field format in the established mapping relationship, the data content parsing result is processed, and the processed data content parsing result is mapped to the normalized data format to obtain a normalized data set.

[0035] Further, the generated data content parsing result is mapped to the normalized data format, including:

[0036] If the inconsistent data field format in the network security data is processed by adding the data field format, the corresponding data of the added data field format also needs to be completed and mapped to the normalized data format to obtain a normalized data set.

[0037] Further, after generating the data content parsing result, the step of verifying the data content parsing result is also included, and the data content parsing result that passes the verification is mapped to the normalized data format.

[0038] Further, the network security device includes:

[0039] Application systems, hosts, storage devices, network devices, security devices, virtualization devices, middleware, and databases.

[0040] Compared with the prior art, the application can achieve at least one of the following beneficial effects:

[0041] The application provides a multi-element data access and preprocessing method based on a network security protection system, breaks through key technologies such as multi-source data collection technology and data preprocessing technology, and realizes a multi-element data access method for the network security protection system.

[0042] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art upon examination of the following or can be learned by practice of the present application. The objects and other advantages of the application will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings. BRIEF DESCRIPTION OF DRAWINGS

[0043] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the principles of the application.

[0044] Figure 1 A network security data normalization processing method based on multi-element network security equipment is provided. DETAILED DESCRIPTION

[0045] The preferred embodiments of the present application will be described in detail below with reference to the drawings, wherein the drawings constitute a part of this application and serve to explain the principles of the embodiments of the present application, but are not intended to limit the scope of the present application.

[0046] In a specific embodiment 1 of the present application, a network security data normalization processing method based on multi-element network security equipment is disclosed, and a flow chart is shown in Figure 1 The method comprises the following steps:

[0047] Step S1: Obtain network security data of each network security equipment; wherein the network security data comprises data name, equipment type, data format, encoding format and data content; each kind of network security equipment corresponds to one kind of equipment type uniquely;

[0048] Exemplarily, the network security equipment in the embodiment comprises an application system, a host, a storage equipment, a network equipment, a security equipment, a virtualization equipment, middleware and a database; wherein,

[0049] The application system can be a mail, an application delivery system, etc.

[0050] The host can be Solaris, AIX (Advanced Interactive eXecutive, a UNIX-like operating system), Linux, Windows, etc.

[0051] The storage device can be EMC (developed by EMC Corporation of the United States), Netapp (developed by Netapp of the United States), etc.

[0052] The network device can be load balancing, switch, etc.

[0053] The security device can be security audit, firewall, bastion host, etc.

[0054] The virtualization device can be Xenserver, KVM (Keyboard Video Mouse), etc.

[0055] The middleware can be Apache, websphere, etc.

[0056] The database can be oracle, HBase, etc.

[0057] Other devices such as network printers, etc. can also be included.

[0058] Preferably, each item of the network security data is described as follows:

[0059] Data name, used to describe data source information of the network security data; the data source information indicates the source of the data, and exemplary examples are port ID, IP address, etc. of the network security device receiving the network security data.

[0060] Device type, used to describe the device type of the network security device; the device type includes application system, host, storage device, network device, security device, virtualization device, middleware and database; for example, the device type corresponding to the switch is the network device, the device type corresponding to the bastion host is the security device, and the data type corresponding to the Xenserver is the virtualization device.

[0061] Data format, used to describe a plurality of data field formats in the log file matched with the device type.

[0062] Data content, used to describe log file information of the network security data; it should be noted that the data format and the data content in the network security data of each network security device are different due to different network security devices, and the data format and the data content in the network security data of each network security device are obtained by using an existing collection method, and the data content herein is not limited in the embodiment.

[0063] Encoding format, used to describe the encoding format of the data content, such as UTF8, CEF, XML, JSOM, etc.

[0064] Step S2: determining a data parsing method corresponding to the network security data of each network security device;

[0065] Preferably, the data parsing method involved in the embodiment is divided into:

[0066] Encoding parsing method, including:

[0067] CEF parsing method: according to the CEF format data, the CEF data content is parsed out;

[0068] XML parsing method: the data content with XML format is parsed out;

[0069] JSON parsing method: the data with JSON format is parsed out;

[0070] Regular parsing method, including:

[0071] Regular expression parsing method: suitable for complex other forms of log that cannot be parsed, and the complex log is parsed by using regular expression;

[0072] Grok regular parsing method: suitable for complex other forms of log that cannot be parsed, and the complex log is parsed by using GROK expression;

[0073] Symbol parsing method, including:

[0074] Delimiter parsing method: when the data format is to separate the content in each piece of data by delimiter, the delimiter style is filled in, the program identifies each delimiter, and the content of each piece of data is extracted, for example, separated by “,”, the program identifies “,” first, and then extracts the data content between them;

[0075] Key-value pair parsing method: each piece of data is separated by field delimiter and key-value delimiter, then fill in the field delimiter and key-value delimiter style, the program identifies the key-value pair data according to the two delimiters;

[0076] And,

[0077] Custom parsing method, according to the encoding mode and data format of the network security data of the network security device, a custom parsing method is generated.

[0078] In this step, the data parsing method corresponding to the network security data of each kind of network security device is determined by performing the following operations, including:

[0079] For each kind of network security device, the historical network security data of the network security device is obtained;

[0080] read the encoding format of the historical network security data,

[0081] If there is a coding analysis method corresponding to the coding format, the data content in the historical network security data is parsed by using the coding analysis method to obtain a historical data content parsing result corresponding to the coding analysis method;

[0082] reading the data field format of the historical network security data,

[0083] If the data field format contains an expression, the data content in the historical network security data is also parsed by using a regular analysis method to obtain a historical data content parsing result corresponding to the regular analysis method;

[0084] If the data field format only contains symbols, the data content in the historical network security data is also parsed by using a symbol analysis method to obtain a historical data content parsing result corresponding to the symbol analysis method;

[0085] The parsing accuracy of the historical data content parsing results obtained by the above analysis methods is compared, and the analysis method with the highest parsing accuracy is selected as the data analysis method corresponding to the network security data of the network security device of this type. The parsing accuracy threshold is adaptively set according to the performance requirements of subsequent network security analysis. Generally, the parsing accuracy threshold is more than 70%.

[0086] If the highest parsing accuracy value is lower than the parsing accuracy threshold, a custom analysis method is generated according to the coding mode and data format of the network security data of the network security device of this type; and the generated custom analysis method is determined as the data analysis method corresponding to the network security data of the network security device of this type.

[0087] The custom analysis method can be generated based on the following process:

[0088] According to the coding mode and data format of the network security data of the network security device of this type, the data content of the historical network security data is divided to obtain the data field content corresponding to each data field format of the coding mode;

[0089] Selecting a corresponding parsing sub-rule adapted to the data field content of each data field format; this process is similar to the data analysis method corresponding to each device type; specifically,

[0090] reading the data field format of the data field content,

[0091] If the data field format contains an expression, the data content in the historical network security data is parsed by using the coding analysis method to obtain a historical data content parsing result corresponding to the coding analysis method;

[0092] If the data field format only contains symbols, the data content in the historical network security data is also parsed by using a symbol analysis method to obtain a historical data content parsing result corresponding to the symbol analysis method;

[0093] The data field content analysis data obtained by the above analysis method is compared with the analysis accuracy, and the data analysis method with the highest analysis accuracy is selected as the data analysis method corresponding to the data field content analysis data.

[0094] The association relationship between all data field formats in the data format is mapped to the association relationship between the corresponding analysis sub-rules.

[0095] All analysis sub-rules and their association relationships are integrated to generate a custom analysis method.

[0096] Step S3: For the network security data of each network security device, the data content is analyzed based on the data analysis method and the data field format, and a data content analysis result is generated.

[0097] To improve the accuracy of the data content analysis result, after generating the data content analysis result, the step of verifying the data content analysis result is also included, and the data content analysis result that passes the verification is mapped to the normalized data format; if the verification fails, the data content is re-analyzed, and the re-analysis can be based on the original data analysis method or replaced by other data analysis method.

[0098] For example, the data content analysis result is verified by performing the following operations:

[0099] The data content analysis result is matched and verified with the corresponding data content before analysis, if the matching is successful, the verification is passed; otherwise, the verification fails.

[0100] For example, the matching verification here means that the data content before analysis and the data content analysis result after analysis corresponding to each data field format are both consistent, if consistent, the matching is successful, and the verification is passed.

[0101] Step S4: Establish the mapping relationship between the data format of each network security device and the normalized data format; based on the mapping relationship, the generated data content analysis result is mapped to the normalized data format to obtain a normalized data set.

[0102] Preferably, the mapping relationship is established by performing the following:

[0103] If the data format of the network security data is inconsistent with the data field format in the normalized data format, the inconsistent data field format in the network security data is processed by adding fields, deleting fields, cropping fields, merging fields, or renaming fields, thereby establishing a one-to-one mapping relationship between the data format of the network security data and the normalized data format; otherwise, a one-to-one mapping relationship between the data format of the network security data and the normalized data format is directly established.

[0104] wherein,

[0105] Adding a field means adding a data field format that does not exist in the data format of the network security data but appears in the data field format of the normalized data format, so that the corresponding field can be automatically added in the parsed data each time.

[0106] Deleting a field means deleting a data field format that does not exist in the data field format of the normalized data format from the data format of the network security data.

[0107] Cutting a field means that if one data field format in the data format of the network security data corresponds to multiple data field formats in the normalized data format, the corresponding data field format in the data format of the network security data is cut according to the multiple data field formats in the normalized data format.

[0108] Merging a field means that if multiple data field formats in the data format of the network security data correspond to one data field format in the normalized data format, the corresponding data field format in the data format of the network security data is merged according to the one data field format in the normalized data format.

[0109] Renaming a field means that if the data field format in the data format of the network security data is consistent with the data field format in the normalized data format in terms of meaning but inconsistent in terms of name, the data field format in the data format of the network security data is adjusted to the corresponding data field format in the normalized data format.

[0110] Preferably, the generated data content parsing result is mapped into the normalized data format, including:

[0111] According to the processing mode of the data field format in the established mapping relationship, the data content parsing result is processed, and the processed data content parsing result is mapped into the normalized data format to obtain a normalized data set.

[0112] Meanwhile, if the inconsistent data field format in the network security data is subjected to the adding data field format processing, the data corresponding to the added data field format also needs to be supplemented and mapped into the normalized data format to obtain a normalized data set.

[0113] It should be noted that the specification of the normalized data format is directly related to the purpose of the normalized data set, which is not limited in the embodiment.

[0114] Exemplarily, the normalized data format can be: parsing rule ID, event summary, event level, application protocol, transmission protocol, initial time, source address, source port, destination address, destination port, device address, occurrence time, receiving time, sending traffic, receiving traffic, duration, original log, vendor, sending packet number, receiving packet number, parsing rule name, five-tuple, flow ID, end time, data source, two-tuple, unit, event content, protocol, total traffic, total packet number, alarm, event classification, event name. Correspondingly, the specific form of the normalized data set is exemplified as follows:

[0115] {"Resolve Rule ID":72190fa7-2a8d-457d-8d83-4985ac8c9b48,"Event Summary":nta_flow,"Event Level":Info,"Application Protocol":rx,"Transport Protocol":UDP,"Initial Time":2020-12-29 23:33:05.384,"Source Address":10.18.68.102,"Source Port":2222,"Destination Address":10.17.4.23,"Destination Port":800,"Device Address"} :10.21.172.175,"Time of Occurrence":2020-12-29 23:33:05.384,"Time of Reception":2020-12-29 17:22:29.069,"Sent Traffic":122496,"Received Traffic":131648,"Duration":21090132,"Raw Log":{"@timestamp":1609255985384,"@end_timestamp":1609277075516,"sens or_id":"7d43b783-eee2-4f49-b3d1-911c4921bd74","flow_id":455332236222203,"protocol":"flow"," src_ip":"10.18.68.102","src_port":2222,"dst_ip":"10.17.4.23","dst_port":800,"proto":"UDP","a djust":0,"ndpi_app_proto":"rx","app_proto":"rx","app_proto_ts":"unknown","app_proto_tc":"un known","flow":{"pkts_toserver":704,"pkts_toclient":704,"bytes_toserver":122496,"bytes_toclie nt":131648,"start":"2020-12-29T23:33:05.384763+0800","end":"2020-12-30T05:24:35.516385+0800","age":21090,"state":"established","reason":"timeout","alerted":false},"nta_dispatcher":"flow"},"vendor":NTA(HanSight),"packets_sent":704,"packets_received":704,"rule_name":nta_dispatcher,"5tuple":10.18.68.102_2222_flow_10.17.4.23_800,"flow_id":455332236222203,"end_time":2020-12-3005:24:35.516,"data_source":NTA(HanSight),"2tuple":10.18.68.102_10.17.4.23,"units":bytes,"event_content":no_data,"protocol":flow,"total_bytes":254144,"total_packets":1408,"alert":false,"event_category":network_access / session_connection,"event_name":network_connection}.

[0116] In summary, the embodiment forms a network security data normalization processing method based on multi-element network security devices for missing and inconsistent errors that are prone to occur in the mass data normalization processing process of multi-element network security devices. The method breaks through key technologies such as multi-source data access and data preprocessing technology, forms a complete and effective multi-data access and preprocessing method, realizes effective processing of network security data of multi-element network security devices in the network security protection process, and effectively improves the accuracy and reliability of multi-data access. Compared with the way of separately processing the network security data of each network security device in the traditional mode, the data normalization processing into a unified format has obvious advantages such as fast subsequent analysis speed, fast information retrieval, fast correlation processing of data, and unified processing of all data. It provides a perfect data processing method, which can improve the multi-data access and preprocessing of the network security protection system in various application scenarios, and realize fast and accurate data analysis and processing of mass log data.

[0117] Those skilled in the art can understand that all or part of the processes of the above-mentioned embodiments can be completed by a computer program instructing related hardware, and the program can be stored in a computer readable storage medium. The computer readable storage medium includes a magnetic disk, an optical disk, a read-only memory, a random access memory, etc.

[0118] The above merely describes preferred specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A network security data normalization processing method based on a multi-element network security device, characterized in that, The method comprises the following steps: obtaining network security data of each network security device; wherein the network security data comprises data name, device type, data format, encoding format and data content; each type of network security device corresponds to a unique device type; determining the data parsing method corresponding to the network security data of each type of network security device; for the network security data of each type of network security device, parsing the data content based on the data parsing method and the data field format to generate a data content parsing result; establishing a mapping relationship between the data format of the network security data of each type of network security device and the normalized data format; based on the mapping relationship, the generated data content parsing result is mapped into the normalized data format to obtain a normalized data set; the data parsing method comprises a custom parsing method, which is generated based on the following process: according to the encoding method and data format of the network security data of this type of network security device, the data content of the historical network security data is divided to obtain the data field content corresponding to each data field format of the encoding method; selecting the corresponding parsing sub-rules suitable for the data field content of each data field format; mapping the association relationship between all data field formats in the data format into the association relationship between the corresponding parsing sub-rules; integrating all parsing sub-rules and their association relationships to generate a custom parsing method; the data parsing method further comprises: encoding parsing method, including: CEF parsing method, XML parsing method and JSON parsing method; regular parsing method, including: regular expression parsing method, Grok regular parsing method; symbol parsing method, including: delimiter parsing method, key-value pair parsing method; determining the data parsing method corresponding to the network security data of each type of network security device comprises: for each type of network security device, obtaining the historical network security data of the network security device; reading the encoding format of the historical network security data, if there is an encoding parsing method corresponding to the encoding format, then the data content in the historical network security data is parsed using the encoding parsing method to obtain the historical data content parsing result corresponding to the encoding parsing method; reading the data field format of the historical network security data, if the data field format contains an expression, then the data content in the historical network security data is also parsed using the regular parsing method to obtain the historical data content parsing result corresponding to the regular parsing method; if the data field format only contains symbols, then the data content in the historical network security data is also parsed using the symbol parsing method to obtain the historical data content parsing result corresponding to the symbol parsing method; comparing the parsing accuracy of the historical data content parsing results obtained by the above parsing methods, and selecting the parsing method with the highest parsing accuracy as the data parsing method corresponding to the network security data of this type of network security device.

2. The network security data normalization processing method based on multiple network security devices according to claim 1, wherein the data name is used to describe the data source information of the network security data; the device type is used to describe the device type of the network security device; The data format is used for describing a plurality of data field formats in a log file matched with the device type. The data content is used for describing a piece of log file information of network security data. The encoding format is used for describing an encoding format of the data content. 3.The network security data normalization processing method based on multiple network security devices according to claim 2, wherein if the highest value of the analysis accuracy is lower than the analysis accuracy threshold, a self-defined analysis method is generated according to the encoding mode and the data format of the network security data of the network security device of the category, and the generated self-defined analysis method is determined as the data analysis method corresponding to the network security data of the network security device of the category. The mapping relationship is established by performing the following:

4. The network security data normalization processing method based on a multi-element network security device according to any one of claims 1-3, characterized in that, If the data format of the network security data is inconsistent with the data field format in the normalized data format, the inconsistent data field format in the network security data is processed by adding a field, deleting a field, cropping a field, merging a field, or renaming a field, thereby establishing a one-to-one mapping relationship between the data format of the network security data and the normalized data format. The generated data content analysis result is mapped into the normalized data format, including: 5.The method for network security data normalization processing based on a multi-element network security device according to claim 4, characterized in that, The data content analysis result is processed according to the processing mode of the data field format in the established mapping relationship, and the processed data content analysis result is mapped into the normalized data format to obtain a normalized data set. The generated data content analysis result is mapped into the normalized data format, including: 6.The method for network security data normalization processing based on a multi-element network security device according to claim 5, characterized in that, If the inconsistent data field format in the network security data is processed by adding a data field format, the corresponding data of the added data field format also needs to be supplemented and mapped into the normalized data format to obtain a normalized data set. 7.The network security data normalization processing method based on multiple network security devices according to claim 6, wherein after the data content analysis result is generated, a step of verifying the data content analysis result is further included, and the data content analysis result that passes the verification is mapped into the normalized data format. The network security devices include: application systems, host computers, storage devices, network devices, security devices, virtualization devices, middleware, and databases. 8.The method for network security data normalization processing based on a multi-element network security device according to claim 7, characterized in that, ​ ​

Images