A format standardization method for network security threat intelligence sharing

Abstract

The application provides a format standardization method for network security threat intelligence sharing, and relates to the technical field of network security. The method comprises the following steps: constructing a mapping table based on field names and attribute names and outputting a mapping dictionary; reading multiple formats of intelligence data uploaded by users and converting the intelligence data into a two-dimensional table; performing data format checking and conversion based on the mapping relationship, and creating multiple SCO objects; filling the SCO objects into AttackPattern and ObservedData objects for serialization, generating threat intelligence conforming to the STIX format; and outputting a JSONL format file and naming the file according to a hash value. The application realizes the automatic and standardized conversion of multi-source heterogeneous threat intelligence data, improves the data interoperability and sharing efficiency, and enhances the network security defense capability.

Description

Technical Field

[0001] This invention relates to the field of network security technology, and in particular to a format standardization method for sharing network security threat intelligence. Background Technology

[0002] With the continuous development of information technology and the escalating nature of cybersecurity threats, cybersecurity threat intelligence (CTI) has become a crucial information resource for enterprises and organizations in defending against and responding to cyberattacks. By collecting, analyzing, and sharing data related to cyberattacks, CTI helps organizations identify potential attacks, vulnerabilities, malware, and attacker behavior, thereby enhancing their network protection capabilities. However, despite its critical role in cybersecurity, challenges remain regarding the format, quality, and sharing of CTI data in practical applications. Currently, CTI data is typically stored in various formats. Due to the lack of a unified standard, the formats of CTI data often differ, leading to poor interoperability between different platforms and systems. Particularly during the sharing and exchange of CTI data, format inconsistencies cause several major problems: lack of a unified standard format, difficulties in data conversion, and limitations on the sharing and circulation of CTI.

[0003] Therefore, there is an urgent need to provide a solution to improve the above problems. Summary of the Invention

[0004] The purpose of this invention is to provide a standardized method for sharing network security threat intelligence, in order to improve the problem of low existing network security defense capabilities.

[0005] The present invention provides a format standardization method for sharing cybersecurity threat intelligence, which adopts the following technical solution:

[0006] A mapping table is constructed based on the field names and attribute names of network security threat intelligence. Each row of data in the mapping table is traversed, and dictionary key-value pairs are constructed according to the mapping relationship. When the mapping table is traversed, the mapping dictionary is output.

[0007] Read the cybersecurity intelligence data uploaded by the user, determine the file format of the intelligence data and then read it to obtain a two-dimensional table containing the intelligence data. The field names of the two-dimensional table are composed of the field names used in the cybersecurity threat intelligence uploaded by the user.

[0008] Based on the mapping relationships in the mapping dictionary, the read network security intelligence data is checked and converted in terms of data format, and multiple SCO objects are created according to the STIX object type corresponding to the field name;

[0009] The SCO objects are filled into AttackPattern objects and ObservedData objects respectively for serialization to obtain SDO objects, and network security threat intelligence conforming to the STIX format is generated;

[0010] The network security threat intelligence conforming to the STIX format is output as a file in the JSONL format, and the file is named according to a hash value;

[0011] The SDO objects are used to describe data and information in various network security events, and focus on expressing high-level structures of entities, behaviors and processes related to network threats.

[0012] Optionally, the mapping table stores field names used in network security threat intelligence uploaded by a user, and specifies a corresponding STIX object name in a STIX standard format and an attribute name in the object for each field name.

[0013] Optionally, the file format includes a comma-separated value mode, a text file format and a JavaScript Object Notation format.

[0014] Optionally, the specific process of creating the plurality of SCO objects includes:

[0015] Row data in the two-dimensional table is extracted by line-by-line traversal, and the extracted row data is subjected to data format checking and conversion.

[0016] After the data format checking is qualified, a plurality of corresponding SCO objects are created according to the types of fields in the two-dimensional table, wherein each SCO object will be filled with corresponding attribute values according to definitions in the mapping dictionary.

[0017] Optionally, the specific process of data format checking and conversion includes:

[0018] According to the mapping relationship in the mapping dictionary, the read network security threat intelligence data content is converted into corresponding attribute values in the SCO object, and it is judged whether the attribute values have format requirements.

[0019] The method for format standardization of network security threat intelligence sharing provided by the present application has the beneficial effects that:

[0020] Firstly, the present application provides a unified format conversion method based on the STIX standard format, which can effectively convert CTI data from different sources and in different formats into data formats conforming to the STIX standard, ensuring the unity of network security threat intelligence, and by adopting a widely recognized standard format, the present application greatly improves the interoperability of data between different platforms and systems, and reduces the complexity in cross-platform sharing and analysis.

[0021] Secondly, the present application can quickly and efficiently convert threat intelligence data in non-standard formats into structured data conforming to the STIX standard through an automated conversion process, reducing the need for manual intervention during the conversion process and improving the efficiency and accuracy of data conversion. Especially when sharing across platforms, the automated processing of the present application greatly reduces the operational complexity and improves the real-time performance of intelligence sharing.

[0022] Thirdly, the present application supports the input of multiple data formats (such as CSV, JSON, Excel, etc.), and no matter what format the user uploads the network security threat intelligence data in, the system can automatically parse and convert it. Compared with the existing technology which usually only supports intelligence data in a fixed format, the present application provides more extensive format support, enabling users to more flexibly select data sources and formats, enhancing the adaptability and scalability of the system.

[0023] Fourthly, through the standardization of the STIX format, the present application promotes the cooperation and information exchange between network security threat intelligence sharing platforms. The standardized data format enables different organizations and platforms to easily share and analyze intelligence, helping network security teams to identify and respond to potential threats in real time, and improving the response capability of the entire network security defense system. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 a flowchart of the method for format standardization of network security threat intelligence sharing provided by the present application;

[0025] Figure 2 a specific flowchart of the mapping dictionary construction provided by the present application;

[0026] Figure 3 a step diagram of the automated data parsing method provided by the present application;

[0027] Figure 4 a flowchart of the data format checking and conversion provided by the present application;

[0028] Figure 5A flow chart showing the SCO object in the STIX standard format output by the present application. DETAILED DESCRIPTION

[0029] For the purposes of the present application, the technical solutions and advantages of the embodiments will be more clearly described below. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the present application. Unless otherwise defined, the technical terms or scientific terms used herein should be understood as the common meanings understood by those skilled in the art. The “comprise” and similar words used in the present application mean that the elements or objects before the word cover the elements or objects listed after the word and their equivalents, and do not exclude other elements or objects.

[0030] Referring to Figure 1 , a flow chart showing a format standardization method for network security threat intelligence sharing provided by the present application, comprising the following steps:

[0031] S1, constructing a mapping table based on the field name and attribute name of network security threat intelligence, traversing each row of data of the mapping table, constructing a dictionary key-value pair according to the mapping relationship, and outputting a mapping dictionary when the mapping table is traversed;

[0032] S2, reading the network security intelligence data uploaded by the user, judging the file format of the intelligence data before reading, obtaining a two-dimensional table containing the intelligence data, and the field name of the two-dimensional table being composed of the field name used by the network security threat intelligence uploaded by the user;

[0033] S3, based on the mapping relationship in the mapping dictionary, performing data format checking and conversion on the read network security intelligence data, and creating a plurality of SCO objects according to the STIX object type corresponding to the field name;

[0034] S4, filling the SCO objects into the AttackPattern object and the ObservedData object respectively for serialization processing, obtaining an SDO object, and generating network security threat intelligence conforming to the STIX format;

[0035] S5, outputting the network security threat intelligence conforming to the STIX format as a file in the JSONL format, and naming the file according to the hash value.

[0036] In some embodiments, when performing step S1, it is considered that the network security threat intelligence data uploaded by the user does not have a unified standard format, and the field names used in the data can be various, for example, for the relevant intelligence of the source IP address, the field name can be "source_ip" or "sourceIP", etc. Therefore, in order to ensure that the present application has good universality and can efficiently and automatically perform data conversion, the present application maintains a mapping table.

[0037] Specifically, the mapping table stores the field names used in the network security threat intelligence uploaded by the user, and specifies the corresponding STIX object name in the STIX standard format and the attribute name in the object for each field name, and the mapping table includes: the field name in the user uploaded data, the STIX object type, and the attribute name in the STIX object type.

[0038] Further, the present application uses a dictionary structure (Dictionary) in Python language to temporarily store the mapping relationship in the mapping table. Specifically, the dictionary structure maps the data field name uploaded by the user to the corresponding object type and attribute name in the STIX format, so as to ensure that each data field can be accurately converted into the standardized STIX format. The advantage of this structure is high efficiency and flexibility, which can quickly match and process different field names and provide reliable support for subsequent conversion work.

[0039] In fact, whenever the user needs to perform conversion operation, the present application will automatically construct a mapping dictionary according to the latest maintained mapping table, and make pre-preparation for subsequent conversion operation.

[0040] Referring to Figure 2 At the beginning, the first row of data in the mapping table is read, the dictionary key-value pair is constructed according to the mapping relationship, it is judged whether the mapping table is traversed, if yes, the mapping dictionary is output, if not, the next row of data in the mapping table is continuously read, the dictionary key-value pair is constructed according to the mapping relationship again, when the mapping table is traversed, the mapping dictionary is output, and the cycle is ended.

[0041] In some embodiments, when performing step S2, considering that users may use different file formats to store network security threat intelligence data, the present application designs a function that can adapt to multiple mainstream file formats to support network security threat intelligence data with different file suffixes. These mainstream formats include but are not limited to: CSV (Comma Separated Values format), TXT (Text file format), JSON (JavaScript Object Notation format), etc. For these different file types, the present application provides an automated data parsing method to ensure that no matter which format of file the user uploads, the system can efficiently and accurately process the data therein. Specifically, the present application uses the open source Pandas library to develop this function, and uses different reading methods for different formats of network security intelligence data uploaded by the user. Through this function, the present application can realize that no matter what format of network security intelligence data the user uploads, a two-dimensional table containing all the network security intelligence data of the original data file can be obtained, thereby realizing efficient data processing and conversion operations.

[0042] Referring to Figure 3 First, the user uploads network security intelligence data, judges the file format, and selects a corresponding reading method for each file format to obtain a two-dimensional table, wherein the field names of the two-dimensional table include source_ip and dst_ip, which are 192.168.1 and 192.168.2, respectively.

[0043] Further, after obtaining the two-dimensional table containing all the network security intelligence data of the original data file, in order to preserve the network security threat intelligence information contained in the file as much as possible in detail and completeness, the present application uses a line-by-line scanning operation to extract the content of each row in the two-dimensional table and performs data conversion operations.

[0044] In some embodiments, when performing step S3, the specific process of creating multiple SCO objects includes:

[0045] S3-1, traversing and extracting row data in the two-dimensional table line by line, and performing data format checking and conversion on the extracted row data;

[0046] S3-2, after the data format checking is qualified, creating multiple corresponding SCO objects according to the types of the fields in the two-dimensional table, wherein each SCO object will fill in the corresponding attribute values according to the definitions in the mapping dictionary.

[0047] Specifically, in the execution of S3-1, the application has successfully read and converted the user uploaded data file into a two-dimensional table, and according to the mapping dictionary obtained in step S1, the field name of each row of data is matched with the object name and attribute name in the STIX standard format. On this basis, this step will convert the read network security threat information data content into the corresponding attribute value in the SCO object according to the mapping relationship in the mapping dictionary.

[0048] Further, when the extracted row data is subjected to data format checking and conversion, the STIX standard has strict data format requirements for the attribute values of the SCO object. For example, the IP address needs to meet the format of the IPv4Address object, the timestamp needs to be in the ISO-8601 format, and the file hash value needs to meet the SHA246 standard. To ensure that all extracted data meets these requirements, the application will perform data format checking on each read field, and format conversion operation on data that does not meet the format requirements.

[0049] Referring to Figure 4 At the beginning, according to the mapping dictionary, the object attribute corresponding to the field is found, and it is judged whether the attribute has format requirements. If not, the cycle is ended, if yes, it is continued to judge whether the numerical value corresponding to the field meets the format requirements, if yes, the cycle is ended, if not, the data format is converted to the required format and the cycle is ended.

[0050] In fact, after steps S3-1 and S3-2, the application has completed the most critical step in the entire network security threat information format standardization process: extracting and outputting the user uploaded network security threat information data into several SCO objects in the STIX standard format, laying the foundation for subsequent construction of SDO objects in the STIX standard format and final output of network security threat information data files conforming to the STIX format.

[0051] Referring to Figure 5 At the beginning, the two-dimensional table is traversed row by row, the row data is extracted and subjected to data format checking and conversion, the corresponding SCO object is constructed according to the mapping dictionary and stored until the two-dimensional table traversal is completed, the stored SCO objects are output, the cycle is ended, and when the two-dimensional table has not been traversed, the two-dimensional table is traversed row by row.

[0052] In some embodiments, in the execution of step S4, the specific process of data format checking and conversion includes:

[0053] According to the mapping relationship in the mapping dictionary, the read network security threat intelligence data content is converted into the corresponding attribute value in the SCO object, and it is judged whether the attribute value has a format requirement. If not, the cycle is ended. If yes, it is judged whether the numerical value corresponding to the field meets the format requirement. If yes, the cycle is ended. If not, the data format is converted into the specified format and the cycle is ended. Wherein, the format needs to meet the standards: the IP address meets the format of IPv4Address object, the timestamp meets the format of ISO-8601, and the file hash value meets the SHA246 standard.

[0054] In some embodiments, when performing step S5, the SDO object is used to describe data and information in various network security events, focusing on expressing the high-level structure of entities, behaviors, and processes related to network threats. The SDO object is used to represent various types of information related to attacker behavior, attack patterns, and threat activities, helping analysts to understand and track attack activities.

[0055] Further, when performing step S5, the JSONL file is used to enhance the circulation and universality of network security threat intelligence on the platform. Using the JSONL format can make the data stored row by row, and each row is a valid STIX object, which is convenient for large-scale data processing, transmission and query, and improves the cross-platform compatibility and use efficiency of intelligence data.

[0056] The system supports saving the generated STIX format file to a specified directory and naming the file. The naming of each output file will be automatically named according to the hash value of the network security threat intelligence data obtained in step S4, to ensure the uniqueness and traceability of the file.

[0057] Although the embodiments of the present application are described in detail above, it is obvious for those skilled in the art that various modifications and changes can be made to these embodiments. However, it should be understood that such modifications and changes are within the scope and spirit of the present application described in the claims. Moreover, the present application described herein can have other embodiments and can be implemented or realized in various ways.

Claims

1. A format standardization method for sharing cybersecurity threat intelligence, characterized in that, include: A mapping table is constructed based on the field names and attribute names of network security threat intelligence. Each row of data in the mapping table is traversed, and dictionary key-value pairs are constructed according to the mapping relationship. When the mapping table is traversed, the mapping dictionary is output. Read the cybersecurity intelligence data uploaded by the user, determine the file format of the intelligence data and then read it to obtain a two-dimensional table containing the intelligence data. The field names of the two-dimensional table are composed of the field names used in the cybersecurity threat intelligence uploaded by the user. Based on the mapping relationships in the mapping dictionary, the read network security intelligence data is checked and converted in terms of data format, and multiple SCO objects are created according to the STIX object type corresponding to the field name; The SCO object is populated into the AttackPattern object and the ObservedData object respectively for serialization processing to obtain the SDO object, generating cybersecurity threat intelligence that conforms to the STIX format; The cybersecurity threat intelligence conforming to the STIX format is output as a JSONL file, and the file is named according to the hash value; The SDO object is used to describe data and information in various network security incidents, focusing on describing the high-level structure of entities, behaviors, and processes related to network threats. The SDO object is used to represent various types of information related to attacker behavior, attack patterns, and threat activities, helping analysts understand and track attack activities.

2. The format standardization method for sharing cybersecurity threat intelligence according to claim 1, characterized in that, The mapping table stores the field names used in the network security threat intelligence uploaded by the user, and specifies the corresponding STIX object name and attribute name in the STIX standard format for each field name. The mapping table includes: field names in user-uploaded data, STIX object types, and attribute names in STIX object types.

3. The format standardization method for sharing cybersecurity threat intelligence according to claim 1, characterized in that, The file formats include comma-separated value mode, text file format, and JavaScript object representation format.

4. The format standardization method for sharing cybersecurity threat intelligence according to claim 1, characterized in that, The specific process of creating multiple SCO objects includes: Iterate through the rows of the two-dimensional table and extract the row data, then check and convert the extracted row data for data format. After the data format check is passed, multiple corresponding SCO objects are created according to the types of the fields in the two-dimensional table. Each SCO object will be filled with the corresponding attribute values ​​according to the definition in the mapping dictionary.

5. A format standardization method for sharing cybersecurity threat intelligence according to claim 4, characterized in that, The specific process of data format checking and conversion includes: Based on the mapping relationships in the mapping dictionary, the read network security threat intelligence data is converted into the corresponding attribute values ​​in the SCO object. It is then determined whether the attribute values ​​have format requirements. If not, the loop ends. If yes, it is determined whether the values ​​corresponding to the fields meet the format requirements. If yes, the loop ends. If not, the data format is converted to the specified format and the loop ends. The format must meet the following standards: IP addresses conform to the format of IPv4Address objects, timestamps conform to the ISO-8601 format, and file hash values ​​conform to the SHA246 standard.

Images