Information extraction methods, devices, electronic equipment and storage media

Abstract

This application discloses an information extraction method, apparatus, electronic device, and storage medium. This application obtains a set of targets to be processed; the set of targets includes multiple asset information strings; based on a pre-constructed asset information mapping table and asset index table, multiple asset information strings in the set of targets are extracted to obtain target substrings; for each asset information string in the set of targets, based on the asset information mapping table, asset index table, and target substring, the structured triplet information of the asset information string is determined. This completes the structured transformation of asset information, clearly defining the asset description object of the asset information string in the form of structured triplet information, reducing the ambiguity of asset reporting results and the cost of subsequent asset management, improving the accuracy and processing efficiency of asset information, and providing a data foundation for further cybersecurity activities.

Description

Technical Field

[0001] This application relates to the field of network security technology, and in particular to information extraction methods, devices, electronic devices and storage media. Background Technology

[0002] Nmap (Network Mapper) is a free and open-source network port scanning tool in the network security field. Integrating Nmap's asset detection results for further asset management, vulnerability detection, and other network security activities is a common practice in the network security field. Internally, Nmap's port service detection capabilities mainly depend on its asset rule configuration file. The asset information field in the configuration file is a text string with implicit structured information, but the asset vendor and product name do not have clear boundaries. This makes it difficult to extract the asset information in a structured way according to vendor and product name, and it is difficult to clarify the asset meaning of the field. In actual business, it is usually necessary to describe this asset information in a structured form for further network security activities. The problem of the asset information in the asset field not being able to be described in a structured way creates a significant obstacle to these subsequent network security management activities.

[0003] In response to the above situation, the relevant technologies involve manually querying the corresponding manufacturer and product name information one by one based on the asset field information to extract the asset information in a structured manner. However, this manual query method is extremely inefficient, and the accuracy of the structured extraction is also low. At the same time, it is difficult to ensure the consistency of the manual output results in complex data situations. Summary of the Invention

[0004] This application provides an information extraction method, apparatus, electronic device, and storage medium to solve the problems of low efficiency and accuracy of manual structured extraction of asset information from asset fields, and difficulty in ensuring consistency of manual output results in complex data situations.

[0005] In a first aspect, embodiments of this application provide an information extraction method, including:

[0006] Obtain a set of targets to be processed; the set of targets to be processed includes multiple asset information strings;

[0007] Based on a pre-built asset information mapping table and asset index table, multiple asset information strings in the target set to be processed are extracted to obtain target substrings; the target substrings are substrings that appear more frequently than a preset frequency threshold among the multiple asset information strings; the asset information mapping table is used to represent the correspondence between manufacturer strings and product name strings in the asset information strings; the asset index table is used to represent the structured triple information of the asset information strings and the index relationship between the asset information strings;

[0008] For each asset information string in the target set to be processed: based on the asset information mapping table, the asset index table, and the target substring, determine the structured triplet information of the asset information string.

[0009] In one possible implementation, the asset information mapping table and the asset index table are constructed according to the following method:

[0010] Acquire business data; and extract information from the business data to obtain the port service string, asset information string, manufacturer string from the general platform enumeration CPE data, and product name string from the CPE data corresponding to the asset information string in the business data;

[0011] Based on the manufacturer string and the product name string, construct the asset information mapping table; and based on the port service string, the asset information string, the manufacturer string, and the product name string, construct the asset index table.

[0012] In one possible implementation, constructing the asset information mapping table based on the manufacturer string and the product name string includes:

[0013] Based on preset mapping rules, the natural strings corresponding to the manufacturer string and the product name string are determined; the preset mapping rules represent the correspondence between formatted characters and natural characters.

[0014] The asset information mapping table is constructed based on the natural string corresponding to the manufacturer string and the natural string corresponding to the product name string.

[0015] In one possible implementation, constructing the asset index table based on the port service string, the asset information string, the manufacturer string, and the product name string includes:

[0016] If the manufacturer string and the product name string are the same, the manufacturer string is included in the asset information string, and the manufacturer string is different from the asset information string, then the characters in the asset information string other than the manufacturer string are set as preset reference characters to obtain the first asset information string; based on the port service string, the manufacturer string, the product name string and the first asset information string, the asset index table is constructed;

[0017] If the manufacturer string and the product name string are different, the manufacturer string and the asset information string have the same part, and the product name string is completely contained in the asset information string, then the characters in the asset information string other than the part that is the same as the manufacturer string and the product name string are set as preset reference characters to obtain the second asset information string; based on the port service string, the manufacturer string, the product name string and the second asset information string, the asset index table is constructed.

[0018] In one possible implementation, the step of extracting multiple asset information strings from the target set to be processed based on a pre-built asset information mapping table and asset index table to obtain target substrings includes:

[0019] For each asset information string in the target set to be processed: based on the asset information string, the asset information mapping table, and the asset index table, the asset information string is divided into multiple substrings in a right-to-left order; the number of substrings is the same as the length of the asset information string.

[0020] For each substring: the frequency of occurrence of the substring is determined based on the number of asset information strings in the target set to be processed and the number of times the substring appears in the corresponding substring in the target set to be processed;

[0021] The substrings that appear more frequently than a preset frequency threshold among the multiple asset information strings are selected as the target substrings.

[0022] In one possible implementation, determining the structured triplet information of the asset information string based on the asset information mapping table, the asset index table, and the target substring includes:

[0023] If the asset information string is not greater than a preset length threshold, then the structured triplet information of the asset information string is determined based on the asset information mapping table and the asset index table;

[0024] If the asset information string is greater than the preset length threshold, then the structured triplet information of the asset information string is determined based on the asset information mapping table, the asset index table, and the target substring.

[0025] In one possible implementation, determining the structured triplet information of the asset information string based on the asset information mapping table and the asset index table includes:

[0026] If the asset index table includes an asset information string that is the same as the asset information string, then the structured triplet information corresponding to the asset information string in the asset index table shall be used as the structured triplet information of the asset information string.

[0027] If the asset index table does not include an asset information string that is the same as the asset information string, and the asset information mapping table includes a product name string that is the same as the asset information string, then the product name string, the manufacturer string corresponding to the product name string, and the port service string in the asset information mapping table are used as the structured triplet information of the asset information string.

[0028] If the asset index table does not include an asset information string that is the same as the asset information string, and the asset information mapping table includes a manufacturer string that is the same as the asset information string, then the manufacturer string, the asset information string, and the port service string in the asset information mapping table are used as the structured triplet information of the asset information string.

[0029] If the asset index table does not contain an asset information string that is the same as the asset information string, and the asset information mapping table does not contain a product name string that is the same as the asset information string, and the asset information mapping table does not contain a manufacturer string that is the same as the asset information string, then the asset information string will be used as the product name string and the manufacturer string; and the product name string, the manufacturer string, and the port service string will be used as the structured triplet information of the asset information string.

[0030] In one possible implementation, determining the structured triplet information of the asset information string based on the asset information mapping table, the asset index table, and the target substring includes:

[0031] If the target substring exists in the asset information string, then delete the target substring from the asset information string to obtain the third asset information string; based on the asset information mapping table and the asset index table, determine the structured triplet information of the third asset information string, and use the structured triplet information of the third asset information string as the structured triplet information of the asset information string;

[0032] If the target substring does not exist in the asset information string, the asset information string is divided into multiple substrings in order from left to right; and based on the asset information mapping table and the asset index table, the structured triple information of the multiple substrings is determined, and the structured triple information of the multiple substrings is used as the structured triple information of the asset information string.

[0033] In one possible implementation, the method further includes:

[0034] If at least one new asset information string is obtained, the at least one new asset information string is added to the target set to be processed, and then the step of extracting multiple asset information strings in the target set to be processed based on the pre-built asset information mapping table and asset index table to obtain target substrings is executed.

[0035] Secondly, embodiments of this application provide an information extraction device, the device comprising:

[0036] An acquisition unit is used to acquire a set of targets to be processed; the set of targets to be processed includes multiple asset information strings;

[0037] The target substring determination unit is used to extract multiple asset information strings from the target set to be processed based on a pre-built asset information mapping table and an asset index table to obtain target substrings; the target substring is a substring among the multiple asset information strings whose frequency of occurrence is greater than a preset frequency threshold; the asset information mapping table is used to represent the correspondence between manufacturer strings and product name strings in the asset information strings; the asset index table is used to represent the structured triple information of the asset information strings and the index relationship between the asset information strings;

[0038] The structured triplet information determination unit is used to determine the structured triplet information of each asset information string in the target set to be processed, based on the asset information mapping table, the asset index table, and the target substring.

[0039] In one possible implementation, the device further includes:

[0040] The construction unit is used to construct the asset information mapping table and the asset index table according to the following methods:

[0041] Acquire business data; and extract information from the business data to obtain the port service string, asset information string, manufacturer string in CPE data, and product name string in CPE data corresponding to the asset information string in the business data;

[0042] Based on the manufacturer string and the product name string, construct the asset information mapping table; and based on the port service string, the asset information string, the manufacturer string, and the product name string, construct the asset index table.

[0043] In one possible implementation, the building unit is specifically used for:

[0044] Based on preset mapping rules, the natural strings corresponding to the manufacturer string and the product name string are determined; the preset mapping rules represent the correspondence between formatted characters and natural characters.

[0045] The asset information mapping table is constructed based on the natural string corresponding to the manufacturer string and the natural string corresponding to the product name string.

[0046] In one possible implementation, the building unit is specifically used for:

[0047] If the manufacturer string and the product name string are the same, the manufacturer string is included in the asset information string, and the manufacturer string is different from the asset information string, then the characters in the asset information string other than the manufacturer string are set as preset reference characters to obtain the first asset information string; based on the port service string, the manufacturer string, the product name string and the first asset information string, the asset index table is constructed;

[0048] If the manufacturer string and the product name string are different, the manufacturer string and the asset information string have the same part, and the product name string is completely contained in the asset information string, then the characters in the asset information string other than the part that is the same as the manufacturer string and the product name string are set as preset reference characters to obtain the second asset information string; based on the port service string, the manufacturer string, the product name string and the second asset information string, the asset index table is constructed.

[0049] In one possible implementation, the target substring determination unit is specifically used for:

[0050] For each asset information string in the target set to be processed: based on the asset information string, the asset information mapping table, and the asset index table, the asset information string is divided into multiple substrings in a right-to-left order; the number of substrings is the same as the length of the asset information string.

[0051] For each substring: the frequency of occurrence of the substring is determined based on the number of asset information strings in the target set to be processed and the number of times the substring appears in the corresponding substring in the target set to be processed;

[0052] The substrings that appear more frequently than a preset frequency threshold among the multiple asset information strings are selected as the target substrings.

[0053] In one possible implementation, the structured triplet information determination unit is specifically used for:

[0054] If the asset information string is not greater than a preset length threshold, then the structured triplet information of the asset information string is determined based on the asset information mapping table and the asset index table;

[0055] If the asset information string is greater than the preset length threshold, then the structured triplet information of the asset information string is determined based on the asset information mapping table, the asset index table, and the target substring.

[0056] In one possible implementation, the structured triplet information determination unit is specifically used for:

[0057] If the asset index table includes an asset information string that is the same as the asset information string, then the structured triplet information corresponding to the asset information string in the asset index table shall be used as the structured triplet information of the asset information string.

[0058] If the asset index table does not include an asset information string that is the same as the asset information string, and the asset information mapping table includes a product name string that is the same as the asset information string, then the product name string, the manufacturer string corresponding to the product name string, and the port service string in the asset information mapping table are used as the structured triplet information of the asset information string.

[0059] If the asset index table does not include an asset information string that is the same as the asset information string, and the asset information mapping table includes a manufacturer string that is the same as the asset information string, then the manufacturer string, the asset information string, and the port service string in the asset information mapping table are used as the structured triplet information of the asset information string.

[0060] If the asset index table does not contain an asset information string that is the same as the asset information string, and the asset information mapping table does not contain a product name string that is the same as the asset information string, and the asset information mapping table does not contain a manufacturer string that is the same as the asset information string, then the asset information string will be used as the product name string and the manufacturer string; and the product name string, the manufacturer string, and the port service string will be used as the structured triplet information of the asset information string.

[0061] In one possible implementation, the structured triplet information determination unit is specifically used for:

[0062] If the target substring exists in the asset information string, then delete the target substring from the asset information string to obtain the third asset information string; based on the asset information mapping table and the asset index table, determine the structured triplet information of the third asset information string, and use the structured triplet information of the third asset information string as the structured triplet information of the asset information string;

[0063] If the target substring does not exist in the asset information string, the asset information string is divided into multiple substrings in order from left to right; and based on the asset information mapping table and the asset index table, the structured triple information of the multiple substrings is determined, and the structured triple information of the multiple substrings is used as the structured triple information of the asset information string.

[0064] In one possible implementation, the device further includes:

[0065] The filling unit is used to add at least one new asset information string to the target set to be processed if at least one new asset information string is obtained, and then execute the step of extracting multiple asset information strings in the target set to be processed based on the pre-built asset information mapping table and asset index table to obtain target substrings.

[0066] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the information extraction method as described in any one of the first aspects above.

[0067] Fourthly, this application provides a computer-readable storage medium that, when the instructions in the computer-readable storage medium are executed by an electronic device, enables the electronic device to perform the steps of the information extraction method as described in any one of the first aspects above.

[0068] Fifthly, this application provides a computer program product, comprising a computer program: when the computer program is executed by a processor, it implements the steps of the information extraction method as described in any one of the first aspects above.

[0069] The technical solutions provided by the embodiments of this application bring at least the following beneficial effects:

[0070] The information extraction method, apparatus, electronic device, and storage medium provided in this application embodiment obtain a set of targets to be processed. This set includes multiple asset information strings. Based on a pre-built asset information mapping table and asset index table, multiple asset information strings in the set are extracted to obtain target substrings. For each asset information string in the set, structured triplet information of the asset information string is determined based on the asset information mapping table, asset index table, and target substring. Compared to existing technologies, the information extraction method provided in this application embodiment completes the structured transformation of asset information reported by Nmap, clearly defining the asset description object of the asset information string in the form of structured triplet information, directly reducing the ambiguity of asset reporting results. This not only reduces the cost of subsequent asset management and improves the accuracy and processing efficiency of asset information, but also has stronger fault tolerance for redundant character information unrelated to structured asset information. The obtained structured asset information can provide a data foundation for further cybersecurity activities.

[0071] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0072] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0073] Figure 1 A flowchart illustrating an information extraction method provided in an embodiment of this application;

[0074] Figure 2 A flowchart illustrating a method for constructing an asset information mapping table provided in an embodiment of this application;

[0075] Figure 3 A flowchart illustrating a method for determining a target substring provided in an embodiment of this application;

[0076] Figure 4 A flowchart illustrating a method for determining structured triplet information provided in an embodiment of this application;

[0077] Figure 5 A schematic diagram of the information extraction device provided in this application embodiment;

[0078] Figure 6 The present application provides a schematic diagram of the structure of an electronic device. Detailed Implementation

[0079] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0080] Furthermore, in the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0081] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined with "first" and "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0082] Nmap is a free and open-source network port scanning tool in the field of network security. Based on contributions and improvements from the open-source community, it has accumulated a wealth of network asset detection rules, forming powerful operating system and port service detection capabilities, enabling effective detection of many asset targets. Integrating Nmap's asset detection results for further asset management, vulnerability detection, and other network security activities is a common practice in the current network security field.

[0083] Internally, Nmap's port service detection capability primarily depends on its asset rule configuration file. This file contains the form of the packet probes, the rules for asset identification, and the corresponding asset and port service information. Nmap's port service detection capability is its core scanning capability. Its basic principle is to construct special packet probes to target the ports of the target host, and then match the reported information of the target port with pre-defined asset identification rules. If identification is successful, the service and asset information on the responding port is reported according to the pre-defined asset rules and asset-service correspondence.

[0084] Typically, the asset information reported by Nmap port probes is described by the ` / p / ` field (asset information string) in the configuration file. This field is supposed to be a concatenation of the asset's vendor string and product name string. However, due to the open-source nature of the data, this convention is not strictly enforceable. This means that while the asset information field in the configuration file is a text string with implicit structured information, the boundaries between the vendor and product name are not clear. This makes it difficult to extract the asset information in a structured manner based on vendor and product name, and to clearly define the asset meaning of the field. For example, in the field "p / ActiveMQ OpenWire transport / ", the actual vendor string is Apache or Apache ActiveMQ, the product name string is ActiveMQ, and the remaining information is just additional description of the port service protocol. However, since the vendor and product name are not clearly defined, it is difficult to extract the asset information in a structured manner based on vendor and product name.

[0085] In practical applications, this asset information typically needs to be described in a structured format to facilitate further cybersecurity activities, such as vulnerability probing based on accurate asset / product names and multi-dimensional asset management based on vendor information. The inability to structure asset information within the asset field poses a significant obstacle to these subsequent cybersecurity management activities.

[0086] To address the above situation, related technologies include: on the one hand, manually querying the corresponding manufacturer and product name information one by one based on the asset field information to extract the asset information in a structured manner. However, this manual query method is extremely inefficient, and the accuracy of structured extraction is also low. At the same time, it is difficult to ensure the consistency of the manual output results in complex data situations. On the other hand, based on pre-constructed asset data, strict matching of asset strings is performed to obtain structured asset information. However, this method is prone to false alarms for redundant character information unrelated to structured asset information, resulting in the inability to structure unknown assets.

[0087] In view of this, this application provides an information extraction method, device, apparatus and medium to solve the problems that the efficiency and accuracy of manual extraction of asset information from asset fields are low, and it is also difficult to ensure the consistency of manual output results in complex data situations.

[0088] The inventive concept of this application can be summarized as follows: This application obtains a set of targets to be processed; the set of targets to be processed includes multiple asset information strings; based on a pre-built asset information mapping table and asset index table, multiple asset information strings in the set of targets to be processed are extracted to obtain target substrings; for each asset information string in the set of targets to be processed: based on the asset information mapping table, asset index table, and target substring, the structured triplet information of the asset information string is determined. This completes the structured transformation of the asset information reported by Nmap, clarifying the asset description object of the asset information string in the form of structured triplet information, reducing the ambiguity of the asset reporting results, and improving the accuracy and processing efficiency of asset information.

[0089] After introducing the main inventive concepts of the embodiments of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of this application provide method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on conventional or non-inventive effort. In steps where there is no logically necessary causal relationship, the execution order of these steps is not limited to the execution order provided in the embodiments of this application.

[0090] See Figure 1This is a flowchart illustrating an information extraction method provided in an embodiment of this application. Figure 1 As shown, the method includes the following steps:

[0091] In step 101, a set of targets to be processed is obtained; the set of targets to be processed includes multiple asset information strings.

[0092] In one possible implementation, if at least one new asset information string is obtained, the at least one new asset information string is added to the target set to be processed, and then steps 102 and 103 are performed based on the added target set to be processed.

[0093] In step 102, based on the pre-built asset information mapping table and asset index table, multiple asset information strings in the target set to be processed are extracted to obtain target substrings.

[0094] The target substring is a substring that appears more frequently than a preset frequency threshold among multiple asset information strings.

[0095] The asset information mapping table is used to represent the correspondence between the manufacturer string and the product name string in the asset information string.

[0096] The asset index table is used to represent the index relationship between the structured triplet information of the asset information string and the asset information string.

[0097] In one possible implementation, the asset information mapping table and asset index table are the foundation for structured asset information extraction, and their richness affects the accuracy and reliability of the structured information extraction. Therefore, in this embodiment, the asset information mapping table and asset index table can be pre-constructed before information extraction. The specific construction method is as follows:

[0098] S11: Obtain business data; and extract information from the business data to obtain the port service string corresponding to the asset information string in the business data, the asset information string, the manufacturer string in the general platform enumeration CPE data, and the product name string in the CPE data.

[0099] Business data includes, but is not limited to, asset information strings related to business and cybersecurity, such as configuration file data from the network scanning tool Nmap.

[0100] S12: Construct an asset information mapping table based on the manufacturer string and product name string; and construct an asset index table based on the port service string, asset information string, manufacturer string, and product name string.

[0101] In specific implementation, the data sources for constructing the asset information mapping table and asset index table in this application embodiment include, but are not limited to, the following three types of data:

[0102] Data 1: Open-source CPE specification data. The CPE specification is a standardized naming method for software applications, operating systems, and hardware platforms. This standard also maintains an open-source, globally available asset CPE data set. Each data point describes a specific asset object in a structured form according to the CPE standard.

[0103] Data 2: Open network data. This mainly refers to network asset or software application data that is publicly available on the internet. When the availability of the data is confirmed, asset entity information is mainly obtained through web crawlers or dedicated data acquisition APIs.

[0104] Data 3: Nmap configuration file. The port service configuration file contains some asset rule information that includes CPE data for the assets. This data is relatively small; however, due to its open-source nature, the completeness of descriptions of the same product submitted by different users may vary. Therefore, when constructing the asset information mapping table and asset index table, priority is given to parsing the correspondence between asset information and CPE in the configuration file to establish the corresponding asset entities.

[0105] In one possible implementation, this application embodiment constructs an asset information mapping table based on the manufacturer string and product name string, which can be executed as follows: Figure 2 The steps shown are as follows:

[0106] In step 201, based on preset mapping rules, the natural strings corresponding to the manufacturer string and the product name string are determined;

[0107] In step 202, an asset information mapping table is constructed based on the natural string corresponding to the manufacturer string and the natural string corresponding to the product name string.

[0108] In practice, information is extracted from the Nmap configuration file data to obtain the vendor string and product name string from the CPE specification data. Then, according to the preset mapping rules, the formatting characters in the vendor string and product name string are converted into natural characters to obtain the natural string corresponding to the vendor string and the natural string corresponding to the product name string. Finally, an asset information mapping table is constructed based on the natural string corresponding to the vendor string and the natural string corresponding to the product name string.

[0109] The preset mapping rules represent the correspondence between formatted characters and natural characters. This means that the formatted characters in the manufacturer string and product name string are reverse-engineered to obtain the natural strings corresponding to the manufacturer string and product name string, respectively. Table 1 shows the correspondence between formatted characters and natural characters.

[0110] Table 1

[0111] ％21 ！ ％2b + ％5b [ ％22 " ％2c , ％5c \ ％23 # ％2f / ％5d ] ％24 $ ％3a : ％5e ^ ％25 ％ ％3b ； 60％ ` ％26 & ％3c < ％7b { ％27 ' ％3d ＝ ％7c | ％28 ( ％3e > ％7d } ％29 ) ％3f ？ ％7e ～ ％2a * 40％ @ _ Space

[0112] For example, the CPE specification data in the Nmap configuration file is as follows:

[0113] cpe: / a:%240.99_kindle_books_project:%240.99_kindle_books:6::~~~android~~"

[0114] Extracting the CPE specification data from the above, we obtain the manufacturer string and product name string from the CPE specification data:

[0115] Manufacturer string: %240.99_kindle_books_project

[0116] Product name string: %240.99_kindle_books

[0117] Then, according to Table 1, the formatting characters in the manufacturer string and product name string are converted into natural characters:

[0118] Manufacturer string: $0.99 Kindle Books Project

[0119] Product name string: $0.99 Kindle books

[0120] Finally, based on the natural strings corresponding to the manufacturer string and the natural strings corresponding to the product name string, an asset information mapping table for the above data is constructed, as shown in Table 2:

[0121] Table 2

[0122] $0.99 Kindle Books Project $0.99 Kindle books

[0123] In one possible implementation, the asset index table is constructed based on the port service string, asset information string, manufacturer string, and product name string in this embodiment of the application, and can be performed as follows:

[0124] If the manufacturer string and product name string are the same, the manufacturer string is included in the asset information string, and the manufacturer string and asset information string are different, then the characters in the asset information string other than the manufacturer string are set as preset reference characters to obtain the first asset information string; based on the port service string, manufacturer string, product name string and first asset information string, an asset index table is constructed;

[0125] If the manufacturer string and product name string are different, the manufacturer string and asset information string have the same part, and the product name string is completely contained in the asset information string, then the characters in the asset information string other than the parts that are the same as the manufacturer string and product name string are set as preset reference characters to obtain the second asset information string; based on the port service string, manufacturer string, product name string and second asset information string, an asset index table is constructed.

[0126] In practice, the preset reference characters are set to (.*?). First, information is extracted from the Nmap configuration file data to obtain the port service string, asset information string, manufacturer string, and product name string from the CPE data. Then, according to Table 1, the formatting characters in the port service string, asset information string, manufacturer string, and product name string are converted into natural characters. Finally, an asset index table can be constructed based on the following three scenarios:

[0127] Scenario 1: When the manufacturer string and product name string are exactly the same, if the manufacturer string is completely contained in the asset information string and the manufacturer string is different from the asset information string, then replace all characters in the asset information string except the manufacturer string with (.*?) to obtain the first asset information string. Then, based on the port service string, manufacturer string, product name string and the first asset information string, construct an asset index table.

[0128] Example 1: The original data is match amanda m|xxxxxxx|p / Amanda backup system indexserver / cpe: / a:amanda:amanda:$2 / ; the port service string, asset information string, manufacturer string, and product name string are extracted as: amanda, Amanda backup system index server, amanda, amanda; at this time, the condition of case one is met, so the asset information string is processed to obtain the first asset information string amanda(.*?), and (amanda, amanda, amanda) is used as the structured triple information corresponding to the first asset information string.

[0129] Scenario 2: When the manufacturer string and product name string are completely different, the manufacturer string and asset information string have the same beginning part, and the product name string is completely contained in the asset information string, then replace all characters in the asset information string except for the parts that are the same as the manufacturer string and product name string with (.*?) to obtain the second asset information string; construct an asset index table based on the port service string, manufacturer string, product name string and second asset information string.

[0130] Example 2: Original data: match afsmain m|xxxxxxx|p / Code-Crafters Ability FTPServer afsmain admin / cpe: / a:code-crafters:ability_ftp_server / ; The port service string, asset information string, manufacturer string, and product name string are extracted as: afsmain, Code CraftersAbility FTP Server afsmain admin, code crafters, and ability ftp server; This satisfies the conditions of Case 2, so the asset information string is processed to obtain the second asset information string code craftersability ftp server(.*?), and (code crafters, ability ftp server, afsmain) is used as the structured triple information corresponding to the second asset information string.

[0131] Example 3: Original data: match activesync m|xxx|sp / Microsoft ActiveSync / o / Windows / cpe: / a:microsoft:activesync / ; The port service string, asset information string, vendor string, and product name string are extracted as: activesync, Microsoft ActiveSync, microsoft, activesync; This satisfies the conditions of Case 2, so (microsoft, activesync, activesync) is used as the structured triplet information corresponding to the asset information string Microsoft ActiveSync.

[0132] Scenario 3: In addition to Scenario 1 and Scenario 2, construct an asset index table directly based on the port service string, vendor string, product name string, and asset information string.

[0133] Example 4: Original data: match 4d-server m|xxxxx|sp / 4th Dimension databaseserver / cpe: / a:4d_sas:4d / ; The port service string, asset information string, manufacturer string, and product name string are extracted as: 4d-server, 4th Dimension database server, 4d_sas, 4d; At this time, the conditions of Case 1 and Case 2 are not met, so (4d_sas, 4d, 4d-server) is used as the structured triple information of the asset information string 4thdimension database server.

[0134] Based on the original data from Examples 1-4 above, the constructed asset index table is shown in Table 3:

[0135] Table 3

[0136] amanda(.*?) (amanda, amanda, amanda) code crafters ability ftp server(.*?) (code crafters, ability ftp server, afsmain) Microsoft ActiveSync (microsoft, activesync, activesync) 4th dimension database server (4d_sas, 4d, 4d-server)

[0137] Therefore, an asset information mapping table and an asset index table can be constructed.

[0138] In one possible implementation, this embodiment of the application extracts multiple asset information strings from the target set to be processed based on a pre-built asset information mapping table and asset index table to obtain target substrings, which can be executed as follows: Figure 3 The steps shown are as follows:

[0139] In step 301, for each asset information string in the target set to be processed: based on the asset information string, the asset information mapping table, and the asset index table, the asset information string is divided into multiple substrings in a right-to-left order; the number of substrings is the same as the length of the asset information string.

[0140] In step 302, for each substring: the frequency of occurrence of the substring is determined based on the number of asset information strings in the target set to be processed and the number of times the substring appears in the corresponding substring in the target set to be processed;

[0141] In step 303, the substrings that appear more frequently than a preset frequency threshold among the multiple asset information strings are selected as target substrings.

[0142] In practice, for each asset information string in the target set to be processed, it is first matched with the asset information mapping table and asset index table constructed in step S12. If the corresponding manufacturer string and product name string of the asset information string can be matched in the asset information mapping table and asset index table, then the string after the maximum position of the manufacturer string or product name string in the asset information string is retained. For example, if the structured triple information corresponding to the asset information string veritas backup exec remote agent in the constructed asset information mapping table and asset index table is (symantec, veritas backup exec, backupexec-remote), then the maximum position of the manufacturer string in the asset information string is 3, and the asset information string is updated to remote agent; otherwise, no operation is performed on the asset information string.

[0143] Then, the asset information string is divided into multiple substrings in a right-to-left order. For example, the string "AB CD" is divided into four substrings in a right-to-left order: 'D', 'CD', 'BCD', and 'ABCD'.

[0144] Next, determine the frequency of each substring using the following formula:

[0145]

[0146] Where freq(subchar) is the frequency of occurrence of the substring; L{subchar} is the number of times the substring appears in the corresponding substring of the target set to be processed; and len(asset_list) is the number of asset information strings in the target set to be processed.

[0147] Finally, substrings that appear more frequently than a preset frequency threshold in multiple asset information strings are selected as target substrings, including:

[0148] First, delete the shortest substring among multiple substrings with the same frequency in the same asset information string, keeping only the longest substring. For example, in the asset information string "infochat remote control", the substrings and their corresponding frequencies are: "infochat remote control": 1 / 6, "remote control": 2 / 6, "control": 2 / 6. In this case, delete the substring "control", keeping only the substrings "infochat remote control" and "remote control". Then, select the substring with a frequency greater than a preset frequency threshold from the remaining substrings as the target substring. For example, if the preset frequency threshold is set to 1 / 6, then only the substring "remote control" from the substrings "infochat remote control" and "remote control" in the asset information string "infochat remote control" will be selected as the target substring.

[0149] For example: Suppose that the multiple asset information strings in the target set to be processed include: bittorrent p2pclient, hp software patrol agent, ninja sphere chromecast driver, infochatremote control, veritas backup exec remote agent, ca unicenter remote control. The substrings and their corresponding frequencies of occurrence in the target set obtained at this point are as follows: bittorrentp2pclient:1 / 6, p2p client:1 / 6, client:1 / 6, hp software patrol agent:1 / 6, softwarepatrol agent:1 / 6, patrol agent:1 / 6, agent:2 / 6, ninja sphere chromecast driver:1 / 6, sphere chromecast driver:1 / 6, chromecast driver:1 / 6, driver:1 / 6, infochatremote control:1 / 6, remote control:2 / 6, control:2 / 6, veritas backup exec remoteagent:1 / 6, backup exec remote agent:1 / 6, exec remote agent:1 / 6, remote agent:1 / 6, ca unicenter remote control:1 / 6, unicenter remote control:1 / 6. Then, setting the preset frequency threshold to 1 / 6, the target substrings in the target set to be processed are "remote control" and "agent".

[0150] Therefore, multiple asset information strings in the target set can be extracted to obtain the target substring.

[0151] In step 103, for each asset information string in the target set to be processed: based on the asset information mapping table, the asset index table, and the target substring, the structured triplet information of the asset information string is determined.

[0152] In one possible implementation, this embodiment of the application determines the structured triplet information of the asset information string based on the asset information mapping table, the asset index table, and the target substring, which can be executed as follows: Figure 4 The steps shown are as follows:

[0153] In step 401, the length of the asset information string is determined.

[0154] In practice, the length of the asset information string is obtained by separating the strings with spaces. For example, the length of the asset information string "bittorrent p2p client" is 3.

[0155] In step 402, it is determined whether the asset information string is greater than a preset length threshold. If the asset information string is not greater than the preset length threshold, then in step 403, the structured triplet information of the asset information string is determined based on the asset information mapping table and the asset index table. If the asset information string is greater than the preset length threshold, then in step 404, the structured triplet information of the asset information string is determined based on the asset information mapping table, the asset index table, and the target substring.

[0156] In the embodiments of this application, the preset length threshold is generally set to 1.

[0157] In one possible implementation, the structured triplet information of the asset information string is determined based on the asset information mapping table and the asset index table in this application embodiment, which can be implemented in the following four ways:

[0158] Implementation Method 1: If the asset index table includes an asset information string that is the same as the asset information string, then the structured triplet information corresponding to the asset information string in the asset index table is used as the structured triplet information of the asset information string.

[0159] Implementation Method 2: If the asset index table does not include an asset information string that is the same as the asset information string, but the asset information mapping table includes a product name string that is the same as the asset information string, then the product name string, the manufacturer string corresponding to the product name string, and the port service string in the asset information mapping table are used as the structured triplet information of the asset information string.

[0160] Implementation Method 3: If the asset index table does not include an asset information string that is the same as the asset information string, but the asset information mapping table includes a manufacturer string that is the same as the asset information string, then the manufacturer string, asset information string, and port service string in the asset information mapping table are used as the structured triplet information of the asset information string.

[0161] Implementation Method 4: If the asset index table does not contain an asset information string that is the same as the asset information string, and the asset information mapping table does not contain a product name string that is the same as the asset information string, and the asset information mapping table does not contain a manufacturer string that is the same as the asset information string, then the asset information string will be used as the product name string and the manufacturer string; and the product name string, the manufacturer string, and the port service string will be used as the structured triplet information of the asset information string.

[0162] In practice, if the asset information string "asset" is not greater than a preset length threshold, the structured triplet information of the asset information string can be determined by the following steps:

[0163] S21: If an asset information string identical to the asset information string can be found in the asset index table, then the structured triplet information corresponding to the asset information string in the asset index table is directly used as the structured triplet information of the asset information string; otherwise, proceed to step S22.

[0164] S22: If a product name string Name that is the same as the asset information string can be found in the asset information mapping table, then the product name string Name, the vendor string corresponding to the product name string Name, and the port service string service in the asset information mapping table are used as the structured triple information (vendor, Name, service) of the asset information string; otherwise, proceed to step S23.

[0165] S23: If the vendor string that is the same as the asset information string can be found in the asset information mapping table, then the vendor string, asset information string, and port service string in the asset information mapping table are used as the structured triple information (vendor, asset, service) of the asset information string; otherwise, proceed to step S24.

[0166] S24: Use the asset information string "asset" as the product name string and the manufacturer string; and use the product name string, manufacturer string, and port service string "service" as the structured triple information (asset, asset, service) of the asset information string.

[0167] In one possible implementation, the structured triplet information of the asset information string is determined based on the asset information mapping table, the asset index table, and the target substring, which can be implemented in the following two ways:

[0168] Implementation Method 1: If a target substring exists in the asset information string, delete the target substring from the asset information string to obtain the third asset information string; based on the asset information mapping table and the asset index table, determine the structured triplet information of the third asset information string, and use the structured triplet information of the third asset information string as the structured triplet information of the asset information string;

[0169] Implementation Method 2: If the target substring does not exist in the asset information string, the asset information string is divided into multiple substrings in order from left to right; and based on the asset information mapping table and the asset index table, the structured triple information of the multiple substrings is determined, and the structured triple information of the multiple substrings is used as the structured triple information of the asset information string.

[0170] In practice, if the asset information string "asset" is longer than a preset length threshold, the structured triplet information of the asset information string can be determined by the following steps:

[0171] S31: If the target substring exists in the asset information string, delete the target substring from the asset information string to obtain the third asset information string, and then perform the following based on the third asset information string: Figure 4 Steps 401-404 are shown; if the target substring does not exist in the asset information string, proceed to step S32;

[0172] S32: If an asset information string identical to the asset information string can be found in the asset index table, then the structured triplet information corresponding to the asset information string in the asset index table is directly used as the structured triplet information of the asset information string; otherwise, proceed to step S33.

[0173] S33: If a product name string Name that is the same as the asset information string can be found in the asset information mapping table, then the product name string Name, the vendor string corresponding to the product name string Name, and the port service string service in the asset information mapping table are used as the structured triple information (vendor, Name, service) of the asset information string; otherwise, proceed to step S34.

[0174] S34: If a vendor string that is the same as the asset information string can be found in the asset information mapping table, then the vendor string, asset information string, and port service string in the asset information mapping table are used as the structured triplet information (vendor, asset, service) of the asset information string; otherwise, proceed to step S35.

[0175] S35: Divide the asset information string into multiple substrings from left to right; then, according to the string length from longest to shortest, execute steps S32-S34 sequentially for the multiple substrings to determine the structured triplet information of the asset information string; if the structured triplet information of the asset information string is not determined, then execute step S36.

[0176] The asset information string is divided into multiple substrings from left to right. For example, the asset information string "netop remote control httpd" is divided into multiple substrings "netop", "netop remote", and "netop remote control".

[0177] S36: Use the asset information string "asset" as the product name string and the manufacturer string; and use the product name string, manufacturer string, and port service string "service" as the structured triple information (asset, asset, service) of the asset information string.

[0178] In this embodiment, the port service string "service" is provided by business data, such as by the configuration file of the network scanning tool Nmap.

[0179] In one possible implementation, the credibility of the structured triplet information of the asset information string can also be set.

[0180] In practice, if the structured triplet information of the asset information string is determined based on the asset index table, the confidence level is set to high; if the product name string with the same asset information string is determined based on the asset information mapping table, and then the structured triplet information of the asset information string is determined, the confidence level is set to high; if the manufacturer string with the same asset information string is determined based on the asset information mapping table, and then the structured triplet information of the asset information string is determined, the confidence level is set to medium; if the asset information string "asset" is directly used as the product name string and manufacturer string, and then the structured triplet information of the asset information string is determined, the confidence level is set to low.

[0181] For example, if the target substring is "remote control" and "agent"; the asset information mapping table is Table 4 and the asset index table is Table 5:

[0182] Table 4

[0183] netop netop remote control netop netop geti2p i2p smtpd opennetworking OpenFlow

[0184] Table 5

[0185] amanda(.*?) (amanda, amanda, amanda) code crafters ability ftp server(.*?) (code crafters, ability ftp server, afsmain) Microsoft ActiveSync (microsoft, activesync, activesync) 4th dimension database server (4d_sas, 4d, 4d-server)

[0186] The asset information strings in the target set to be processed are openflow, i2p smtpd, netop remotecontrol httpd, code crafters ability ftp server afsmain ftpd config;

[0187] 1) Process the asset information string "openflow", and the port service string is "http":

[0188] The string length is 1. Step S21: No asset information string with the same name as the asset information string "openflow" is found in Table 5. Step S22: A product name string with the same name as the asset information string "openflow" is found in Table 4. Then the structured triple information of the asset information string "openflow" is (opennetworking, openflow, http), and the credibility is set to high.

[0189] 2) Process the asset information string i2p smtpd, and the port service string is http:

[0190] The string length is 2. Step S31: No target substring exists. Step S32: No asset information string identical to the asset information string i2p smtpd is found in Table 5. Step S33: A product name string identical to the asset information string i2p smtpd is found in Table 4. Therefore, the structured triplet information of the asset information string i2p smtpd is (geti2p, i2p smtpd, http), and the credibility is set to high.

[0191] 3) Process the asset information string "netop remote control httpd", and the port service string is "http".

[0192] The string length is 4. Step S31: No target substring exists. Step S32: No asset information string identical to the asset information string "netop remote control httpd" is found in Table 5. Steps S33 and S34: A product name string and manufacturer string identical to the asset information string "netop remote control httpd" are found in Table 4. Step S35: Multiple substrings "netop", "netop remote", and "netopremote control" are obtained. A product name string identical to the substring "netop remote control" is found in Table 4. Therefore, the structured triplet information of the asset information string "netop remote control httpd" is (netop, netopremote control, http), and its credibility is set to high.

[0193] 4) Process the asset information strings "code crafters ability ftp server afsmain ftpdconfig", with the port service string being "afsmain".

[0194] The string length is 8. Step S31: No target substring exists. Step S32: In Table 5, an asset information string that is the same as the asset information string "code crafters ability ftp server afsmain ftpd config" is found. The structured triplet information of the asset information string "code crafters ability ftp server afsmain ftpdconfig" is (code crafters, ability ftp server, afsmain), and the confidence level is set to high.

[0195] Based on the foregoing description, this application embodiment obtains a set of targets to be processed; the set of targets to be processed includes multiple asset information strings; based on a pre-built asset information mapping table and asset index table, multiple asset information strings in the set of targets to be processed are extracted to obtain target substrings; for each asset information string in the set of targets to be processed: based on the asset information mapping table, asset index table, and target substring, the structured triplet information of the asset information string is determined. Compared with the prior art, the information extraction method provided by this application embodiment completes the structured transformation of the asset information reported by Nmap, and clarifies the asset description object of the asset information string in the structured form of structured triplet information, directly reducing the ambiguity of the asset reporting results. This not only reduces the cost of subsequent asset management and improves the accuracy and processing efficiency of asset information, but also has stronger fault tolerance for redundant character information unrelated to structured asset information. The obtained structured asset information can provide a data foundation for further cybersecurity activities.

[0196] Based on the same inventive concept, embodiments of this application also provide an information extraction device. For example... Figure 5 The diagram shown is a structural schematic of the information extraction device provided in this application embodiment, which may include:

[0197] The acquisition unit 501 is used to acquire a set of targets to be processed; the set of targets to be processed includes multiple asset information strings;

[0198] The target substring determination unit 502 is used to extract multiple asset information strings from the target set to be processed based on a pre-built asset information mapping table and an asset index table to obtain target substrings; the target substring is a substring among the multiple asset information strings whose frequency of occurrence is greater than a preset frequency threshold; the asset information mapping table is used to represent the correspondence between manufacturer strings and product name strings in the asset information strings; the asset index table is used to represent the structured triple information of the asset information strings and the index relationship between the asset information strings;

[0199] The structured triplet information determination unit 503 is used to determine the structured triplet information of each asset information string in the target set to be processed, based on the asset information mapping table, the asset index table, and the target substring.

[0200] In one possible implementation, the device further includes:

[0201] The construction unit is used to construct the asset information mapping table and the asset index table according to the following methods:

[0202] Acquire business data; and extract information from the business data to obtain the port service string, asset information string, manufacturer string, and product name string corresponding to the asset information string in the business data;

[0203] Based on the manufacturer string and the product name string, construct the asset information mapping table; and based on the port service string, the asset information string, the manufacturer string, and the product name string, construct the asset index table.

[0204] In one possible implementation, the building unit is specifically used for:

[0205] Based on preset mapping rules, the natural strings corresponding to the manufacturer string and the product name string are determined; the preset mapping rules represent the correspondence between formatted characters and natural characters.

[0206] The asset information mapping table is constructed based on the natural string corresponding to the manufacturer string and the natural string corresponding to the product name string.

[0207] In one possible implementation, the building unit is specifically used for:

[0208] If the manufacturer string and the product name string are the same, the manufacturer string is included in the asset information string, and the manufacturer string is different from the asset information string, then the characters in the asset information string other than the manufacturer string are set as preset reference characters to obtain the first asset information string; based on the port service string, the manufacturer string, the product name string and the first asset information string, the asset index table is constructed;

[0209] If the manufacturer string and the product name string are different, the manufacturer string and the asset information string have the same part, and the product name string is completely contained in the asset information string, then the characters in the asset information string other than the part that is the same as the manufacturer string and the product name string are set as preset reference characters to obtain the second asset information string; based on the port service string, the manufacturer string, the product name string and the second asset information string, the asset index table is constructed.

[0210] In one possible implementation, the target substring determination unit 502 is specifically used for:

[0211] For each asset information string in the target set to be processed: based on the asset information string, the asset information mapping table, and the asset index table, the asset information string is divided into multiple substrings in a right-to-left order; the number of substrings is the same as the length of the asset information string.

[0212] For each substring: the frequency of occurrence of the substring is determined based on the number of asset information strings in the target set to be processed and the number of times the substring appears in the corresponding substring in the target set to be processed;

[0213] The substrings that appear more frequently than a preset frequency threshold among the multiple asset information strings are selected as the target substrings.

[0214] In one possible implementation, the structured triplet information determination unit 503 is specifically used for:

[0215] If the asset information string is not greater than a preset length threshold, then the structured triplet information of the asset information string is determined based on the asset information mapping table and the asset index table;

[0216] If the asset information string is greater than the preset length threshold, then the structured triplet information of the asset information string is determined based on the asset information mapping table, the asset index table, and the target substring.

[0217] In one possible implementation, the structured triplet information determination unit 503 is specifically used for:

[0218] If the asset index table includes an asset information string that is the same as the asset information string, then the structured triplet information corresponding to the index in the asset index table shall be used as the structured triplet information of the asset information string;

[0219] If the asset index table does not include an asset information string that is the same as the asset information string, and the asset information mapping table includes a product name string that is the same as the asset information string, then the product name string, the manufacturer string corresponding to the product name string, and the port service string in the asset information mapping table are used as the structured triplet information of the asset information string.

[0220] If the asset index table does not include an asset information string that is the same as the asset information string, and the asset information mapping table includes a manufacturer string that is the same as the asset information string, then the manufacturer string, the asset information string, and the port service string in the asset information mapping table are used as the structured triplet information of the asset information string.

[0221] If the asset index table does not contain an asset information string that is the same as the asset information string, and the asset information mapping table does not contain a product name string that is the same as the asset information string, and the asset information mapping table does not contain a manufacturer string that is the same as the asset information string, then the asset information string will be used as the product name string and the manufacturer string; and the product name string, the manufacturer string, and the port service string will be used as the structured triplet information of the asset information string.

[0222] In one possible implementation, the structured triplet information determination unit 503 is specifically used for:

[0223] If the target substring exists in the asset information string, then delete the target substring from the asset information string to obtain the third asset information string; based on the asset information mapping table and the asset index table, determine the structured triplet information of the third asset information string, and use the structured triplet information of the third asset information string as the structured triplet information of the asset information string;

[0224] If the target substring does not exist in the asset information string, the asset information string is divided into multiple substrings in order from left to right; and based on the asset information mapping table and the asset index table, the structured triple information of the multiple substrings is determined, and the structured triple information of the multiple substrings is used as the structured triple information of the asset information string.

[0225] In one possible implementation, the device further includes:

[0226] The filling unit is used to add at least one new asset information string to the target set to be processed if at least one new asset information string is obtained, and then execute the step of extracting multiple asset information strings in the target set to be processed based on the pre-built asset information mapping table and asset index table to obtain target substrings.

[0227] Based on the same technical concept, this application also provides an electronic device 600. Referring below... Figure 6 To describe an electronic device 600 according to this embodiment of the present application. Figure 6 The electronic device 600 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0228] like Figure 6 As shown, the electronic device 600 is presented in the form of a general-purpose electronic device. The components of the electronic device 600 may include, but are not limited to: at least one processor 601, at least one memory 602, and a bus 603 connecting different system components (including memory 602 and processor 601).

[0229] Bus 603 represents one or more of several bus structures, including a memory bus or memory controller, peripheral bus, processor, or local bus using any of the various bus structures.

[0230] The memory 602 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 6021 and / or cache memory 6022, and may further include read-only memory (ROM) 6023.

[0231] The memory 602 may also include a program / utility 6025 having a set (at least one) of program modules 6024, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0232] Electronic device 600 can also communicate with one or more external devices 604 (e.g., keyboard, pointing device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other electronic devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 605. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 606. As shown, network adapter 606 communicates with other modules used in electronic device 600 via bus 603. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0233] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 602 including instructions, which can be executed by a processor 601 to complete the information extraction method described above. Optionally, the storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0234] In an exemplary embodiment, a computer program product is also provided, including a computer program that, when executed by a processor 601, implements any of the information extraction methods provided in this application.

[0235] In an exemplary embodiment, various aspects of the information extraction method provided in this application can also be implemented as a program product, which includes program code. When the program product is run on a computer device, the program code is used to cause the computer device to perform the steps in the information extraction method according to the various exemplary embodiments of this application described above.

[0236] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0237] The program product for the information extraction method according to the embodiments of this application can be a portable compact disc read-only memory (CD-ROM) and include program code, and can run on an electronic device. However, the program product of this application is not limited thereto. In this document, the readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0238] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take many forms, including—but not limited to—electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0239] The program code contained on the readable medium may be transmitted using any suitable medium, including—but not limited to—wireless, wired, fiber optic, RF, etc., or any suitable combination thereof.

[0240] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's electronic device, partially on the user's device, as a standalone software package, partially on the user's electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In cases involving remote electronic devices, the remote electronic device can be connected to the user's electronic device via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0241] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0242] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0243] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0244] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable image scaling device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable image scaling device, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0245] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable image scaling device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0246] These computer program instructions can also be loaded onto a computer or other programmable image scaling device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0247] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0248] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An information extraction method, characterized in that, The method includes: Obtain a set of targets to be processed; the set of targets to be processed includes multiple unstructured asset information strings; the asset information is the asset information reported by Nmap port probe; each asset information string is composed of a manufacturer string and a product name string, and there is no clear boundary between the manufacturer string and the product name string; For each asset information string in the set of targets to be processed: Based on the asset information string, a pre-built asset information mapping table, and a pre-built asset index table, the asset information string is divided into multiple substrings from right to left; the number of substrings is the same as the length of the asset information string; the asset information mapping table represents the correspondence between the manufacturer string and the product name string in the asset information string; the asset index table represents the index relationship between the structured triple information of the asset information string and the asset information string; the data sources for constructing the asset information mapping table and the asset index table are CPE specification data, publicly available network asset or software application data on the network, and Nmap configuration files; before dividing the asset information string into multiple substrings from right to left, the asset information string is matched with the asset information mapping table and the asset index table. If the corresponding manufacturer string or product name string can be matched in the asset information mapping table and the asset index table, the string after the maximum position of the manufacturer string or product name string in the asset information string is retained and used as the updated asset information string; For each substring: the frequency of occurrence of the substring is determined based on the number of asset information strings in the target set to be processed and the number of times the substring appears in the corresponding substring in the target set to be processed; The substrings that appear more frequently than a preset frequency threshold in the plurality of asset information strings are selected as target substrings; the selection of the substrings that appear more frequently than the preset frequency threshold in the plurality of asset information strings as target substrings includes: deleting the shorter string from the plurality of substrings that appear with the same frequency in the same asset information string, and keeping only the longest string; selecting the substrings that appear more frequently than the preset frequency threshold from the remaining substrings as target substrings; Based on the asset information mapping table, the asset index table, and the target substring, the structured triplet information of the asset information string is determined; the structured triplet information includes the manufacturer string, the product name string, and the port service string.

2. The method as described in claim 1, characterized in that, The asset information mapping table and the asset index table are constructed according to the following methods: Acquire business data; and extract information from the business data to obtain the port service string, asset information string, manufacturer string from the general platform enumeration CPE data, and product name string from the CPE data corresponding to the asset information string in the business data; Based on the manufacturer string and the product name string, construct the asset information mapping table; and based on the port service string, the asset information string, the manufacturer string, and the product name string, construct the asset index table.

3. The method as described in claim 2, characterized in that, The step of constructing the asset information mapping table based on the manufacturer string and the product name string includes: Based on preset mapping rules, the natural strings corresponding to the manufacturer string and the product name string are determined; the preset mapping rules represent the correspondence between formatted characters and natural characters. The asset information mapping table is constructed based on the natural string corresponding to the manufacturer string and the natural string corresponding to the product name string.

4. The method as described in claim 2, characterized in that, The step of constructing the asset index table based on the port service string, the asset information string, the manufacturer string, and the product name string includes: If the manufacturer string and the product name string are the same, the manufacturer string is included in the asset information string, and the manufacturer string is different from the asset information string, then the characters in the asset information string other than the manufacturer string are set as preset reference characters to obtain the first asset information string; based on the port service string, the manufacturer string, the product name string and the first asset information string, the asset index table is constructed; If the manufacturer string and the product name string are different, the manufacturer string and the asset information string have the same part, and the product name string is completely contained in the asset information string, then the characters in the asset information string other than the part that is the same as the manufacturer string and the product name string are set as the preset reference characters to obtain the second asset information string; based on the port service string, the manufacturer string, the product name string and the second asset information string, the asset index table is constructed.

5. The method as described in claim 1, characterized in that, The step of determining the structured triplet information of the asset information string based on the asset information mapping table, the asset index table, and the target substring includes: If the asset information string is not greater than a preset length threshold, then the structured triplet information of the asset information string is determined based on the asset information mapping table and the asset index table; If the asset information string is greater than the preset length threshold, then the structured triplet information of the asset information string is determined based on the asset information mapping table, the asset index table, and the target substring.

6. The method as described in claim 5, characterized in that, The step of determining the structured triplet information of the asset information string based on the asset information mapping table and the asset index table includes: If the asset index table includes an asset information string that is the same as the asset information string, then the structured triplet information corresponding to the asset information string in the asset index table shall be used as the structured triplet information of the asset information string. If the asset index table does not include an asset information string that is the same as the asset information string, and the asset information mapping table includes a product name string that is the same as the asset information string, then the product name string, the manufacturer string corresponding to the product name string, and the port service string in the asset information mapping table are used as the structured triplet information of the asset information string. If the asset index table does not include an asset information string that is the same as the asset information string, and the asset information mapping table includes a manufacturer string that is the same as the asset information string, then the manufacturer string, the asset information string, and the port service string in the asset information mapping table are used as the structured triplet information of the asset information string. If the asset index table does not contain an asset information string that is the same as the asset information string, and the asset information mapping table does not contain a product name string that is the same as the asset information string, and the asset information mapping table does not contain a manufacturer string that is the same as the asset information string, then the asset information string will be used as the product name string and the manufacturer string; and the product name string, the manufacturer string, and the port service string will be used as the structured triplet information of the asset information string.

7. The method as described in claim 6, characterized in that, The step of determining the structured triplet information of the asset information string based on the asset information mapping table, the asset index table, and the target substring includes: If the target substring exists in the asset information string, then delete the target substring from the asset information string to obtain the third asset information string; based on the asset information mapping table and the asset index table, determine the structured triplet information of the third asset information string, and use the structured triplet information of the third asset information string as the structured triplet information of the asset information string; If the target substring does not exist in the asset information string, the asset information string is divided into multiple substrings in order from left to right; and based on the asset information mapping table and the asset index table, the structured triple information of the multiple substrings is determined, and the structured triple information of the multiple substrings is used as the structured triple information of the asset information string.

8. The method as described in claim 1, characterized in that, The method further includes: If at least one new asset information string is obtained, the at least one new asset information string is added to the target set to be processed. Then, the step of dividing the asset information string into multiple substrings in the order from right to left for each asset information string in the target set to be processed is performed, based on the asset information string, the pre-built asset information mapping table and the pre-built asset index table.

9. An information extraction device, characterized in that, The device includes: The acquisition unit is used to acquire a set of targets to be processed; the set of targets to be processed includes multiple unstructured asset information strings; the asset information is the asset information reported by Nmap port probe; each asset information string is composed of a manufacturer string and a product name string, and there is no clear boundary between the manufacturer string and the product name string; The target substring determination unit is used to, for each asset information string in the target set to be processed, divide the asset information string into multiple substrings in a right-to-left order based on the asset information string, a pre-built asset information mapping table, and a pre-built asset index table; the number of substrings is the same as the length of the asset information string; the asset information mapping table is used to represent the correspondence between the manufacturer string and the product name string in the asset information string; the asset index table is used to represent the structured triple information of the asset information string and the index relationship between the asset information string; the data sources for constructing the asset information mapping table and the asset index table are CPE specification data, publicly available network asset or software application data on the network, and Nmap configuration files; before dividing the asset information string into multiple substrings in a right-to-left order, the asset information string is mapped to the asset information mapping table. The asset information mapping table and the asset index table are matched. If the manufacturer string and product name string corresponding to the asset information string can be matched in the asset information mapping table and the asset index table, the string after the maximum position of the manufacturer string or product name string in the asset information string is retained and used as the updated asset information string. For each substring: the occurrence frequency of the substring is determined based on the number of asset information strings in the target set to be processed and the number of times the substring appears in the corresponding substring in the target set to be processed. The substrings with an occurrence frequency greater than a preset frequency threshold among the multiple asset information strings are selected as target substrings. The target substring determination unit is used to: delete the shorter string among multiple substrings with the same occurrence frequency in the same asset information string, and only retain the longest string; select the substrings with an occurrence frequency greater than a preset frequency threshold from the remaining substrings as target substrings. The structured triple information determination unit is used to determine the structured triple information of the asset information string based on the asset information mapping table, the asset index table, and the target substring; the structured triple information includes a vendor string, a product name string, and a port service string.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the information extraction method as described in any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the information extraction method as described in any one of claims 1 to 8.

Images