Fingerprint rule generation method, electronic equipment, storage medium and program product

Abstract

The embodiment of the invention provides a fingerprint rule generation method, electronic equipment, a storage medium and a program product. The method comprises the following steps: acquiring network response data and a product label; determining label data corresponding to the network response data according to the network response data; according to each piece of label data, determining importance parameter information of the label data; determining key information in the label data according to the product label; determining target fingerprint label data in the label data according to the key information and the importance parameter information; according to the target fingerprint label data and the product label, the fingerprint rule corresponding to the network response data is generated, the fingerprint rule is stored in the fingerprint database, the label related to the product is effectively extracted by calculating the importance parameter information of the HTML label, and the accuracy of rule generation is greatly improved by comparing the key information of the product. And the quality of the rule is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method for generating fingerprint rules, an electronic device, a storage medium, and a program product. Background Art

[0002] Computer networks have penetrated into every aspect of social life, and inevitably brought many security issues, such as frequent hacker attacks and information theft. Monitoring nodes in the network, timely discovering their vulnerabilities, assessing risk status, and targeted repairs are the basic ways to ensure network security. When conducting risk assessment, if you can first identify its important attributes, such as operating system, device type, running services and versions, you can conduct more targeted vulnerability assessments and more accurately assess its security status. The goal of fingerprint recognition is to quickly and accurately discover relevant information about components, so as to discover vulnerabilities in the site based on this information. Depending on the object to be identified, fingerprints are generally divided into host system fingerprint recognition, port service fingerprint recognition, web application fingerprint recognition, etc. The basic principle is to make judgments based on different features in the response.

[0003] In the prior art, a neural network model can be used to identify the fingerprint of a component. However, for some components, the fingerprint data cannot be accurately identified, such as https protocol data. Therefore, how to accurately identify the fingerprint data of https protocol data is a problem that urgently needs to be solved. Summary of the invention

[0004] The purpose of some embodiments of the present application is to provide a method for generating fingerprint rules, an electronic device, a storage medium and a program product. Through the technical solutions of the embodiments of the present application, network response data and product labels are obtained; label data corresponding to the network response data is determined according to the network response data; importance parameter information of the label data is determined according to each label data; key information in the label data is determined according to the product label; target fingerprint label data in the label data is determined according to the key information and the importance parameter information; fingerprint rules corresponding to the network response data are generated according to the target fingerprint label data and the product label, and the fingerprint rules are stored in a fingerprint library. By calculating the importance parameter information of HTML tags, labels related to products are effectively extracted, and by comparing product key information, the accuracy of generated rules is greatly improved, which helps to improve the quality of rules.

[0005] In a first aspect, some embodiments of the present application provide a method for generating a fingerprint rule, including:

[0006] Get network response data and product labels;

[0007] Determining, according to the network response data, label data corresponding to the network response data;

[0008] Determine, according to each of the label data, the importance parameter information of the label data;

[0009] According to the product label, determining key information in the label data;

[0010] Determining target fingerprint label data in the label data according to the key information and the importance parameter information;

[0011] A fingerprint rule corresponding to the network response data is generated according to the target fingerprint tag data and the product tag, and the fingerprint rule is stored in a fingerprint library.

[0012] Some embodiments of the present application calculate the importance parameter information of HTML tags, effectively extract tags related to products, and compare product key information, thereby greatly improving the accuracy of generated rules and helping to improve the quality of rules.

[0013] Optionally, determining the importance parameter information of the label data according to each of the label data includes:

[0014] The TF-IDF algorithm is used to calculate the TF-IDF value of each tag data.

[0015] Some embodiments of the present application respectively calculate the TF-IDF value of each tag data by adopting the TF-IDF algorithm, so as to select the tag data with high TF-IDF value.

[0016] Optionally, determining the target fingerprint label data in the label data according to the key information and the importance parameter information includes:

[0017] According to the size of the TF-IDF value, obtain the first label data corresponding to the first N TF-IDF values;

[0018] Determining whether the first tag data contains key information;

[0019] From the remaining label data, obtain the second label data containing the key information;

[0020] When the key information does not exist in the first label data, replacing the first label data with the second label data;

[0021] The replaced first label data is determined as the target fingerprint label data.

[0022] Some embodiments of the present application replace the label data that does not contain product key information and has the lowest TF-IDF value among the first N label data with the second label data. In this way, the obtained label data may be label data that contains product keywords and has a higher TF-IDF value, thereby improving the accuracy of label data recognition.

[0023] Optionally, generating a fingerprint rule corresponding to the network response data according to the target fingerprint tag data and the product tag includes:

[0024] Determine, according to the target fingerprint tag data, a regular expression corresponding to the target fingerprint tag data;

[0025] A fingerprint rule corresponding to the network response data is generated according to a regular expression corresponding to the target fingerprint tag data and the product tag, wherein the product tag includes at least a manufacturer name, a product name, and a product type.

[0026] Some embodiments of the present application generate fingerprint rules corresponding to network response data based on regular expressions and product labels corresponding to target fingerprint tag data, so as to make the query process more accurate, improve the efficiency of obtaining the queried web pages, and reduce the errors generated in the query process.

[0027] Optionally, determining a regular expression corresponding to the target fingerprint label data according to the target fingerprint label data includes:

[0028] If there are multiple pieces of label data containing the key information in the target fingerprint label data, fingerprint rules corresponding to the label data are generated according to each piece of label data, and the fingerprint rules are in an OR relationship.

[0029] In some embodiments of the present application, if there are multiple label data containing product keywords in the target fingerprint label data, each label data is generated as a fingerprint rule, and the rules are in an or relationship.

[0030] Optionally, determining a regular expression corresponding to the target fingerprint label data according to the target fingerprint label data includes:

[0031] If the key information does not exist in any of the first label data, the first label data are sequentially spliced ​​to obtain spliced ​​label data;

[0032] Generate a corresponding regular expression according to the concatenated label data.

[0033] In some embodiments of the present application, if the first label data do not contain product keywords, the extracted first label data can be spliced ​​into a regular expression as a fingerprint rule. In this case, the specific sequence structure between multiple labels can be used as a fingerprint feature.

[0034] Optionally, the method further comprises:

[0035] If the key information does not exist in the first tag data, the first tag data is deleted.

[0036] In some embodiments of the present application, if one or more of the first N tag data contain product keywords, the remaining tag data that do not contain product keywords are discarded to reduce memory space usage.

[0037] In a second aspect, some embodiments of the present application provide a device for generating fingerprint rules, including:

[0038] The acquisition module is used to obtain network response data and product labels;

[0039] A first determining module, configured to determine, according to the network response data, label data corresponding to the network response data;

[0040] A second determination module, used to determine the importance parameter information of the label data according to each of the label data;

[0041] A third determination module, used to determine key information in the label data according to the product label;

[0042] A fourth determination module, used to determine target fingerprint label data in the label data according to the key information and the importance parameter information;

[0043] A generation module is used to generate a fingerprint rule corresponding to the network response data according to the target fingerprint tag data and the product tag, and store the fingerprint rule in a fingerprint library.

[0044] Some embodiments of the present application calculate the importance parameter information of HTML tags, effectively extract tags related to products, and compare product key information, thereby greatly improving the accuracy of generated rules and helping to improve the quality of rules.

[0045] Optionally, the second determining module is used to:

[0046] The TF-IDF algorithm is used to calculate the TF-IDF value of each tag data.

[0047] Some embodiments of the present application respectively calculate the TF-IDF value of each tag data by adopting the TF-IDF algorithm, so as to select the tag data with high TF-IDF value.

[0048] Optionally, the fourth determining module is used to:

[0049] According to the size of the TF-IDF value, obtain the first label data corresponding to the first N TF-IDF values;

[0050] Determining whether the first tag data contains key information;

[0051] From the remaining label data, obtain the second label data containing the key information;

[0052] When the key information does not exist in the first label data, replacing the first label data with the second label data;

[0053] The replaced first label data is determined as the target fingerprint label data.

[0054] Some embodiments of the present application replace the label data that does not contain product key information and has the lowest TF-IDF value among the first N label data with the second label data. In this way, the obtained label data may be label data that contains product keywords and has a higher TF-IDF value, thereby improving the accuracy of label data recognition.

[0055] Optionally, the generating module is used to:

[0056] Determine, according to the target fingerprint tag data, a regular expression corresponding to the target fingerprint tag data;

[0057] A fingerprint rule corresponding to the network response data is generated according to a regular expression corresponding to the target fingerprint tag data and the product tag, wherein the product tag includes at least a manufacturer name, a product name, and a product type.

[0058] Some embodiments of the present application generate fingerprint rules corresponding to network response data based on regular expressions and product labels corresponding to target fingerprint tag data, so as to make the query process more accurate, improve the efficiency of obtaining the queried web pages, and reduce the errors generated in the query process.

[0059] Optionally, the generating module is used to:

[0060] If there are multiple pieces of label data containing the key information in the target fingerprint label data, fingerprint rules corresponding to the label data are generated according to each piece of label data, and the fingerprint rules are in an OR relationship.

[0061] In some embodiments of the present application, if there are multiple label data containing product keywords in the target fingerprint label data, each label data is generated as a fingerprint rule, and the rules are in an or relationship.

[0062] Optionally, the generating module is used to:

[0063] If the key information does not exist in any of the first label data, the first label data are sequentially spliced ​​to obtain spliced ​​label data;

[0064] Generate a corresponding regular expression according to the concatenated label data.

[0065] In some embodiments of the present application, if the first label data do not contain product keywords, the extracted first label data can be spliced ​​into a regular expression as a fingerprint rule. In this case, the specific sequence structure between multiple labels can be used as a fingerprint feature.

[0066] Optionally, the generating module is used to:

[0067] If the key information does not exist in the first tag data, the first tag data is deleted.

[0068] In some embodiments of the present application, if one or more of the first N tag data contain product keywords, the remaining tag data that do not contain product keywords are discarded to reduce memory space usage.

[0069] In a third aspect, some embodiments of the present application provide an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for generating fingerprint rules as described in any embodiment of the first aspect can be implemented.

[0070] In a fourth aspect, some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the method for generating fingerprint rules as described in any embodiment of the first aspect.

[0071] In a fifth aspect, some embodiments of the present application provide a computer program product, wherein the computer program product includes a computer program, wherein when the computer program is executed by a processor, it can implement the method for generating fingerprint rules as described in any embodiment of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the technical solutions of some embodiments of the present application, the drawings required for use in some embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0073] Figure 1 A schematic diagram of a flow chart of a method for generating fingerprint rules provided in an embodiment of the present application;

[0074] Figure 2 A flowchart of another method for generating fingerprint rules provided in an embodiment of the present application;

[0075] Figure 3 A schematic diagram of the structure of a fingerprint rule generation device provided in an embodiment of the present application;

[0076] Figure 4 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0077] The technical solutions in some embodiments of the present application will be described below in conjunction with the drawings in some embodiments of the present application.

[0078] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0079] Computer networks have penetrated into every aspect of social life, and inevitably brought many security issues, such as frequent hacker attacks and information theft. Monitoring nodes in the network, timely discovering their vulnerabilities, assessing risk status, and targeted repairs are the basic ways to ensure network security. When conducting risk assessment, if you can first identify its important attributes, such as operating system, device type, running services and versions, you can conduct more targeted vulnerability assessments and more accurately assess its security status. The goal of fingerprint recognition is to quickly and accurately discover relevant information about components, so as to discover vulnerabilities in the site based on this information. Depending on the object to be identified, fingerprints are generally divided into host system fingerprint recognition, port service fingerprint recognition, web application fingerprint recognition, etc. The basic principle is to make judgments based on different features in the response.

[0080] In the prior art, a neural network model can be used to identify the fingerprint of a component. However, for some components, the fingerprint data cannot be accurately identified, such as https protocol data. Therefore, how to accurately identify the fingerprint data of https protocol data is a problem that needs to be solved urgently. In view of this, some embodiments of the present application provide a method for generating fingerprint rules, which includes obtaining network response data and product labels; determining label data corresponding to the network response data based on the network response data; determining importance parameter information of the label data based on each label data; determining key information in the label data based on the product label; determining target fingerprint label data in the label data based on the key information and importance parameter information; generating fingerprint rules corresponding to the network response data based on the target fingerprint label data and the product label, and storing the fingerprint rules in a fingerprint library. By calculating the importance parameter information of HTML tags, labels related to the products are effectively extracted, and by comparing the product key information, the accuracy of the generated rules is greatly improved, which helps to improve the quality of the rules.

[0081] like Figure 1 As shown, an embodiment of the present application provides a method for generating fingerprint rules, the method comprising:

[0082] S101, obtaining network response data and product labels;

[0083] Specifically, data is downloaded from an open asset search engine, or scan data is obtained from other systems, and the network response data at least includes response data of HTTP and HTTPS protocols and product labels.

[0084] The terminal device parses and identifies the network response data, and obtains the corresponding product label data based on the characteristic information in the network response data. The network response data usually contains characteristic information, and the product can be identified by writing identification rules based on this characteristic information. This characteristic information is generally included in the response header, URL, file MD5, HTML text data, HTML tag data, etc. In general, it is necessary to manually analyze the characteristics and manually write identification rules containing characteristic information, and use the identification rules to match the response data. If a match is found, the target of the response data can be identified as the product.

[0085] S102, determining label data corresponding to the network response data according to the network response data;

[0086] Specifically, the terminal device preprocesses the network response data and extracts the HTML tag data. Specifically, it first performs preliminary processing, that is, deletes erroneous responses and deletes response data with error codes and redirection codes; then uses regular expressions to match and extract HTML tag data sequences, and deletes obviously featureless tag data, such as,, etc., and finally obtains tag data corresponding to the network response data.

[0087] S103, determining importance parameter information of each tag data according to the tag data;

[0088] Specifically, the terminal device analyzes each tag data to obtain the importance parameter information of the tag data, for example, using the TF-IDF algorithm to calculate the TF-IDF value of each tag data.

[0089] S104, determining key information in the label data according to the product label;

[0090] Specifically, the terminal device identifies the key information in the product label to obtain the key information of each product. For example, a neural network model can be used to identify the key information in the product label, or other algorithms can be used to identify the key information in the product label, which is not specifically limited in the embodiments of the present application.

[0091] S105, determining target fingerprint label data in the label data according to the key information and importance parameter information;

[0092] Specifically, the terminal device judges the key information and the importance parameter information, and selects label data that contains product keywords and has a higher TF-IDF value as target fingerprint label data in the label data.

[0093] S106: Generate fingerprint rules corresponding to the network response data according to the target fingerprint tag data and the product tag, and store the fingerprint rules in a fingerprint library.

[0094] Specifically, the terminal device generates a corresponding regular expression for each target fingerprint tag data, and then combines it with the corresponding product tag to generate a fingerprint rule corresponding to the network response data, and stores all fingerprint rules in the fingerprint library. The fingerprint rules are automatically generated, saving a lot of manpower and time for manually writing fingerprint rules, and quickly expanding the existing fingerprint rule library; the generated regular expression rules are closer to the manually written rules, have a certain generalization ability, and are more conducive to target identification; the obtained HTML tag fingerprint rules are, to a certain extent, more accurate than the rules extracted from text alone.

[0095] Some embodiments of the present application calculate the importance parameter information of HTML tags, effectively extract tags related to products, and compare product key information, thereby greatly improving the accuracy of generated rules and helping to improve the quality of rules.

[0096] Another embodiment of the present application further supplements the method for generating fingerprint rules provided in the above embodiment.

[0097] Optionally, according to each tag data, importance parameter information of the tag data is determined, including:

[0098] The TF-IDF algorithm is used to calculate the TF-IDF value of each tag data.

[0099] Some embodiments of the present application use a TF-IDF algorithm to calculate the TF-IDF value of each tag data respectively, so as to select tag data with a high TF-IDF value.

[0100] Optionally, determining target fingerprint label data in the label data according to the key information and importance parameter information includes:

[0101] According to the size of the TF-IDF value, obtain the first label data corresponding to the first N TF-IDF values;

[0102] Determine whether the first tag data contains key information;

[0103] From the remaining label data, obtain the second label data containing the key information;

[0104] When the key information does not exist in the first label data, the first label data is replaced by the second label data;

[0105] The replaced first label data is determined as the target fingerprint label data.

[0106] Some embodiments of the present application replace the label data that does not contain product key information and has the lowest TF-IDF value among the first N label data with the second label data. In this way, the obtained label data may be label data that contains product keywords and has a higher TF-IDF value, thereby improving the accuracy of label data recognition.

[0107] Optionally, generating fingerprint rules corresponding to the network response data according to the target fingerprint tag data and the product tag includes:

[0108] According to the target fingerprint label data, determining a regular expression corresponding to the target fingerprint label data;

[0109] A fingerprint rule corresponding to the network response data is generated according to a regular expression corresponding to the target fingerprint tag data and a product tag, wherein the product tag includes at least a manufacturer name, a product name and a product type.

[0110] Some embodiments of the present application generate fingerprint rules corresponding to network response data based on regular expressions and product labels corresponding to target fingerprint tag data, so as to make the query process more accurate, improve the efficiency of obtaining the queried web pages, and reduce the errors generated in the query process.

[0111] Optionally, determining a regular expression corresponding to the target fingerprint label data according to the target fingerprint label data includes:

[0112] If there are multiple pieces of label data containing key information in the target fingerprint label data, fingerprint rules corresponding to the label data are generated according to each piece of label data, and the fingerprint rules are in an OR relationship with each other.

[0113] In some embodiments of the present application, if there are multiple label data containing product keywords in the target fingerprint label data, each label data is generated as a fingerprint rule, and the rules are in an or relationship.

[0114] Optionally, determining a regular expression corresponding to the target fingerprint label data according to the target fingerprint label data includes:

[0115] If the key information does not exist in the first label data, the first label data are sequentially spliced ​​to obtain spliced ​​label data;

[0116] Generate the corresponding regular expression based on the concatenated label data.

[0117] In some embodiments of the present application, if the first label data do not contain product keywords, the extracted first label data can be spliced ​​into a regular expression as a fingerprint rule. In this case, the specific sequence structure between multiple labels can be used as a fingerprint feature.

[0118] Optionally, the method further comprises:

[0119] If the key information does not exist in the first tag data, the first tag data is deleted.

[0120] In some embodiments of the present application, if one or more of the first N tag data contain product keywords, the remaining tag data that do not contain product keywords are discarded to reduce memory space usage.

[0121] like Figure 2 As shown, the embodiment of the present application provides a method for automatically generating fingerprint rules based on HTML tags, including the following three processes:

[0122] Process (1): data preprocessing process;

[0123] Process (2): Extracting HTML tags;

[0124] Process (3): rule generation process.

[0125] Process (1) Data preprocessing process:

[0126] First, download data from an open asset search engine, or obtain scan data from other systems, including response data of http and https protocols.

[0127] Then preprocess the response data and extract the HTML tag data:

[0128] Initial processing: delete the error response, delete the response data of the error code and redirection code;

[0129] Use regular expressions to match and extract HTML tag data sequences;

[0130] Delete obviously featureless label data, such as,,, etc.

[0131] Process (2) Extracting HTML tags:

[0132] For the above HTML tag data sequence, calculate the TF-IDF value of each tag data and extract the top N tag data with higher TF-IDF values. Specific steps:

[0133] The TF-IDF algorithm is used to calculate the TF-IDF value of each HTML tag data, and its delimiter is set to a regular expression that recognizes the <> tag; TF-IDF (term frequency-inverse document frequency) is a commonly used weighting technology for information retrieval and data mining. TF is term frequency, and IDF is inverse document frequency.

[0134] Extract the top N HTML tags with higher TF-IDF values, such as the top 5.

[0135] HTML tag data with a high TF-IDF value does not necessarily contain product keywords, but HTML tag data containing product keywords can generally identify the product. Therefore, it is necessary to check whether the unextracted tag data contains product keywords. If so, replace the tag data that does not contain product keywords and has the lowest TF-IDF value among the first N tag data with the above tag data. In this way, it is ensured that the extracted tag data contains product keywords and has a high TFIDF value as much as possible.

[0136] Process (3) Rule generation process:

[0137] For the first N HTML tag data obtained above, check whether each tag data contains the product keyword, and then generate a regular expression, and associate the regular expression with the product information to form a fingerprint rule.

[0138] If a piece of extracted label data contains product keywords, it means that the label data itself can identify the product, and it is directly converted into a regular expression as a fingerprint rule. If there are multiple pieces of label data containing product keywords, each piece of label data is generated as a fingerprint rule, and the relationship between the rules is or.

[0139] If one or more of the first N label data contain product keywords, the remaining label data that do not contain product keywords are discarded.

[0140] If the first N tag data do not contain the product keyword, the first N tag data extracted can be spliced ​​into a regular expression as a fingerprint rule. At this time, the specific sequence structure between multiple tags can be used as a fingerprint feature.

[0141] In steps 1) and 3), when converting label data into regular expressions, the following processing is required: for special characters, the escape character "\" needs to be added; for pure numeric strings, they can be replaced with regular expressions, such as "\d+", "[\d\.]+"; for multiple label data, "[\w\W]*?" can be used for splicing.

[0142] Compare the response data with the manufacturer database, extract the manufacturer (vendor), and then obtain the product type (type) according to the product type set when downloading the data, and complete the product information triple (vendor, product, type).

[0143] Specifically: first collect a list of common manufacturers in each category to form a manufacturer database; segment the product name and compare it with the manufacturer database to find the manufacturer; if it fails, try to find the manufacturer by comparing the keywords of the response information.

[0144] Finally, the generated fingerprint rules are associated with the product triplet information (vendor, product, type).

[0145] After the above three processes, a fingerprint rule that associates the regular expression with the product information is finally generated, and the fingerprint rule is stored in the rule base for subsequent product identification.

[0146] For example, 1. Download the target data from the asset search engine. The downloaded data includes information such as product type (type), protocol, product, banner (response data), certificate, etc., where the product field value is the product name recognized by the asset search engine. Each time, a large number of response data of many products are processed. After extracting HTML tags from each response data, they are calculated together as a TF-IDF corpus. A data sample of the open source Web application framework Web2py obtained from the asset search engine is shown below.

[0147]

[0148]

[0149]

[0150] 2. Extract the HTML tag data sequence from the banner field. As described in detail: delete tags without obvious features and extract the HTML tag data sequence. As follows: [

[0152] ' <meta name＝"description"content＝"Gestion de Telefonía e Internet"> ',

[0153] ' <meta name＝"generator"content＝"Web2py Web Framework"> ',

[0154] ' <script src＝" gestion static js web2py.js"type＝"text javascript">',

[0155] '<link href＝" / Gestion / static / css / web2py.css"rel＝"stylesheet"type＝"text / css" / >',

[0156] '<link href＝" / Gestion / static / css / bootstrap.min.css"rel＝"stylesheet"type＝"text / css" / >',

[0157] '<link href＝" / Gestion / static / css / bootstrap-responsive.min.css"rel＝"stylesheet"type＝"text / css" / >',

[0158] '<link href＝" / Gestion / static / css / web2py_bootstrap.css"rel＝"stylesheet"type＝"text / css" / >',

[0159] '<link href＝" / Gestion / static / css / DT_bootstrap.css"rel＝"stylesheet"type＝"text / css" / >',

[0160] '<link href＝" / Gestion / static / css / dataTables.tableTools.min.css"rel＝"stylesheet"type＝"text / css" / >',

[0161] '<link href＝" / Gestion / static / css / web2py_bootstrap_nojs.css"rel＝"stylesheet"type＝"text / css" / >',

[0162] '<li class＝"web2py-menu-first">',

[0163] '<div id＝"web2py_user_form">',

[0164] '<a href＝"http: / / www.web2py.com / ">',

[0165] '<script src＝" / Gestion / static / js / web2py_bootstrap.js">',......]

[0168] 3、计算该条数据HTML标签的TF-IDF值，获取前N个标签(如前5个)。如详细阐述中处理：先提取TF-IDF值较高的前5个标签，再检查未提取的标签数据是否包含产品关键字，如果包含，则将前5个标签数据中不包含产品关键字且TF-IDF值最低的标签数据替换为上述标签数据。得到如下5个标签数据：[

[0170] '<meta name＝"generator"content＝"web2py web framework" / >',

[0171] '<script src＝" / gestion / static / js / web2py.js"type＝"text / javascript">',

[0172] '<link href＝" / gestion / static / css / web2py.css"rel＝"stylesheet"type＝"text / css" / >',

[0173] '<link href＝" / gestion / static / css / web2py_bootstrap.css"rel＝"stylesheet"type＝"text / css" / >',

[0174] '<link href＝" / gestion / static / css / web2py_bootstrap_nojs.css"rel＝"stylesheet"type＝"text / css" / >']

[0176] 4、根据提取的前N个(5个)标签生成规则。如详细阐述中处理：上述5个标签中第一个标签就包含产品关键字"web2py”，所以该标签本身可识别产品，可作为一条指纹规则，而其他标签也包含产品关键字，也可生成一条指纹规则。故该产品可提取5条指纹规则。根据第一条标签数据生成一条正则表达式如下：

[0177] <meta name＝\\\"generator\\\"content＝\\\"web2py web framework\\\"\\ / >

[0178] 5、根据厂商库比对响应数据，提取厂商(vendor)，再根据下载数据时设置的产品类型，获取产品类型(type)，补全产品信息三元组(vendor,product,type)。这里未提取到厂商(vendor)。将正则表达式与产品三元组信息(vendor,product,type)进行关联，形成指纹规则。如下，banner_clRegexp字段为第一条指纹规则。也可将5条规则用"or”合并为一条规则。

[0179]

[0180] 本申请实施例自动生成指纹规则，节省大量手动编写指纹规则的人力和时间，快速扩充现有的指纹规则库；生成的正则表达式规则更加趋近于手动编写的规则，具备一定的泛化能力，更有利于识别目标；获取的HTML标签指纹规则，一定程度上，比单从文本提取的规则更加准确，可过滤掉一些纯描述的Web网站，大大提升识别的准确性。

[0181] 本申请实施例在于对响应数据提取HTML标签并自动生成指纹规则，通过计算HTML标签的TF-IDF值，有效提取与产品相关的标签，并通过产品关键字比对，大大提高了生成规则的准确性，有助于提高了规则的质量。提取的HTML标签生成正则时，将纯数字等替换为正则匹配，与原始的字符串相比，使规则具备了一定的泛化能力，有助于提高规则的识别能力。通过计算TF-IDF值、提取标签、再生成正则等一系列逼近人工分析的自动化过程，有效提升了规则开发工作的效率。

[0182] 本申请实施例提供的一种基于HTML标签自动生成指纹规则的方法是，根据资产搜索引擎识别的产品标签(product、厂商vendor、产品类型type)和目标返回的响应数据，对响应数据进行预处理，提取HTML标签数据；再计算所有标签数据的TF-IDF值，提取前N个标签数据；最后将前N个标签数据转换为正则表达式，将生成的正则表达式与产品三元组信息(vendor,product,type)进行关联，形成指纹规则，并加入规则库中。

[0183] 需要说明的是，本实施例中各可实施的方式可以单独实施，也可以在不冲突的情况下以任意组合方式结合实施本申请不做限定。

[0184] 本申请另一实施例提供一种指纹规则的生成装置，用于执行上述实施例提供的指纹规则的生成方法。

[0185] 如图3所示，为本申请实施例提供的指纹规则的生成装置的结构示意图。该指纹规则的生成装置包括获取模块301、第一确定模块302、第二确定模块303、第三确定模块304、第四确定模块305和生成模块306，其中：

[0186] 获取模块301用于获取网络响应数据和产品标签；

[0187] 第一确定模块302用于根据网络响应数据，确定与网络响应数据对应的标签数据；

[0188] 第二确定模块303用于根据每一个标签数据，确定标签数据的重要性参数信息；

[0189] 第三确定模块304用于根据产品标签，确定标签数据中的关键信息；

[0190] 第四确定模块305用于根据关键信息和重要性参数信息，确定标签数据中的目标指纹标签数据；

[0191] 生成模块306用于根据目标指纹标签数据和产品标签，生成与网络响应数据对应的指纹规则，并将指纹规则存储在指纹库中。

[0192] 关于本实施例中的装置，其中各个模块执行操作的具体方式已经在有关该方法的实施例中进行了详细描述，此处将不做详细阐述说明。

[0193] 本申请的一些实施例通过计算HTML标签的重要性参数信息，有效提取与产品相关的标签，并通过产品关键信息比对，大大提高了生成规则的准确性，有助于提高了规则的质量。

[0194] 本申请又一实施例对上述实施例提供的指纹规则的生成装置做进一步补充说明。

[0195] 可选地，第二确定模块，用于：

[0196] 采用TF-IDF算法，分别计算每一个标签数据的TF-IDF值。

[0197] 本申请的一些实施例通过采用TF-IDF算法，分别计算每一个标签数据的TF-IDF值，用于选择TF-IDF值高的标签数据。

[0198] 可选地，第四确定模块用于：

[0199] 根据TF-IDF值的大小，获取前N个TF-IDF值对应的第一标签数据；

[0200] 判断第一标签数据是否存在关键信息；

[0201] 在剩下的标签数据中，获取包含关键信息的第二标签数据；

[0202] 在第一标签数据中不存在关键信息的情况下，将第二标签数据替换第一标签数据；

[0203] 将替换后的第一标签数据，确定为目标指纹标签数据。

[0204] 本申请的一些实施例将前N个标签数据中不包含产品关键信息且TF-IDF值最低的标签数据替换为第二标签数据，这样，得到的标签数据为可能是包含产品关键字的且TF-IDF值较高的标签数据，提高标签数据识别的准确性。

[0205] 可选地，生成模块用于：

[0206] 根据目标指纹标签数据，确定与目标指纹标签数据对应的正则表达式；

[0207] 根据与目标指纹标签数据对应的正则表达式和产品标签，生成与网络响应数据对应的指纹规则，其中，产品标签至少包括厂商名称、产品名称和产品类型。

[0208] 本申请的一些实施例，根据与目标指纹标签数据对应的正则表达式和产品标签，生成与网络响应数据对应的指纹规则，使得查询过程更加精准，并且提高得到查询网页的效率，减少查询过程中产生的误差。

[0209] 可选地，生成模块用于：

[0210] 若目标指纹标签数据中存在多条包含关键信息的标签数据，根据每条标签数据，分别生成为与标签数据对应的指纹规则，且各个指纹规则之间为或的关系。

[0211] 本申请的一些实施例若目标指纹标签数据存在多条包含产品关键字的标签数据，则每条标签数据分别生成为一条指纹规则，规则之间为或的关系。

[0212] 可选地，生成模块用于：

[0213] 若第一标签数据中都不存在关键信息，将第一标签数据进行依次拼接，得到拼接后的标签数据；

[0214] 根据拼接后的标签数据，生成对应的正则表达式。

[0215] 本申请的一些实施例如果第一标签数据都不包含产品关键字，那么可将提取的第一标签数据拼接为正则表达式，作为指纹规则，此时多个标签之间的特定顺序结构，即可作为指纹特征。

[0216] 可选地，生成模块用于：

[0217] 若第一标签数据中不存在关键信息，将第一标签数据进行删除。

[0218] 本申请的一些实施例如果前N个标签数据中存在某一个或多个包含产品关键字，则剩余不包含产品关键字的标签数据丢弃，减少内存空间的占用。

[0219] 关于本实施例中的装置，其中各个模块执行操作的具体方式已经在有关该方法的实施例中进行了详细描述，此处将不做详细阐述说明。

[0220] 需要说明的是，本实施例中各可实施的方式可以单独实施，也可以在不冲突的情况下以任意组合方式结合实施本申请不做限定。

[0221] 本申请实施例还提供了一种计算机可读存储介质，其上存储有计算机程序，程序被处理器执行时可实现如上述实施例提供的指纹规则的生成方法中的任意实施例所对应方法的操作。

[0222] 本申请实施例还提供了一种计算机程序产品，的计算机程序产品包括计算机程序，其中，的计算机程序被处理器执行时可实现如上述实施例提供的指纹规则的生成方法中的任意实施例所对应方法的操作。

[0223] 如图4所示，本申请的一些实施例提供一种电子设备400，该电子设备400包括：存储器410、处理器420以及存储在存储器410上并可在处理器420上运行的计算机程序，其中，处理器420通过总线430从存储器410读取程序并执行程序时可实现如上述指纹规则的生成方法包括的任意实施例的方法。

[0224] 处理器420可以处理数字信号，可以包括各种计算结构。例如复杂指令集计算机结构、结构精简指令集计算机结构或者一种实行多种指令集组合的结构。在一些示例中，处理器420可以是微处理器。

[0225] 存储器410可以用于存储由处理器420执行的指令或指令执行过程中相关的数据。这些指令和 / 或数据可以包括代码，用于实现本申请实施例描述的一个或多个模块的一些功能或者全部功能。本公开实施例的处理器420可以用于执行存储器410中的指令以实现上述所示的方法。存储器410包括动态随机存取存储器、静态随机存取存储器、闪存、光存储器或其它本领域技术人员所熟知的存储器。

[0226] 以上仅为本申请的实施例而已，并不用于限制本申请的保护范围，对于本领域的技术人员来说，本申请可以有各种更改和变化。凡在本申请的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本申请的保护范围之内。应注意到：相似的标号和字母在下面的附图中表示类似项，因此，一旦某一项在一个附图中被定义，则在随后的附图中不需要对其进行进一步定义和解释。

[0227] 以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应所述以权利要求的保护范围为准。

[0228] 需要说明的是，在本文中，诸如第一和第二等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来，而不一定要求或者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。而且，术语"包括”、"包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者设备所固有的要素。在没有更多限制的情况下，由语句"包括一个……”限定的要素，并不排除在包括所述要素的过程、方法、物品或者设备中还存在另外的相同要素。< / script>

Claims

1. A method for generating fingerprint rules, characterized in that: The method comprises: Get network response data and product labels; Determining, according to the network response data, label data corresponding to the network response data; Determine importance parameter information of each tag data according to the tag data; According to the product label, determining key information in the label data; Determining target fingerprint label data in the label data according to the key information and the importance parameter information; A fingerprint rule corresponding to the network response data is generated according to the target fingerprint tag data and the product tag, and the fingerprint rule is stored in a fingerprint library.

2. The method for generating fingerprint rules according to claim 1, characterized in that: The step of determining the importance parameter information of the tag data according to each of the tag data includes: The TF-IDF algorithm is used to calculate the TF-IDF value of each tag data.

3. The method for generating fingerprint rules according to claim 2, characterized in that: The step of determining the target fingerprint label data in the label data according to the key information and the importance parameter information includes: According to the size of the TF-IDF value, obtain the first label data corresponding to the first N TF-IDF values; Determining whether the first tag data contains key information; From the remaining label data, obtain the second label data containing the key information; When the key information does not exist in the first label data, replacing the first label data with the second label data; The replaced first label data is determined as the target fingerprint label data.

4. The method for generating fingerprint rules according to claim 1, characterized in that: The step of generating a fingerprint rule corresponding to the network response data according to the target fingerprint tag data and the product tag includes: Determine, according to the target fingerprint tag data, a regular expression corresponding to the target fingerprint tag data; A fingerprint rule corresponding to the network response data is generated according to a regular expression corresponding to the target fingerprint tag data and the product tag, wherein the product tag includes at least a manufacturer name, a product name, and a product type.

5. The method for generating fingerprint rules according to claim 4, characterized in that: The step of determining a regular expression corresponding to the target fingerprint label data according to the target fingerprint label data includes: If there are multiple pieces of label data containing the key information in the target fingerprint label data, fingerprint rules corresponding to the label data are generated according to each piece of label data, and the fingerprint rules are in an OR relationship.

6. The method for generating fingerprint rules according to claim 4, characterized in that: The step of determining a regular expression corresponding to the target fingerprint label data according to the target fingerprint label data includes: If the key information does not exist in any of the first label data, the first label data are sequentially spliced ​​to obtain spliced ​​label data; Generate a corresponding regular expression according to the concatenated label data.

7. The method for generating fingerprint rules according to claim 4, characterized in that: The method further comprises: If the key information does not exist in the first tag data, the first tag data is deleted.

8. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the method for generating fingerprint rules according to any one of claims 1 to 7 can be implemented when the processor executes the program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the program is executed by a processor, the method for generating fingerprint rules according to any one of claims 1 to 7 can be implemented.

10. A computer program product, comprising a computer program, wherein: When the computer program is executed by a processor, the method for generating fingerprint rules described in any one of claims 1 to 7 can be implemented.

Images