A method and device for generating a webpage fingerprint rule based on a resource search engine

Abstract

Embodiments of the present application provide a method and device for generating a webpage fingerprint rule based on a resource search engine, electronic equipment and a storage medium, wherein the method comprises: obtaining asset attribute information and response data of the resource search engine; analyzing the response data to obtain a keyword information sequence; supplementing the asset attribute information according to the keyword information sequence to obtain an asset attribute set; associating the keyword information sequence with the asset attribute set to obtain a fingerprint rule set; and storing the fingerprint rule set in a fingerprint database. By implementing the embodiments of the present application, the range of the fingerprint database can be expanded, the fingerprint database does not need to be added manually, the identification range of the fingerprint database is improved, and the human resources and time cost are saved.

Description

Technical Field

[0001] This application relates to the field of data analysis technology, and more specifically, to a method, apparatus, electronic device, and computer-readable storage medium for generating webpage fingerprint rules based on a resource search engine. Background Technology

[0002] Existing fingerprint recognition tools primarily rely on manually adding fingerprints, which typically requires expanding the coverage of fingerprint recognition. For example, whatweb identifies more web applications by writing more plugins; FOFA, on the other hand, identifies fingerprints by adding rules.

[0003] It is evident that current technologies enrich fingerprint databases manually, which requires a significant amount of time and manpower. Furthermore, many existing fingerprint recognition tools have their own independent fingerprint databases, some of which are not publicly accessible. This presents a considerable challenge to these tools, as each database requires the input of more fingerprint rules to expand its recognition range. Summary of the Invention

[0004] The purpose of this application is to provide a method, apparatus, electronic device, and computer-readable storage medium for generating webpage fingerprint rules based on a resource search engine. This can expand the scope of the fingerprint database, eliminate the need for manual addition of fingerprint databases, improve the recognition range of the fingerprint database, and save human resources and time costs.

[0005] In a first aspect, embodiments of this application provide a method for generating webpage fingerprint rules based on a resource search engine, the method comprising:

[0006] Obtain asset attribute information and response data from the resource search engine;

[0007] The response data is parsed to obtain a sequence of keyword information;

[0008] The asset attribute information is supplemented based on the keyword information sequence to obtain an asset attribute set;

[0009] The keyword information sequence is associated with the asset attribute set to obtain a fingerprint rule set;

[0010] The fingerprint rule set is stored in the fingerprint database.

[0011] In the above implementation process, keyword information sequences are obtained by parsing the response data, web pages are queried based on the keyword information sequences and asset attribute sets are extracted, and the keyword information sequences are associated with the asset attribute sets to obtain fingerprint rules. This can expand the scope of the fingerprint database, eliminate the need to manually add fingerprint databases, improve the recognition range of the fingerprint database, and save human resources and time costs.

[0012] Further, the step of parsing the response data to obtain the keyword information sequence includes:

[0013] The response data is cleaned to obtain multiple keywords;

[0014] Calculate the TF-IDF values ​​of the plurality of keywords;

[0015] The keyword information sequence is obtained based on the TF-IDF value.

[0016] In the above implementation process, by cleaning the response data and deleting erroneous data, the time for obtaining the keyword information sequence is saved and the efficiency is improved.

[0017] Further, the step of supplementing the asset attribute information based on the keyword information sequence to obtain the asset attribute set includes:

[0018] Determine whether the response data contains product information;

[0019] If so, the asset attribute information is supplemented based on the product information and the keyword information sequence to obtain the asset attribute set;

[0020] If not, supplement the asset attribute information according to the keyword information sequence to obtain the asset attribute set.

[0021] In the above implementation process, the asset attribute information is supplemented based on the product information and keyword information sequence, so that the asset attribute information contains more information and is more accurate, which facilitates the subsequent acquisition of fingerprint rules.

[0022] Further, the step of associating the keyword information sequence with the asset attribute set to obtain the fingerprint rule set includes:

[0023] Determine whether the response data contains product information;

[0024] If so, the fingerprint rule set is obtained based on the product information, the keyword information sequence, and the asset attribute set;

[0025] If not, the fingerprint rule set is obtained based on the keyword information sequence and the asset attribute set.

[0026] In the above implementation process, if product information can be identified, a fingerprint rule set is obtained based on product information, keyword information sequence, and asset attribute set, so that the fingerprint rule set contains more information and the obtained fingerprint rules are more accurate.

[0027] Further, the step of obtaining the fingerprint rule set based on the product information, the keyword information sequence, and the asset attribute set includes:

[0028] A query is performed by combining the product information and the keyword information sequence to obtain the query webpage;

[0029] The product information is compared with the query webpage, and the asset attribute set is completed based on the manufacturer information and product type information extracted from the query webpage to obtain the fingerprint rule set.

[0030] In the above implementation process, the combination of product information and keyword information sequence for querying makes the query process more accurate, improves the efficiency of obtaining the query webpage, and reduces the errors generated during the query process.

[0031] Further, the step of comparing the queried webpage with the asset attribute set to obtain the fingerprint rule set includes:

[0032] If the first asset attribute in the asset attribute set is extracted from the query webpage, a first fingerprint rule is generated.

[0033] If the first asset attribute in the asset attribute set is not extracted from the query webpage, a second fingerprint rule is generated.

[0034] The fingerprint rule set is obtained based on the first fingerprint rule and the second fingerprint rule.

[0035] In the above implementation process, a first fingerprint rule and a second fingerprint rule are generated based on the asset attributes in the asset attribute set, so that the fingerprint rule set contains more fingerprint rules and expands the scope of the fingerprint rule set.

[0036] Secondly, embodiments of this application also provide an apparatus for generating webpage fingerprint rules based on a resource search engine, the apparatus comprising:

[0037] The acquisition module is used to acquire asset attribute information and response data from the resource search engine;

[0038] The parsing module is used to parse the response data to obtain a sequence of keyword information;

[0039] The asset attribute set acquisition module is used to supplement the asset attribute information based on the keyword information sequence to obtain the asset attribute set;

[0040] The association module is used to associate the keyword information sequence with the asset attribute set to obtain a fingerprint rule set;

[0041] The storage module is used to store the fingerprint rule set into the fingerprint database.

[0042] In the above implementation process, keyword information sequences are obtained by parsing the response data, web pages are queried based on the keyword information sequences and asset attribute sets are extracted, and the keyword information sequences are associated with the asset attribute sets to obtain fingerprint rules. This can expand the scope of the fingerprint database, eliminate the need to manually add fingerprint databases, improve the recognition range of the fingerprint database, and save human resources and time costs.

[0043] Furthermore, the parsing module is also used for:

[0044] The response data is cleaned to obtain multiple keywords;

[0045] Calculate the TF-IDF values ​​of the plurality of keywords;

[0046] The keyword information sequence is obtained based on the TF-IDF value.

[0047] In the above implementation process, by cleaning the response data and deleting erroneous data, the time for obtaining the keyword information sequence is saved and the efficiency is improved.

[0048] Thirdly, an electronic device provided in this application includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method as described in any of the first aspects.

[0049] Fourthly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the method described in any of the first aspects.

[0050] Fifthly, embodiments of this application provide a computer program product that, when run on a computer, causes the computer to perform the method described in any of the first aspects.

[0051] Other features and advantages of this disclosure will be set forth in the following description, or some features and advantages may be inferred from the description or determined without doubt, or may be learned by practicing the techniques described above.

[0052] It can be implemented in accordance with the contents of the specification. The preferred embodiments of this application are described in detail below with reference to the accompanying drawings. Attached Figure Description

[0053] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 A flowchart illustrating a method for generating webpage fingerprint rules based on a resource search engine, provided in an embodiment of this application;

[0055] Figure 2 A schematic diagram illustrating the structural composition of the apparatus for generating webpage fingerprint rules based on a resource search engine, provided in an embodiment of this application;

[0056] Figure 3 This is a schematic diagram of the structural composition of the electronic device provided in the embodiments of this application. Detailed Implementation

[0057] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0058] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0059] The specific embodiments of this application will be described in further detail below with reference to the accompanying drawings and examples. The following examples are used to illustrate this application, but are not intended to limit the scope of this application.

[0060] Example 1

[0061] Figure 1 This is a flowchart illustrating the method for generating webpage fingerprint rules based on a resource search engine, as provided in an embodiment of this application. Figure 1 As shown, the method includes:

[0062] S1, obtain asset attribute information and resource search engine response data;

[0063] S2, parse the response data to obtain the keyword information sequence;

[0064] S3, supplement the asset attribute information based on the keyword information sequence to obtain the asset attribute set;

[0065] S4. Associate the keyword information sequence with the asset attribute set to obtain the fingerprint rule set;

[0066] S5 stores the fingerprint rule set into the fingerprint database.

[0067] In the above implementation process, keyword information sequences are obtained by parsing the response data, web pages are queried based on the keyword information sequences and asset attribute sets are extracted, and the keyword information sequences are associated with the asset attribute sets to obtain fingerprint rules. This can expand the scope of the fingerprint database, eliminate the need to manually add fingerprint databases, improve the recognition range of the fingerprint database, and save human resources and time costs.

[0068] Computer networks have permeated all aspects of social life, bringing with them numerous security challenges, such as frequent hacker attacks and information theft. Monitoring network nodes, promptly identifying their vulnerabilities, assessing their risk status, and implementing targeted remediation are fundamental ways to ensure network security. When conducting risk assessments, identifying key attributes such as operating system, device type, running services, and versions allows for more targeted vulnerability assessments and a more accurate evaluation of the security status. Fingerprinting aims to quickly and accurately discover relevant information about components, thereby identifying vulnerabilities in web pages. Depending on the object being identified, fingerprints are generally categorized into host system fingerprinting, port service fingerprinting, and web application fingerprinting, all based on the fundamental principle of judging based on different characteristics in the response.

[0069] Commonly used fingerprint recognition methods include:

[0070] 1) Keyword matching found in normal or error pages;

[0071] 2) Keyword matching in the request header information;

[0072] 3) Keyword matching contained in some Uniform Resource Locators (URLs);

[0073] 4) Matching the MD5 value of a specific file.

[0074] Some common web application fingerprinting tools include whatweb, wapplyzer, fofa, and shodan.

[0075] WhatWeb is an open-source website fingerprinting software that can identify website information including Content Management System (CMS) types, blog platforms, middleware, web framework modules, web servers, JavaScript libraries, and many other web applications.

[0076] Wappalyzer's function is to identify the fingerprint of a single URL. Its principle is to send an HTTP request to a specified URI, obtain the response header and response body, and match them according to fingerprint rules.

[0077] FOFA is a cyberspace asset search engine that periodically and continuously performs in-depth scanning and detection of exposed assets on the global internet. It uses various methods to search for assets, comprehensively discovering exposed internet assets and managing their profiles. It helps users quickly match cyber assets, such as analyzing the scope of vulnerability impact, statistically analyzing application distribution, and ranking applications by popularity.

[0078] Shodan is a search engine used to help discover vulnerabilities in major internet systems, including routers, switches, and industrial control systems. Search results provide information such as: geographic location, hostname, operating system, service name, application name, and application version information.

[0079] Typical applications contain distinctive codes in their HTML, JS, and CSS files. For example, if WordPress isn't intentionally hidden, it will include something like "wp-admin" in its robots.txt file, `generator=wordpress 3.xx` followed by the version number in its index.php file, and the page will contain paths to `wp-content`, etc. These are all sequences of keyword information within the webpage. Similarly, Apache will have the keyword "Apache" appear in the Server field of the header, or... <title> It appeared in the middle.< / title>

[0080] This application provides a method for generating webpage fingerprint rules. Based on the response data returned by the webpage application and target identified by the asset search engine, the method extracts the keyword information sequence from the response data, completes the manufacturer information and product type information based on the keyword information sequence, forms an asset attribute set, and generates fingerprint rules.

[0081] In S1, data is downloaded from a resource search engine or scanned from other systems; this data is the response data.

[0082] Furthermore, S2 includes:

[0083] The response data was cleaned to obtain multiple keywords;

[0084] Calculate the commonly used weighted (Term Frequency–Inverse Document Frequency, TF-IDF) value for information retrieval and data mining using multiple keywords;

[0085] Obtain the keyword information sequence based on the TF-IDF value.

[0086] In the above implementation process, by cleaning the response data and deleting erroneous data, the time for obtaining the keyword information sequence is saved and the efficiency is improved.

[0087] Cleaning response data includes removing erroneous response data, error codes, and redirection codes; for File Transfer Protocol (FTP) response data, removing response packets containing common keywords such as "filezilla" and "service-u"; for Telnet response data, removing character sequences with a specific code (IAC0xFF) used to negotiate communication between different operating systems; and for Hypertext Transfer Protocol (HTTP) response data, removing hyperlinks, field names, timestamps, script blocks, and symbols (such as...). And <\p> etc.; remove dictionary words from the response data (the name models of manufacturers and products are usually non-dictionary words).

[0088] The TF-IDF value is calculated using the Neuro-Linguistic Programming (NLP) toolkit. If the TF-IDF score is high, the keyword is considered to be more relevant to the webpage.

[0089] Extract the top 5 keywords (e.g., the first 5) with the highest TF-IDF values ​​from among multiple keywords and use them to form the keyword information sequence of the response data.

[0090] Furthermore, S3 includes:

[0091] Determine whether the response data contains product information;

[0092] If so, supplement the asset attribute information based on the product information and keyword information sequence to obtain the asset attribute set;

[0093] If not, supplement the asset attribute information based on the keyword information sequence to obtain the asset attribute set.

[0094] In the above implementation process, the asset attribute information is supplemented based on the product information and keyword information sequence, so that the asset attribute information contains more information and is more accurate, which facilitates the subsequent acquisition of fingerprint rules.

[0095] The asset search engine identifies product information and keyword information sequences to form a query sequence. Based on the query sequence, the asset search engine queries product-related web pages through its application programming interface (API). Combining the manufacturer information database and product type information database, it effectively extracts asset attribute information such as product manufacturer information and product type information, supplements the asset attribute information, and forms an asset attribute set.

[0096] To improve the accuracy of extracting manufacturer and product type information when searching relevant web pages based on the query sequence, it is necessary to compile commonly used manufacturer and product type information databases. The manufacturer database includes the manufacturer's common name (Chinese and English), full Chinese name, and full English name. Commonly used manufacturers include: NSFOCUS, Topsec, Venustech, Iridium, and NetSec. The product type information database includes the product type's common name (Chinese and English), full Chinese name, and full English name, and is similar to the manufacturer database.

[0097] Furthermore, S4 includes:

[0098] Determine whether the response data contains product information;

[0099] If so, obtain the fingerprint rule set based on product information, keyword information sequence, and asset attribute set;

[0100] If not, obtain the fingerprint rule set based on the keyword information sequence and the asset attribute set.

[0101] In the above implementation process, if product information is present in the response data, a fingerprint rule set is obtained based on the product information, keyword information sequence, and asset attribute set, so that the fingerprint rule set contains more information and the obtained fingerprint rules are more accurate.

[0102] Furthermore, the steps for obtaining the fingerprint rule set based on product information, keyword information sequences, and asset attribute sets include:

[0103] The query webpage is obtained by combining product information and keyword information sequences.

[0104] The product information is compared with the query webpage, and the asset attribute set is completed based on the manufacturer information and product type information extracted from the query webpage, resulting in a fingerprint rule set.

[0105] In the above implementation process, the combination of product information and keyword information sequence for querying makes the query process more accurate, improves the efficiency of obtaining the query webpage, and reduces the errors generated during the query process.

[0106] Further, the step of comparing the queried webpage with the asset attribute set to obtain the fingerprint rule set includes:

[0107] If the first asset attribute in the asset attribute set is extracted from the query webpage, the first fingerprint rule is generated;

[0108] If the first asset attribute in the asset attribute set cannot be extracted from the query webpage, generate a second fingerprint rule;

[0109] A fingerprint rule set is obtained based on the first fingerprint rule and the second fingerprint rule.

[0110] In the above implementation process, a first fingerprint rule and a second fingerprint rule are generated based on the asset attributes in the asset attribute set, so that the fingerprint rule set contains more fingerprint rules and expands the scope of the fingerprint rule set.

[0111] If the response data contains product information, combining the product information and keyword information sequences identified by the asset search engine can quickly and effectively retrieve product-related manufacturer and product type information. If the retrieved webpage can extract the first asset attribute, such as specific manufacturer and product type information, then a first fingerprint rule can be generated: keywords->(product, type, vendor); if the first asset attribute, such as specific manufacturer and product type information, cannot be extracted, then a second fingerprint rule can be generated: keywords->(product, ,); optionally, if only manufacturer or product type information can be retrieved, then the rule can be generated: keywords->(product, type,) or keywords->(product, , vendor).

[0112] If the response data does not contain product information, a search will be performed directly based on the keyword sequence extracted from the response data. If the product name can be extracted from the retrieved webpage, a corresponding fingerprint rule can be generated based on whether manufacturer information and product type information can be extracted; if the product name cannot be extracted from the retrieved webpage, no fingerprint rule will be generated. Since identifying a product name solely from keywords extracted from the query page is generally inaccurate, confirmation based on manufacturer and product type information on the page is necessary. Therefore, in this case, a keywords->(product,,) rule generally will not occur.

[0113] The system downloads response data from an asset search engine, extracts keyword information sequences from the response data, and then uses these keyword information sequences and product information to search and complete product manufacturer and product type information, thereby obtaining an asset attribute set. This forms fingerprint rules between the product information, keyword information sequences, and the asset attribute set. Fingerprint rules are generated quickly and automatically, expanding the existing fingerprint rule library.

[0114] Example 2

[0115] To execute the method corresponding to Embodiment 1 above and achieve the corresponding functions and technical effects, an apparatus for generating webpage fingerprint rules based on a resource search engine is provided below, such as... Figure 2 As shown, the device includes:

[0116] Module 1 is used to acquire asset attribute information and response data from the resource search engine;

[0117] Parsing module 2 is used to parse the response data to obtain a sequence of keyword information;

[0118] The asset attribute set acquisition module 3 is used to supplement the asset attribute information based on the keyword information sequence to obtain the asset attribute set;

[0119] Module 4 is used to associate the keyword information sequence with the asset attribute set to obtain the fingerprint rule set;

[0120] Module 5 is used to store the fingerprint rule set into the fingerprint database.

[0121] In the above implementation process, keyword information sequences are obtained by parsing the response data. The keyword information sequences are actively associated with the asset attribute set to obtain fingerprint rules. This can expand the scope of the fingerprint database, eliminate the need to manually add fingerprints, improve the recognition range of the fingerprint database, and save human resources and time costs.

[0122] Furthermore, parsing module 2 is also used for:

[0123] The response data was cleaned to obtain multiple keywords;

[0124] Calculate the TF-IDF values ​​of multiple keywords;

[0125] Obtain the keyword information sequence based on the TF-IDF value.

[0126] In the above implementation process, by cleaning the response data and deleting erroneous data, the time for obtaining the keyword information sequence is saved and the efficiency is improved.

[0127] Furthermore, the asset attribute set acquisition module 3 is also used for:

[0128] Determine whether the response data contains product information;

[0129] If so, supplement the asset attribute information based on the product information and keyword information sequence to obtain the asset attribute set;

[0130] If not, supplement the asset attribute information based on the keyword information sequence to obtain the asset attribute set.

[0131] In the above implementation process, the asset attribute information is supplemented based on the product information and keyword information sequence, so that the asset attribute information contains more information and is more accurate, which facilitates the subsequent acquisition of fingerprint rules.

[0132] Furthermore, the association module 4 is also used for:

[0133] Determine whether the response data contains product information;

[0134] If so, obtain the fingerprint rule set based on product information, keyword information sequence, and asset attribute set;

[0135] If not, obtain the fingerprint rule set based on the keyword information sequence and the asset attribute set.

[0136] In the above implementation process, if product information can be identified, a fingerprint rule set is obtained based on product information, keyword information sequence, and asset attribute set, so that the fingerprint rule set contains more information and the obtained fingerprint rules are more accurate.

[0137] Furthermore, the association module 4 is also used for:

[0138] The query webpage is obtained by combining product information and keyword information sequences.

[0139] The product information is compared with the query webpage, and the asset attribute set is completed based on the manufacturer information and product type information extracted from the query webpage, resulting in a fingerprint rule set.

[0140] In the above implementation process, the combination of product information and keyword information sequence for querying makes the query process more accurate, improves the efficiency of obtaining the query webpage, and reduces the errors generated during the query process.

[0141] Furthermore, the association module 4 is also used for:

[0142] If the first asset attribute in the asset attribute set is extracted from the query webpage, the first fingerprint rule is generated;

[0143] If the first asset attribute in the asset attribute set cannot be extracted from the query webpage, generate a second fingerprint rule;

[0144] A fingerprint rule set is obtained based on the first fingerprint rule and the second fingerprint rule.

[0145] In the above implementation process, a first fingerprint rule and a second fingerprint rule are generated based on the asset attributes in the asset attribute set, so that the fingerprint rule set contains more fingerprint rules and expands the scope of the fingerprint rule set.

[0146] The apparatus described above for generating webpage fingerprint rules based on a resource search engine can implement the method of Embodiment 1. The options in Embodiment 1 also apply to this embodiment, and will not be detailed here.

[0147] The remaining contents of this embodiment can be referred to the contents of Embodiment 1 above, and will not be repeated in this embodiment.

[0148] Example 3

[0149] This application provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor runs the computer program to enable the electronic device to execute the method of generating webpage fingerprint rules based on a resource search engine, as described in Embodiment 1.

[0150] Alternatively, the aforementioned electronic device may be a server.

[0151] Please see Figure 3 , Figure 3 This is a schematic diagram illustrating the structural composition of an electronic device provided in an embodiment of this application. The electronic device may include a processor 31, a communication interface 32, a memory 33, and at least one communication bus 34. The communication bus 34 is used to enable direct communication between these components. In this embodiment, the communication interface 32 is used for signaling or data communication with other node devices. The processor 31 may be an integrated circuit chip with signal processing capabilities.

[0152] The processor 31 described above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor, or the processor 31 can be any conventional processor.

[0153] The memory 33 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc. The memory 33 stores computer-readable instructions. When these computer-readable instructions are executed by the processor 31, the device can perform the aforementioned operations. Figure 1 The various steps involved in the method implementation examples.

[0154] Optionally, the electronic device may also include a storage controller and an input / output unit. The memory 33, storage controller, processor 31, peripheral interface, and input / output unit are electrically connected directly or indirectly to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses 34. The processor 31 is used to execute executable modules stored in the memory 33, such as software function modules or computer programs included in the device.

[0155] Input / output units are used to enable users to create tasks and set optional start periods or preset execution times for those tasks, facilitating user-server interaction. Input / output units can be, but are not limited to, a mouse and keyboard.

[0156] Understandable. Figure 3 The structure shown is for illustrative purposes only; the electronic device may also include components that are more advanced than those shown. Figure 3 The more or fewer components shown, or having the same Figure 3 The different configurations shown. Figure 3 The components shown can be implemented using hardware, software, or a combination thereof.

[0157] In addition, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method for generating webpage fingerprint rules based on a resource search engine as described in Embodiment 1.

[0158] This application also provides a computer program product that, when run on a computer, causes the computer to perform the method described in the method embodiment.

[0159] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using dedicated hardware-based apparatus that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0160] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0161] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, external hard drives, ROM, RAM, magnetic disks, or optical disks.

[0162] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0163] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0164] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for generating webpage fingerprint rules based on a resource search engine, characterized in that, The method includes: Obtain asset attribute information and response data from the resource search engine; The response data is parsed to obtain a sequence of keyword information; The asset attribute information is supplemented based on the keyword information sequence to obtain an asset attribute set; The keyword information sequence is associated with the asset attribute set to obtain a fingerprint rule set; The fingerprint rule set is stored in the fingerprint database; The step of parsing the response data to obtain the keyword information sequence includes: The response data is cleaned to obtain multiple keywords; Calculate the TF-IDF values ​​of the plurality of keywords; The keyword information sequence is obtained based on the TF-IDF value; The step of supplementing the asset attribute information based on the keyword information sequence to obtain the asset attribute set includes: Determine whether the response data contains product information; If so, the asset attribute information is supplemented based on the product information and the keyword information sequence to obtain the asset attribute set; If not, supplement the asset attribute information according to the keyword information sequence to obtain the asset attribute set; The step of associating the keyword information sequence with the asset attribute set to obtain the fingerprint rule set includes: Determine whether the response data contains product information; If so, the fingerprint rule set is obtained based on the product information, the keyword information sequence, and the asset attribute set; If not, obtain the fingerprint rule set based on the keyword information sequence and the asset attribute set; The step of obtaining the fingerprint rule set based on the product information, the keyword information sequence, and the asset attribute set includes: A query is performed by combining the product information and the keyword information sequence to obtain the query webpage; The product information is compared with the query webpage, and the asset attribute set is completed based on the manufacturer information and product type information extracted from the query webpage to obtain the fingerprint rule set. The step of comparing the queried webpage with the asset attribute set to obtain the fingerprint rule set includes: If the first asset attribute in the asset attribute set is extracted from the query webpage, a first fingerprint rule is generated. If the first asset attribute in the asset attribute set is not extracted from the query webpage, a second fingerprint rule is generated. The fingerprint rule set is obtained based on the first fingerprint rule and the second fingerprint rule.

2. A device for generating webpage fingerprint rules based on a resource search engine, characterized in that, The device includes: The acquisition module is used to acquire asset attribute information and response data from the resource search engine; The parsing module is used to parse the response data to obtain a sequence of keyword information; The asset attribute set acquisition module is used to supplement the asset attribute information based on the keyword information sequence to obtain the asset attribute set; The association module is used to associate the keyword information sequence with the asset attribute set to obtain a fingerprint rule set; The storage module is used to store the fingerprint rule set into the fingerprint database; The parsing module is also used for: The response data is cleaned to obtain multiple keywords; Calculate the TF-IDF values ​​of the plurality of keywords; The keyword information sequence is obtained based on the TF-IDF value; The asset attribute set acquisition module is also used for: Determine whether the response data contains product information; If so, the asset attribute information is supplemented based on the product information and the keyword information sequence to obtain the asset attribute set; If not, supplement the asset attribute information according to the keyword information sequence to obtain the asset attribute set; The association module is also used for: Determine whether the response data contains product information; If so, the fingerprint rule set is obtained based on the product information, the keyword information sequence, and the asset attribute set; If not, obtain the fingerprint rule set based on the keyword information sequence and the asset attribute set; A query is performed by combining the product information and the keyword information sequence to obtain the query webpage; The product information is compared with the query webpage, and the asset attribute set is completed based on the manufacturer information and product type information extracted from the query webpage to obtain the fingerprint rule set. If the first asset attribute in the asset attribute set is extracted from the query webpage, a first fingerprint rule is generated. If the first asset attribute in the asset attribute set is not extracted from the query webpage, a second fingerprint rule is generated. The fingerprint rule set is obtained based on the first fingerprint rule and the second fingerprint rule.

3. An electronic device, characterized in that, The device includes a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to cause the electronic device to perform the method for generating webpage fingerprint rules based on a resource search engine according to claim 1.

4. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method for generating webpage fingerprint rules based on a resource search engine as described in claim 1.

Images