Website application fingerprint computing system and method based on home page code structure

Abstract

The invention belongs to the technical field of website identification, and particularly relates to a website application fingerprint calculation system and method based on a home page code structure, and the method comprises the steps: initiating an access request for a home page of a target website, and obtaining the home page response content of the target website; analyzing the home page response content, and extracting information of the website, including page structure information, page hyperlink information and page rendering information; calculating an accurate fingerprint, a loose fingerprint and a rough fingerprint of the target website according to a preset field and a sorting rule based on the extracted information; and forming a fingerprint set containing the three types of fingerprints. According to the website application fingerprint calculation method and system based on home page response content analysis, only one network request is needed, and the efficiency, stability and applicability of fingerprint generation are improved by generating accurate, loose and rough fingerprints.

Description

Technical Field

[0001] The present invention belongs to the technical field of website identification, and in particular relates to a website application fingerprint calculation system and method based on a homepage code structure. Background Art

[0002] Network asset mapping is an important network security activity that aims to help organizations gain a comprehensive understanding of all assets and resources existing in their networks, including hardware devices, software applications, services, configurations, vulnerabilities, etc. Network asset mapping helps organizations identify and manage their network environment to better protect their network assets and data.

[0003] In network asset mapping, "Web Application Fingerprint" is a technology that is often used to identify a specific website application or a category of website applications through fingerprints. Using "Web Application Fingerprint" to identify the relevance of website applications has also become an important technology in network asset mapping.

[0004] In the context of red-blue confrontation or vulnerability mining, "website application fingerprint" plays an important role. For example, when a website application vulnerability is discovered, the scope of the vulnerability is determined by application fingerprint identification. In other words, we can use application fingerprint technology to determine how many websites are affected by this vulnerability. This method helps to avoid the adverse effects that the use of vulnerability verification (PoC) or vulnerability scanning may have on the security or stability of the target website.

[0005] At present, network asset mapping products usually collect a variety of information about website objects and comprehensively calculate website fingerprints. The collected website information includes: web server type, application port, web application framework, front-end technology, back-end technology, database system, application version, third-party components and plug-ins, etc. Combining several of these information, the hash fingerprint corresponding to the website is calculated through the manufacturer's customized algorithm. The fingerprint form is often a string composed of uppercase and lowercase letters, numbers and other common characters, such as: ZmlkPSJnMkJweF34OEXoQXo1em1WMktE. This calculation method relies on too many calculation information elements and usually requires sending more network requests. When the target website makes some changes, such as blocking specific detection requests or modifications, different fingerprints will be generated. The difference in fingerprints will also directly lead to errors in other calculations based on fingerprints. Summary of the invention

[0006] Purpose of the invention: The purpose of the present invention is to address the deficiencies in the prior art and provide a website application fingerprint calculation system and method based on the homepage code structure. Only one network request needs to be initiated for the target website application. The website application fingerprint calculation method and system based on the homepage response content analysis improves the efficiency, stability and applicability of fingerprint generation by generating three types of fingerprints: precise, loose and rough.

[0007] Technical solution: The website application fingerprint calculation system based on the homepage code structure of the present invention includes: An input module, used for receiving one or more sets of website domain names; The page response module is configured to initiate a real access request to each website domain name and obtain the homepage response content; A response structure parsing module is configured to parse the home page response content and extract page structure information, page hyperlink information and page rendering information; A fingerprint calculation model configured to synthesize the information extracted by the response structure analysis module and calculate the precise fingerprint, loose fingerprint and rough fingerprint of the target website according to preset fields and sorting rules; The output module is used to combine the three fingerprints obtained by the fingerprint calculation model to form a fingerprint set of the corresponding website.

[0008] Furthermore, the fingerprint calculation module combines the fields contained in the extracted page structure information, page hyperlink information, and page rendering information, and calculates the fingerprint using a cryptographic hash function, wherein the cryptographic hash function includes MD5, SHA-1, and SHA-256.

[0009] A method for calculating website application fingerprints by a website application fingerprint calculation system based on a homepage code structure comprises the following steps: S1. Initiate an access request to the homepage of the target website and obtain the response content of the homepage of the target website; S2, parsing the homepage response content, extracting website information, including: page structure information, page hyperlink information and page rendering information; S3, based on the extracted information, according to the preset fields and sorting rules, calculate the precise fingerprint, loose fingerprint and rough fingerprint of the target website; S4. Form a fingerprint set including the three types of fingerprints.

[0010] To further improve the above technical solution, the home page response content includes the home page URL path, HTML response content, and HTTP header information; the page structure information includes the sub-tags of the head tag and the body tag in the HTML response content and their order; the page hyperlink information includes the path, parameters, and file type; the page rendering information includes the CSS class name and script function call information.

[0011] Furthermore, the step of extracting the page hyperlink information includes: Extract all src and href attribute values ​​from the response content; Identify and count paths, parameters, and file types from the extracted attribute values ​​and sort them by frequency of occurrence; Get the top 3 values ​​of each type of information including paths, parameters, and file types.

[0012] Furthermore, the step of extracting the page rendering information includes: Extract the class name field of the page CSS structure from the response content, sort it by frequency of occurrence, and select the top 3 values ​​with the highest frequency of occurrence; Extract the page script function call names from the response content, sort them by frequency of occurrence, and select the top 3 values ​​by frequency of occurrence.

[0013] Furthermore, the target website's precise fingerprint, loose fingerprint and rough fingerprint calculation process includes: combining the fields contained in the extracted page structure information, page hyperlink information, and page rendering information, and using an encrypted hash function to calculate the fingerprint; and the fields contained in the precise fingerprint, loose fingerprint and rough fingerprint decrease in sequence, but the matching scenarios covered increase in sequence.

[0014] Furthermore, the precise fingerprint is calculated based on all fields; the loose fingerprint is calculated based on the fields included in the page hyperlink information, page rendering information, and the homepage URL path; and the rough fingerprint is calculated based on the fields included in the page hyperlink information and the homepage URL path.

[0015] Furthermore, the method further comprises the following steps: Select fingerprints with different accuracy for matching according to different application scenarios; When an exact match is required, select precise fingerprint; when false positives are allowed but false negatives are not allowed, select loose fingerprint; when only large-scale features need to be identified, select rough fingerprint.

[0016] Beneficial effects: Compared with the prior art, the advantages of the present invention are: only one network request needs to be initiated for the target website application, and by obtaining the homepage response information, combined with analysis and feature extraction, a multi-level website application fingerprint is generated, which has the characteristics of high efficiency, low interference, and strong stability, and is suitable for a variety of scenarios, providing efficient and reliable technical support for network asset mapping and vulnerability assessment. The fingerprint calculation method based on the website page content has less dependence; the fingerprint focuses more on the website application itself, rather than the website; by calculating three types of fingerprints, precise, loose, and rough, to describe the website application, it is adapted to more retrieval scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a principle block diagram of the website application fingerprint calculation system based on the homepage code structure of the present invention. DETAILED DESCRIPTION

[0018] The technical solution of the present invention is described in detail below with reference to the accompanying drawings, but the protection scope of the present invention is not limited to the embodiments.

[0019] Embodiment 1: The website application fingerprint calculation method based on the homepage code structure provided by the present invention comprises the following steps: S1. Initiate an access request to the homepage of the target website and obtain the homepage response content of the target website, including: homepage URL path (homepage), response HTML content, and HTTP header information; S2. parse the homepage response content and extract the website information, including: page structure information, page hyperlink information and page rendering information. The page structure element information extracts the subtags of the head tag and body tag in the HTML response and their arrangement order. The page hyperlink information extracts the src and href attributes, and counts the path, parameter, and file type, and sorts them by frequency. The page rendering information extracts the CSS class name and script function call name, and sorts them by frequency. S3, based on the extracted information, according to the preset fields and sorting rules, calculate the precise fingerprint, loose fingerprint and rough fingerprint of the target website; S4. Form a fingerprint set including the three types of fingerprints.

[0020] Specifically, S3 calculates the fingerprint by combining the fields contained in the extracted page structure information, page hyperlink information, and page rendering information using an encrypted hash function. The precise fingerprint includes fields such as page structure elements, paths, parameters, file types, CSS class names, and script functions. The loose fingerprint is generated using some fields (such as paths, parameters, file types, CSS class names, and script functions). The rough fingerprint is generated only using path, parameters, and file type fields.

[0021] Match the generated fingerprint with the stored fingerprint database; select different types of fingerprints for retrieval according to the matching rules: when an exact match is required, select the precise fingerprint; when false positives are allowed but false negatives are not allowed, select the loose fingerprint; when only the broad features need to be identified, select the rough fingerprint.

[0022] Example 2: If Figure 1 As shown, the system for implementing the website application fingerprint calculation method based on the homepage code structure provided in Example 1 includes: An input module, used for receiving one or more sets of website domain names; The page response module is configured to initiate a real access request to each website domain name, obtain the real URL of the website homepage, homepage file information and website response content; A response structure parsing module is configured to parse the website response content and extract page structure information, page hyperlink information, and page rendering information; A fingerprint calculation model configured to synthesize the information extracted by the response structure analysis module and calculate the precise fingerprint, loose fingerprint and rough fingerprint of the target website according to preset fields and sorting rules; The output module is used to combine the three fingerprints produced by the fingerprint calculation model to form a fingerprint set of the corresponding website.

[0023] Among them: the page response module initiates a real access request to the website domain name through HTTP or HTTPS. The page structure elements extracted by the response structure analysis module include the sorting information of the neutron elements and the sorting information of the neutron elements. The page hyperlink information extracted by the response structure analysis module includes extracting the hyperlink elements in the website page, including the website path, path URL parameters, and the file type of the path URL. The elements are extracted as features, and the top 3 ranking of the frequency of occurrence of each element is taken. If the top ranking is less than 3, as many as there are, the data can also be empty. The page rendering information extracted by the response structure analysis module includes the name field of the CSS identifier and the high-frequency function call information in the script page code. The fingerprint calculation model uses MD5 hash to calculate precise fingerprints, loose fingerprints, and rough fingerprints.

[0024] The three types of fingerprints, precise, loose, and rough, are defined based on the type of selected fields. Different fingerprint types will select different fields and the order of the fields will also be different. The more precise the fingerprint, the more fields it selects, and the more the selected fields can reflect the uniqueness of the website application. The selected fields are less likely to be modified during website deployment, fine-tuning, or even secondary development.

[0025] For example, the web page field information and sorting method selected for accurate fingerprint calculation are as follows: 1. head_finger: other http tags and tag order in the http head tag in the html response content. Example values ​​are: meta, title, meta, meta, link 2. body_finger: The method is the same as the head field feature. The content is taken from the sub-tag feature of the http body tag in the html response content. Example values ​​are: div,div,frame,image,div 3. path_list: sort the root path structures of website applications counted from the main web page content by the frequency of occurrence of the web page structure, and then take the top three website paths for splicing. Example values ​​are: / web / , / web / images / , / web / images / productMenu / 4. param_list: After sorting the frequency of occurrence of the parameters extracted from the website URL counted in the main webpage content, take the top three parameters for splicing. Example values ​​are: id, productId, solutionId 5. url_file_type: Website language information collected from the website page content. Example value: jsp 6. homepage_url: The root page information of the website counted from the root path of the website. The example value is: / index.jsp 7. css_class: extract all the class names of the css structure from the page code of the main web page, sort them by frequency of occurrence, and concatenate the three values ​​with the highest frequency of occurrence. The example values ​​are: fontSize_16, fontSize_18, fontSize_20 8. script_func: Extract the function name of all scripts from the page code of the main web page, sort them by frequency of occurrence, and concatenate the three values ​​with the highest frequency of occurrence. Example values ​​are: on, parent, val.

[0026] Example 3: The method provided in Example 1 is used for a specific application, including an input and an output. The input is: a website domain name set, which can be multiple or a single website, such as: www.test.com; the output is: the fingerprint set information of the current website. It mainly includes four processes, from the data flow direction: page response, response structure analysis, fingerprint calculation model, website fingerprint set, respectively: 1. Page response: The program initiates a real access request to the computing website through HTTP or HTTPS to the port of the web application.

[0027] Get the real URL of the website homepage (such as: http: / / www.test.com / web / index / home.html), the homepage file information and the website's response content, and enter the subsequent calculations. Sample data such as: homepage: home.html 2. Response structure analysis: Structural analysis of the website's response content, extracting the website's main elements and structural information, including: (1) Page structure elements: Different website applications usually have different main arrangement orders of HTML web page elements on the homepage, which can be an important basis for distinction. Such as the sorting information of neutron elements, the sorting information of neutron elements. Example data such as: res_head: meta,meta,meta,title,link,link,link,link,link,link,link res_body: style,div,style,div,div,div (2) Page hyperlink information: Extract hyperlinks in website pages, such as src, href and other hyperlink addresses in web pages. In website pages, src and href are two commonly used attributes used to reference different types of resources.

[0028] The src attribute is often used to reference external resources, such as images, audio, video, and embedded content. For example: <img src=""image.jpg”" alt=""图片描述”"> , which will embed an image in the web page. The href attribute is usually used to specify a hyperlink, which is used to navigate to other resources or pages, usually to external or internal resources. For example: <link rel=""stylesheet”" href="" / file / upload / styles.css”"> The src attribute is usually used to load external resources, such as media files or embedded content, while the href attribute is mainly used to specify hyperlinks and reference external style sheets. Both can enrich page content and provide navigation functions.

[0029] Since the same website application usually has the same website path, path URL parameters, and path URL file type, these elements are used as features here. The website path, parameters, file type, etc. in the hyperlink address are extracted separately. In order to eliminate interference, the top ranking of the frequency of each type is taken here, such as Top3. Example data is as follows: path_list: [' / article / detail / ', ' / article / index / id / ', ' / style / images / '] param_list: ['fileid', 'newsid', 'style'] file_type: php (3) Page rendering information: Extract information such as CSS and script from the website page. CSS is used to define the style and layout of the web page, thereby determining the appearance of the web page, while scripts are used to add interactivity and dynamic functions, allowing users to interact with the web page. Together, these two create the unique richness of the page and a unique user experience.

[0030] CSS: Also known as Cascading Style Sheets, it is a style description language used to control the appearance and layout of a web page. It allows developers to specify how to display HTML elements, including style attributes such as color, font, size, margin, spacing, etc. Its main function is to separate content (HTML) and style (appearance), which helps improve the maintainability and reusability of the website. It enables developers to better control and customize the appearance of web pages to provide a better user experience. CSS is also used for responsive design to adapt to different screen sizes and device types.

[0031] Script: A programming code, usually written in JavaScript, used to add interactivity and dynamic functionality to web pages. The main purpose is to make web pages interactive, allowing users to interact with the page and perform various operations. For example, scripts can be used to validate form data, create animations, load asynchronous content, process user input, etc. JavaScript is the most commonly used scripting language for developing modern web applications.

[0032] In the present invention, css mainly extracts the name field of the page css tag, and script mainly extracts the high-frequency function call information in the page js code. Example data such as: css_class: ['content', 'link list_br', 'link title_br'] script_func: ['append', 'find', 'html'] 3. Fingerprint calculation model: By integrating the above field information: res_head, res_body, path_list, param_list, file_type, homepage, css_class, script_func. According to the accuracy of the match, three fingerprint information are calculated respectively. The calculation method is: Precise fingerprint hash_precision = md5(res_head, res_body, path_list, param_list, file_type, homepage, css_class,script_func) Loose fingerprint hash_sketchy = md5(path_list, param_list, file_type,homepage, css_class, script_func) Rough fingerprint hash_wide = md5(path_list, param_list, file_type, homepage) 4. Website fingerprint set: The three types of fingerprints produced by the comprehensive fingerprint calculation model are used to form the fingerprint information of the corresponding website. webfinger = {hash_precision, hash_sketchy, hash_wide}, sample data such as: webfinger={hash_precision:'9eb87954eb809db0450b2fc3683b5630',hash_sketchy:''a 66ffa28005b47ad6a75053f15882b05,hash_wide:'02d19212a5b481233fd62c897e06eb27'} Embodiment 4: The method provided in Embodiment 1 is applied, including: network asset mapping, searching for all websites of the same type through fingerprints; big data analysis, analyzing the proportion of website applications in the Internet; vulnerability impact assessment, when a vulnerability is found, analyzing the impact scope of the same vulnerability.

[0033] Retrieval scenarios such as: when searching for the same website deployed using the same website application developed by the same supplier through cyberspace mapping, the method of the present invention can be used to calculate the three fingerprints of each website. When an exact match is required and false positives are not allowed, websites with the same exact fingerprints can be selected. When a small number of false positives are allowed but omissions are not allowed, website applications with the same loose fingerprints are selected.

[0034] As described above, although the present invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the present invention itself. Various changes in form and details may be made without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A website application fingerprint calculation system based on the homepage code structure, characterized in that: include: An input module, used for receiving one or more sets of website domain names; The page response module is configured to initiate a real access request to each website domain name and obtain the homepage response content; A response structure parsing module is configured to parse the home page response content and extract page structure information, page hyperlink information and page rendering information; A fingerprint calculation model configured to synthesize the information extracted by the response structure analysis module and calculate the precise fingerprint, loose fingerprint and rough fingerprint of the target website according to preset fields and sorting rules; The output module is used to combine the three fingerprints obtained by the fingerprint calculation model to form a fingerprint set of the corresponding website.

2. According to claim 1, the website application fingerprint calculation system based on the homepage code structure is characterized in that: The fingerprint calculation module combines the fields contained in the extracted page structure information, page hyperlink information, and page rendering information, and calculates the fingerprint using a cryptographic hash function, wherein the cryptographic hash function includes MD5, SHA-1, and SHA-256.

3. A method for calculating website application fingerprints using the website application fingerprint calculation system based on the homepage code structure of claim 1, characterized in that: The following steps are involved: S1. Initiate an access request to the homepage of the target website and obtain the response content of the homepage of the target website; S2, parsing the homepage response content, extracting website information, including: page structure information, page hyperlink information and page rendering information; S3, based on the extracted information, according to the preset fields and sorting rules, calculate the precise fingerprint, loose fingerprint and rough fingerprint of the target website; S4. Form a fingerprint set including the three types of fingerprints.

4. The method according to claim 3, characterized in that: The home page response content includes the home page URL path, HTML response content, and HTTP header information; the page structure information includes the sub-tags of the head tag and body tag in the HTML response content and their order; the page hyperlink information includes the path, parameters, and file type; the page rendering information includes CSS class name and script function call information.

5. The method according to claim 4, characterized in that The step of extracting the page hyperlink information comprises: Extract all src and href attribute values ​​from the response content; Identify and count paths, parameters, and file types from the extracted attribute values ​​and sort them by frequency of occurrence; Get the top 3 values ​​of each type of information including paths, parameters, and file types.

6. The method according to claim 4, characterized in that The step of extracting the page rendering information includes: Extract the class name field of the page CSS structure from the response content, sort it by frequency of occurrence, and select the top 3 values ​​with the highest frequency of occurrence; Extract the page script function call names from the response content, sort them by frequency of occurrence, and select the top 3 values ​​by frequency of occurrence.

7. The method according to claim 3, characterized in that The target website's precise fingerprint, loose fingerprint and rough fingerprint calculation process includes: combining the fields contained in the extracted page structure information, page hyperlink information and page rendering information, and using an encrypted hash function to calculate the fingerprint; and the fields contained in the precise fingerprint, loose fingerprint and rough fingerprint are reduced in sequence.

8. The method according to claim 7, characterized in that The precise fingerprint is calculated based on all fields; the loose fingerprint is calculated based on the fields included in the page hyperlink information, page rendering information, and the homepage URL path; the rough fingerprint is calculated based on the fields included in the page hyperlink information and the homepage URL path.

9. The method according to claim 5, characterized in that The following steps are also included: Select fingerprints with different accuracy for matching according to different application scenarios; When an exact match is required, select precise fingerprint; when false positives are allowed but false negatives are not allowed, select loose fingerprint; when only large-scale features need to be identified, select rough fingerprint.

10. Application of the method according to claim 5 in network asset mapping, big data analysis, and vulnerability impact assessment.

Images