Method and device for identifying similar structures of webpages

Abstract

The invention discloses a webpage similar structure identification method and device, and relates to the technical field of computer data processing, and the method comprises the following steps: storing webpage data in a byte array form; sequentially traversing each byte in the byte array, determining an initial position of each HTML tag of the webpage in the byte array, and writing a byte data value of the initial position of each HTML tag and a byte data value corresponding to an HTML tag name into a hash function; calling a hash function to generate hash abstracts, and taking a set of the hash abstracts as a webpage structure ID; judging the similarity between the two web pages by comparing the web page structure IDs of the two web pages; or identifying the Web application associated with the target webpage by comparing the matching degree between the target webpage structure ID and the corresponding webpage structure ID stored in the Web application fingerprint database. The structural features of the web pages are efficiently and robustly extracted, the unique ID is generated, and the pages with similar structures in massive web pages are quickly identified.

Description

Technical Field

[0001] The present invention relates to the technical field of computer data processing, and in particular to a method and device for identifying similar structures of web pages. Background Art

[0002] Currently, in the processing of massive web data, quickly identifying and removing web pages with similar structures is an important step. Existing web page similarity detection methods mainly include: (1) Method based on DOM tree comparison. This method completely parses the HTML document, constructs a DOM (Document Object Model) tree, and then compares the structure and node information of the two DOM trees. The advantage of this method is its high accuracy and ability to accurately reflect the page structure. The disadvantage is that DOM parsing itself is very time-consuming, especially for large and complex web pages. Its computational overhead is huge. For pages or page fragments that do not conform to the HTML specification, the DOM parser may make errors or fail to parse them completely, resulting in the inability to perform subsequent comparisons. At the same time, building the DOM tree requires a large memory overhead.

[0003] (2) Methods based on web page content hashing. This type of method focuses on the similarity of text content on web pages, extracting text features and calculating hash values ​​for comparison. This method is effective for identifying pages with duplicate content, but it is not effective for identifying pages with similar structures but different content, such as product listing pages that use the same template but have different data.

[0004] (3) Methods based on specific tag or path features. Some methods extract specific HTML tag combinations or CSS selector paths as features. These methods may be faster than DOM tree comparison methods, but the universality and coverage of feature selection are difficult to guarantee and are easily affected by subtle adjustments to the page.

[0005] In summary, existing technologies often have deficiencies in efficiency, robustness, and resource consumption when dealing with large-scale web page structural similarity judgment, and are unable to meet the requirements of fast and accurate recognition. Summary of the Invention

[0006] In view of the above-mentioned defects or deficiencies in the prior art, the present invention provides a method and device for identifying similar structures of web pages, which can solve the above-mentioned technical problems mentioned in the background technology.

[0007] In one aspect of the present invention, a method for identifying similar structures of web pages is provided, comprising the following steps: storing web page data in the form of a byte array; sequentially traversing each byte in the byte array to determine the starting position of each HTML tag of the web page in the byte array, and writing the byte data value of the starting position of each HTML tag and the byte data value corresponding to the HTML tag name into a predetermined hash function; calling the hash function to generate a hash digest of each HTML tag, and using the set of hash digests as a web page structure ID; determining the similarity between two web pages by comparing the web page structure IDs of the two web pages; or identifying the web application associated with the target web page by comparing the matching degree of the target web page structure ID with the corresponding web page structure ID stored in a web application fingerprint database.

[0008] In another aspect of the present invention, a device for identifying similar structures of web pages is provided, comprising: a first module for storing web page data in the form of a byte array; a second module for sequentially traversing each byte in the byte array, determining the starting position of each HTML tag of the web page in the byte array, and writing the byte data value of the starting position of each HTML tag and the byte data value corresponding to the HTML tag name into a predetermined hash function; a third module for calling the hash function to generate a hash digest of each HTML tag, and using the set of hash digests as a web page structure ID; and a fourth module for determining the similarity between two web pages by comparing their web page structure IDs; or identifying the web application associated with the target web page by comparing the matching degree between the target web page structure ID and the corresponding web page structure ID stored in a web application fingerprint database.

[0009] The method and device for identifying similar web page structures provided by the present invention focuses solely on the features of the HTML tags that form the web page skeleton, specifically the HTML tag start character and HTML tag name, while ignoring variable content such as HTML tag attributes, text, and comments. The HTML tag features are serialized and then input into a hash function to generate a unique web page structure ID. Compared to existing technologies, the present invention has the following significant technical advantages: (1) High efficiency The present invention performs only a single linear scan of the input byte stream, with a time complexity close to O(n), where n is the byte length of the page. This avoids time-consuming DOM parsing, tree construction, and complex tree comparison algorithms. As a result, the present invention offers extremely fast computational speeds and is suitable for processing very large web page datasets.

[0010] (2) Low resource consumption There is no need to build a complete DOM tree in memory, and the memory usage is extremely small. It only needs to store the hash state of the current HTML tag and a small number of intermediate variables.

[0011] (3) Strong robustness Because the present invention directly operates on the byte stream and focuses only on the tag start feature consisting of the HTML tag start symbol and the HTML tag name (i.e., only the left angle bracket "<" followed by an English letter), it is insensitive to whether HTML strictly complies with W3C standards. It can effectively process HTML pages with irregular formats, unclosed tags, incorrectly nested tags, and HTML fragments. As long as the page contains a recognizable tag start feature, its structural information can be extracted.

[0012] (4) Structural sensitivity This method accurately captures the order and type of key page tags (such as div, p, table, h1-h6, ul, li, etc.). It is insensitive to tag attributes (such as class, id, style), inter-tag text, comments, script content, CSS styles, and so on, focusing solely on the underlying structure. Therefore, even for pages with the same template but different data (such as news listings and e-commerce product pages), it can accurately identify structural similarities. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings: Figure 1 This is a flow chart of a method for identifying similar structures of web pages provided by one embodiment of the present application; Figure 2 This is a schematic diagram of the structure of a device for identifying similar structures of web pages provided by one embodiment of the present application; Figure 3 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0014] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0015] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the embodiments of the present invention are also intended to include plural forms, unless the context clearly indicates other meanings.

[0016] It should be understood that although the terms first, second, third, etc. may be used to describe the acquisition modules in the embodiments of the present invention, the acquisition modules should not be limited to these terms. These terms are only used to distinguish the acquisition modules from each other.

[0017] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0018] It should be noted that the directional terms such as "upper," "lower," "left," and "right" described in the embodiments of the present invention are described from the perspectives shown in the accompanying drawings and should not be construed as limiting the embodiments of the present invention. Furthermore, in the context, it should be understood that when an element is referred to as being formed "on" or "under" another element, it can be formed not only directly "on" or "under" the other element, but also indirectly "on" or "under" the other element through an intermediate element.

[0019] An embodiment of the present application provides a method for identifying similar structures of web pages. The method only focuses on the HTML tags that constitute the skeleton of the web page, ignores variable content such as attributes, text, and comments, and serializes the features of the HTML tags and inputs them into a hash function to generate a unique web page structure ID.

[0020] See also Figure 1 ,The method for identifying similar structures of web pages includes the following steps: Step S101: store the web page data in the form of a byte array.

[0021] Specifically, the webpage byte stream is received as input and stored as a byte array, each containing a number of bytes. Furthermore, a hash function instance is selected, such as MD5, SHA1, or SHA256. This step preserves the original HTML encoding of the webpage through the byte array, avoiding parsing errors. By selecting a suitable hash function instantiation, it ensures that feature data can be incrementally written and hashed later.

[0022] Step S102, traverse each byte in the byte array in turn, determine the starting position of each HTML tag in the web page in the byte array, and write the byte data value of the starting position of each HTML tag and the byte data value corresponding to the HTML tag name into a predetermined hash function.

[0023] This step is used to realize byte stream traversal and web page structure feature extraction. Its technical concept is to find the starting position of HTML tags Extract HTML tag features Hash HTML tag features.

[0024] First, let's introduce the basic terminology of HTML documents. HTML documents include tags, content, comments, document type declarations, and other components. HTML tags include left and right angle brackets, tag names, tag attributes, and so on. For example: HTML tags , where "div" is the HTML tag name, class="example" is the tag attribute, "<" is the left angle bracket, and ">" is the right angle bracket.

[0025] To improve scanning efficiency, reduce time complexity, and increase robustness, this step focuses on specific patterns in HTML tags: the HTML tag start character and the HTML tag name. Alternatively, consider the pattern of a left angle bracket (<) followed by an English letter. This step identifies the characteristics of HTML tags by recognizing these patterns. As long as the recognizable HTML tag features are present on a page, the page's structural information can be extracted.

[0026] First, start checking from the first byte in the byte array data to determine whether the current byte value r = data[i] is a left angle bracket "<", where i is the byte index. If so, check whether the next adjacent byte value data[i+1] is an English letter "a to z" or "A to Z". If the next adjacent byte value data[i+1] is an English letter, use the byte corresponding to the left angle bracket "<" as the starting position of the HTML tag in the byte array. Otherwise, do not use the byte corresponding to the left angle bracket "<" as the starting position of the HTML tag in the byte array, and continue to traverse the next byte value to see if it is a left angle bracket "<". Repeat the above steps until all bytes in the byte array have been traversed.

[0027] Then, the byte value r=data[i] corresponding to the starting position of each HTML tag in the byte array is first written into a predetermined hash function (that is, the left angle bracket is written into the hash function), where the hash algorithm used by the hash function can be MD5, SHA256, etc. Then, the subsequent bytes i+1, i+2, ..., i+n at the starting position are traversed in sequence. If the subsequent byte value r=data[i+1] is an English letter, the subsequent byte value r=data[i+1] is written into the predetermined hash function, and so on, until the subsequent byte value contains the end symbol of the HTML tag name, such as: space, right angle bracket >, number or other symbols, which indicates that the HTML tag name has ended and the following are attributes or tag content. At this time, the traversal stops.

[0028] Through the above method, the most representative feature of HTML tags, "left angle bracket + HTML tag name", is identified and hashed, ignoring tag attributes, text content between tags, comments, scripts, styles, etc., thereby focusing on the macro-structure sequence of the page. This brings strong robustness and structural sensitivity when comparing web pages.

[0029] Step S103: calling the hash function to generate a hash digest of each HTML tag, and using the set of hash digests as the web page structure ID.

[0030] Specifically, the Sum(nil) method of the hash function may be called to generate the final hash digest of each HTML tag (but not limited to this, other methods may also be called). The set of hash digests of all HTML tags in the web page is used as a unique web page structure ID.

[0031] Step S104 , determining the similarity between two web pages or two websites by comparing the web page structure IDs of the web pages; or identifying the web application associated with the target web page by comparing the matching degree between the target web page structure ID and the corresponding web page structure ID stored in the web application fingerprint database.

[0032] Specifically, after obtaining the webpage structure ID, the webpage structure ID can be used in the following two implementation scenarios: (1) Determination of similar pages or similar websites based on structural ID Suppose there are three HTML snippets below: I. Page A HTML <!DOCTYPE html> <title> My Page Title< / title> <h1> Welcome< / h1> This is a paragraph. <script>alert('hello');< / script> II. Page B (same structure as Page A, but different content and attributes) HTML <!DOCTYPE html> <title> Another Title< / title> <h1> Greetings< / h1> Some other text here. III. Page C (different structure) HTML <!DOCTYPE html> <title> Different Structure< / title> <article> <h2> A Post< / h2> Some content. < / article> In this implementation scenario, MD5 is selected as the hashing algorithm of the hash function.

[0033] Processing page A: The following byte sequences (or the characters they represent) are sequentially fed into the MD5 hash function: <h t m l <h e a d <t i t l e <b o d y <d i v (when encountering a space, the class attribute is ignored) <h (when encountering 1, since 1 is not an English letter, only h is taken, that is, the tag <h1>1 in it is ignored) <p (When encountering a space, the content is ignored) <s c r i p t (The script content is ignored) Call the hash function, and finally generate an MD5 hash value, such as ID_A.

[0034] Process page B: Although the title content of page B, the attributes of the div (id = "container" and class = "main - content"), the paragraph content, and the comments are different from those of page A, the core part of its label feature sequence is the same. Therefore, the byte sequence fed into the MD5 hash function will be exactly the same as that of page A. The finally generated MD5 hash value ID_B will be equal to ID_A.

[0035] Process page C: Feed the following byte sequences into the MD5 hash function in sequence: <h t m l <h e a d <t i t l e <b o d y <a r t i c l e <h (from the label< / h1> <h2>) This sequence is significantly different from the sequences of Pages A and B. Therefore, the generated MD5 hash value ID_C will be different from ID_A and ID_B.

[0036] In a massive webpage library, if the calculated structural ID of page X is identical to page A's ID_A, it can be quickly determined that page X and page A (and page B) are highly similar in macroscopic structure, even if their specific text content, images, CSS styles, or JavaScript scripts may be completely different. Alternatively, a first set of structural IDs is generated for a collection of webpages on a first website and a second set of structural IDs is generated for a collection of webpages on a second website. The number of identical structural IDs in the first and second sets of structural IDs is compared. If the number of identical structural IDs is greater than or equal to a preset threshold, the first and second websites are determined to have similar structures. The above method has significant technical benefits for identifying website templates, removing duplicates, or detecting structural plagiarism.

[0037] (2) Web application fingerprint recognition based on structure ID Web fingerprinting is an important technology in fields such as network security, asset management, and market analysis. It aims to identify the web server, backend framework, content management system (CMS), front-end library, or other specific technologies used by a target website by analyzing its specific characteristics. Traditional fingerprinting methods may rely on server response headers, specific file paths, HTML comment content, JavaScript variable names, or specific CSS class names. These methods can sometimes fail due to website customization, version updates, or security measures. Because many web applications or CMSs use fixed templates with specific HTML tag structures when generating pages, this implementation scenario utilizes the web page structure ID proposed in this invention as a novel fingerprint feature for identifying web applications.

[0038] First, we selected a number of known and representative web applications as the source of the fingerprint library. For example, we selected different versions of WordPress and its default themes, different versions of mainstream CMSs such as Joomla and Drupal, common e-commerce platforms, specific SaaS applications, and applications built with specific front-end frameworks and a unified shell structure.

[0039] Then, for each selected known web application sample, crawl the HTML content of one or more representative pages. For example, for WordPress, you can crawl its homepage, a typical article page, category list page, etc.

[0040] ​Next, for each captured HTML page, the web page similar structure identification method or program of the present invention is used to generate its structure ID.

[0041] Next, a fingerprint database (e.g., a key-value store or a relational database) is created to associate the calculated structure ID with the corresponding web application information (e.g., application name, version, page type, etc.) and store it.

[0042] Finally, one or more target URLs are input. For each target URL, the HTML content of its page is crawled. The structure ID of the target page's HTML content is calculated using the web page similarity structure identification method or program of the present invention. The calculated structure ID of the target page is compared with the structure ID stored in the fingerprint database. If the structure ID of the target page completely or largely matches the structure ID of one or more entries in the database, it can be preliminarily determined that the target website or page is using the web application, CMS, or framework associated with the ID. If no match is found, it is determined to be an unknown structure or the fingerprint library does not contain the fingerprint of the application.

[0043] See also Figure 2 Another embodiment of the present invention further provides a device 200 for identifying similar structures of web pages, including a first module 201, a second module 202, a third module 203 and a fourth module 204. The device 200 can execute the method for identifying similar structures of web pages in the method embodiment.

[0044] Specifically, the apparatus 200 for identifying similar structures of web pages includes: The first module 201 is used to store web page data in the form of a byte array; The second module 202 is configured to sequentially traverse each byte in the byte array, determine the starting position of each HTML tag of the webpage in the byte array, and write the byte data value of the starting position of each HTML tag and the byte data value corresponding to the HTML tag name into a predetermined hash function; The third module 203 is configured to call the hash function to generate a hash digest of each HTML tag, and use the set of hash digests as the webpage structure ID; The fourth module 204 is used to determine the similarity between two web pages or two websites by comparing the web page structure IDs of the web pages; or to identify the web application associated with the target web page by comparing the matching degree between the target web page structure ID and the corresponding web page structure ID stored in the web application fingerprint database.

[0045] The second module 202 is further configured to: start checking from the first byte in the byte array to determine whether the current byte value is a left angle bracket; if so, check whether the next adjacent byte value is an English letter; if so, use the byte corresponding to the left angle bracket as the starting position of the HTML tag in the byte array; otherwise, do not use the byte corresponding to the left angle bracket as the starting position of the HTML tag in the byte array, and continue traversing to determine whether the next byte value is a left angle bracket; and perform the above steps until all bytes in the byte array have been traversed.

[0046] The second module 202 is further used to: first write the byte data value corresponding to the starting position of each HTML tag in the byte array into a predetermined hash function, and then traverse the subsequent bytes of the starting position in sequence. If the subsequent byte value is an English letter, write the subsequent byte value into the predetermined hash function until the subsequent byte value contains the end symbol of the HTML tag name, and then stop traversing.

[0047] The fourth module 204 is further used to: generate a first web page structure ID and a second web page structure ID, compare whether the first web page structure ID and the second web page structure ID are the same, and if they are the same, determine that the first web page and the second web page structures are similar; or generate a first group of structure IDs for the web page set of the first website and a second group of structure IDs for the web page set of the second website, compare the number of identical structure IDs in the first group of structure IDs and the second group of structure IDs, and if the number of identical structure IDs is greater than or equal to a preset threshold, determine that the first website and the second website have similar structures.

[0048] The fourth module 204 is further configured to: obtain multiple typical web page structure IDs of web application samples; associate the multiple typical web page structure IDs of each web application sample with corresponding web application feature information and store them in a web application fingerprint database; obtain the web page structure ID of a target web page, determine whether there is a web page structure ID matching the web page structure ID of the target web page in the web application fingerprint database, and if so, determine the web application associated with the target web page based on the web application feature information corresponding to the matching web page structure ID in the web application fingerprint database.

[0049] It should be noted that the apparatus 200 for identifying similar structures of web pages provided in this embodiment corresponds to a technical solution that can be used to execute various method embodiments. Its implementation principle and technical effects are similar to those of the method and will not be described in detail here.

[0050] Figure 3 This is a schematic diagram of the structure of an electronic device 300 provided in another embodiment of the present invention, which is used to implement the method for identifying similar structures in web pages in the method embodiment. The electronic device 300 in this embodiment of the present invention may include, but is not limited to, a smartphone, a tablet computer, a PC, a laptop computer, a server, etc. Figure 3 The electronic device 300 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0051] like Figure 3 As shown, electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes to implement the methods of the embodiments of the present invention according to programs stored in read-only memory (ROM) 302 or programs loaded from storage device 308 into random access memory (RAM) 303. RAM 303 also stores various programs and data required for the operation of electronic device 300. Processing device 301, ROM 302, and RAM 303 are interconnected via bus 305. Input / output (I / O) interface 304 is also connected to bus 305.

[0052] Typically, the following devices may be connected to the I / O interface 304: an input device 306 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0053] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention further provides a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart, thereby implementing the method for identifying similar structures of web pages as described above. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of the embodiment of the present invention are performed.

[0054] The above description is merely a preferred embodiment of the present invention. Those skilled in the art should understand that the scope of the present invention is not limited to technical solutions formed by specific combinations of the above-mentioned technical features. It also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents, without departing from the above-mentioned disclosure. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this invention.< / h2>

Claims

1. A method for identifying similar structures of web pages, characterized in that: The steps include: Store web page data in the form of byte array; Traversing each byte in the byte array in sequence, determining the starting position of each HTML tag in the webpage in the byte array, and writing the byte data value of the starting position of each HTML tag and the byte data value corresponding to the HTML tag name into a predetermined hash function; Calling the hash function to generate a hash digest of each HTML tag, and using a set of the hash digests as a web page structure ID; By comparing the webpage structure IDs of webpages, the similarity between two webpages or two websites is determined; or by comparing the matching degree between the target webpage structure ID and the corresponding webpage structure ID stored in the web application fingerprint database, the web application associated with the target webpage is identified.

2. A method for identifying similar structures of web pages according to claim 1, characterized in that: The step of sequentially traversing each byte in the byte array to determine the starting position of each HTML tag of the webpage in the byte array includes: Starting from the first byte in the byte array, checking whether the current byte value is a left angle bracket, if so, checking whether the next adjacent byte value is an English letter, if the next adjacent byte value is an English letter, using the byte corresponding to the left angle bracket as the starting position of the HTML tag in the byte array, otherwise, not using the byte corresponding to the left angle bracket as the starting position of the HTML tag in the byte array, and continuing to traverse to see if the next byte value is a left angle bracket; executing the above steps until all bytes in the byte array have been traversed.

3. A method for identifying similar structures of web pages according to claim 2, characterized in that: The step of writing the byte data value at the starting position of each HTML tag and the byte data value corresponding to the HTML tag name into a predetermined hash function includes: First, the byte data value corresponding to the starting position of each HTML tag in the byte array is written into a predetermined hash function, and then the subsequent bytes of the starting position are traversed in sequence. If the subsequent byte value is an English letter, the subsequent byte value is written into the predetermined hash function until the subsequent byte value appears the end symbol of the HTML tag name, and then the traversal stops.

4. A method for identifying similar structures of web pages according to claim 3, characterized in that: The step of determining the similarity between two web pages or two websites by comparing the web page structure IDs of the web pages includes: generating a first webpage structure ID and a second webpage structure ID, and comparing the first webpage structure ID and the second webpage structure ID to determine whether they are the same; if they are the same, determining that the first webpage and the second webpage structures are similar; Alternatively, a first group of structural IDs for a set of web pages of a first website and a second group of structural IDs for a set of web pages of a second website are generated, and the number of identical structural IDs in the first group of structural IDs and the second group of structural IDs are compared. If the number of identical structural IDs is greater than or equal to a preset threshold, it is determined that the structures of the first website and the second website are similar.

5. The method for identifying similar structures of web pages according to claim 3, characterized in that: The step of identifying the Web application associated with the target web page by comparing the matching degree between the target web page structure ID and the corresponding web page structure ID stored in the Web application fingerprint database includes: Obtain multiple typical web page structure IDs of the web application sample; Associating multiple typical web page structure IDs of each web application sample with corresponding web application feature information and storing them in a web application fingerprint database; Obtain the webpage structure ID of the target webpage, determine whether there is a webpage structure ID matching the webpage structure ID of the target webpage in the web application fingerprint database, and if so, determine the web application associated with the target webpage based on the web application feature information corresponding to the matching webpage structure ID in the web application fingerprint database.

6. A device for identifying similar structures of web pages, characterized in that: include: The first module is used to store web page data in the form of byte array; The second module is configured to sequentially traverse each byte in the byte array, determine the starting position of each HTML tag of the webpage in the byte array, and write the byte data value of the starting position of each HTML tag and the byte data value corresponding to the HTML tag name into a predetermined hash function; The third module is used to call the hash function to generate a hash digest of each HTML tag, and use the set of hash digests as the web page structure ID; The fourth module is used to determine the similarity between two web pages or two websites by comparing the web page structure IDs of the web pages; or to identify the web application associated with the target web page by comparing the matching degree between the target web page structure ID and the corresponding web page structure ID stored in the web application fingerprint database.

7. The device for identifying similar structures of web pages according to claim 6, characterized in that: The second module is further configured to: Starting from the first byte in the byte array, checking whether the current byte value is a left angle bracket, if so, checking whether the next adjacent byte value is an English letter, if the next adjacent byte value is an English letter, using the byte corresponding to the left angle bracket as the starting position of the HTML tag in the byte array, otherwise, not using the byte corresponding to the left angle bracket as the starting position of the HTML tag in the byte array, and continuing to traverse to see if the next byte value is a left angle bracket; executing the above steps until all bytes in the byte array have been traversed.

8. The device for identifying similar structures of web pages according to claim 7, characterized in that: The second module is further configured to: First, the byte data value corresponding to the starting position of each HTML tag in the byte array is written into a predetermined hash function, and then the subsequent bytes of the starting position are traversed in sequence. If the subsequent byte value is an English letter, the subsequent byte value is written into the predetermined hash function until the subsequent byte value appears the end symbol of the HTML tag name, and then the traversal stops.

9. The device for identifying similar structures of web pages according to claim 8, characterized in that: The fourth module is further configured to: generating a first webpage structure ID and a second webpage structure ID, and comparing the first webpage structure ID and the second webpage structure ID to determine whether they are the same; if they are the same, determining that the first webpage and the second webpage structures are similar; Alternatively, a first group of structural IDs for a set of web pages of a first website and a second group of structural IDs for a set of web pages of a second website are generated, and the number of identical structural IDs in the first group of structural IDs and the second group of structural IDs are compared. If the number of identical structural IDs is greater than or equal to a preset threshold, it is determined that the structures of the first website and the second website are similar.

10. The device for identifying similar structures of web pages according to claim 8, characterized in that: The fourth module is further configured to: Obtain multiple typical web page structure IDs of the web application sample; Associating multiple typical web page structure IDs of each web application sample with corresponding web application feature information and storing them in a web application fingerprint database; Obtain the webpage structure ID of the target webpage, determine whether there is a webpage structure ID matching the webpage structure ID of the target webpage in the web application fingerprint database, and if so, determine the web application associated with the target webpage based on the web application feature information corresponding to the matching webpage structure ID in the web application fingerprint database.

Images