A method and device for identifying similar structures of web pages

Abstract

The application discloses a webpage similar structure identification method and device, and relates to the technical field of computer data processing, which comprises the following steps: storing webpage data in the form of a byte array; sequentially traversing each byte in the byte array to determine the starting position of each HTML tag of the webpage in the byte array, and writing the byte data value of the starting position of each HTML tag and the byte data value corresponding to the HTML tag name into a hash function; calling the hash function to generate a hash digest, and taking the set of the hash digest as a webpage structure ID; comparing the webpage structure IDs of two webpages to determine the similarity between the two; or comparing the matching degree of the target webpage structure ID and the corresponding webpage structure ID stored in a Web application fingerprint database to identify the Web application associated with the target webpage. The structure characteristics of the webpage are efficiently and robustly extracted, a unique ID is generated, and the structurally similar pages in a large number of webpages are quickly identified.

Description

Technical Field

[0001] This invention relates to the field of computer data processing technology, and in particular to a method and apparatus for identifying similar structures on web pages. Background Technology

[0002] Currently, in the processing of massive amounts of web data, quickly identifying and deduplicating structurally similar web pages is a crucial step. Existing web page similarity detection methods mainly include:

[0003] (1) DOM tree-based comparison method. This method involves completely parsing the HTML document, constructing a DOM (Document Object Model) tree, and then comparing the structure and node information of the two DOM trees. The advantage of this method is its high accuracy, which can accurately reflect the page structure. The disadvantage is that DOM parsing itself is very time-consuming, especially for large and complex web pages, where the computational overhead is huge. For pages or page fragments that do not conform to HTML specifications, the DOM parser may make mistakes or fail to parse them completely, making subsequent comparison impossible. At the same time, constructing the DOM tree requires a large memory overhead.

[0004] (2) Methods based on webpage content hashing. This type of method mainly focuses on the similarity of text content on webpages, comparing them by extracting text features and calculating hash values. This method is effective in identifying pages with duplicate content, but it cannot effectively identify pages with similar structures but different content, such as product list pages that use the same template but are filled with different data.

[0005] (3) Methods based on specific tags or path features. Some methods extract specific combinations of HTML tags or CSS selector paths as features. These methods may be faster than DOM tree comparison, but the universality and coverage of feature selection are difficult to guarantee, and they are easily affected by minor page adjustments.

[0006] In summary, existing technologies often fall short in terms of efficiency, robustness, and resource consumption when dealing with large-scale webpage structural similarity assessments, making it difficult to meet the requirements for rapid and accurate identification. Summary of the Invention

[0007] In view of the above-mentioned defects or deficiencies in the prior art, the present invention provides a method and apparatus for identifying similar web page structures, which can solve the above-mentioned technical problems mentioned in the background art.

[0008] One aspect of the present invention provides a method for identifying similar structures of web pages, comprising the following steps: storing web page data as a byte array; sequentially traversing each byte in the byte array to determine the starting position of each HTML tag in the byte array, and writing the byte data value of the starting position of each HTML tag and the byte data value corresponding to the HTML tag name into a predetermined hash function; calling the hash function to generate a hash digest of each HTML tag, and using the set of hash digests as a web page structure ID; determining the similarity between two web pages by comparing their web page structure IDs; or, identifying the web application associated with the target web page by comparing the matching degree between the target web page structure ID and the corresponding web page structure ID stored in a web application fingerprint database.

[0009] In another aspect, the present invention provides a device for identifying webpage similar structures, comprising: a first module for storing webpage data as a byte array; a second module for sequentially traversing each byte in the byte array, determining the starting position of each HTML tag in the byte array, and writing the byte data value of the starting position of each HTML tag and the byte data value corresponding to the HTML tag name into a predetermined hash function; a third module for calling the hash function to generate a hash digest of each HTML tag, and using the set of hash digests as a webpage structure ID; and a fourth module for determining the similarity between two webpages by comparing their webpage structure IDs; or, identifying the web application associated with the target webpage by comparing the matching degree between the target webpage structure ID and the corresponding webpage structure ID stored in a web application fingerprint database.

[0010] The method and apparatus for identifying similar webpage structures provided by this invention focus only on the features of HTML tags that constitute the skeleton of a webpage. Specifically, it only considers the HTML tag start characters and HTML tag names, ignoring variable content such as HTML tag attributes, text, and comments. The features of the HTML tags are serialized and input into a hash function to generate a unique webpage structure ID. Compared with existing technologies, this invention has the following significant technical advantages:

[0011] (1) High efficiency

[0012] This invention performs only one linear scan of the input byte stream, with a time complexity close to O(n), where n is the page length in bytes. This avoids time-consuming DOM parsing, tree construction, and complex tree comparison algorithms. Therefore, this invention is extremely fast and suitable for processing very large-scale web page datasets.

[0013] (2) Low resource consumption

[0014] There is no need to build a complete DOM tree in memory, resulting in minimal memory usage. It only needs to store the hash state of the current HTML tag and a small number of intermediate variables.

[0015] (3) Strong robustness

[0016] Because it directly manipulates byte streams and focuses only on the tag start characteristics consisting of HTML tag start characters and HTML tag names (i.e., only focusing on the tag start characteristics formed by the left angle bracket "<" followed by an English letter), this invention is insensitive to whether the HTML strictly conforms to the W3C standard. It can effectively handle HTML pages with non-standard formatting, unclosed tags, and incorrect nesting, as well as HTML fragments. As long as there are identifiable tag start characteristics in the page, its structural information can be extracted.

[0017] (4) Structural sensitivity

[0018] This invention can accurately capture the order and type of main page tags (such as div, p, table, h1-h6, ul, li, etc.). It is insensitive to tag attributes (class, id, style, etc.), text content between tags, comments, script content, CSS styles, etc., focusing solely on the skeletal structure. Therefore, it can accurately identify the structural similarity of pages with the same template but different data filling (such as news list pages and e-commerce product pages). Attached Figure Description

[0019] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0020] Figure 1 This is a flowchart illustrating a method for identifying similar webpage structures according to an embodiment of this application;

[0021] Figure 2 This is a schematic diagram of the structure of a webpage similarity recognition device provided in one embodiment of this application;

[0022] Figure 3 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” used in the embodiments of this invention are also intended to include the plural forms unless the context clearly indicates otherwise.

[0025] It should be understood that although the terms first, second, third, etc., may be used to describe the acquisition modules in the embodiments of the present invention, these acquisition modules should not be limited to these terms. These terms are only used to distinguish the acquisition modules from each other.

[0026] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0027] It should be noted that the directional terms such as "upper," "lower," "left," and "right" used in the embodiments of the present invention are used to describe the angles shown in the accompanying drawings and should not be construed as limiting the embodiments of the present invention. Furthermore, in the context, it should be understood that when it is mentioned that an element is formed "upper" or "lower" of another element, it can not only be formed directly "upper" or "lower" of the other element, but also indirectly "upper" or "lower" of the other element through an intermediate element.

[0028] One embodiment of this application provides a method for identifying similar web page structures. This method focuses only on the HTML tags that constitute the skeleton of the web page, ignoring variable content such as attributes, text, and comments. The features of the HTML tags are serialized and input into a hash function to generate a unique web page structure ID.

[0029] See Figure 1 The method for identifying similar structures on web pages includes the following steps:

[0030] Step S101: Store the web page data as a byte array.

[0031] Specifically, the process receives the input webpage byte stream and stores it as a byte array, with each array containing several bytes. Further, a hash function instance is selected, such as MD5, SHA1, or SHA256. This step preserves the original HTML encoded data of the webpage completely through the byte array, avoiding parsing errors. By selecting an appropriate hash function instantiation, it ensures that feature data can be incrementally written and hashed subsequently.

[0032] Step S102: Iterate through each byte in the byte array sequentially, determine the starting position of each HTML tag in the byte array, and write the byte data value of the starting position of each HTML tag and the byte data value corresponding to the HTML tag name into a predetermined hash function.

[0033] This step is used to traverse the byte stream and extract webpage structural features. The technical concept is to find the starting position of HTML tags. Extracting HTML tag features Hash HTML tag features.

[0034] First, let's introduce the basic terminology of an HTML webpage document. An HTML webpage document consists of tags, content, comments, document type declarations, etc. HTML tags include angle brackets, tag names, tag attributes, etc. For example: HTML tags In this text, "div" is the HTML tag name, class="example" is the tag attribute, "<" is the left angle bracket, and ">" is the right angle bracket.

[0035] To improve scanning efficiency, reduce time complexity, and increase robustness, this step focuses only on specific patterns of HTML tags, namely: HTML tag start characters and HTML tag names. Alternatively, it can be considered as focusing only on the left angle bracket "<" followed by an English letter. This step identifies the characteristics of HTML tags by recognizing this specific pattern. As long as identifiable HTML tag characteristics exist on the page, the page's structural information can be extracted.

[0036] First, starting from the first byte of the byte array `data`, check if the current byte value `r = data[i]` is a left angle bracket "<", where `i` is the byte index. If it is, check if the next adjacent byte value `data[i+1]` is an English letter "a to z" or "A to Z". If the next adjacent byte value `data[i+1]` is an English letter, then the byte corresponding to the left angle bracket "<" is used as the starting position of the HTML tag in the byte array. Otherwise, the byte corresponding to the left angle bracket "<" is not used as the starting position of the HTML tag in the byte array, and the process continues to traverse the next byte value to check if it is a left angle bracket "<". Repeat these steps until all bytes in the byte array have been traversed.

[0037] Next, the byte value r=data[i] corresponding to the starting position of each HTML tag in the byte array is written into a predetermined hash function (that is, the left angle bracket is written into the hash function). The hash function can use a hash algorithm such as MD5 or SHA256. Then, the subsequent bytes i+1, i+2...i+n are traversed sequentially. If the subsequent byte value r=data[i+1] is an English letter, then the subsequent byte value r=data[i+1] is written into the predetermined hash function. This process continues until the subsequent byte value contains an HTML tag name ending symbol, such as a space, a right angle bracket >, a number, or other symbols. These symbols indicate that the HTML tag name has ended and what follows are attributes or tag content. At this point, the traversal stops.

[0038] By using the above method, the most representative feature of HTML tags, "left angle bracket + HTML tag name", is identified and hashed, ignoring tag attributes, text content between tags, comments, scripts, styles, etc., thus focusing on the macro-structure sequence of the page. This brings strong robustness and structural sensitivity when comparing web pages.

[0039] Step S103: Call the hash function to generate a hash digest for each HTML tag, and use the set of hash digests as the webpage structure ID.

[0040] Specifically, the Sum(nil) method of the hash function can be called to generate the final hash digest of each HTML tag (but this is not limited to this; other methods can also be called). The collection of hash digests of all HTML tags in the webpage is used as a unique webpage structure ID.

[0041] Step S104: By comparing the webpage structure IDs of the webpages, determine the similarity between two webpages or two websites; or, by comparing the matching degree between the target webpage structure ID and the corresponding webpage structure ID stored in the Web application fingerprint database, identify the Web application associated with the target webpage.

[0042] Specifically, after obtaining the webpage structure ID, the webpage structure ID can be used in the following two implementation scenarios:

[0043] (1) Determination of basic similar pages or similar websites based on structure ID

[0044] Suppose we have the following three HTML snippets:

[0045] Page A

[0046] HTML

[0047] <!DOCTYPE html>

[0048]

[0049]

[0050] <title> My Page Title< / title>

[0051]

[0052]

[0053]

[0054] <h1> Welcome< / h1>

[0055] This is a paragraph.

[0056]

[0057] <script>alert('hello');< / script>

[0058]

[0059]

[0060] II. Page B (same structure as page A, but different content and attributes)

[0061] HTML

[0062] <!DOCTYPE html>

[0063]

[0064]

[0065] <title> Another Title< / title>

[0066]

[0067]

[0068]

[0069] <h1> Greetings< / h1>

[0070] Some other text is here.

[0071]

[0072]

[0073]

[0074] III. Page C (Different Structure)

[0075] HTML

[0076] <!DOCTYPE html>

[0077]

[0078]

[0079] <title> Different Structure< / title>

[0080]

[0081]

[0082] <article>

[0083] <h2> A Post< / h2>

[0084] Some content.

[0085] < / article>

[0086]

[0087]

[0088] This implementation scenario selects MD5 as the hashing algorithm of the hash function.

[0089] Process Page A:

[0090] Send the following byte sequences (or the characters they represent) to the MD5 hash function in sequence:

[0091] <h t m l

[0092] <h e a d

[0093] <t i t l e

[0094] <b o d y

[0095] <d i v (When encountering a space, the class attribute is ignored)

[0096] <h (When encountering 1, since 1 is not an English letter, only h is taken, that is, the tag <h1>1 in it is ignored)

[0097] <p (When encountering a space, the content is ignored)

[0098] <s c r i p t (The script content is ignored)

[0099] Call the hash function, and finally generate an MD5 hash value, such as ID_A.

[0100] Process page B:

[0101] Although the title content of page B, the attributes of the div (id = "container" and class = "main - content"), the paragraph content, and the comments are different from those of page A, the core part of its tag feature sequence is the same. Therefore, the byte sequence sent into the MD5 hash function will be exactly the same as that of page A. The finally generated MD5 hash value ID_B will be equal to ID_A.

[0102] Process page C:

[0103] Send the following byte sequences into the MD5 hash function in sequence:

[0104] <h t m l

[0105] <h e a d

[0106] <t i t l e

[0107] <b o d y

[0108] <a r t i c l e

[0109] <h (from the tag< / h1> <h2>)

[0110]

[0111] This sequence is significantly different from the sequences of page A and page B. Therefore, the generated MD5 hash value ID_C will be different from ID_A and ID_B.

[0112] In a massive webpage database, if the calculated structure ID of page X is the same as the ID_A of page A, it can be quickly determined that page X and page A (and page B) are highly similar in macroscopic structure, even if their specific text content, images, CSS styles, or JavaScript scripts may be completely different. Alternatively, generate a first set of structure IDs for the webpage set of a first website and a second set of structure IDs for the webpage set of a second website, and compare the number of identical structure IDs in the first and second sets of structure IDs. If the number of identical structure IDs is greater than or equal to a preset threshold, then the first and second websites are determined to be structurally similar. The above methods have significant technical effects on identifying website templates, deduplicating, or detecting structural plagiarism.

[0113] (2) Web application fingerprinting based on structure ID

[0114] Web fingerprinting is an important technology in fields such as cybersecurity, asset management, and market analysis. It aims to identify the web server, backend framework, content management system (CMS), frontend library, or other specific technologies used by a target website by analyzing its specific characteristics. Traditional fingerprinting methods may rely on server response headers, specific file paths, HTML comment content, JavaScript variable names, or specific CSS class names. These methods can sometimes fail due to website customization, version updates, or security measures. Since many web applications or CMSs use fixed templates when generating pages, and these templates have specific HTML tag structures, this implementation utilizes the webpage structure ID proposed in this invention as a novel fingerprint feature for identifying web applications.

[0115] First, select a batch of known and representative web applications as the source of the fingerprint database. For example: different versions of WordPress and their default themes, different versions of mainstream CMS such as Joomla and Drupal, common e-commerce platforms, specific SaaS applications, applications built using specific front-end frameworks and with a unified shell structure, etc.

[0116] Then, for each selected known web application sample, scrape the HTML content of one or more representative pages. For example, for WordPress, you could scrape its homepage, a typical post page, category list pages, etc.​

[0117] Next, for each captured HTML page, the webpage similarity structure recognition method or program of the present invention is used to generate its structure ID.

[0118] Next, a fingerprint database (e.g., a key-value store or relational database) is created to associate the calculated structure ID with the corresponding web application information (such as application name, version, page type, etc.) and store it.

[0119] Finally, one or more target URLs are input. For each target URL, the HTML content of its page is crawled. Using the webpage similarity structure identification method or program of this invention, the structure ID of the target page's HTML content is calculated. The calculated structure ID of the target page is compared with the structure IDs stored in the fingerprint database. If the structure ID of the target page completely or mostly matches the structure IDs of one or more entries in the database, it can be preliminarily determined that the target website or page is using the web application, CMS, or framework associated with that ID. If no match is found, it is determined to be an unknown structure or the fingerprint of the application is not contained in the fingerprint database.

[0120] See Figure 2 Another embodiment of the present invention provides a webpage similarity structure identification device 200, including a first module 201, a second module 202, a third module 203 and a fourth module 204. The device 200 is capable of executing the webpage similarity structure identification method in the method embodiment.

[0121] Specifically, the webpage similarity recognition device 200 includes:

[0122] The first module 201 is used to store web page data as a byte array.

[0123] The second module 202 is used to sequentially traverse each byte in the byte array, determine the starting position of each HTML tag in the byte array, and write the byte data value of the starting position of each HTML tag and the byte data value corresponding to the HTML tag name into a predetermined hash function.

[0124] The third module 203 is used to call the hash function to generate a hash digest for each HTML tag, and use the set of hash digests as the webpage structure ID;

[0125] The fourth module 204 is used to determine the similarity between two web pages or two websites by comparing the web page structure IDs of the web pages; or, by comparing the matching degree between the target web page structure ID and the corresponding web page structure ID stored in the Web application fingerprint database, to identify the Web application associated with the target web page.

[0126] The second module 202 is further configured to: start checking from the first byte in the byte array, determine whether the current byte value is a left angle bracket, if so, check whether the next adjacent byte value is an English letter, if the next adjacent byte value is an English letter, then take the byte corresponding to the left angle bracket as the starting position of the HTML tag in the byte array, otherwise do not take the byte corresponding to the left angle bracket as the starting position of the HTML tag in the byte array, and continue to traverse the next byte value to see if it is a left angle bracket; execute the above steps until all bytes in the byte array have been traversed.

[0127] The second module 202 is further configured to: first write the byte data value corresponding to the starting position of each HTML tag in the byte array into a predetermined hash function, then traverse the subsequent bytes of the starting position in sequence. If the value of the subsequent byte is an English letter, then write the value of the subsequent byte into the predetermined hash function until the value of the subsequent byte contains the end symbol of the HTML tag name, then stop traversing.

[0128] The fourth module 204 is further configured to: generate a first webpage structure ID and a second webpage structure ID; compare whether the first webpage structure ID and the second webpage structure ID are the same; if they are the same, determine that the first webpage and the second webpage have similar structures; or, generate a first set of structure IDs for the webpage set of the first website and a second set of structure IDs for the webpage set of the second website; compare the number of identical structure IDs in the first set of structure IDs and the second set of structure IDs; if the number of identical structure IDs is greater than or equal to a preset threshold, determine that the first website and the second website have similar structures.

[0129] The fourth module 204 is further configured to: obtain multiple typical webpage structure IDs of the Web application sample; associate and store the multiple typical webpage structure IDs of each Web application sample with the corresponding Web application feature information in the Web application fingerprint database; obtain the webpage structure ID of the target webpage; determine whether there is a webpage structure ID in the Web application fingerprint database that matches the webpage structure ID of the target webpage; if so, determine the Web application associated with the target webpage based on the Web application feature information corresponding to the matching webpage structure ID in the Web application fingerprint database.

[0130] It should be noted that the technical solutions corresponding to the webpage similarity structure recognition device 200 provided in this embodiment, which can be used to execute various method embodiments, have similar implementation principles and technical effects to the methods, and will not be repeated here.

[0131] Figure 3 This is a schematic diagram of the structure of an electronic device 300 provided in another embodiment of the present invention. The electronic device 300 is used to implement the webpage similarity structure recognition method in the method embodiment. The electronic device 300 in the embodiments of the present invention may include, but is not limited to, smartphones, tablet computers, PCs, laptops, servers, etc. Figure 3 The electronic device 300 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0132] like Figure 3 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes to implement the methods of the embodiments described herein, based on a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing device 301, ROM 302, and RAM 303 are interconnected via a bus 305. An input / output (I / O) interface 304 is also connected to the bus 305.

[0133] Typically, the following devices can be connected to I / O interface 304: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0134] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention also provide a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts, thereby implementing the webpage similarity structure recognition method as described above. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 309, or installed from a storage device 308, or installed from a ROM 302. When the computer program is executed by the processing device 301, it performs the functions defined in the methods of the embodiments of the present invention.

[0135] The above description is merely a preferred embodiment of the present invention. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to the specific combination of the above-described technical features, but should also cover other technical solutions formed by any combination of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.< / h2>

Claims

1. A method for identifying similar structures on web pages, characterized in that, Includes the following steps: Store web page data as a byte array; The process involves iterating through each byte in the byte array to determine the starting position of each HTML tag in the byte array. First, the byte data value corresponding to the starting position of each HTML tag in the byte array is written into a predetermined hash function. Then, the subsequent bytes at the starting position are iterated through sequentially. If the value of the subsequent byte is an English letter, the value of the subsequent byte is written into the predetermined hash function. The process continues until the value of the subsequent byte contains the end symbol of the HTML tag name, at which point the iteration stops. Call the hash function to generate a hash digest for each HTML tag, and use the set of hash digests as the webpage structure ID; The similarity between two web pages or two websites can be determined by comparing their web page structure IDs; alternatively, the web application associated with the target web page can be identified by comparing the matching degree between the target web page structure ID and the corresponding web page structure ID stored in the web application fingerprint database.

2. The method for identifying similar webpage structures according to claim 1, characterized in that, The step of sequentially traversing each byte in the byte array to determine the starting position of each HTML tag in the byte array includes: Starting from the first byte in the byte array, check if the current byte value is a left angle bracket. If it is, check if the next adjacent byte value is an English letter. If the next adjacent byte value is an English letter, then the byte corresponding to the left angle bracket is taken as the starting position of the HTML tag in the byte array. Otherwise, the byte corresponding to the left angle bracket is not taken as the starting position of the HTML tag in the byte array, and the process continues to traverse the next byte value to check if it is a left angle bracket. Repeat the above steps until all bytes in the byte array have been traversed.

3. The method for identifying similar webpage structures according to claim 1, characterized in that, The step of determining the similarity between two web pages or two websites by comparing their web page structure IDs includes: Generate a first webpage structure ID and a second webpage structure ID, and compare whether the first webpage structure ID and the second webpage structure ID are the same. If they are the same, the first webpage and the second webpage are determined to be similar in structure. Alternatively, generate a first set of structure IDs for the webpage set of the first website and a second set of structure IDs for the webpage set of the second website, compare the number of identical structure IDs in the first set of structure IDs and the second set of structure IDs, and if the number of identical structure IDs is greater than or equal to a preset threshold, then determine that the first website and the second website have similar structures.

4. The method for identifying similar webpage structures according to claim 1, characterized in that, The step of identifying the web application associated with the target webpage by comparing the matching degree between the target webpage structure ID and the corresponding webpage structure ID stored in the web application fingerprint database includes: Obtain multiple typical webpage structure IDs from a web application sample; Multiple typical webpage structure IDs of each web application sample are associated with and stored in the web application fingerprint database along with the corresponding web application feature information; Obtain the webpage structure ID of the target webpage, determine whether there is a webpage structure ID in the Web application fingerprint database that matches the webpage structure ID of the target webpage, and if so, determine the Web application associated with the target webpage based on the Web application feature information corresponding to the matching webpage structure ID in the Web application fingerprint database.

5. A device for recognizing similar structures on web pages, characterized in that, include: The first module is used to store web page data as a byte array. The second module is used to sequentially traverse each byte in the byte array, determine the starting position of each HTML tag in the byte array, first write the byte data value corresponding to the starting position of each HTML tag in the byte array into a predetermined hash function, and then sequentially traverse the subsequent bytes at the starting position. If the value of the subsequent byte is an English letter, then write the value of the subsequent byte into the predetermined hash function until the value of the subsequent byte contains the end symbol of the HTML tag name, then stop traversing. The third module is used to call the hash function to generate a hash digest for each HTML tag, and use the set of hash digests as the webpage structure ID; The fourth module is used to determine the similarity between two web pages or two websites by comparing their web page structure IDs; or, by comparing the matching degree between the target web page structure ID and the corresponding web page structure ID stored in the Web application fingerprint database, to identify the Web application associated with the target web page.

6. The webpage similarity structure recognition device according to claim 5, characterized in that, The second module is further used for: Starting from the first byte in the byte array, check if the current byte value is a left angle bracket. If it is, check if the next adjacent byte value is an English letter. If the next adjacent byte value is an English letter, then the byte corresponding to the left angle bracket is taken as the starting position of the HTML tag in the byte array. Otherwise, the byte corresponding to the left angle bracket is not taken as the starting position of the HTML tag in the byte array, and the process continues to traverse the next byte value to check if it is a left angle bracket. Repeat the above steps until all bytes in the byte array have been traversed.

7. The webpage similarity structure recognition device according to claim 5, characterized in that, The fourth module is further used for: Generate a first webpage structure ID and a second webpage structure ID, and compare whether the first webpage structure ID and the second webpage structure ID are the same. If they are the same, the first webpage and the second webpage are determined to be similar in structure. Alternatively, generate a first set of structure IDs for the webpage set of the first website and a second set of structure IDs for the webpage set of the second website, compare the number of identical structure IDs in the first set of structure IDs and the second set of structure IDs, and if the number of identical structure IDs is greater than or equal to a preset threshold, then determine that the first website and the second website have similar structures.

8. The webpage similarity structure recognition device according to claim 5, characterized in that, The fourth module is further used for: Obtain multiple typical webpage structure IDs from a web application sample; Multiple typical webpage structure IDs of each web application sample are associated with and stored in the web application fingerprint database along with the corresponding web application feature information; Obtain the webpage structure ID of the target webpage, determine whether there is a webpage structure ID in the Web application fingerprint database that matches the webpage structure ID of the target webpage, and if so, determine the Web application associated with the target webpage based on the Web application feature information corresponding to the matching webpage structure ID in the Web application fingerprint database.

Images