A method for discriminating dynamic and static web page types based on web page structure

Abstract

This application relates to the field of data processing technology, and discloses a method for discriminating dynamic and static web page types based on web page structure. The method includes: first collecting dynamic and static web page samples and analyzing their structural features, constructing dynamic and static web page recognition instructions based on this, and generating a web page type discriminator; then using the discriminator to execute two types of recognition instructions on the target structure of the web page to be recognized, and respectively obtaining the confidence levels of it belonging to dynamic and static web pages; finally, based on the level of confidence or comparison with a preset threshold, accurately discriminating the type of the web page to be recognized, providing an accurate basis for multi-scenario applications. This application breaks through the limitations of traditional methods that only rely on URL rules or simple request screening, constructs recognition instructions and a discriminator starting from the web page structure features, conducts targeted structural analysis on dynamic and static web pages, effectively avoids misjudgments caused by special situations such as pseudo-static web pages, and improves the accuracy of discrimination.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular, to a method for discriminating dynamic and static web page types based on web page structure. Background Art

[0002] With the rapid development of Internet technology, web pages have become an important carrier for information dissemination, interaction, and application deployment. Currently, web pages are mainly divided into two categories: static web pages and dynamic web pages. Among them, static web pages are usually fixed Html files, and the content of the web page remains basically unchanged after creation and can be directly transmitted to the client browser without server-side processing; while dynamic web pages generate content dynamically through server-side rendering technologies (such as PHP, ASP.NET, JSP) or client-side dynamic scripts (such as JavaScript), and their content can be updated in real time according to user behavior or other conditions.

[0003] In practical applications, the demand for discriminating web page types in various scenarios is gradually increasing. For example, in data collection, it is necessary to distinguish between dynamic and static web page types to adjust the collection strategy and optimize the allocation of collection resources; in search engine optimization (SEO), it is necessary to distinguish between dynamic web pages and static web pages to optimize the crawling efficiency of crawlers; in a network content distribution system, it is necessary to adopt different strategies to store and cache these two types of web pages; in the field of security detection, it is also necessary to focus on analyzing the vulnerabilities and threats that may be introduced during the processing of dynamic web pages. Therefore, quickly and accurately determining whether a web page is a static web page or a dynamic web page has become an important technical issue.

[0004] In the prior art, the distinction between dynamic and static web pages usually depends on web page URL rules (such as whether it contains dynamic parameters) or simple screening of web page requests. These methods lack a comprehensive analysis of the web page structure itself and are easily affected by misjudgment. Especially in some websites using pseudo-static technology, traditional discrimination methods are difficult to meet the requirements. Summary of the Invention

[0005] In view of this, an embodiment of this application provides a method for discriminating dynamic and static web page types based on web page structure.

[0006] According to one aspect of this application, a method for discriminating dynamic and static web page types based on web page structure is provided, including:

[0007] Obtaining a plurality of dynamic web page samples corresponding to the dynamic web page type and a plurality of static web page samples corresponding to the static web page type, and respectively performing feature analysis on the dynamic web page samples and the static web page samples to determine the dynamic web page structure features corresponding to the dynamic web page type and the static web page structure features corresponding to the static web page type;

[0008] Construct dynamic web page recognition instructions corresponding to the dynamic web page structure features and static web page recognition instructions corresponding to the static web page structure features, and generate a web page type discriminator based on the dynamic web page recognition instructions and the static web page recognition instructions;

[0009] Through the web page type discriminator, respectively execute the dynamic web page recognition instructions and the static web page recognition instructions on the target web page structure corresponding to the web page to be recognized, so as to determine the dynamic web page confidence level of the web page to be recognized belonging to the dynamic web page type and the static web page confidence level of belonging to the static web page type;

[0010] Discriminate the web page type of the web page to be recognized according to the dynamic web page confidence level and the static web page confidence level.

[0011] According to another aspect of the present application, a storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method for discriminating dynamic and static web page types based on the web page structure as described above is implemented.

[0012] According to still another aspect of the present application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, and when the processor executes the program, the method for discriminating dynamic and static web page types based on the web page structure as described above is implemented.

[0013] By means of the above technical solutions, a method for discriminating dynamic and static web page types based on the web page structure provided by the embodiments of the present application first collects dynamic and static web page samples and analyzes their structure features, constructs dynamic and static web page recognition instructions based on this and generates a web page type discriminator; then uses this discriminator to execute two types of recognition instructions on the target structure of the web page to be recognized, and respectively obtains its confidence levels of belonging to dynamic and static web pages; finally, based on the high or low confidence levels or comparison with a preset threshold, accurately discriminates the web page type to be recognized, providing an accurate basis for multi-scenario applications. The present application breaks through the limitations of traditional methods that only rely on URL rules or simple request screening, starts from the web page structure features, constructs recognition instructions and discriminators, conducts targeted structure analysis on dynamic and static web pages, effectively avoids misjudgments caused by special situations such as pseudo-static web pages, and improves the discrimination accuracy.

[0014] The above description is only an overview of the technical solutions of the present application. In order to be able to understand the technical means of the present application more clearly, it can be implemented according to the content of the specification. And in order to make the above and other purposes, features and advantages of the present application more obvious and understandable, the specific embodiments of the present application are specifically described below. Brief Description of the Drawings

[0015] The accompanying drawings described herein are used to provide a further understanding of the present application and form a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings:

[0016] Figure 1 A flowchart showing a method for discriminating dynamic and static web page types based on web page structure provided by an embodiment of the present application is shown. Detailed implementation manners

[0017] The present application will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments. It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments may be combined with each other.

[0018] In this embodiment, a method for discriminating dynamic and static web page types based on web page structure is provided. As Figure 1 shown, the method includes:

[0019] Step 101: Obtain a plurality of dynamic web page samples corresponding to the dynamic web page type and a plurality of static web page samples corresponding to the static web page type, and perform feature analysis on the dynamic web page samples and the static web page samples respectively to determine the dynamic web page structure features corresponding to the dynamic web page type and the static web page structure features corresponding to the static web page type.

[0020] Step 102: Construct a dynamic web page recognition instruction corresponding to the dynamic web page structure feature and a static web page recognition instruction corresponding to the static web page structure feature, and generate a web page type discriminator based on the dynamic web page recognition instruction and the static web page recognition instruction.

[0021] Step 103: Through the web page type discriminator, execute the dynamic web page recognition instruction and the static web page recognition instruction on the target web page structure corresponding to the web page to be recognized respectively to determine the dynamic web page confidence corresponding to the web page to be recognized belonging to the dynamic web page type and the static web page confidence corresponding to the static web page type.

[0022] Step 104: Discriminate the web page type of the web page to be recognized according to the dynamic web page confidence and the static web page confidence.

[0023] The embodiment of the present application provides a method for accurately discriminating dynamic and static web page types based on web page structure to solve the problem in the prior art that methods such as relying on web page URL rules or simple request filtering are prone to misjudgment in scenarios such as pseudo-static technology websites due to the lack of a comprehensive analysis of the web page structure itself.

[0024] First, collect multiple sample web pages corresponding to dynamic web pages and static web pages respectively. Dynamic web page samples are web pages that dynamically generate content through server-side rendering technologies (such as PHP, ASP.NET, JSP) or client-side dynamic scripts (such as JavaScript); static web page samples are Html files with basically fixed content that can be transmitted to the client browser without server-side processing. Perform feature analysis on the collected dynamic web page samples and static web page samples respectively to find out the structural features that can represent the characteristics of dynamic web pages and static web pages respectively. For example, dynamic web pages may contain structural features such as specific script tags and interface calls for dynamically loading content; static web pages may have relatively fixed layout tags and simple document structures and other features. Through feature analysis, determine the dynamic web page structural features corresponding to the dynamic web page type and the static web page structural features corresponding to the static web page type.

[0025] Next, according to the determined dynamic web page structural features, construct corresponding dynamic web page recognition instructions, which are a series of rules or operations used to determine whether a web page conforms to the dynamic web page structural features, such as checking whether there are specific script tags in the web page and whether it contains API calls for dynamic data loading. Similarly, according to the static web page structural features, construct corresponding static web page recognition instructions, which are used to determine whether a web page conforms to the static web page structural features, such as checking whether the layout structure of the web page conforms to the common patterns of static web pages. Based on the constructed dynamic web page recognition instructions and static web page recognition instructions, generate a web page type discriminator, and use these recognition instructions to perform structural feature analysis on the input web page, so as to determine the web page type.

[0026] Finally, for the web page to be recognized, through the web page type discriminator, respectively execute the dynamic web page recognition instructions and static web page recognition instructions on the target web page structure corresponding to the web page to be recognized. After executing the dynamic web page recognition instructions, according to the matching situation of the instructions, determine the confidence level of the web page to be recognized belonging to the dynamic web page type. The confidence level is a value between 0 and 1, and the higher the value, the more the web page to be recognized conforms to the structural features of the dynamic web page, and the greater the possibility of belonging to the dynamic web page. After executing the static web page recognition instructions, similarly according to the matching situation of the instructions, determine the confidence level of the web page to be recognized belonging to the static web page type. Thus, according to the dynamic web page confidence level and the static web page confidence level, determine the web page type of the web page to be recognized. For example, a threshold can be set. When the dynamic web page confidence level is higher than this threshold, determine that the web page to be recognized is a dynamic web page; when the static web page confidence level is higher than this threshold, determine that the web page to be recognized is a static web page. It is also possible to compare the magnitudes of the dynamic web page confidence level and the static web page confidence level, and take the web page type corresponding to the higher confidence level as the type of the web page to be recognized.

[0027] By applying the technical solution of this embodiment, first collect dynamic and static web page samples and analyze their structural characteristics, and accordingly construct dynamic and static web page recognition instructions and generate a web page type discriminator; then use this discriminator to execute two types of recognition instructions on the target structure of the web page to be recognized, and respectively obtain the confidence levels of its belonging to dynamic and static web pages; finally, based on the high or low confidence levels or comparison with a preset threshold, accurately discriminate the type of the web page to be recognized, providing an accurate basis for multi-scenario applications. The embodiment of this application breaks through the limitation of traditional methods that only rely on URL rules or simple request filtering. Starting from the structural characteristics of web pages, it constructs recognition instructions and a discriminator, conducts targeted structural analysis on dynamic and static web pages, effectively avoids misjudgments caused by special situations such as pseudo-static web pages, and improves the accuracy of discrimination.

[0028] In the embodiment of this application, optionally, perform feature analysis on the static web page samples to determine the static web page structural characteristics corresponding to the static web page type, including: perform feature analysis on the static web page sample structures of each static web page sample to determine at least one unified static feature possessed by the static web page sample structures; wherein, the unified static feature is a feature with a probability of occurrence greater than a first preset value in each static web page sample structure; the unified static feature includes a standard feature and a variable feature, the standard feature is a fixed feature, and the variable feature includes a fixed-length feature and a non-fixed-length feature; determine the static web page structural characteristics corresponding to the static web page type according to the unified static feature.

[0029] In this embodiment, feature analysis is carried out on the web page structures of each collected static web page sample, and structural features that can represent the static web page type are mined from these sample structures to determine the static web page structural features. When analyzing the static web page sample structure, features with an occurrence probability greater than the first preset value in each static web page sample structure are found, that is, unified static features. That is to say, these features are prevalent in most static web page samples and have a certain representativeness. The unified static features are further subdivided into standard features and variable features. The standard features are fixed and unchanged. No matter in which static web page, as long as it conforms to the static web page type, this feature remains consistent and has absolute stability. The variable features include fixed-length features and non-fixed-length features. Although the fixed-length features have variations in their manifestation forms, their lengths are fixed. For example, for some specific tag combinations, the number of elements they contain is fixed; the non-fixed-length features have variations in both their manifestation forms and lengths, but they still belong to the common features in the static web page structure. For example, for some blocks used to display content, the content length will vary depending on the web page content, but the structural pattern belongs to the category of static web page features. For example, the unified static feature includes ABAC, where A represents the standard feature, B represents the fixed-length feature, and C represents the non-fixed-length feature. According to the obtained unified static features (including standard features and variable features), the static web page structural features corresponding to the static web page type are comprehensively determined, and these features will be used as the key basis for subsequent discrimination of static web pages. By deeply analyzing the static web page sample structure in this embodiment of the application, unified static features with a high occurrence probability are mined and carefully classified into standard features and variable features, and then the static web page structural features are determined. On the one hand, the standard features ensure the stability of the basis for static web page discrimination, and the variable features fully consider the diversity of the static web page structure. The combination of the two enables the determined static web page structural features to comprehensively and accurately reflect the essential structural characteristics of static web pages; on the other hand, discrimination based on these accurate and comprehensive features can effectively improve the accuracy and reliability of static web page discrimination.

[0030] In an embodiment of the present application, optionally, feature analysis is performed on the dynamic web page sample to determine the dynamic web page structure features corresponding to the dynamic web page type, including: dividing the dynamic web page sample content of each dynamic web page sample into dynamic web page sample content segments matching each dynamic web page structure tag according to a plurality of dimensions of preset dynamic web page structure tags, where the dynamic web page structure tags include URL tags and multiple page source code tags, and the dynamic web page sample content includes a dynamic web page sample URL and a dynamic web page sample source code; for each dynamic web page structure tag, performing feature analysis on the respective dynamic web page sample content segments corresponding to the dynamic web page structure tag to obtain the dynamic web page structure tag features corresponding to the dynamic web page structure tag; and summarizing the dynamic web page structure tag features of each dynamic web page structure tag to obtain the dynamic web page structure features corresponding to the dynamic web page type.

[0031] In this embodiment, a plurality of dimensions of dynamic web page structure tags are preset, and these tags are used to describe and analyze the dynamic web page structure from different perspectives. The dynamic web page structure tags in this embodiment include URL tags and a plurality of page source code tags, while the dynamic web page sample content covers the dynamic web page sample URL and the dynamic web page sample source code. According to these preset dynamic web page structure tags, the content of each dynamic web page sample is disassembled into content segments that match the respective dynamic web page structure tags. For example, according to the URL tag, the part related to the URL in the sample is extracted as a content segment; according to different page source code tags (such as script tags, data loading interface tags, etc.), the corresponding parts in the sample source code are respectively extracted as different content segments. Further, for each dynamic web page structure tag, feature analysis is carried out on the respective dynamic web page sample content segments corresponding to the tag. For example, for the content segment corresponding to the URL tag, analyze whether the URL contains dynamic parameters (such as the form of?id=123), the number and type of parameters, etc.; for the content segment corresponding to the script tag, analyze the type of the script (such as different libraries of JavaScript), the position of the script in the page, the specific functions called in the script, etc.; for the content segment corresponding to the data loading interface tag, analyze the call method of the interface (such as AJAX call), the data format returned by the interface, etc. Through such analysis, the dynamic web page structure tag features corresponding to each dynamic web page structure tag are obtained. Finally, the dynamic web page structure tag features corresponding to the respective dynamic web page structure tags are summarized and integrated, so as to obtain the dynamic web page structure features that can comprehensively represent the dynamic web page type. In the embodiment of the present application, by presetting multi-dimensional dynamic web page structure tags, the dynamic web page sample content is accurately divided into content segments corresponding to different tags, and feature analysis is respectively carried out on each tag content segment, and finally the dynamic web page structure features are summarized. Therefore, with multi-dimensional tag analysis and comprehensive feature summarization, it can flexibly respond and accurately identify dynamic web pages, providing reliable and accurate feature support for subsequent applications in scenarios such as data collection, web page classification, and security detection.

[0032] In a specific application scenario, the page source code tags include dynamic page technology tags, front-end framework tags, page resource import tags, Body node tags, Script tags, and dynamic script execution tags. By analyzing the dynamic web page structure features corresponding to the dynamic web page structure tags, the following characteristics can be summarized: 1. By obtaining the content fragment corresponding to the dynamic page technology tag, the page source code fragment corresponding to the DOM structure can be obtained. When analyzing that it contains dynamic technology page tags such as jsp, jstl, php, asp of the dynamic page technology, the web page may belong to a dynamic page. 2. By obtaining the content fragment corresponding to the front-end framework tag and obtaining the page source code fragment corresponding to the DOM structure analysis, when analyzing that it contains front-end framework tags such as Vue and React, the web page may belong to a dynamic page. At this time, the corresponding element attributes and dynamic behaviors can be further analyzed to determine whether it has dynamic rendering characteristics. If it meets the requirements, it is determined as a dynamic page. 3. By obtaining the content fragment corresponding to the page resource import tag, when analyzing that there are front-end framework script import characteristics such as vue.js and react.js, the web page may belong to a dynamic page. 4. By obtaining the content fragment corresponding to the Body node tag and analyzing the Body node in the DOM, it can be known that if the content in the Body is empty, the Meta tag contains obfuscation attributes, and the page source code contains an inlined JavaScript obfuscated Base64 encrypted string, then the page meets the client-side rendering characteristics of dynamic data encryption and asynchronous content loading in SPA, and the web page may belong to a dynamic page. 5. By obtaining the content fragment corresponding to the Script tag and analyzing the Script tag in the DOM, if there are characteristics of data interaction using technologies such as XMLHttpRequest and Ajax in the inlined Script script, the web page may belong to a dynamic page. 6. By obtaining the content fragment corresponding to the dynamic script execution tag and analyzing the dynamic script execution situation, for example, there are behavioral characteristics of dynamic DOM node operations: methods such as document.write(), appendChild(), and insertBefore(). If such behaviors exist, it can be determined that the page has dynamic rendering capabilities on the client side, and the web page may belong to a dynamic page.

[0033] In an embodiment of the present application, optionally, feature analysis is performed on each dynamic web page sample content segment corresponding to the dynamic web page structure tag to obtain the dynamic web page structure tag features corresponding to the dynamic web page structure tag, including: performing feature analysis on each dynamic web page sample content segment corresponding to the dynamic web page structure tag to determine at least one unified dynamic feature in each dynamic web page sample content segment and at least one associated feature corresponding to the unified dynamic feature; wherein, the unified dynamic feature is a feature with an occurrence probability greater than a second preset value in each dynamic web page sample content segment; the unified dynamic feature includes a standard feature and a variable feature; the associated feature is a feature with an occurrence probability greater than a third preset value at the context position of the unified dynamic feature; and determining the dynamic web page structure features corresponding to the dynamic web page type according to the unified dynamic feature and the corresponding associated feature.

[0034] In this embodiment, feature analysis is carried out on each dynamic web page sample content segment corresponding to the dynamic web page structure tag. During the analysis process, features with an occurrence probability greater than the second preset value in each dynamic web page sample content segment are found, and these features are defined as unified dynamic features. This means that these features are generally present in the corresponding content segments of most dynamic web page samples and have high representativeness. The unified dynamic features are further divided into standard features and variable features. For each unified dynamic feature, features with an occurrence probability greater than the third preset value at its context position are further analyzed, and these features are called associated features. The associated features are closely related to the unified dynamic features and usually appear around the unified dynamic features to assist in describing the structural characteristics of the dynamic web page. For example, a specific delimiter or parameter value type (associated feature) following a certain dynamic parameter (unified dynamic feature). By synthesizing the unified dynamic features and the corresponding associated features, the structural features of the dynamic web page type in the dimension of this dynamic web page structure tag are determined. These features together constitute the structural feature set of the dynamic web page under this tag, providing a basis for comprehensively discriminating dynamic web pages subsequently. In the embodiment of the present application, the unified dynamic features ensure the stability of the basis for discriminating dynamic web pages, while the associated features further refine the understanding of the context of the dynamic web page structure. The combination of the two enables the determined dynamic web page structure features to more comprehensively and accurately reflect the essential structural characteristics of the dynamic web page in this tag dimension, improving the accuracy and reliability of dynamic web page discrimination and avoiding misjudgment or missed judgment caused by single or incomplete features.

[0035] In an embodiment of the present application, optionally, determining the static web page structure features corresponding to the static web page type according to the unified static features, and determining the dynamic web page structure features corresponding to the dynamic web page type according to the unified dynamic features includes: deleting the same features in the unified static features and the unified dynamic features, and performing the following operations based on the remaining unified static features and the remaining unified dynamic features: for each unified static feature, according to the unified static feature, each static web page sample structure, and each dynamic web page sample structure, statistically calculate the first probability that the web page structure where the unified static feature appears belongs to the static web page type; according to the first probability and the probability of the unified static feature appearing in each static sample structure, determine the credibility of the unified static feature; according to the unified static feature and the corresponding credibility, determine the static web page structure features corresponding to the static web page type; for each unified dynamic feature, according to the unified dynamic feature, the content fragments corresponding to the unified dynamic feature in each static web page sample structure, and the content fragments of each dynamic web page sample corresponding to the unified dynamic feature, statistically calculate the second probability that the web page structure where the unified dynamic feature appears belongs to the dynamic web page type; according to the second probability and the probability of the unified dynamic feature appearing in each dynamic web page sample content fragment, determine the credibility of the unified dynamic feature; according to the unified dynamic feature, the corresponding associated features, and the corresponding credibility, determine the dynamic web page structure features corresponding to the dynamic web page type.

[0036] In this embodiment, first, the same features in the unified static features and the unified dynamic features are deleted to avoid interference of duplicate features on subsequent analysis. The remaining unified static features and unified dynamic features will be used for the structural feature analysis of static web pages and dynamic web pages respectively. For each remaining unified static feature, according to this feature, the structures of each static web page sample, and the structures of each dynamic web page sample, the first probability that the web page structure where this unified static feature appears belongs to the static web page type is statistically calculated. This probability reflects the exclusivity and discrimination of this feature in static web pages. Combining the first probability and the probability that this unified static feature appears in the structures of each static sample, the credibility of this feature is calculated. The higher the credibility, the stronger the representativeness of this feature in static web pages and the lower the possibility of misjudgment. According to the unified static features and their corresponding credibilities, the structural features of the static web page type are comprehensively determined. For each remaining unified dynamic feature, according to this feature, the content segments corresponding to this feature in the structures of each static web page sample, and the content segments of each dynamic web page sample corresponding to this feature, the second probability that the web page structure where this unified dynamic feature appears belongs to the dynamic web page type is statistically calculated. This probability reflects the exclusivity and discrimination of this feature in dynamic web pages. Combining the second probability and the probability that this unified dynamic feature appears in the content segments of each dynamic web page sample, the credibility of this feature is calculated. According to the unified dynamic features, their corresponding associated features, and the corresponding credibilities, the structural features of the dynamic web page type are comprehensively determined. Features with high credibility and their associated features will be key considerations, thus ensuring the accuracy and comprehensiveness of the structural features of dynamic web pages. Deleting duplicate features in the embodiments of the present application avoids feature redundancy, ensures that the structural features of static and dynamic web pages are independent and have high discrimination, and reduces the possibility of misjudgment. By statistically calculating the probability of feature appearance and combining its performance in different types of web pages to calculate the credibility, more representative features are screened out, making the determined structural features of static and dynamic web pages more accurate and reliable.

[0037] In the embodiments of the present application, optionally, dynamic web page recognition instructions corresponding to the structural features of the dynamic web page and static web page recognition instructions corresponding to the structural features of the static web page are constructed, and a web page type discriminator is generated based on the dynamic web page recognition instructions and the static web page recognition instructions, including: sorting each unified static feature in the structural features of the static web page according to the credibility of each unified static feature, and constructing static web page recognition instructions based on the sorted unified static features and the corresponding credibilities; sorting each unified dynamic feature in the structural features of the dynamic web page according to the credibility of each unified dynamic feature, and constructing dynamic web page recognition instructions based on the sorted unified dynamic features, the corresponding associated features, and the corresponding credibilities; generating a web page type discriminator based on the dynamic web page recognition instructions, the static web page recognition instructions, and the dynamic web page structure tags.

[0038] In this embodiment, for each unified static feature in the static web page structure features, they are sorted according to their credibility, with features having high credibility ranked first and features having low credibility ranked last. Based on the sorted unified static features and their corresponding credibility, a static web page recognition instruction is constructed. This instruction checks, in the order of the credibility of the features, whether the structure of the web page to be recognized conforms to these static features and makes a comprehensive judgment in combination with the credibility. For each unified dynamic feature in the dynamic web page structure features, they are also sorted according to their credibility, with features having high credibility taking precedence. Based on the sorted unified dynamic features, their corresponding associated features, and the corresponding credibility, a dynamic web page recognition instruction is constructed. This instruction not only checks the dynamic features themselves but also considers their associated features to more comprehensively verify the structural characteristics of the dynamic web page and makes a judgment in combination with the credibility. The constructed dynamic web page recognition instruction, static web page recognition instruction, and the predefined dynamic web page structure tags are combined to generate a complete web page type discriminator. This discriminator can, according to the input web page structure, sequentially apply the static and dynamic recognition instructions, conduct a comprehensive analysis in combination with the structure tags, and finally determine the web page type. In the embodiment of the present application, according to the order of feature checking sorted by credibility, high-credibility features are verified first, improving the discrimination efficiency. Making a comprehensive judgment in combination with credibility, associated features, and structure tags can more comprehensively and accurately reflect the structural characteristics of the web page, reducing the possibility of misjudgment and missed judgment.

[0039] In an embodiment of the present application, optionally, the web page type discriminator is used to execute the dynamic web page recognition instruction and the static web page recognition instruction on the target web page structure corresponding to the web page to be recognized respectively, so as to determine the dynamic web page confidence level of the web page to be recognized belonging to the dynamic web page type and the static web page confidence level of the web page to be recognized belonging to the static web page type, including: obtaining the target web page structure of the web page to be recognized through the web page type discriminator, where the target web page structure includes a target URL and a target page source code; dividing the target web page structure into target dynamic web page content segments matching each dynamic web page structure tag; performing feature recognition on the target page source code in the order of each unified static feature through the static web page recognition instruction; if the target page source code hits a unified static feature with a credibility of 1, determining that the static web page confidence level of the web page to be recognized belonging to the static web page type is 1 and stopping the recognition; otherwise, continuing the feature recognition and recording the unified static features hit by the target page source code and the corresponding credibility, and determining the static web page confidence level of the web page to be recognized belonging to the static web page type according to the credibility corresponding to the hit unified static feature; performing feature recognition on the corresponding target dynamic web page content segments in the order of each unified dynamic feature through the dynamic web page recognition instruction; if the target dynamic web page content segment hits a unified dynamic feature with a credibility of 1, determining that the dynamic web page confidence level of the web page to be recognized belonging to the dynamic web page type is 1 and stopping the recognition; otherwise, continuing the feature recognition and recording the unified dynamic features hit by the target dynamic web page content segment and the corresponding credibility, and determining the dynamic web page confidence level of the web page to be recognized belonging to the dynamic web page type according to the credibility corresponding to the hit unified dynamic feature.

[0040] In this embodiment, the target web page structure of the web page to be recognized is obtained through a web page type discriminator, and the structure includes a target URL and target page source code. The target web page structure is divided into target dynamic web page content segments matching each tag according to preset dynamic web page structure tags for subsequent dynamic feature recognition. Further, according to the sorting of unified static features in the static web page recognition instruction (from high to low credibility), feature recognition is sequentially performed on the target page source code. If the target page source code hits a unified static feature with a credibility of 1, it is immediately determined that the web page to be recognized belongs to the static web page type, and the static web page confidence is 1, and further recognition is stopped. This is because the feature with a credibility of 1 has absolute representativeness and does not need to be further verified. If the feature with a credibility of 1 is not hit, continue the recognition and record the hit unified static features and their credibility. Finally, according to the credibility of all hit features, the static web page confidence that the web page to be recognized belongs to the static web page type is comprehensively calculated. At the same time, according to the sorting of unified dynamic features in the dynamic web page recognition instruction (from high to low credibility), feature recognition is sequentially performed on the corresponding target dynamic web page content segments. If the target dynamic web page content segment hits a unified dynamic feature with a credibility of 1, it is immediately determined that the web page to be recognized belongs to the dynamic web page type, and the dynamic web page confidence is 1, and further recognition is stopped. If the feature with a credibility of 1 is not hit, continue the recognition and record the hit unified dynamic features and their credibility, and consider their associated features at the same time. Finally, according to the credibility of all hit features and their associated features, the dynamic web page confidence that the web page to be recognized belongs to the dynamic web page type is comprehensively calculated. In the embodiment of the present application, features with high credibility are preferentially checked. Once hit, the web page type can be immediately determined, avoiding unnecessary subsequent recognition steps. For features with high credibility, the result is directly determined to ensure accuracy. For features with low credibility, the credibility of multiple features is comprehensively considered for determination, reducing the possibility of misjudgment.

[0041] In the embodiment of the present application, optionally, according to the dynamic web page confidence and the static web page confidence, the web page type of the web page to be recognized is discriminated, including: if the dynamic web page confidence is greater than a preset confidence or the static web page confidence is greater than the preset confidence, and the absolute value of the difference between the dynamic web page confidence and the static web page confidence is greater than a first preset difference, then it is determined that the web page type greater than the preset confidence is the web page type of the web page to be recognized; if the dynamic web page confidence is not greater than the preset confidence and the static web page confidence is not greater than the preset confidence, then the web page type of the web page to be recognized is predicted through a trained random forest model; otherwise, based on the recognition information of the web page to be recognized by the web page type discriminator, a web page type recognition request is generated, and the web page type recognition request is sent to an expert system for web page type recognition.

[0042] In this embodiment, if one of the dynamic web page confidence or the static web page confidence is greater than a preset confidence, and the absolute value of the difference between the dynamic web page confidence and the static web page confidence is greater than a first preset difference, then directly determine that the web page type greater than the preset confidence is the web page type of the to-be-identified web page. This condition applies to the situation where the confidence difference is significant, and it can quickly and clearly determine the web page type without further processing. If both the dynamic web page confidence and the static web page confidence are not greater than the preset confidence, it means that neither of them is sufficient to determine the web page type alone. At this time, a trained random forest model is used for prediction. If both the dynamic web page confidence and the static web page confidence do not meet the direct discrimination conditions, for example, both the dynamic and static web page confidences are greater than the preset confidence at the same time, or both are not greater than the preset confidence and the absolute value of the confidence difference is not greater than the first preset difference, then a web page type recognition request is generated based on the recognition information of the web page type discriminator, and this request is sent to the expert system for further recognition. The expert system can use domain knowledge and experience to deeply analyze complex or ambiguous cases and provide recognition results. In the embodiment of the present application, for the situation where the confidence difference is significant, the result can be directly determined, avoiding unnecessary calculations and delays. In the case where the confidences are all low or the difference is small, prediction and recognition are performed through the random forest model and the expert system to ensure the accuracy and reliability of the result.

[0043] In the embodiment of the present application, optionally, the training process of the random forest model includes: constructing decision trees respectively according to the unified dynamic features with a credibility not equal to 1 and the unified static features with a credibility not equal to 1 to form a to-be-trained random forest model, constructing a training set by using each static web page sample and each dynamic web page sample, and training the to-be-trained random forest model through the training set to obtain a trained random forest model;

[0044] Predicting the web page type of the to-be-identified web page through the trained random forest model includes: predicting the dynamic prediction confidence that the to-be-identified web page belongs to the dynamic web page type and the static prediction confidence that it belongs to the static web page type through the trained random forest model; if the difference between the dynamic prediction confidence and the static prediction confidence is greater than a second preset difference, and the dynamic web page confidence is greater than the static web page confidence, then determine that the to-be-identified web page is the dynamic web page type; if the difference between the static prediction confidence and the dynamic prediction confidence is greater than the second preset difference, and the static web page confidence is greater than the dynamic web page confidence, then determine that the to-be-identified web page is the static web page type; otherwise, based on the recognition information of the to-be-identified web page by the web page type discriminator and the recognition information of the to-be-identified web page by the random forest model, generate a web page type recognition request, and send the web page type recognition request to the expert system for web page type recognition;

[0045] The method further includes: receiving the web page type recognition result of the expert system for the web page to be recognized, and optimizing the web page type discriminator and the random forest model based on the web page type recognition result.

[0046] In this embodiment, decision trees are respectively constructed according to the unified dynamic features with a credibility not equal to 1 and the unified static features with a credibility not equal to 1. These features are used as model inputs to distinguish between dynamic and static web page types. Thus, a random forest model to be trained is formed by using multiple decision trees. Further, a training set is constructed using each static web page sample and each dynamic web page sample to ensure that the model can learn the feature patterns of different web page types. The random forest model to be trained is trained using the training set, and the model parameters are adjusted to optimize the prediction performance, and finally a trained random forest model is obtained. When predicting the web page type through the random forest model, specifically, the dynamic prediction confidence that the web page to be recognized belongs to the dynamic web page type and the static prediction confidence that it belongs to the static web page type are predicted using the trained random forest model. If the difference between the dynamic prediction confidence and the static prediction confidence is greater than the second preset difference, and the dynamic web page confidence is greater than the static web page confidence, it is determined that the web page to be recognized is of the dynamic web page type. If the difference between the static prediction confidence and the dynamic prediction confidence is greater than the second preset difference, and the static web page confidence is greater than the dynamic web page confidence, it is determined that the web page to be recognized is of the static web page type. If the difference in prediction confidence is not greater than the second preset difference, or the confidence size relationship is inconsistent with the size relationship given by the web page type discriminator, a web page type recognition request is generated based on the recognition information of the web page type discriminator and the random forest model, and sent to the expert system for further recognition. Additionally, based on the recognition result of the expert system, the web page type discriminator and the random forest model can be optimized, including adjusting feature weights, updating model parameters, or adding new training samples, to improve the accuracy of future predictions. Thus, by combining the random forest model and the expert system, efficient, accurate, and robust web page type recognition is achieved, providing reliable support for various web page processing tasks.

[0047] In the embodiment of the present application, optionally, the page source code tags include dynamic page technology tags, front-end framework tags, page resource import tags, Body node tags, Script tags, and dynamic script execution tags; predicting the web page type of the to-be-identified web page through the trained random forest model includes: if the dynamic web page recognition instruction recognizes that the target dynamic web page content segment hits the unified dynamic feature corresponding to the dynamic script execution tag, performing a simulated interactive access on the to-be-identified web page, and analyzing whether there is a two-way data flow feature in the to-be-identified page according to the DOM change of the to-be-identified page after the simulated interactive access. When it is confirmed that there is a two-way data flow feature, directly determining the to-be-identified web page as the dynamic web page type; otherwise, predicting the web page type of the to-be-identified web page through the trained random forest model.

[0048] In this embodiment, the page source code tags include multiple types, such as dynamic page technology tags (such as PHP, ASP, etc.), front-end framework tags (such as React, Vue, etc.), page resource import tags (such as CSS, JS file import), Body node tags, Script tags, and dynamic script execution tags (such as tags for executing JavaScript). When the web page type discriminator executes the dynamic web page recognition instruction, it checks whether the target dynamic web page content segment hits the unified dynamic feature corresponding to the dynamic script execution tag. If the unified dynamic feature of the dynamic script execution tag is recognized, a simulated interactive access is performed on the to-be-identified web page, for example, simulating the interaction between the user and the web page (such as clicking a button, filling out a form, etc.) to observe the response behavior of the web page. After the simulated interactive access, the DOM (Document Object Model) change of the to-be-identified page is analyzed. The two-way data flow feature usually manifests as the web page being able to dynamically update content after user interaction without completely reloading the page. If it is confirmed that there is a two-way data flow feature, the to-be-identified web page is directly determined as the dynamic web page type. This is because the two-way data flow is a typical feature of dynamic web pages, indicating that the web page has a high degree of interactivity and dynamic content update ability. If it is not confirmed that there is a two-way data flow feature, the web page type of the to-be-identified web page is predicted through the trained random forest model, and the random forest model can make a comprehensive judgment based on other features. In the embodiment of the present application, by identifying the dynamic script execution tag and performing a simulated interactive access, the core features of dynamic web pages (such as two-way data flow) can be directly captured, thereby improving the accuracy of dynamic web page recognition.

[0049] The embodiments of the present application further provide a computer device, which may specifically be a personal computer, a server, a network device, etc. The computer device includes a bus, a processor, a memory, and a communication interface, and may further include an input / output interface and a display device. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store location information. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the steps in the method embodiments are implemented.

[0050] Those skilled in the art can understand that the structure of the above computer device is only a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components, or combine certain components, or have different component arrangements.

[0051] In one embodiment, a computer-readable storage medium is provided. The computer-readable storage medium may be non-volatile or volatile, and stores a computer program. When the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0052] In one embodiment, a computer program product is provided, including a computer program. When the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0053] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or fully authorized by all parties.

[0054] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in the present application can be general-purpose processors, graphics processors, digital signal processors, programmable logics, data processing logics based on quantum computing, etc., without limitation.

[0055] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0056] The above-described embodiments merely represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for discriminating dynamic and static web page types based on web page structure, characterized in that Including: Obtain multiple dynamic web page samples corresponding to the dynamic web page type and multiple static web page samples corresponding to the static web page type, and perform feature analysis on the dynamic web page samples and the static web page samples respectively to determine the dynamic web page structure features corresponding to the dynamic web page type and the static web page structure features corresponding to the static web page type; Construct a dynamic web page recognition instruction corresponding to the dynamic web page structure features and a static web page recognition instruction corresponding to the static web page structure features, and generate a web page type discriminator based on the dynamic web page recognition instruction and the static web page recognition instruction; Through the web page type discriminator, execute the dynamic web page recognition instruction and the static web page recognition instruction on the target web page structure corresponding to the web page to be recognized respectively to determine the dynamic web page confidence corresponding to the web page to be recognized belonging to the dynamic web page type and the static web page confidence corresponding to the static web page type; According to the dynamic web page confidence and the static web page confidence, discriminate the web page type of the web page to be recognized, including: if the dynamic web page confidence is greater than the preset confidence or the static web page confidence is greater than the preset confidence, and the absolute value of the difference between the dynamic web page confidence and the static web page confidence is greater than the first preset difference, then determine that the web page type greater than the preset confidence is the web page type of the web page to be recognized; if the dynamic web page confidence is not greater than the preset confidence and the static web page confidence is not greater than the preset confidence, then predict the web page type of the web page to be recognized through the trained random forest model; otherwise, generate a web page type recognition request based on the recognition information of the web page to be recognized by the web page type discriminator, and send the web page type recognition request to the expert system for web page type recognition; The training process of the random forest model includes: respectively construct decision trees according to the unified dynamic features with a credibility not equal to 1 and the unified static features with a credibility not equal to 1 to form a random forest model to be trained, construct a training set by using each static web page sample and each dynamic web page sample, and train the random forest model to be trained through the training set to obtain the trained random forest model; Predicting the web page type of the to-be-identified web page through the trained random forest model, including: predicting the dynamic prediction confidence that the to-be-identified web page belongs to the dynamic web page type and the static prediction confidence that it belongs to the static web page type through the trained random forest model; if the difference between the dynamic prediction confidence and the static prediction confidence is greater than a second preset difference, and the dynamic web page confidence is greater than the static web page confidence, then determine that the to-be-identified web page is of the dynamic web page type; if the difference between the static prediction confidence and the dynamic prediction confidence is greater than the second preset difference, and the static web page confidence is greater than the dynamic web page confidence, then determine that the to-be-identified web page is of the static web page type; otherwise, based on the identification information of the to-be-identified web page by the web page type discriminator and the identification information of the to-be-identified web page by the random forest model, generate a web page type identification request, and send the web page type identification request to the expert system for web page type identification.

2. The method according to claim 1, wherein Performing feature analysis on the static web page samples to determine the static web page structure features corresponding to the static web page type, including: Performing feature analysis on the static web page sample structures of each static web page sample to determine at least one unified static feature possessed by the static web page sample structures; wherein, the unified static feature is a feature with an appearance probability greater than a first preset value in each static web page sample structure; the unified static feature includes a standard feature and a variable feature, the standard feature is a fixed feature, and the variable feature includes a feature with a fixed length and a feature with an unfixed length; Determining the static web page structure features corresponding to the static web page type according to the unified static feature.

3. The method according to claim 2, wherein Performing feature analysis on the dynamic web page samples to determine the dynamic web page structure features corresponding to the dynamic web page type, including: According to the dynamic web page structure tags of a preset plurality of dimensions, dividing the dynamic web page sample content of each dynamic web page sample into dynamic web page sample content segments matching each dynamic web page structure tag, wherein the dynamic web page structure tags include URL tags and a plurality of page source code tags, and the dynamic web page sample content includes a dynamic web page sample URL and a dynamic web page sample source code; For each dynamic web page structure tag, performing feature analysis on the respective dynamic web page sample content segments corresponding to the dynamic web page structure tag to obtain the dynamic web page structure tag features corresponding to the dynamic web page structure tag; Summarizing the dynamic web page structure tag features of each dynamic web page structure tag to obtain the dynamic web page structure features corresponding to the dynamic web page type.

4. The method according to claim 3, wherein Performing feature analysis on the respective dynamic web page sample content segments corresponding to the dynamic web page structure tag to obtain the dynamic web page structure tag features corresponding to the dynamic web page structure tag, including: Perform feature analysis on each dynamic web page sample content segment corresponding to the dynamic web page structure tag to determine at least one unified dynamic feature in each dynamic web page sample content segment and at least one associated feature corresponding to the unified dynamic feature; wherein, the unified dynamic feature is a feature with an occurrence probability greater than a second preset value in each dynamic web page sample content segment; the unified dynamic feature includes a standard feature and a variable feature; the associated feature is a feature with an occurrence probability greater than a third preset value at the context position of the unified dynamic feature. Determine the dynamic web page structure feature corresponding to the dynamic web page type according to the unified dynamic feature and the corresponding associated feature.

5. The method according to claim 4, wherein Determining the static web page structure feature corresponding to the static web page type according to the unified static feature, and determining the dynamic web page structure feature corresponding to the dynamic web page type according to the unified dynamic feature, includes: Delete the same features in the unified static feature and the unified dynamic feature, and perform the following operations based on the remaining unified static feature and the remaining unified dynamic feature: For each unified static feature, according to the unified static feature, each static web page sample structure, and each dynamic web page sample structure, count the first probability that the web page structure where the unified static feature appears belongs to the static web page type; according to the first probability and the occurrence probability of the unified static feature in each static sample structure, determine the credibility of the unified static feature. Determine the static web page structure feature corresponding to the static web page type according to the unified static feature and the corresponding credibility. For each unified dynamic feature, according to the unified dynamic feature, the content segment corresponding to the unified dynamic feature in each static web page sample structure, and each dynamic web page sample content segment corresponding to the unified dynamic feature, count the second probability that the web page structure where the unified dynamic feature appears belongs to the dynamic web page type; according to the second probability and the occurrence probability of the unified dynamic feature in each dynamic web page sample content segment, determine the credibility of the unified dynamic feature. Determine the dynamic web page structure feature corresponding to the dynamic web page type according to the unified dynamic feature, the corresponding associated feature, and the corresponding credibility.

6. The method according to claim 5, wherein Construct a dynamic web page recognition instruction corresponding to the dynamic web page structure feature and a static web page recognition instruction corresponding to the static web page structure feature, and generate a web page type discriminator based on the dynamic web page recognition instruction and the static web page recognition instruction, including: Sort each unified static feature based on the credibility of each unified static feature in the static web page structure feature, and construct a static web page recognition instruction based on the sorted unified static feature and the corresponding credibility. Sort each unified dynamic feature based on the credibility of each unified dynamic feature in the dynamic web page structure feature, and construct a dynamic web page recognition instruction based on the sorted unified dynamic feature, the corresponding associated feature, and the corresponding credibility. Generate a web page type discriminator based on the dynamic web page recognition instruction, the static web page recognition instruction, and the dynamic web page structure tag.

7. The method according to claim 6, wherein Through the web page type discriminator, respectively execute the dynamic web page recognition instruction and the static web page recognition instruction on the target web page structure corresponding to the web page to be recognized, to determine the dynamic web page confidence level of the web page to be recognized belonging to the dynamic web page type and the static web page confidence level of the web page to be recognized belonging to the static web page type, including: Through the web page type discriminator, obtain the target web page structure of the web page to be recognized, where the target web page structure includes a target URL and a target page source code; divide the target web page structure into target dynamic web page content segments that match each dynamic web page structure tag; Perform feature recognition on the target page source code in the order of each unified static feature through the static page recognition instruction; if the target page source code hits a unified static feature with a credibility of 1, determine that the static web page confidence level of the web page to be recognized belonging to the static web page type is 1 and stop the recognition; otherwise, continue with the feature recognition and record the unified static features hit by the target page source code and the corresponding credibility, and determine the static web page confidence level of the web page to be recognized belonging to the static web page type according to the credibility corresponding to the hit unified static feature; Perform feature recognition on the corresponding target dynamic web page content segments in the order of each unified dynamic feature through the dynamic page recognition instruction; if the target dynamic web page content segment hits a unified dynamic feature with a credibility of 1, determine that the dynamic web page confidence level of the web page to be recognized belonging to the dynamic web page type is 1 and stop the recognition; otherwise, continue with the feature recognition and record the unified dynamic features hit by the target dynamic web page content segment and the corresponding credibility, and determine the dynamic web page confidence level of the web page to be recognized belonging to the dynamic web page type according to the credibility corresponding to the hit unified dynamic feature.

8. The method according to claim 7, characterized in that, The method further includes: Receiving the web page type recognition result of the web page to be recognized by the expert system, and optimizing the web page type discriminator and the random forest model based on the web page type recognition result.

9. The method according to claim 8, wherein The page source code tags include dynamic page technology tags, front-end framework tags, page resource introduction tags, Body node tags, Script tags, dynamic script running tags; Predicting the web page type of the web page to be recognized through the trained random forest model, including: If the dynamic page recognition instruction recognizes that the target dynamic web page content segment hits the unified dynamic feature corresponding to the dynamic script running tag, perform a simulated interactive access on the web page to be recognized, and analyze whether there are two-way data flow features in the web page to be recognized according to the DOM change of the web page to be recognized after the simulated interactive access. When it is confirmed that there are two-way data flow features, directly determine the web page to be recognized as the dynamic web page type, otherwise predict the web page type of the web page to be recognized through the trained random forest model.

Images