Crawler algorithm configuration validity evaluation method and system oriented to network security measurement

Abstract

The invention provides a crawler algorithm configuration validity evaluation method and system oriented to network security measurement, and belongs to the technical field of network security measurement. Defining adjustable configuration and network security measurement indexes of a crawler algorithm; collecting and inputting a representative to-be-measured website set; generating a to-be-evaluated algorithm configuration scheme; performing two-stage page crawling on all the configuration schemes, wherein the two-stage page crawling comprises reference page crawling aiming at a search strategy and network measurement index data crawling aiming at configuration parameters; establishing a multi-dimensional evaluation model for network measurement index data, wherein the multi-dimensional evaluation model comprises index quantitative effectiveness analysis, index data similarity analysis and index parameter sensitivity analysis; and in combination with a multi-dimensional security measurement index, generating an optimal series of crawler configuration parameters. The core defects of poor technical scene adaptability and single measurement index dimension in existing crawler algorithm configuration evaluation are solved, and high-precision matching of algorithm parameter configuration and network security measurement requirements is realized.

Description

Technical Field

[0001] This invention relates to the field of network security measurement technology, and in particular to a method and system for evaluating the effectiveness of web crawler algorithm configuration for network security measurement. Background Technology

[0002] Network security measurement relies on web crawler technology to systematically measure and analyze the security behavior of large-scale websites. A web crawler is an automated data collection tool that iteratively traverses website links and collects information according to a preset algorithm configuration. It can obtain multi-dimensional state information of a website, providing crucial data support for in-depth analysis of web server security configurations, evaluation of the effectiveness of defense mechanisms, and research on tracking technologies such as cookies and web fingerprints.

[0003] With the continuous development of modern web technologies, dynamic content and complex front-end logic place higher demands on the page exploration capabilities of web crawlers. Web crawler algorithms involve complex parameter configurations, including but not limited to: page search strategies, browser state management methods, user interaction simulation methods, and crawling depth settings. In large-scale network measurement applications, using inefficient crawler algorithm configurations may lead to insufficient target page coverage or omission of critical page states with security vulnerabilities, ultimately compromising the accuracy and reliability of network security measurements. Therefore, establishing a systematic crawler configuration effectiveness evaluation mechanism and quantitatively analyzing the performance of various configuration schemes in real dynamic network environments has become a key technical prerequisite for improving the value of network measurement data and the reliability of results.

[0004] Therefore, in order to effectively improve the credibility and application value of network security measurement by selecting the optimal crawler configuration scheme through multi-dimensional evaluation methods, it is urgent to propose a crawler algorithm configuration effectiveness evaluation method and system for practical security scenarios. Summary of the Invention

[0005] In view of the above background, the present invention aims to provide a method and system for evaluating the effectiveness of web crawler algorithm configuration for network security measurement. The method and system evaluate the effectiveness of web crawler algorithm configuration by constructing web crawler configuration combinations covering a variety of typical parameters, performing large-scale empirical measurement experiments, and collecting multi-dimensional measurement index data, which can effectively improve the reliability, repeatability, and efficiency of network security measurement.

[0006] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides a method for evaluating the effectiveness of crawler algorithm configuration for network security measurement, comprising the following steps:

[0008] (1) Define the adjustable parameter configuration of the crawler algorithm and its value space, and define a set of network security measurement indicators for evaluating the effectiveness of the crawler configuration; the adjustable parameter configuration of the crawler algorithm includes at least the crawler search strategy, the browser state management strategy, and the user interaction simulation strategy.

[0009] (2) Input a set of websites to be measured, consisting of a large number of target website domain names;

[0010] (3) Based on the parameter configuration and values ​​defined in step (1), generate a set of crawler configuration schemes to be evaluated by combining Cartesian products;

[0011] (4) Perform a phased crawling operation on the set of websites to be measured, including:

[0012] Benchmark crawling phase: For each crawler search strategy, crawl the website under benchmark conditions of stateless management and no user interaction, collect raw measurement data in real time, and generate a set of benchmark pages and an ordered sequence of URLs;

[0013] Re-crawl phase: For each crawler configuration scheme, based on the baseline page set corresponding to its search strategy, browser state management strategy and user interaction simulation strategy are applied to re-crawl, and raw measurement data is collected in real time;

[0014] (5) Establish a multidimensional evaluation model to analyze the original measurement data in step (4) and generate evaluation results; the multidimensional evaluation model includes quantitative effectiveness analysis, similarity and uniqueness analysis, and parameter sensitivity analysis; the quantitative effectiveness analysis results include the strongest crawler search strategy and the crawler configuration scheme with the best collection capability for each indicator; the similarity and uniqueness analysis results include the crawler configuration scheme with the best collection diversity for each indicator; the parameter sensitivity analysis is to select the optimal crawler configuration scheme for each indicator based on the comprehensive quantitative effectiveness analysis, similarity and uniqueness analysis and output the optimal crawler search depth and page crawling timeout duration;

[0015] (6) Generate a comprehensive evaluation report based on the evaluation results, and output the optimal crawler configuration scheme and resource optimization parameters.

[0016] Furthermore, the adjustable parameters of the crawler algorithm also include crawler search depth and page crawling timeout duration.

[0017] Furthermore, the crawler search strategy includes a breadth-first search strategy, a depth-first search strategy, and a breadth-limited depth-first search strategy.

[0018] Furthermore, during the baseline crawling phase, invalid pages are skipped, and the uniqueness and order of the baseline page set are ensured.

[0019] Furthermore, a set of cybersecurity measurement metrics includes:

[0020] HTTP request behavior, including all HTTP request domains issued by the web pages accessed during the crawling process;

[0021] JavaScript Web API call behavior, including all browser JavaScript Web APIs called by the web pages accessed during the crawling process;

[0022] Cookie settings and modification behavior, including all cookies set, modified, or deleted by the visited web pages during the crawling process.

[0023] Furthermore, when collecting raw measurement data, for each type of indicator, the behavior triggered on the accessed webpage is fully recorded, including the timestamp, event type, and related context information (the specific object of the HTTP request, the name of the API called, and the cookies set or changed). All collected raw network measurement indicator data, as well as the detailed timestamp and context information of each behavior, are structured, stored, and automatically input into the next step.

[0024] Furthermore, the quantitative validity analysis includes:

[0025] For each crawler search strategy, based on the baseline page set corresponding to that search strategy, count the number of valid pages n in each baseline page set. i and with the domain name u i Forming binary pairs <n i ,u i >, where u i Let n represent the domain name of the i-th website. i This represents the number of valid pages crawled under the i-th website domain by a certain crawler search strategy; plot the cumulative distribution function of the number of pages covered by the crawler search strategy to evaluate the page discovery efficiency and determine the crawler search strategy with the strongest page discovery efficiency.

[0026] Furthermore, the total number of security measurement indicators collected from all website domains under each crawler configuration scheme and the median number of security measurement indicators were statistically analyzed. The total number of security measurement indicators includes the total number of HTTP request domains, the total number of JavaScript WebAPI calls, and the total number of Cookies set and modified. The relationship between the total number of data collected and the median under all configuration parameters for each security measurement indicator was plotted to obtain the crawler configuration scheme with the optimal collection capability for each indicator.

[0027] Furthermore, similarity and uniqueness analysis includes:

[0028] Calculate the similarity percentage between the two configuration schemes for a single domain:

[0029]

[0030] Among them, S k (c i ,u i ), S k (c j ,u i ) represent two web crawler configuration parameter schemes c respectively. i With c j In a single domain u i The measurement results of the safety measurement index k collected above, J k (c i ,c j ,u u ) represents a single domain name u i The two configuration schemes above c i c j The percentage of similarity;

[0031] Calculate the median similarity across all domains as the global domain similarity for that metric category:

[0032]

[0033] Among them, Sim k (c i ,c j () represents two web crawler configuration parameter schemes c i With c j In a single domain u i The similarity of security measurement index k collected above, where U represents the set of domain names in the set of websites to be measured;

[0034] Construct a similarity matrix M for the index k. k , where matrix element M k [i,j]=Sim k (c i ,c j The heatmap of the matrix is ​​generated, and the configuration scheme with low similarity to other configurations is extracted based on the heatmap. This configuration scheme can collect more diverse security measurement results, and the crawler configuration scheme with the optimal collection diversity for each index is obtained.

[0035] Furthermore, based on the results of the quantitative effectiveness analysis and the similarity and uniqueness analysis, a comprehensive score is calculated for each configuration scheme:

[0036]

[0037] Where Q(c)i ,k) indicates configuration c i The total number of collections for index k, S(c i ,k) indicates configuration c i The average similarity between the configuration and all other configurations, where ω is a weighting factor representing preference;

[0038] For a given safety measurement index k, the optimal configuration scheme is selected. for:

[0039]

[0040] Furthermore, the parameter sensitivity analysis includes:

[0041] For configuration The raw measurement data under the crawler settings are used to extract the changes of three types of security measurement indicators with crawler search depth and page crawling timeout duration. The static threshold quantification analysis method is used to evaluate the impact of these two types of parameters on the integrity and efficiency of data collection, and output the parameters that minimize resource consumption while ensuring the integrity of measurement indicator collection.

[0042] Secondly, the present invention provides a web crawler algorithm configuration effectiveness evaluation system for network security measurement, which is used to implement the above-mentioned web crawler algorithm configuration effectiveness evaluation method for network security measurement.

[0043] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0044] (1) This invention proposes a method and system for evaluating the effectiveness of web crawler algorithm configuration for network security measurement. The method and system proposed in this invention evaluate the effectiveness of the core parameter configuration of web crawler algorithms. It can objectively and accurately reflect the data capture capability and effectiveness of various crawler configurations in real, complex and dynamic network environments. It solves the problems of single evaluation scenarios and insufficient indicator dimensions in existing technologies. At the same time, the evaluation results obtained by this method have time stability and robustness.

[0045] (2) This invention proposes a phased large-scale website crawling technology. This technology effectively reduces the computational and network resources consumed by repeatedly crawling massive websites with multiple configurations through two core stages: baseline page set collection and indicator data collection. It also eliminates measurement biases introduced by page differences and ensures the comparability, reproducibility, and credibility of evaluation results between configuration schemes.

[0046] (3) This invention proposes a comprehensive evaluation model for crawler effectiveness based on multi-dimensional security measurement indicators. This model evaluates the configuration effectiveness from multiple dimensions, including the quantitative effectiveness of security indicator collection, similarity and uniqueness, and sensitivity analysis of key parameters. It can generate evaluation results and crawler configuration suggestions that are highly matched with the actual needs of security measurement, and effectively guide the selection of the optimal methodology based on the specific measurement objectives of network security. Attached Figure Description

[0047] Figure 1 A flowchart of a web crawler algorithm configuration effectiveness evaluation method for network security measurement according to the present invention;

[0048] Figure 2 This is a flowchart illustrating the phased page crawling process for a large-scale website collection according to the present invention.

[0049] Figure 3 Flowchart of the multidimensional evaluation model guided by the safety measurement indicators of this invention;

[0050] Figure 4 This is a diagram illustrating the overall architecture of a web crawler algorithm configuration effectiveness evaluation system for network security measurement, as described in this invention. Detailed Implementation

[0051] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the embodiments described below are intended to facilitate the understanding of the present invention and do not limit it in any way.

[0052] This invention provides a method for evaluating the effectiveness of web crawler algorithm configurations for network security measurement. The specific process is as follows: Figure 1 As shown, the specific implementation of each step is as follows:

[0053] Step 0: Define the adjustable parameter configuration of the crawler algorithm to be evaluated in this invention and its corresponding value space, and define a set of measurement indicators for quantitatively evaluating website network security risks.

[0054] Specifically, this embodiment aims to evaluate five core parameter configurations that influence crawler behavior and measurement results, including crawler search strategy, browser state management strategy, user interaction simulation strategy, crawler search depth, and page crawling timeout duration. The crawler algorithm parameter configurations to be evaluated can also be designed based on the needs of those skilled in the art and the network measurement task.

[0055] The crawler search strategy refers to the control algorithm by which the crawler program starts from the website homepage and traverses and accesses pages within the site according to specific rules. In this embodiment, the selectable values ​​of the strategy parameters include the following three: Breadth-First Search (BFS), which prioritizes accessing all pages at the current level before moving to the next level; Depth-First Search (DFS), which delves deeper along a branch path until the termination condition is triggered before backtracking; and Breadth-Limited Depth-First Search (BFSL), which sets breadth-limiting conditions on the basis of depth-first access to control the expansion range of the crawling path. In this embodiment, the breadth-limiting factor of the BFSL algorithm is set to 3.

[0056] The browser state management strategy refers to whether the crawler retains the browser runtime state when the page jumps or switches during the exploration and information collection process of pages under the same domain. In this embodiment, the selectable values ​​of the strategy parameter include the following two: stateful, that is, the browser context state is continuously maintained during the page switching process; stateless, that is, the browser environment is reinitialized every time a page is accessed, and the state information of the previous page is not retained.

[0057] The user interaction simulation strategy refers to whether the crawler, after the page is loaded, simulates real user actions to trigger the loading of dynamic content and the response of third-party components on the webpage. In this embodiment, the optional values ​​of the strategy parameter include the following two: interaction, that is, simulating user actions after the page is loaded. In this embodiment, the simulated user actions include scrolling the page and periodically sending the Tab key to switch focus; no interaction, that is, only statically loading and analyzing the page content without performing any form of user behavior simulation.

[0058] The crawler search depth refers to the maximum number of internal pages crawled under each domain name. In this embodiment, it is set to a constant value of 50.

[0059] The page crawling timeout refers to the time the crawler stays on each page; in this embodiment, it is set to a constant value of 30 seconds.

[0060] Specifically, this embodiment uses three types of measurement indicators to quantitatively assess website network security risks: HTTP request behavior, JavaScript Web API call behavior, and Cookies setting and modification behavior.

[0061] The HTTP request behavior refers to all HTTP requests initiated by the page itself and its loaded resources (including HTML files, images, and JavaScript scripts) when a web crawler accesses a webpage. This metric is used to comprehensively capture and analyze HTTP requests.

[0062] The JavaScript Web API call behavior refers to the various Web APIs called by the JavaScript script of a webpage during execution. This metric is used to monitor Web API calls that are used to illegally track security or user behavior.

[0063] The cookie setting and modification behavior refers to various cookie operations performed by page scripts or HTTP response headers when a crawler accesses a webpage. This metric uses a unique "name-domain pair" as the basic unit of analysis and uses a machine learning classifier to identify and analyze cookies used for advertising or cross-site tracking purposes.

[0064] Step 1: Collect and input the dataset of the website to be tested.

[0065] In one specific embodiment of the present invention, in order to ensure that the evaluation process is representative and universal, this embodiment selects the top 10,000 website domains in the latest statistical period based on the Chrome User Experience Report (CrUX) to construct a dataset of websites to be tested. This dataset can accurately reflect the mainstream usage of current Internet websites and provide a representative test basis for subsequent network security measurements.

[0066] Step 2: Configure the core parameters of the three types of crawler algorithms defined in this embodiment, including all values ​​of the crawler search strategy, browser state management strategy and user interaction simulation strategy, and perform Cartesian product combination to generate a total of 12 different crawler configuration schemes (3×2×2=12). Input these schemes into the next step to construct crawler instances with different behavioral characteristics.

[0067] Step 3: Based on the three types of network security measurement indicators defined in Step 0, configure the automated measurement platform OpenWPM and its integrated Firefox browser, and combine the 12 crawler configuration schemes generated in Step 2 to perform phased page crawling operations on the dataset of the website under test, and collect the raw measurement data corresponding to each crawler configuration scheme and input it into the next step.

[0068] In a specific embodiment of the present invention, the staged page crawling process used in this step includes the following sub-steps, such as... Figure 2 As shown:

[0069] Sub-step 301: Use a benchmark search strategy to crawl the set of benchmark pages corresponding to each website domain in the dataset of websites to be tested;

[0070] Specifically, for each search strategy (breadth-first search, depth-first search, and breadth-first search with limited breadth), a unified stateless and non-interactive initial parameter configuration was used for large-scale crawling on the test website dataset. There were three crawler configuration schemes that met the conditions. During the crawling process, starting from the homepage of a website domain, for each valid internal page visited, the corresponding valid URL was recorded in real time, and the order in which the pages were visited was recorded to form a unique ordered sequence of URLs for that domain. The set of valid URLs and the URL sequence constitute the baseline page set obtained from crawling that domain under a specific crawler configuration scheme. This baseline page set serves as the unique input for all crawler configuration schemes using the same search strategy in subsequent measurements.

[0071] In this embodiment, if the accessed page is an invalid page, including 404, redirection to an external page, or page that cannot be loaded, the statistics are skipped, and only the URLs of valid and reachable internal pages are recorded; for URLs that appear multiple times, only the order of the first access is retained to ensure the uniqueness and order of the baseline page set.

[0072] In sub-step 302, for the remaining 9 crawler configuration schemes generated in step 2, firstly, based on the search strategy corresponding to each crawler configuration scheme, the baseline page set generated in step 301 for that search strategy is loaded. Then, for each crawler configuration scheme, the corresponding environment configuration is performed in the crawler framework according to its preset browser state parameters (e.g., stateful or stateless) and user interaction mode (e.g., simulating user scrolling, Tab key, no interaction, etc.). Next, for each valid URL in the baseline page set, each page is accessed sequentially according to a unique URL order and re-crawled. During the re-crawl process, the automated measurement framework monitors and records the raw data of the measurement indicators related to the three types of measurement indicators defined in step 1 in real time, including: all HTTP request domains issued by the accessed web pages during the crawling process, all browser JavaScript Web APIs called by the accessed web pages during the crawling process, and all cookies set, modified, or deleted by the accessed web pages during the crawling process. For each type of behavior triggered in the accessed web pages, the timestamp, event type, and related context information (e.g., the specific object of the HTTP request, the name of the API called, and the cookies set or modified) are fully recorded. All collected raw network measurement data, as well as detailed timestamps and contextual information for each behavior, are structured, stored, and automatically input into the next step of the analysis process.

[0073] Step 4: Establish a multi-dimensional evaluation model for the raw data of measurement indicators collected from the 12 crawler configuration schemes, and conduct a horizontal comparative analysis between the crawler configuration schemes.

[0074] In a specific embodiment of the present invention, the multi-dimensional evaluation model described in this step includes the following sub-steps, such as... Figure 3 As shown:

[0075] Sub-step 401 involves performing a quantitative effectiveness analysis on the quantifiable security measurement indicators in the raw measurement data. For the raw data of measurement indicators collected under different crawler configuration schemes, data visualization processing and systematic quantitative comparison are performed to evaluate the advantages and disadvantages of different crawler configuration schemes in terms of page discovery efficiency and security indicator collection capabilities.

[0076] Specifically, it includes the following steps:

[0077] (1) For each crawler search strategy (DFS, BHS and BFSL), based on the baseline page set obtained in step 301 for all website domains, count the number of valid pages n in each baseline page set. i and with the domain name u i Forming binary pairs <n i ,u i >, where u i Let n represent the domain name of the i-th website. i This represents the number of valid pages crawled under the i-th website domain by a certain crawler search strategy; it constructs a set of quantified page coverage results corresponding to this search strategy for all domains.<n1,u1> ,<n2,u2> ,…, <n n ,u n >|Search=BFS|DFS|BFSL}.

[0078] Subsequently, a cumulative distribution function graph of the number of pages covered was plotted for each crawler search strategy. By comparing the curves corresponding to different crawler search strategies, the efficiency of each crawler search strategy in finding effective pages in large-scale crawling was evaluated, and the search strategy with the strongest page discovery efficiency was determined.

[0079] (2) For the 12 crawler configuration schemes generated in this embodiment, based on the raw data of the measurement indicators collected in step 302, the total number of security measurement indicators collected from all website domains under each crawler configuration scheme and the median number of security measurement indicators are calculated. The total statistics include the total number of HTTP request domains, the total number of JavaScript Web API calls, and the total number of Cookies set and modified. The relationship between the total number of data collected and the median under all configuration parameters is plotted for each security measurement indicator, the security measurement indicator collection capabilities under different configuration schemes are compared, and the crawler configuration scheme with the optimal security indicator collection capability for each security indicator is identified.

[0080] Sub-step 402 involves performing similarity and uniqueness analysis on comparable security measurement indicators in the original measurement data. For each type of measurement indicator defined in step 0, the pairwise similarity of the indicator sets collected by the twelve crawler configuration schemes is calculated. Through data visualization methods, the overlap and differences in measurement results of different configuration schemes are systematically examined.

[0081] Specifically, for a selected security measurement index category, this embodiment uses the Jaccard similarity coefficient to evaluate the similarity of the data collected by two crawler configuration schemes on a single domain. The specific calculation method is as follows: for security measurement index k, for two crawler configuration parameter schemes c i With c j The crawler is on a single domain u i The S of this type of safety measurement index collected above k (c i ,u i ) and S k (c j ,u i If the similarity percentage between these two configuration schemes on this domain is defined as follows:

[0082]

[0083] Specifically, based on the above-mentioned method for calculating the similarity of a single domain name's configuration collection indicators, this embodiment further calculates the median similarity across all domain names as the global domain name similarity under that indicator category:

[0084]

[0085] Specifically, based on the above calculations, a 12x12 similarity matrix M is constructed for the index k. k , where matrix element M k [i,j]=Sim k (c i ,c j The heatmap of the matrix is ​​generated, and configuration schemes with low similarity to other configurations are extracted based on the heatmap. These configuration schemes can collect more diverse security measurement results.

[0086] Sub-step 403: Based on the evaluation results of sub-steps 401 and 402, calculate the comprehensive score for each configuration scheme for each of the three security measurement index categories. Defined as:

[0087]

[0088] Where Q(c) i ,k) indicates configuration ci The total number of collections for index k, S(c i ,k) indicates configuration c i The average similarity between the configuration and all other configurations, where ω is a weighting factor representing preference, set to 0.5 in this embodiment. The optimal configuration scheme is selected for a security measurement metric category k. for:

[0089]

[0090] For configuration The raw measurement data under the crawler settings are used to extract the changes of three types of security measurement indicators with crawler search depth and page crawling timeout duration. The static threshold quantification analysis method is used to evaluate the impact of these two types of parameters on the integrity and efficiency of data collection, and output the parameters that minimize resource consumption while ensuring the integrity of measurement indicator collection.

[0091] Specifically, the static threshold quantification analysis methods proposed in this step for the parameters of crawler search depth and page crawling timeout are as follows:

[0092] (1) In this embodiment, the crawler search depth is set to a baseline value D. max =50. For any crawler search depth d(0 <d≤D max ), any security measurement index k, any domain name u i With a fixed page crawling timeout of 30 seconds, the single-domain error rate is e. k,ui (i) is defined as:

[0093]

[0094] in This indicates that when the maximum search depth of the crawler is set to i, it will search in domain u. i The above data collection results for safety measurement index k, e k,d (i) indicates the proportion of missed category k index collection results when crawling stops at depth i; define the global metric error rate E on the security measurement index k. k (i) For all domain names The median of:

[0095]

[0096] Furthermore, the global metric error rate E is plotted for each metric k. k (d) The change with d determines whether E is satisfied. k (d) < minimum depth value of δ In this embodiment, δ is set to 0.05; Output Minimum crawler search depth to satisfy the integrity requirements of security metric category k.

[0097] (2) In this embodiment, the page crawling timeout is set to a baseline value T. max =30 seconds. This embodiment defines three page crawling timeout durations: t1 = 5s, t2 = 10s, t3 = 20s; for any security measurement index k, any domain u i Similar to defining a fixed crawler search depth of 50, the single-domain metric error rate is defined. Compared with the global metric error rate E k (t); Plot the global metric error rate E for each metric k. k (t) changes with the three values ​​of t to determine whether E is satisfied. k The minimum time value of (t) < ε In this embodiment, ε is set to 0.1; output The shortest page crawling timeout duration to meet the security metric category k collection integrity requirements.

[0098] Step 5: Systematically synthesize the analysis results of the above embodiment, and output the optimal parameter configuration scheme according to the requirements, providing a quantifiable optimal configuration scheme for specific measurement targets, so as to achieve a balance between measurement integrity and resource efficiency.

[0099] Specifically, based on the evaluation results of sub-steps 401 and 402, this step outputs the search strategy with the highest page discovery efficiency for general network security measurement tasks; and for specialized measurement tasks with specific metrics as the core, it outputs a set of mapping relationships between all security measurement metric categories and the optimal configuration.

[0100] Specifically, this step, based on the static threshold quantification analysis method in sub-step 403, collects the maximum permissible error rates δ and ε according to the user-input indicators, and outputs the minimum resource consumption parameter combination that satisfies the constraints. These are constant parameters for crawler search depth and page crawling timeout.

[0101] Based on the same inventive concept, this embodiment also provides a system for evaluating the effectiveness of web crawler algorithm configurations for network security measurement, such as... Figure 4 Specifically, it includes the following modules:

[0102] Input module: This module is used to read the dataset from the website to be tested.

[0103] Data Preprocessing Module: This module is used to define the crawler algorithm configuration items, define the network security measurement indicators to be analyzed, generate the configuration parameter combinations to be evaluated, and set up the automated network measurement framework based on the configuration parameters.

[0104] Baseline Page Crawling Module: This module crawls website datasets based on basic search strategies and generates a set of baseline pages.

[0105] Measurement index re-crawl module: This module loads the corresponding set of benchmark pages generated by the benchmark page crawling module according to each specific configuration scheme generated by the data preprocessing module, and re-crawls according to the parameters of the current scheme to collect raw measurement data related to various security indicators.

[0106] Configuration Validity Analysis Module: This module is used to compare and analyze the original measurement data, and to perform quantitative validity analysis, data similarity analysis and parameter sensitivity analysis of the configuration schemes based on the original measurement data.

[0107] The adaptive report generation module integrates the results from the evaluation and analysis module to generate a final evaluation report and provide optimization suggestions, optimal crawler algorithm configuration schemes, and computing resource optimization strategy parameters.

[0108] For the system embodiments, since they basically correspond to the method embodiments, relevant details can be found in the descriptions of the method embodiments; the implementation methods of the remaining modules will not be repeated here. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0109] The system embodiments of the present invention can be applied to any device with data processing capabilities, such as a computer or other similar device. The system embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution.

[0110] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for evaluating the effectiveness of crawler algorithm configuration for network security measurement, characterized in that, Includes the following steps: (1) Define the adjustable parameter configuration of the crawler algorithm and its value space, and define a set of network security measurement indicators for evaluating the effectiveness of the crawler configuration; the adjustable parameter configuration of the crawler algorithm includes at least the crawler search strategy, the browser state management strategy, and the user interaction simulation strategy. (2) Input a set of websites to be measured, consisting of a large number of target website domain names; (3) Based on the parameter configuration and values ​​defined in step (1), generate a set of crawler configuration schemes to be evaluated by combining Cartesian products; (4) Perform a phased crawling operation on the set of websites to be measured, including: Benchmark crawling phase: For each crawler search strategy, crawl the website under benchmark conditions of stateless management and no user interaction, collect raw measurement data in real time, and generate a set of benchmark pages and an ordered sequence of URLs; Re-crawl phase: For each crawler configuration scheme, based on the baseline page set corresponding to its search strategy, browser state management strategy and user interaction simulation strategy are applied to re-crawl, and raw measurement data is collected in real time; (5) Establish a multidimensional evaluation model to analyze the original measurement data in step (4) and generate evaluation results; the multidimensional evaluation model includes quantitative effectiveness analysis, similarity and uniqueness analysis, and parameter sensitivity analysis; the quantitative effectiveness analysis results include the strongest crawler search strategy and the crawler configuration scheme with the best collection capability for each indicator; the similarity and uniqueness analysis results include the crawler configuration scheme with the best collection diversity for each indicator; the parameter sensitivity analysis is to select the optimal crawler configuration scheme for each indicator based on the comprehensive quantitative effectiveness analysis, similarity and uniqueness analysis and output the optimal crawler search depth and page crawling timeout duration; (6) Generate a comprehensive evaluation report based on the evaluation results, and output the optimal crawler configuration scheme and resource optimization parameters.

2. The method for evaluating the effectiveness of crawler algorithm configuration for network security measurement according to claim 1, characterized in that, Adjustable parameters for the crawler algorithm also include crawler search depth and page crawling timeout.

3. The method for evaluating the effectiveness of crawler algorithm configuration for network security measurement according to claim 1, characterized in that, During the baseline crawling phase, invalid pages are skipped, and the uniqueness and order of the baseline page set are ensured.

4. The method for evaluating the effectiveness of crawler algorithm configuration for network security measurement according to claim 1, characterized in that, A set of cybersecurity measurement metrics includes: HTTP request behavior, including all HTTP request domains issued by the web pages accessed during the crawling process; JavaScript Web API call behavior, including all browser JavaScript Web APIs called by the web pages accessed during the crawling process; Cookie settings and modification behavior, including all cookies set, modified, or deleted by the visited web pages during the crawling process.

5. The method for evaluating the effectiveness of crawler algorithm configuration for network security measurement according to claim 4, characterized in that, When collecting raw measurement data, for each type of indicator, the timestamp, event type and related context information of the behavior triggered on the accessed webpage are fully recorded. The related context information includes the specific object of the HTTP request, the name of the API called, and the cookies that were set or changed. All collected raw network measurement data, along with detailed timestamps and context information for each behavior, are structured, stored, and automatically input into the next step.

6. The method for evaluating the effectiveness of crawler algorithm configuration for network security measurement according to claim 4, characterized in that, Quantitative validity analysis includes: For each crawler search strategy, based on the baseline page set corresponding to that search strategy, count the number of valid pages n in each baseline page set. i and with the domain name u i Forming binary pairs <n i ,u i >, where u i Let n represent the domain name of the i-th website. i This represents the number of valid pages crawled under the i-th website domain by a certain crawler search strategy; plot the cumulative distribution function of the number of pages covered by the crawler search strategy to evaluate the page discovery efficiency and determine the crawler search strategy with the strongest page discovery efficiency. Furthermore, the total number of security measurement indicators collected from all website domains under each crawler configuration scheme and the median number of security measurement indicators were statistically analyzed. The total number of security measurement indicators includes the total number of HTTP request domains, the total number of JavaScript Web API calls, and the total number of Cookies set and modified. The relationship between the total number of data collected and the median under all configuration parameters was plotted for each security measurement indicator, thus obtaining the crawler configuration scheme with the optimal collection capability for each indicator.

7. The method for evaluating the effectiveness of crawler algorithm configuration for network security measurement according to claim 4, characterized in that, Similarity and uniqueness analysis includes: Calculate the similarity percentage between the two configuration schemes for a single domain: Among them, S k (c i ,u i ), S k 9c j ,u i ) represent two web crawler configuration parameter schemes c respectively. i With c j In a single domain u i The measurement results of the safety measurement index k collected above, J k (c i ,c j ,u i ) represents a single domain name u i The two configuration schemes above c i c j The percentage of similarity; Calculate the median similarity across all domains as the global domain similarity for this metric category: Among them, Sim k (c i ,c j () represents two web crawler configuration parameter schemes c i With c j In a single domain u i The similarity of security measurement index k collected above, where U represents the set of domain names in the set of websites to be measured; Construct a similarity matrix M for the index k. k , where matrix element M k [i,j]=Sim k (c i ,c j The heatmap of the matrix is ​​generated, and the configuration scheme with low similarity to other configurations is extracted based on the heatmap. This configuration scheme can collect more diverse security measurement results, and the crawler configuration scheme with the optimal collection diversity for each index is obtained.

8. The method for evaluating the effectiveness of crawler algorithm configuration for network security measurement according to claim 1, characterized in that, Based on the results of the quantitative effectiveness analysis and the similarity and uniqueness analysis, a comprehensive score is calculated for each configuration scheme: Where Q(c) i ,k) indicates configuration c i The total number of collections for index k, S(c i ,k) indicates configuration c i The average similarity between the configuration and all other configurations, where ω is a weighting factor representing preference; For a given safety measurement index k, the optimal configuration scheme is selected. for:

9. The method for evaluating the effectiveness of crawler algorithm configuration for network security measurement according to claim 8, characterized in that, Parameter sensitivity analysis includes: For configuration The raw measurement data under the crawler settings are used to extract the changes of three types of security measurement indicators with crawler search depth and page crawling timeout duration. The static threshold quantification analysis method is used to evaluate the impact of these two types of parameters on the integrity and efficiency of data collection, and output the parameters that minimize resource consumption while ensuring the integrity of measurement indicator collection.

10. A web crawler algorithm configuration effectiveness evaluation system for network security measurement, used to implement the web crawler algorithm configuration effectiveness evaluation method of claim 1, characterized in that, The system includes: The data preprocessing module is used to define the adjustable parameter configuration of the crawler algorithm and network security measurement indicators, and to generate a set of crawler configuration schemes; The input module is used to input a set of websites to be measured, consisting of a large number of target website domains; The baseline page crawling module is used to perform baseline crawling operations to generate a baseline page set; The measurement index re-crawl module is used to perform re-crawl operations and collect raw measurement data; Configure a validity analysis module to analyze raw measurement data based on a multidimensional evaluation model. The adaptive report generation module is used to integrate evaluation results and output the optimal configuration scheme.

Images