A method and system for adaptive crawler scheduling and anti-crawling confrontation based on a dynamic strategy pool

Abstract

The application discloses a kind of based on dynamic strategy pool's self-adapting crawler scheduling and anti-crawling confrontation method and system, belong to network data acquisition and intelligent scheduling technical field.The application is by dual mode mode acquisition crawler intelligent agent's multidimensional operation data calculation comprehensive environment situation value, and constructs history situation value sequence;Based on current environment situation value from two-level strategy pool candidate adaptive strategy set is screened;According to expected utility value proportion distribution executes quota and records actual execution utility;According to history situation value sequence and actual execution utility, update utility archives, eliminate detection to low-efficiency strategy, generate new strategy by crossover variation and gene injection, complete strategy pool synchronous update.The application realizes the automatic adaptation and continuous evolution of strategy combination, improves the data acquisition success rate under anti-crawling confrontation environment.

Description

Technical Field

[0001] This invention belongs to the field of network data acquisition and intelligent scheduling technology, specifically relating to an adaptive crawler scheduling and anti-crawling countermeasure method and system based on a dynamic policy pool, which is applicable to the intelligent policy selection and dynamic scheduling of distributed crawler systems in complex and ever-changing network anti-crawling environments. Background Technology

[0002] With the explosive growth of internet data, web crawling technology has become an important means of obtaining publicly available data. However, to protect their data resources, target websites have generally deployed increasingly complex anti-crawling mechanisms, including but not limited to request frequency limits, user agent detection, IP address blocking, CAPTCHA pop-ups, and browser fingerprint recognition. To cope with these anti-crawling mechanisms, existing crawling systems typically use fixed policy configurations or periodic policy switching for policy scheduling.

[0003] However, existing technologies still have the following shortcomings: First, fixed strategy configuration methods operate with a single strategy combination. When the anti-scraping strategy of the target site changes, the fixed strategy cannot adaptively adjust, leading to a significant decrease in data acquisition success rate. Second, while periodic strategy switching methods attempt to counter anti-scraping by rotating strategy combinations at regular intervals, the timing of these switches is not synchronized with changes in the target site's anti-scraping strategy, resulting in blind strategy selection and increased system overhead due to invalid requests from the switching. Third, existing feedback-based strategy adjustment methods lack a systematic recording and evaluation mechanism for the historical utility of strategy combinations, failing to effectively identify which strategy combinations performed well under similar environmental conditions, and strategy selection lacks effective support from historical data. Fourth, the updating of the strategy pool in existing methods relies on manual intervention or simple rules, lacking a utility-driven automated evolution mechanism, and thus failing to achieve intergenerational updates and continuous optimization of strategy combinations.

[0004] To address the aforementioned issues, this invention provides an adaptive crawler scheduling and anti-crawler countermeasure method and system based on a dynamic policy pool. Through multi-dimensional environmental situation awareness, a two-level policy pool architecture, nonlinear utility evaluation, and a policy evolution mechanism with gene injection, it achieves automatic adaptation, dynamic scheduling, and continuous optimization of crawler policy combinations. Summary of the Invention

[0005] In view of the shortcomings of the prior art, the purpose of this invention is to provide an adaptive crawler scheduling and anti-crawler countermeasure method and system based on a dynamic strategy pool, so as to realize the automatic adaptation and continuous evolution of crawler strategy combinations and improve the data acquisition success rate in dynamic adversarial environments.

[0006] To achieve the above objectives, the present invention adopts the following technical solution;

[0007] An adaptive crawler scheduling and anti-crawling method based on a dynamic policy pool includes:

[0008] Collect real-time operational data of each crawler agent, calculate the comprehensive environmental situation value in a dual-mode approach of periodic and event-driven methods based on request success rate, average response latency, interception status code frequency and data quality indicators, store the current environmental situation value in the historical situation database, and form a historical situation value sequence in chronological order of each record.

[0009] Based on the current environmental situation value, the utility profiles of each strategy combination are obtained from the two-level strategy pool currently maintained by the system. The distance between the current environmental situation value and the record interval of each profile is calculated. The strategy combination with the closest distance and the utility value higher than the average threshold of the corresponding strategy pool is selected to form a set of candidate adapted strategies.

[0010] For each candidate strategy in the candidate adaptation strategy set, the expected utility value is calculated based on its distance from the current environmental situation value. The execution quota is allocated according to the proportion of the expected utility value. The strategy combination with the highest quota is selected for execution, and the actual execution utility is recorded.

[0011] Based on the historical situation value sequence and actual execution effectiveness, update the effectiveness file of each strategy combination, eliminate inefficient strategy combinations, generate new strategy combinations and inject them into the strategy pool, complete the synchronization and collaborative update of the two-level strategy pool, and output the updated two-level strategy pool.

[0012] As a further aspect of the present invention, the calculation of the comprehensive environmental situation value using a dual-mode approach driven by both periodicity and event specifically includes:

[0013] A macro-cycle mode is executed with a fixed time period to statistically analyze the request success rate, average response latency, interception status code frequency and data quality indicators of each crawler agent within a unit time window, and to calculate the comprehensive environmental situation value through non-linear weighted fusion.

[0014] Within the macro cycle, a sliding window is maintained. At the end of each sliding window, the comprehensive environmental situation value within the window is calculated. When the deterioration difference between the comprehensive environmental situation value within the window and the situation value of the previous sliding window exceeds a preset threshold, the candidate strategy reselection process is immediately triggered.

[0015] As a further embodiment of the present invention, the two-level policy pool includes a global public policy pool and a site-specific policy pool; the global public policy pool stores policy combinations shared by all crawler agents, while the site-specific policy pool is maintained independently for each target site.

[0016] When accessing a specific site, policies are first selected from the target site's dedicated policy pool. If the number of candidate policies in the dedicated pool is less than the preset value, they are selected from the global policy pool to supplement them.

[0017] As a further aspect of the present invention, the utility file is a lookup table maintained for each strategy combination, which divides the range of values ​​of the comprehensive environmental situation value into multiple sub-intervals, and each sub-interval is mapped to the historical average utility value of the strategy combination under the situation interval.

[0018] The utility value is calculated by weighting four dimensions: request success rate, response latency, interception frequency, and data quality after the strategy combination is executed.

[0019] As a further aspect of the present invention, the step of selecting the strategy combination that is closest in distance and has a utility value higher than the average threshold of the corresponding strategy pool specifically includes:

[0020] Based on the current overall environmental situation, calculate the distance between the central value of each record interval in the utility file of each strategy combination and record the minimum distance of each strategy combination.

[0021] Strategy combinations whose minimum distance is less than a preset distance threshold and whose average utility value in similar historical environments is higher than the global average utility value of the strategy pool are selected as candidate adaptive strategy sets.

[0022] As a further aspect of the present invention, the expected utility value is calculated using an inverse distance weighting method based on the minimum distance of the strategy combination; the smaller the minimum distance, the higher the expected utility value.

[0023] The execution quota ratio is allocated according to the proportion of the expected utility value of each strategy combination to the total expected utility value, and the strategy combination with the highest expected utility value receives the highest execution weight.

[0024] As a further aspect of the present invention, the elimination detection of inefficient strategy combinations specifically includes:

[0025] Calculate the recent average utility value of each strategy portfolio over the most recent multiple recording periods, as well as the global average utility value and standard deviation of all strategy portfolios in the strategy pool;

[0026] When the recent average utility of a strategy portfolio is lower than the global average utility minus the product of the elimination factor and the standard deviation, the strategy portfolio is marked as to be eliminated.

[0027] The marked strategy enters the observation period. If the recent average utility of the strategy rises above the threshold during the observation period, the elimination mark is removed; otherwise, it is permanently removed from the strategy pool.

[0028] As a further aspect of the present invention, the generation of new strategy combinations injected into the strategy pool specifically includes:

[0029] When the total number of strategy combinations in the strategy pool is lower than the preset minimum holding amount, or when the variance of the utility values ​​of the latest multiple strategy combinations in the strategy pool is lower than the preset diversity threshold, a new strategy combination is generated.

[0030] The generation adopts a dual-track system, which includes: selecting two strategies from the top-ranked strategies in the current strategy pool and performing cross-mutation to generate offspring strategies; and extracting atomic action fragments from an external strategy gene library and injecting them into the gene to generate offspring strategies.

[0031] The newly generated strategy combination is injected into the corresponding strategy pool using an optimistic initialization strategy.

[0032] As a further aspect of the present invention, the synchronization and collaborative update of the two-level policy pool specifically includes:

[0033] The central coordinator periodically broadcasts updates to the global policy pool to each crawler agent;

[0034] Each agent periodically reports the top-performing strategy combinations from its local site-specific strategy pool to the central coordinator, which then injects them into the global strategy pool after review.

[0035] When communication with the central coordinator is interrupted, each agent automatically enters an independent operating mode, and merges with the central coordinator after communication is restored.

[0036] Another aspect of this invention discloses an adaptive crawler scheduling and anti-crawling system based on a dynamic policy pool, comprising:

[0037] The dual-mode environmental situation awareness module is used to collect real-time operational data of each crawler agent. It uses two modes: periodic collection and event-driven. It calculates the comprehensive environmental situation value based on the request success rate, average response latency, interception status code frequency and data quality indicators. The current environmental situation value is stored in the historical situation database, and the records are arranged in chronological order to form a historical situation value sequence.

[0038] The two-level strategy adaptation and filtering module is used to maintain the global shared strategy pool and the site-specific strategy pool, obtain the utility profiles of each strategy combination, calculate the distance between the current environmental status value and the record interval of each profile, and filter the strategy combination that is closest in distance and whose utility value is higher than the average threshold of the corresponding strategy pool to form a candidate adaptation strategy set.

[0039] The dynamic quota allocation module is used to calculate the expected utility value of each candidate strategy in the candidate adaptation strategy set based on its distance from the current environmental situation value, allocate execution quotas according to the proportion of expected utility value, select the strategy combination with the highest quota for execution, and record the actual execution utility.

[0040] The strategy evolution module is used to update the utility profile of each strategy combination based on the historical situation value sequence and actual execution utility, eliminate inefficient strategy combinations, generate new strategy combinations through crossover mutation and gene injection, and maintain the synchronization and collaborative update of the global strategy pool and the site-specific strategy pool.

[0041] In summary, due to the adoption of the above technical solutions, the beneficial technical effects of the invention are as follows: Compared with the prior art, the project group situation awareness and personalized recommendation method for scientific and technological project executors provided by the present invention has the following beneficial effects:

[0042] By constructing an enhanced situational indicator system encompassing four dimensions—request success rate, average response latency, interception status code frequency, and data quality metrics—and employing a non-linear weighted fusion method to calculate the comprehensive environmental situational value, the system can accurately quantify the smoothness of the current crawler task execution environment. Simultaneously, a dual-mode perception mechanism combining periodic data collection and event-driven approaches is introduced. This mechanism monitors drastic fluctuations in the situational value within a 30-second sliding window and promptly triggers strategy reselection. This addresses the issues of asynchronous strategy switching timing and changes in target site anti-crawling strategies, as well as the blind nature of switching in existing technologies, enabling the system to quickly respond to sudden changes in anti-crawling strategies.

[0043] By establishing a multidimensional utility profile for each strategy combination, recording its historical average utility value under different environmental situations, and screening candidate strategies based on the distance between the current situation value and the recorded intervals of each profile, the strategy selection is based on historical experience that has been verified to be effective in similar environments. This solves the problem in existing technologies of lacking a systematic recording and evaluation mechanism for the historical utility of strategy combinations and being unable to effectively identify which strategy combinations have performed well under similar environmental situations, thus significantly improving the accuracy and reliability of strategy selection.

[0044] By establishing a two-tier architecture of a global shared policy pool and a site-specific policy pool, general policy knowledge and site-specific adaptation experience complement each other and are managed in layers. This solves the problem that a single policy pool loses optimal adaptability due to policy averaging when dealing with the differentiated anti-crawling characteristics of multiple sites.

[0045] By allocating execution quotas according to the expected utility value using an inverse distance weighting function, the strategy combination best suited to the current environment receives the highest execution weight. Simultaneously, through weighted moving average updates of utility profiles, elimination detection and observation period confirmation of statistically significantly inefficient strategies, and new strategy generation using a dual-track system of crossover mutation and gene injection, a complete closed-loop evolution mechanism for the strategy pool is constructed. This solves the problems of existing technologies where strategy pool updates rely on manual intervention or simple rules and lack utility-driven automated evolution capabilities. It enables the strategy pool to continuously update itself as the environment changes, effectively avoiding homogeneous convergence of strategies. Attached Figure Description

[0046] Figure 1 A flowchart of an adaptive crawler scheduling and anti-crawling countermeasure method based on a dynamic policy pool;

[0047] Figure 2A flowchart of an adaptive crawler scheduling and anti-crawling countermeasure method S100 based on a dynamic policy pool;

[0048] Figure 3 A flowchart of an adaptive crawler scheduling and anti-crawling countermeasure method S200 based on a dynamic policy pool;

[0049] Figure 4 S300 is a flowchart of an adaptive crawler scheduling and anti-crawling countermeasure method based on a dynamic policy pool.

[0050] Figure 5 The flowchart shows the S400, an adaptive crawler scheduling and anti-crawling countermeasure method based on a dynamic policy pool. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0053] The core technical logic of this method is as follows:

[0054] Environmental Situation Awareness: This step addresses the environmental situation encountered by the crawler task during execution. It involves constructing a distributed swarm of crawler agents and collecting real-time data on four dimensions within a unit time window: request success rate, average response latency, interception status code frequency, and data quality indicators. A comprehensive environmental situation value is calculated through non-linear weighted fusion, and an event-driven mechanism is introduced for rapid response. This step performs situation awareness at fixed intervals, outputting the current comprehensive environmental situation value after each update. This value serves as a selection criterion for the second step and is stored in a historical situation database. The records, arranged chronologically, form a historical situation value sequence, providing an evolutionary data foundation for the fourth step.

[0055] Candidate Strategy Selection: This step addresses the question of which strategy combinations are optimally suited to the current environmental situation. A two-tiered strategy pool architecture is established: a global public strategy pool and a site-specific strategy pool. A utility profile is created for each strategy combination, recording its historical average utility value across different environmental situation intervals. The system uses the current environmental situation value as a benchmark, calculates the distance between the current situation and each recorded interval in the strategy combination profile, and selects the strategy combination with the closest distance and a historical average utility value higher than the global average threshold of the strategy pool as candidate suitable strategies. Priority is given to selecting from the site-specific strategy pool; if insufficient, the global strategy pool is used to supplement the selection.

[0056] Execution quota allocation: This step addresses the problem of optimally allocating execution weights. It involves fine-tuning the weight allocation of candidate strategy combinations, using the distance between the current environmental situation value and each recorded interval in the strategy file as the calculation basis. An inverse distance weighting function is used to calculate the expected utility value of each candidate strategy combination in the current environment. Execution quotas are allocated according to the proportion of each strategy combination's expected utility value to the total expected utility value, ensuring that the strategy combination with the highest expected utility receives the most execution opportunities. Simultaneously, the actual execution utility is recorded as the basis for the evolution in the fourth step.

[0057] Policy Pool Evolution: This stage addresses the challenges of evaluating feedback and ensuring the continuous evolution of the policy pool after policy execution. Based on historical situational value sequences and actual execution utility, this stage sequentially performs utility file updates, inefficient policy elimination detection and observation period confirmation, and a dual-track new policy generation system involving crossover mutation and gene injection at fixed intervals. This process synchronizes and coordinates the updates of the two-level policy pool, outputting the updated policy pool for subsequent scheduling.

[0058] like Figure 1 As shown, this application illustrates an exemplary adaptive crawler scheduling and anti-crawling method based on a dynamic policy pool, specifically including the following steps:

[0059] S100. Collect real-time running data of each crawler agent, calculate the comprehensive environmental situation value in a dual-mode approach of periodic and event-driven based on request success rate, average response latency, interception status code frequency and data quality indicators, store the current environmental situation value in the historical situation database, and the records in time order constitute a historical situation value sequence.

[0060] S200. Based on the current environmental situation value, obtain the utility files of each strategy combination from the two-level strategy pool currently maintained by the system, calculate the distance between the current environmental situation value and the record interval of each file, and filter the strategy combination that is closest and whose utility value is higher than the average threshold of the corresponding strategy pool to form a set of candidate adapted strategies.

[0061] S300. For each candidate strategy in the candidate adaptation strategy set, calculate the expected utility value based on its distance from the current environmental situation value, allocate execution quotas according to the proportion of expected utility values, select the strategy combination with the highest quota for execution, and record the actual execution utility.

[0062] S400. Based on the historical situation value sequence and actual execution utility, update the utility file of each strategy combination, eliminate inefficient strategy combinations, generate new strategy combinations and inject them into the strategy pool, complete the synchronization and collaborative update of the two-level strategy pool, and output the updated two-level strategy pool.

[0063] In the adaptive crawler scheduling and anti-crawling countermeasure method based on dynamic policy pool, the technical purpose of S100 is to shift the measurement object from static fixed policy configuration to dynamic and perceptible environmental situation entity. By analyzing the multi-dimensional operation data of the crawler agent within a unit time window, four key indicators are obtained: request success rate, average response latency, interception status code frequency, and data quality indicators. Then, a comprehensive environmental situation value is calculated as a quantitative benchmark for subsequent policy selection and quota allocation.

[0064] To achieve its technical objectives, the specific steps of S100 are as follows: Figure 2 As shown, it includes:

[0065] S110. Establish an enhanced multi-dimensional situational indicator system. The environmental situational indicator system consists of four core indicators, each of which describes the adversarial situation of the current network environment from different dimensions:

[0066] Request success rate metric, denoted as Request success rate is defined as the proportion of requests by an agent to successfully obtain target data within a unit time window out of all requests. A successful request means that the HTTP status code is between 200 and 299 and the returned data content passes the integrity check, i.e., the data length is not less than 80% of the expected length. The request success rate ranges from [0, 1], and a higher value indicates a higher tolerance of the current environment for crawler requests.

[0067] The average response delay metric is denoted as , defined as the arithmetic mean of the response times of all successful requests made by the agent within a unit time window. The average response latency ranges from [0, +∞). A lower value indicates a lower current load on the target site or less interference with the crawler.

[0068] Interception status code frequency index, denoted as , is defined as the proportion of the number of times anti-scraping related HTTP status codes received by the agent within a unit time window to the total number of requests; the frequency of intercepted status codes ranges from [0, 1], and the higher the value, the more active the anti-scraping system is.

[0069] Among them, anti-scraping related HTTP status codes include, but are not limited to, 403 Forbidden, 429 Too Many Requests, 418 I'm a Pot, and 503 Service Unavailable.

[0070] Data quality indicators, denoted as This is defined as the percentage of successful requests in which the returned data not only passes integrity checks but also successfully extracts key fields such as the title and main content area of ​​the text. The data quality metric ranges from [0, 1], with higher values ​​indicating a more stable data structure and higher parsing effectiveness for the target site. This metric can capture scenarios such as website redesigns or data format changes where requests succeed but parsing fails.

[0071] Once established, the above four metrics serve as the system's basic measurement standards and are stored in the platform's metric configuration library. Their definitions remain stable during system operation and are only expanded and adjusted when new anti-scraping techniques or data parsing requirements change.

[0072] S120. Dual-mode data acquisition and indicator calculation: This involves acquiring the operational log data of each crawler agent, including the timestamp of each request, target site identifier, target URL, HTTP status code, response time, returned data length, and key field extraction results. The operational logs are written to a distributed message queue in real-time by each agent after each request, for consumption by the situational assessment module.

[0073] This step employs a dual-mode data acquisition mechanism, taking into account both macro-trend analysis and real-time anomaly response:

[0074] I. Macro Cycle Model: The platform performs a complete data collection and indicator calculation once at a fixed time period (5 minutes in this example). For each agent, all its request records within the time window are counted:

[0075] Total number of requests: The number of successful requests is A successful request is defined as a status code between 200 and 299 and successful data integrity verification. The number of successful extractions of key fields is [number missing]. ,exist Key fields were fully extracted based on the basic data. The number of times the relevant status codes were intercepted was [number missing]. The status code is any one of 403 Forbidden, 429 Too Many Requests, 418 I'm available, or 503 Service Unavailable. The total response time for all successful requests is... .

[0076] The four indicators are calculated as follows:

[0077] (Formula 1);

[0078] (Formula 2);

[0079] For request success rate metrics; This is an average response delay metric.

[0080] when If no successful requests are made within that time window, Take the value from the previous window period; if it is still 0, take the system default value, which is 500ms.

[0081] (Formula 3);

[0082] (Formula 4);

[0083] For intercepting status code frequency indicators; For data quality indicators;

[0084] The above four indicators are combined into an enhanced environmental situation observation vector. Store the current period's situation record.

[0085] II. Micro-event mode: Within each macro-cycle, the system maintains a sliding window with a relatively short time span; in this embodiment, it is set to 30 seconds. At the end of each sliding window, the system automatically calculates the values ​​of each indicator within the window:

[0086] The four indicators within the sliding window are calculated as follows:

[0087] (Formula 5);

[0088] (Formula 6);

[0089] This refers to the request success rate metric within the sliding window. This refers to the average response latency metric within the sliding window. This represents the total number of requests within the sliding window. This represents the number of successful requests; a successful request is defined as a status code between 200 and 299 and a successful data integrity check. This is the sum of the response times for all successful requests;

[0090] when If no successful requests are received within that time window, Take the value from the previous window period; if it is still 0, take the system default value, which is 500ms.

[0091] (Formula 7);

[0092] (Formula 8);

[0093] This refers to the frequency of intercepted status codes within the sliding window. For data quality metrics within the sliding window; Extract the number of successful attempts for the key field. To intercept the occurrence of the relevant status code.

[0094] Comprehensive environmental situation value within the sliding window The calculation formula is:

[0095] (Formula 9);

[0096] in, The normalized delay value within the sliding window. To intercept normalized values ​​within the sliding window, , , , These are the weighting coefficients; and The formula is expressed as:

[0097] (Formula 10);

[0098] (Formula 11);

[0099] In this embodiment, the steepness coefficient is used. ; This embodiment uses the delay tolerance threshold. millisecond;

[0100] The comprehensive environmental situation value within the sliding window was calculated. Then, the situation value compared to the previous sliding window. Comparison. When minus When the threshold is exceeded, i.e. the environmental situation deteriorates significantly within 30 seconds, the system immediately triggers the S200 emergency strategy reselection process; in this embodiment, the threshold is 0.2.

[0101] S130. Calculate the nonlinear comprehensive environmental situation value. Based on the environmental situation observation vector obtained in S120, calculate the comprehensive environmental situation value of the agent within the current time window. :

[0102] Comprehensive environmental situation value The calculation is as follows:

[0103] (Formula 12);

[0104] , , , For weighting coefficients; in this embodiment, it is set to... =0.30, =0.25, =0.25, =0.20. The system can pre-configure different weight combinations for different target site types.

[0105] This is a delayed normalized value; To intercept normalized values;

[0106] (Formula 13);

[0107] (Formula 14);

[0108] Comprehensive environmental situation value The value range is [0, 1]. According to formula (12), when the request success rate... The higher the delay normalization value Approaching 1, intercepting normalized values Approaching 1, data quality At higher levels, A value close to 1 indicates a smoother execution environment for the crawler task and a higher tolerance for crawler requests from the target site. Conversely, a value close to 1 indicates a more favorable environment. A value close to 0 indicates that the current crawler execution environment is more severe and the target site has a lower tolerance for crawler requests.

[0109] S140. Update the comprehensive environmental situation value according to a preset cycle and construct a historical situation value sequence. Repeat the operations from S120 (macro-cycle mode) to S130 at a fixed time period. In this embodiment, the preset cycle is every 5 minutes. After each cycle, the system re-acquires the latest operation logs of each agent and updates the four indicator values ​​and the comprehensive environmental situation value.

[0110] Comprehensive environmental situation value obtained after each update Along with the corresponding periodic timestamp The target site identifier is also stored in the historical situation database. For any target site, as the cycle progresses, the records are arranged in chronological order to form a historical situation value sequence for that site. ,in This represents the total number of periods currently recorded.

[0111] at the same time, It is used immediately for policy filtering in S200 after each periodic update.

[0112] When an agent does not record any requests within a period, the comprehensive environmental situation value for that period is marked as null and does not participate in subsequent calculations, but the timestamp record is retained to ensure the temporal continuity of the sequence.

[0113] In the adaptive crawler scheduling and anti-crawling countermeasure method based on dynamic strategy pool, the technical objective of S200 is to establish a multi-dimensional utility profile for each strategy combination and maintain a two-level strategy pool that is globally shared and site-specific, and to select strategy combinations that are closest to the current target site environment and have historically had positive benefits under that environment as candidate adaptation strategies, so as to ensure that the selected strategy combinations have proven effectiveness in the current environment.

[0114] To achieve its technical objectives, the specific steps of S200 are as follows: Figure 3 As shown, it includes:

[0115] S210. Establish a two-tiered policy pool and utility profile. A policy combination is defined as a set of atomic actions. Each atomic action is an independently executable basic operation, including setting download latency, constructing request headers, switching proxy protocols, enabling or disabling cookie persistence, and enabling or disabling request retry mechanisms. The download latency is an integer between 50 milliseconds and 5000 milliseconds. The request header includes the User-Agent field, the Accept-Language field, and the Referer field. Proxy protocols include HTTP proxy, HTTPS proxy, and SOCKS proxy. A policy combination is a complete execution plan composed of selecting one of the above atomic actions.

[0116] Steps to establish a two-tier policy pool architecture:

[0117] I. Global Common Policy Pool: Contains policy combinations shared by all crawler agents and stored in the central coordinator. The policy combinations in this pool are applicable to general scenarios and represent a collection of common knowledge across different sites.

[0118] II. Site-Specific Policy Pool: This is a policy pool maintained independently for each high-frequency target site, stored locally in each agent's local storage or in the site partition of the central coordinator. The dedicated pool only records historical utility data for accessing the corresponding site, and its evolution occurs independently.

[0119] The two-level policy pool is initialized when the system starts up. The initial policy combination is generated by random sampling of the Cartesian product of atomic actions.

[0120] For each strategy portfolio, a utility profile is maintained in its respective strategy pool. The utility profile is a lookup table that maps the range of comprehensive environmental situation values ​​that the strategy portfolio has previously run to the corresponding historical average utility values.

[0121] Comprehensive environmental situation value The range of values ​​is evenly divided into The number of sub-intervals, the first The range of each interval is:

[0122] (Formula 15);

[0123] No. The center value of each interval for:

[0124] (Formula 16);

[0125] When a strategy combination has no historical records within a certain interval, the corresponding utility value is recorded as NULL;

[0126] Real utility value Defined as the comprehensive benefit obtained after the execution of the strategy combination, its calculation formula is as follows: It comprehensively considers the combined impact of four dimensions—request success rate, response latency, interception frequency, and data quality—on the strategy execution effect.

[0127] (Formula 17);

[0128] , , As positive coefficients, these are set to 0.5, 1.0, and 0.8 in this embodiment. The function's characteristics are: when latency or interception is high, the utility value decays exponentially; when data quality is lacking, the utility value decreases linearly. Actual utility value. The value range is (-∞, 1], and the higher the value, the better the execution effect of the strategy combination in this environment.

[0129] When an agent needs to access a specific site, it prioritizes selecting policies from that site's dedicated policy pool. If the number of candidate policies in the dedicated pool is less than a preset value, the agent will proceed with the selection process. If so, then supplementation will be selected from the global policy pool.

[0130] S220. Calculate the situational distance, using the current comprehensive environmental situational value updated in S140. Based on this, for each policy combination in the policy pool, firstly, the record intervals in its utility profile where the utility value is not NULL are searched, i.e., under which environmental situations the policy combination has been run and historical utility records are kept. Then, for each found interval with records... The interval center value defined in S210 is adopted. Calculate the current situation and the center value of the interval Distance between :

[0131] (Formula 18);

[0132] The smaller the distance, the closer the current environmental situation is to the environmental conditions that the strategy combination has experienced in the past.

[0133] Record the minimum distance for each strategy combination and its corresponding historical average utility value;

[0134] S230. Filter candidate adaptation strategies. For each strategy combination in the strategy pool, select strategy combinations that simultaneously meet the following two conditions as the candidate adaptation strategy set:

[0135] Condition 1: The minimum distance is less than a preset distance threshold. In this embodiment, the preset distance threshold is 0.15, which means that the strategy combination has been run in an environment with a difference of no more than 15% from the current environment, ensuring that the historical effectiveness experience of the strategy combination is referential under the current environmental conditions.

[0136] Condition 2: The average utility of this strategy portfolio under historically similar conditions is higher than the global average utility of all strategy portfolios in the strategy pool. Here, the average utility of the strategy portfolio under historically similar conditions is the historical average utility of a single strategy portfolio within the interval corresponding to its minimum distance, which is directly read from the utility profile of that strategy portfolio; the global average utility... The calculation method is as follows:

[0137] (Formula 19);

[0138] in This represents the total number of strategy combinations in the strategy pool. For the first The historical average utility value of a strategy portfolio within its most recent effective record interval.

[0139] All strategy combinations that simultaneously meet the above two conditions are gathered together to form a candidate adaptation strategy set;

[0140] If the number of candidate strategies selected by the dedicated pool is less than the preset value of candidate strategies. Then, supplement and filter from the global pool according to the same logic until the candidate strategy preset value is reached. .

[0141] In the adaptive crawler scheduling and anti-crawling countermeasure method based on dynamic policy pool, the technical purpose of S300 is to perform fine weight allocation on candidate adaptation policies so that the policy combination that best suits the current environment gets the highest execution quota, thereby maximizing the execution efficiency of the current round of tasks.

[0142] To achieve its technical objectives, the specific steps of S300 are as follows: Figure 4 As shown, it includes:

[0143] S310. Calculate the expected utility value. For each candidate strategy combination selected in S230, calculate its corresponding minimum distance. Based on the calculation, the expected utility of this strategy combination in the current environment is calculated using an inverse distance weighting function. :

[0144] (Formula 20);

[0145] in, This is a very small positive number; in this embodiment, it is set to 0.001 to prevent calculation overflow caused by a zero denominator. The function's value range is (0, +∞). When When the value approaches 0, meaning the current environment is almost identical to the environment in which this strategy combination has been validated, Approaching Divide by , thus gaining extremely high weight. When When it increases, It decays in an inverse proportional relationship.

[0146] The technical meaning of this function is that the better the historical performance of the strategy combination in similar environments, and the closer the current environment is to that historical environment, the greater the expected utility value of the strategy combination in the current environment.

[0147] S320. Calculate the execution quota ratio, sum the expected utility values ​​of all strategy combinations in the candidate suitable strategy set, and obtain the total expected utility. :

[0148] (Formula 21);

[0149] For each strategy combination Its implementation quota ratio for:

[0150] (Formula 22);

[0151] The value range of is (0, 1), and all candidate strategies are... The sum equals 1;

[0152] S330. Allocate execution quotas and record execution results: according to... Sort all candidate strategy combinations from highest to lowest, and select the top-ranked ones. (Candidate strategy preset values) strategy combinations form an execution strategy set. The strategy combinations in the execution strategy set are then arranged according to their... Proportional allocation of execution quotas: If the total number of requests for a task is [number missing], then [the quota is missing]. Then it is assigned to the strategy combination. The number of requests is In actual execution, the system polls and schedules each strategy combination in the task queue according to the above quota ratio.

[0153] After each strategy combination is executed, the system collects the actual operational data for the current cycle and calculates the actual request success rate for the current cycle. Actual average response delay Actual interception status code frequency and actual data quality indicators Then, according to the comprehensive environmental situation value calculation formula (Formula 12) defined in S130, the above actual measured values ​​are substituted to calculate the actual environmental situation value for this period. Simultaneously, according to the actual utility value calculation formula (Formula 17) defined in S210, the above actual measured values ​​are substituted to calculate the actual environmental situation value for this period. Calculate the actual utility value of each strategy combination. .Will Target site identifier Selected strategy combination identifier Actual utility value Combining into a quadruple Write it to the distributed message queue as the basis for S400 policy evolution.

[0154] In the adaptive crawler scheduling and anti-crawling countermeasure method based on dynamic strategy pool, the technical purpose of S400 is to perform the following in a fixed period: update the utility file, detect the elimination of strategy combinations, generate new strategy combinations, and coordinate the two-level strategy pools according to the historical situation value sequence accumulated by S100 and the actual execution utility recorded by S330, so as to realize the continuous self-updating and generational evolution of the strategy pool.

[0155] To achieve its technical objectives, the S400 specific steps are as follows: Figure 5 As shown, it includes:

[0156] S410. Record the historical utility triplet sequence, and read the quadruple data sent to the distributed message queue in S330 at preset intervals. In this embodiment, each time window is 5 minutes, which is one recording period. For any strategy combination From the quadruple Extract environmental status values Strategy combination identifier and actual utility value Forming record triples The historical utility sequence of the strategy combination in its corresponding strategy pool is constructed in chronological order. After each update of the comprehensive environmental situation value in S140, the system automatically stores the triplet record of the current period into the historical utility database and stores it according to the strategy pool to which the strategy combination belongs.

[0157] S420. Utility profile update: Based on the historical utility sequence obtained in S410, update the utility profile for each strategy combination. The utility file is updated. The utility file is divided according to S210. Given several intervals, iterate through all intervals and check if any new utility observations fall into each interval. If the intervals are... New observations Then, the utility value for that interval is updated using a weighted moving average method:

[0158] (Formula 23);

[0159] in, The learning rate is set to 0.15 in this embodiment. When the interval... When there were no records before, NULL Take directly The utility profiles for the global pool and the private pool use the same update logic, but their data sources are isolated from each other. The updated utility profiles serve as the basis for S430 phase-out detection and S440 new policy generation decisions.

[0160] S430. Strategy Combination Elimination Detection: Based on the utility profile updated in S420, the following detection logic is executed independently for the global policy pool and each site-specific policy pool:

[0161] For each strategy combination in the strategy pool Calculate its most recent Global average utility value over a recording period This embodiment Equals 20:

[0162] (Formula 24);

[0163] in, This is the current cycle number. For strategy combination In the The actual utility value per cycle.

[0164] Calculate the global average utility value of all policy combinations within the policy pool. and standard deviation :

[0165] (Formula 25);

[0166] (Formula 26);

[0167] For the first A combination of strategies in recent The average utility value within each recording period is calculated using the same formula as formula (24).

[0168] when Less than minus Multiply When, combine strategies Marked as pending elimination. The elimination factor is 1.5 in this embodiment.

[0169] The marked policy enters The observation period of each cycle, Option 5. If the recent average utility of the strategy rises above the threshold during the observation period, the elimination flag is removed; otherwise, it is permanently removed from the strategy pool. The elimination result is used for the synchronization operation of S450.

[0170] S440. New strategy combination generation, using the utility profile updated in S420 and the strategy pool after S430's phase-out as the operation objects, executes the following generation logic independently on the global strategy pool and each site-specific strategy pool respectively:

[0171] Triggering condition: When the total number of strategy combinations in the strategy pool is lower than the preset minimum retention amount (in this embodiment, the global pool is 40 and the private pool is 20), or when the latest strategy combination in the strategy pool... The variance of the utility value of the strategy combination is lower than the preset diversity threshold. When this occurs, a new strategy combination generation process is triggered, as described in this embodiment. It equals 20.

[0172] Utility variance The calculation method is as follows:

[0173] (Formula 27);

[0174] in The number of the latest strategy combinations used for calculation. For the first The recent average utility value of the strategy combination. For this The mean utility of a combination of strategies.

[0175] The new strategy generation adopts a dual-track system:

[0176] The first method is cross-mutation generation: randomly select two policy combinations from the top 5 policies in the current policy pool, swap the values ​​of the atomic actions at corresponding positions with a 50% probability, generate two new offspring policy combinations, and then randomly change the value of a certain atomic action in the offspring policy combination with a 5% probability.

[0177] The second step is gene injection generation: with a 15% probability, entirely new atomic action fragments are extracted from an external policy gene library to replace the corresponding atomic actions in the offspring policy combinations. The external policy gene library contains the latest User-Agent list collected from the open-source community, new proxy protocol configurations, and the latest fingerprint features of mainstream browsers, and is updated regularly.

[0178] The newly generated strategy portfolio adopts an optimistic initialization strategy: the initial utility value of each situation interval is set to 1.1 times the current global average utility value of the strategy pool, so that it obtains a certain initial weight in subsequent screening. The new strategy portfolio is immediately injected into the corresponding strategy pool.

[0179] S450. Synchronization and collaborative updates of the global and private policy pools, taking the eviction results of S430 and the new policies of S440 as the operation objects, perform the following synchronization operations:

[0180] The first step is regular synchronization: Every 10 minutes, the central coordinator broadcasts the updated global policy pool to all crawler agents, including policy combinations that have been phased out or removed, and newly injected policy combinations. Upon receiving the broadcast, each agent synchronously updates its local global policy pool.

[0181] The second aspect is knowledge contribution: Each agent reports the top 10% most effective strategy combinations from its local site's proprietary strategy pool to the central coordinator daily. The central coordinator reviews the reported strategy combinations and injects those that meet the criteria into the global strategy pool, thus achieving the convergence of proprietary knowledge into common knowledge.

[0182] Third is the disaster recovery mechanism: When the central coordinator is unable to broadcast due to network failure, each agent automatically enters independent operation mode after three consecutive synchronization cycles, continuing to use its local global policy pool and dedicated policy pool to perform scheduling and updates. After communication is restored, each agent merges differences with the central coordinator.

[0183] Example 2:

[0184] An adaptive crawler scheduling and anti-crawling system based on a dynamic policy pool includes:

[0185] The dual-mode environmental situation awareness module is used to collect real-time operating data of each crawler agent. It uses two modes: periodic collection and event-driven. It calculates a nonlinear comprehensive environmental situation value based on request success rate, average response latency, interception status code frequency and data quality indicators. The current environmental situation value is stored in the historical situation database, and the records are arranged in chronological order to form a historical situation value sequence.

[0186] The two-level strategy adaptation and filtering module is used to maintain the global shared strategy pool and the site-specific strategy pool, obtain the utility profiles of each strategy combination, calculate the distance between the current environmental status value and the record interval of each profile, and filter the strategy combination that is closest in distance and whose utility value is higher than the average threshold of the corresponding strategy pool to form a candidate adaptation strategy set.

[0187] The dynamic quota allocation module is used to calculate the expected utility value of each candidate strategy in the candidate adaptation strategy set based on its distance from the current environmental situation value, allocate execution quotas according to the proportion of expected utility value, select the strategy combination with the highest quota for execution, and record the actual execution utility.

[0188] The strategy evolution module is used to update the utility profile of each strategy combination based on the historical situation value sequence and actual execution utility, eliminate inefficient strategy combinations, generate new strategy combinations through crossover mutation and gene injection, and maintain the synchronization and collaborative update of the global strategy pool and the site-specific strategy pool.

[0189] Example 3:

[0190] System Configuration: A distributed crawler architecture is adopted, with a central coordinator deployed on a cloud server, responsible for maintaining the global policy pool and synchronous broadcasting. Each crawler agent is deployed on multiple independent servers, with each server running 4 agent instances, for a total of 20 agents. Each agent is configured with an independent proxy pool containing 200 proxy IPs and a request header library containing User-Agents for 50 commonly used browsers. The time window is set to 5 minutes, with each agent initiating approximately 300 requests within each window.

[0191] Policy Pool Configuration: The global policy pool initially contains 40 policy combinations, randomly sampled from the Cartesian product of atomic actions. Atomic actions include: five download latency levels (100ms, 300ms, 500ms, 1000ms, 3000ms); four User-Agent types; three proxy protocol types (HTTP proxy, HTTPS proxy, SOCKS proxy); two Cookie policy states (enabled and disabled); and two retry policy states (enabled and disabled). The theoretical combination space is 5 × 4 × 3 × 2 × 2 = 240 combinations, from which the initial policy pool is filled with 40 evenly selected combinations. Each high-frequency target site maintains an independent dedicated policy pool, initially containing 20 policy combinations.

[0192] Test Plan: Five news and information websites with publicly known anti-scraping policies were selected as target sites, each with varying anti-scraping strength. Site A had low strength, restricting only high-frequency requests. Site B had medium strength, including User-Agent detection and frequency restrictions. Site C had high strength, including browser fingerprint detection, CAPTCHA pop-ups, and IP blocking. Site D had dynamic strength, with high strength on weekdays and low strength on weekends. Site E had random strength, with anti-scraping policies switching irregularly between various modes. The system ran continuously for 72 hours, and the data acquisition success rate and average request latency for each site were statistically analyzed.

[0193] Evaluation metrics: Data retrieval success rate and average request latency are the core evaluation metrics. Data retrieval success rate refers to the percentage of requests that successfully retrieve complete target data out of all requests. Average request latency refers to the average time interval between sending a request and receiving a complete response.

[0194] The sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0195] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. An adaptive crawler scheduling and anti-crawling method based on a dynamic policy pool, characterized in that, include: Collect real-time operational data of each crawler agent, calculate the comprehensive environmental situation value in a dual-mode approach of periodic and event-driven methods based on request success rate, average response latency, interception status code frequency and data quality indicators, store the current environmental situation value in the historical situation database, and form a historical situation value sequence in chronological order of each record. Based on the current environmental situation value, the utility profiles of each strategy combination are obtained from the two-level strategy pool currently maintained by the system. The distance between the current environmental situation value and the record interval of each profile is calculated. The strategy combination with the closest distance and the utility value higher than the average threshold of the corresponding strategy pool is selected to form a set of candidate adapted strategies. For each candidate strategy in the candidate adaptation strategy set, the expected utility value is calculated based on its distance from the current environmental situation value. The execution quota is allocated according to the proportion of the expected utility value. The strategy combination with the highest quota is selected for execution, and the actual execution utility is recorded. Based on the historical situation value sequence and actual execution effectiveness, update the effectiveness file of each strategy combination, eliminate inefficient strategy combinations, generate new strategy combinations and inject them into the strategy pool, complete the synchronization and collaborative update of the two-level strategy pool, and output the updated two-level strategy pool.

2. The adaptive crawler scheduling and anti-crawling method based on a dynamic policy pool according to claim 1, characterized in that, The calculation of the comprehensive environmental situation value using a dual-mode approach driven by both periodicity and event specifically includes: A macro-cycle mode is executed with a fixed time period to statistically analyze the request success rate, average response latency, interception status code frequency and data quality indicators of each crawler agent within a unit time window, and to calculate the comprehensive environmental situation value through non-linear weighted fusion. Within the macro cycle, a sliding window is maintained. At the end of each sliding window, the comprehensive environmental situation value within the window is calculated. When the deterioration difference between the comprehensive environmental situation value within the window and the situation value of the previous sliding window exceeds a preset threshold, the candidate strategy reselection process is immediately triggered.

3. The adaptive crawler scheduling and anti-crawling method based on a dynamic policy pool according to claim 1, characterized in that, The two-level policy pool includes a global public policy pool and a site-specific policy pool; the global public policy pool stores policy combinations shared by all crawler agents, while the site-specific policy pool is maintained independently for each target site. When accessing a specific site, policies are first selected from the target site's dedicated policy pool. If the number of candidate policies in the dedicated pool is less than the preset value, they are selected from the global policy pool to supplement them.

4. The adaptive crawler scheduling and anti-crawling method based on a dynamic policy pool according to claim 1, characterized in that, The utility profile is a lookup table maintained for each strategy portfolio, which divides the range of the comprehensive environmental situation value into multiple sub-intervals, and each sub-interval is mapped to the historical average utility value of the strategy portfolio under the situation interval. The utility value is calculated by weighting four dimensions: request success rate, response latency, interception frequency, and data quality after the strategy combination is executed.

5. The adaptive crawler scheduling and anti-crawling method based on a dynamic policy pool according to claim 1, characterized in that, The selection of strategy combinations that are closest in distance and have a utility value higher than the average threshold of the corresponding strategy pool specifically includes: Based on the current overall environmental situation, calculate the distance between the central value of each record interval in the utility file of each strategy combination and record the minimum distance of each strategy combination. Strategy combinations whose minimum distance is less than a preset distance threshold and whose average utility value in similar historical environments is higher than the global average utility value of the strategy pool are selected as candidate adaptive strategy sets.

6. The adaptive crawler scheduling and anti-crawling method based on a dynamic policy pool according to claim 1, characterized in that, The expected utility value is calculated using an inverse distance weighting method based on the minimum distance of the strategy combination; the smaller the minimum distance, the higher the expected utility value. The execution quota ratio is allocated according to the proportion of the expected utility value of each strategy combination to the total expected utility value, and the strategy combination with the highest expected utility value receives the highest execution weight.

7. The adaptive crawler scheduling and anti-crawling method based on a dynamic policy pool according to claim 1, characterized in that, The elimination detection of inefficient strategy combinations specifically includes: Calculate the recent average utility value of each strategy portfolio over the most recent multiple recording periods, as well as the global average utility value and standard deviation of all strategy portfolios in the strategy pool; When the recent average utility of a strategy portfolio is lower than the global average utility minus the product of the elimination factor and the standard deviation, the strategy portfolio is marked as to be eliminated. The marked strategy enters the observation period. If the recent average utility of the strategy rises above the threshold during the observation period, the elimination mark is removed; otherwise, it is permanently removed from the strategy pool.

8. The adaptive crawler scheduling and anti-crawling method based on a dynamic policy pool according to claim 1, characterized in that, The generation of new strategy combinations and their injection into the strategy pool specifically includes: When the total number of strategy combinations in the strategy pool is lower than the preset minimum holding amount, or when the variance of the utility values ​​of the latest multiple strategy combinations in the strategy pool is lower than the preset diversity threshold, a new strategy combination is generated. The generation adopts a dual-track system, which includes: selecting two strategies from the top-ranked strategies in the current strategy pool and performing cross-mutation to generate offspring strategies; and extracting atomic action fragments from an external strategy gene library and injecting them into the gene to generate offspring strategies. The newly generated strategy combination is injected into the corresponding strategy pool using an optimistic initialization strategy.

9. The adaptive crawler scheduling and anti-crawling method based on a dynamic policy pool according to claim 1, characterized in that, The synchronization and collaborative update of the two-level policy pools specifically include: The central coordinator periodically broadcasts updates to the global policy pool to each crawler agent; Each agent periodically reports the top-performing strategy combinations from its local site-specific strategy pool to the central coordinator, which then injects them into the global strategy pool after review. When communication with the central coordinator is interrupted, each agent automatically enters an independent operating mode, and merges with the central coordinator after communication is restored.

10. An adaptive crawler scheduling and anti-crawling system based on a dynamic policy pool, characterized in that, include: The dual-mode environmental situation awareness module is used to collect real-time operational data of each crawler agent. It uses two modes: periodic collection and event-driven. It calculates the comprehensive environmental situation value based on the request success rate, average response latency, interception status code frequency and data quality indicators. The current environmental situation value is stored in the historical situation database, and the records are arranged in chronological order to form a historical situation value sequence. The two-level strategy adaptation and filtering module is used to maintain the global shared strategy pool and the site-specific strategy pool, obtain the utility profiles of each strategy combination, calculate the distance between the current environmental status value and the record interval of each profile, and filter the strategy combination that is closest in distance and whose utility value is higher than the average threshold of the corresponding strategy pool to form a candidate adaptation strategy set. The dynamic quota allocation module is used to calculate the expected utility value of each candidate strategy in the candidate adaptation strategy set based on its distance from the current environmental situation value, allocate execution quotas according to the proportion of expected utility value, select the strategy combination with the highest quota for execution, and record the actual execution utility. The strategy evolution module is used to update the utility profile of each strategy combination based on the historical situation value sequence and actual execution utility, eliminate inefficient strategy combinations, generate new strategy combinations through crossover mutation and gene injection, and maintain the synchronization and collaborative update of the global strategy pool and the site-specific strategy pool.

Images