Anti-crawler method and system based on dynamic request frequency modulation and pseudo-signal interference

Abstract

The application discloses a kind of based on dynamic request frequency modulation and anti-crawler method and system of pseudo-signal interference, it is related to network security technical field.The application enhances the dynamic nature of protection by adjusting request time interval in real time through nonlinear function;By injecting pseudo-signal to induce crawler response, active identification automation behavior;Combined with the dual verification of time interval and pseudo-signal response, the precision of crawler detection is greatly improved;According to network-server data dynamic adjustment protection strategy, ensure that system adaptability is strong and increase protection effect;Through lightweight mathematical model and pseudo-signal injection, provide efficient anti-crawler ability, applicable to high concurrency scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network security, and particularly relates to a crawler prevention method and system based on dynamic request frequency modulation and pseudo-signal interference. BACKGROUND

[0002] In existing network applications, network crawlers cause a huge burden on website servers through high-frequency and large-scale requests, and traditional protection methods (such as IP blocking, verification code verification, etc.) have been difficult to cope with the automated behavior of advanced crawlers. The existing technology has the problems of fixed protection mechanism and inability to cope with dynamic attacks. SUMMARY

[0003] The present application aims to provide a crawler prevention method and system based on dynamic request frequency modulation and pseudo-signal interference to improve the above technical problems.

[0004] In order to achieve the above application purposes, the embodiments of the present application provide the following technical solutions:

[0005] A crawler prevention method based on dynamic request frequency modulation and pseudo-signal interference, comprising:

[0006] Adjusting frequency modulation parameters and pseudo-signal generation parameters based on historical request activity data of a user and network-server data at a current timestamp, and determining a protection strategy;

[0007] Generating an allowed request time interval for a current request through a request interval nonlinear function based on the frequency modulation parameters;

[0008] Proposing a current request through a client based on the allowed request time interval, calculating an actual request time interval of a previous request and the current request;

[0009] Randomly generating a pseudo-signal based on the pseudo-signal generation parameters, judging whether the current request is responded, and obtaining a response result of the current request;

[0010] Determining a request type of the current request and executing the protection strategy based on the actual request time interval of the current request and the response result.

[0011] The adjusting of the frequency modulation parameters and the pseudo-signal generation parameters comprises:

[0012] Filtering historical user behavior data and historical request frequency, determining historical suspected abnormal behavior data and historical suspected abnormal request frequency; and taking historical access time periods corresponding to the historical abnormal behavior data and the historical abnormal request frequency as historical suspected abnormal access time periods;

[0013] The historical suspected abnormal behavior data, the historical suspected abnormal request frequency, and the historical suspected abnormal access time period are analyzed to obtain a comprehensive correlation analysis result.

[0014] Based on the comprehensive correlation analysis result, a frequency modulation parameter in a previous round of requests is dynamically adjusted through an adaptive adjustment algorithm.

[0015] Based on activity data of a current request, a pseudo-signal generation parameter in the previous round of requests is dynamically adjusted. The pseudo-signal generation parameter includes an injection frequency and complexity of the pseudo-signal.

[0016] Further, the historical user behavior data includes historical access logs, historical request time intervals, historical request header information, and historical access paths. The historical access logs include historical IP addresses and historical request URL parameters. The historical access paths include deep pages and resource requests. The historical request frequency includes the number or speed of user requests.

[0017] Further, the network-server data includes user network traffic conditions and server load data.

[0018] The determination of the protection strategy for the current request includes:

[0019] Based on the user network traffic conditions, an overall bandwidth occupancy rate is calculated. Based on the overall bandwidth occupancy rate, it is determined whether the current request is in a peak period. If yes, the protection strategy for the previous request is adjusted to obtain an initial protection strategy. Otherwise, the protection strategy for the previous request is taken as the initial protection strategy.

[0020] Based on the server load data, a resource utilization rate of the server is calculated. Based on the resource utilization rate, the initial protection strategy is adjusted again to obtain the protection strategy for the current request.

[0021] Further, the screening of the historical user behavior data and the historical request frequency to determine the historical suspected abnormal behavior data and the historical suspected abnormal request frequency includes:

[0022] It is identified whether the historical request frequency in different historical access time periods exceeds a request threshold condition. If yes, the historical request frequency is taken as the historical suspected abnormal request frequency. Otherwise, the historical request frequency is taken as a historical normal request frequency.

[0023] It is identified whether the historical request time intervals in different historical access time periods are the same. If yes, the historical request time interval is taken as the historical suspected abnormal behavior data. Otherwise, the historical request time interval is taken as historical normal behavior data.

[0024] Identify whether the historical access log in different historical access time period meets any one of the conditions that the repetition rate of historical IP address reaches the repetition rate threshold, the geographical location distribution of historical IP address changes, and the historical request URL parameter is completely same; if yes, the historical access log is regarded as historical suspected abnormal behavior data; otherwise, the historical access log is regarded as historical normal behavior data.

[0025] Identify whether the historical access path in different historical access time period meets the condition that the historical access path is fixed; if yes, the historical access path is regarded as historical suspected abnormal behavior data; otherwise, the historical access path is regarded as historical normal behavior data.

[0026] Identify whether the historical request header information in different historical access time period is abnormal or lacks browser identification; if yes, the historical request header information is regarded as historical suspected abnormal behavior data; otherwise, the historical request header information is regarded as historical normal behavior data.

[0027] Identify whether the historical number of application pages in different historical access time period reaches the page number threshold; if yes, the historical number of application pages is regarded as historical suspected abnormal behavior data; otherwise, the historical number of application pages is regarded as historical normal behavior data.

[0028] Further, the current request is proposed based on the allowed request time interval by the client, and the actual request time interval of the previous request and the current request is calculated, including:

[0029] The allowed request time interval for receiving the current request is judged by the client, and the current request is proposed;

[0030] It is judged whether the client sends the current request after the allowed request time interval; if yes, the actual request time interval of the previous request and the current request is calculated; otherwise, it is directly determined that the application type of the current request is illegal application, the applicant is crawler activity, the current request is rejected and data protection is started, and the anti-crawler mechanism is triggered.

[0031] Further, the request type of the current request is determined based on the actual request time interval and the response result of the current request, and the protection strategy is executed, including:

[0032] It is judged whether the actual request time interval of the current request meets any one of the conditions that the request interval is nonlinear function and the response result is non-response; if yes, it is determined that the request type of the current request is legal request, the applicant is real user, and the current request is agreed; otherwise, it is determined that the application type of the current request is illegal application, the applicant is crawler activity, the current request is rejected and data protection is started, and the anti-crawler mechanism is triggered.

[0033] An anti-crawler system based on dynamic request frequency modulation and pseudo-signal interference, comprising:

[0034] request activity data collection module, configured to collect historical request activity data of a user and network-server data at a current timestamp;

[0035] parameter setting module, configured to adjust frequency modulation parameters and pseudo-signal generation parameters based on the historical request activity data of the user and the network-server data at the current timestamp;

[0036] protection policy determination module, configured to calculate an overall bandwidth occupancy rate based on the network-server data at the current timestamp, and determine a protection policy for a current request;

[0037] allowed request time interval calculation module, configured to generate an allowed request time interval for the current request based on the frequency modulation parameters and through a request interval nonlinear function;

[0038] actual request time interval calculation module, configured to calculate an actual request time interval of a previous request and the current request based on the allowed request time interval through the client making the current request;

[0039] pseudo-signal generation and response module, configured to randomly generate a pseudo-signal based on the pseudo-signal generation parameters, and determine whether the current request is responded, to obtain a response result of the current request;

[0040] data protection module, configured to determine a request type of the current request and execute the protection policy based on the actual request time interval of the current request and the response result.

[0041] The present application has the following beneficial effects:

[0042] The present application adjusts the request time interval in real time through a nonlinear function, enhances the dynamic nature of protection, induces the response of a crawler through the injection of a pseudo-signal, actively identifies automated behavior, combines the dual verification of time interval and pseudo-signal response, greatly improves the accuracy of crawler detection, dynamically adjusts the protection policy according to network-server data, ensures strong system adaptability and increases protection effect, provides efficient anti-crawler capability through a lightweight mathematical model and pseudo-signal injection, and is suitable for high-concurrency scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0044] Figure 1 The present application is a method flowchart in the embodiments.

[0045] Figure 2 Figure 1 is a system structure diagram in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0047] Referring to Figure 1 The embodiment provides a crawler prevention method based on dynamic request frequency modulation and pseudo-signal interference, which comprises the following steps.

[0048] S1, adjusting frequency modulation parameters and pseudo-signal generation parameters based on historical request activity data of a user and network-server data at a current timestamp, and determining a protection strategy.

[0049] The S1 comprises the following steps.

[0050] S1-1, collecting historical request activity data of a user and network-server data at a current timestamp;

[0051] The historical request activity data comprises historical user behavior data, historical request frequency and historical number of requested pages of historical requests in different historical access time periods. The historical user behavior data comprises historical access logs, historical request time intervals, historical request header information (such as User-Agent, Referer, Accept, etc.), and historical access paths. The historical access logs comprise historical IP addresses and historical request URL parameters. The historical access paths comprise deep pages and resource requests. The historical request frequency comprises the number or speed of user requests. The historical access time periods comprise peak periods and trough periods.

[0052] The network-server data comprises network traffic conditions and server load data of a user. The server load data comprises CPU load, memory occupancy and response time of a server.

[0053] S1-2, calculating overall bandwidth occupancy based on the network-server data, and determining a protection strategy for a current request;

[0054] Based on the user's network traffic, the overall bandwidth occupancy is calculated; based on the overall bandwidth occupancy, it is judged whether the current request is in the peak period; if so, the protection strategy of the last request is adjusted to obtain the initial protection strategy, the modulation function base request interval C0 and the modulation amplitude A are increased, the request interval is prolonged, the high-frequency request is limited, the modulation phase φ is adjusted, and the user request interval difference is increased; otherwise, the protection strategy of the last request is taken as the initial protection strategy;

[0055] Based on the server load data, the resource utilization rate of the server is calculated; based on the resource utilization rate, the initial protection strategy is adjusted again to obtain the protection strategy of the current request.

[0056] According to the actual demand, the first resource utilization rate threshold, the second resource utilization rate threshold and the resource utilization rate threshold are set;

[0057] If the resource utilization rate is greater than or equal to the first resource utilization rate threshold, the request interval and the page access path in the protection strategy are increased, the complexity of the crawler identification is increased, the crawling process of the crawler is further disturbed, and the flexibility and robustness of data protection are improved.

[0058] If the resource utilization rate is less than the first resource utilization rate threshold and greater than or equal to the second load threshold range, a plurality of virtual users operating in parallel are simulated and added to the protection strategy, which increases the analysis difficulty of the crawler and disturbs its normal crawling.

[0059] If the resource utilization rate is less than the third resource utilization rate threshold, the operation corresponding to the IP proxy pool is set to the protection strategy, the request source is dispersed through the proxy pool, and the single IP address frequent request is avoided.

[0060] Using the network-server data at the current timestamp, the protection strategy of the current request is adjusted in real time, which not only helps to avoid the impact of strict protection measures on the access experience of normal users, but also formulates the protection strategy suitable for the current network situation in real time, which is beneficial to quickly respond and timely handle the crawler activity.

[0061] S1-3, the historical user behavior data and the historical request frequency are filtered to determine the historical suspected abnormal behavior data and the historical suspected abnormal request frequency; the historical access time period corresponding to the historical abnormal behavior data and the historical abnormal request frequency is taken as the historical suspected abnormal access time period;

[0062] It is judged whether the historical request frequency in different historical access time periods exceeds the request threshold condition; if so, the historical request frequency is taken as the historical suspected abnormal request frequency; otherwise, the historical request frequency is taken as the historical normal request frequency; in this embodiment, the request threshold condition is that the request frequency of accessing the same resource in a short time reaches the frequency threshold; the frequency threshold is determined according to the actual specific requirements.

[0063] Identify whether the historical request time interval in different historical access time periods is the same; if so, the historical request time interval is taken as the historical suspected abnormal behavior data; otherwise, the historical request time interval is taken as the historical normal behavior data; because the fixed request time interval is a typical feature of crawler activity, the historical user behavior data can be analyzed from the perspective of fixed request time interval to determine whether it is abnormal data.

[0064] Identify whether the historical access log in different historical access time periods meets any one of the conditions that the repetition rate of historical IP address reaches the repetition rate threshold, the geographical location distribution of historical IP address changes, and the historical request URL parameter is completely the same; if so, the historical access log is taken as the historical suspected abnormal behavior data; otherwise, the historical access log is taken as the historical normal behavior data.

[0065] Identify whether the historical access path in different historical access time periods meets the condition of fixed historical access path; if so, the historical access path is taken as the historical suspected abnormal behavior data; otherwise, the historical access path is taken as the historical normal behavior data.

[0066] Identify whether the historical request header information in different historical access time periods is abnormal or lacks common browser identification; if so, the historical request header information is taken as the historical suspected abnormal behavior data; otherwise, the historical request header information is taken as the historical normal behavior data.

[0067] Identify whether the historical number of requested pages in different historical access time periods reaches the page number threshold; if so, the historical number of requested pages is taken as the historical suspected abnormal behavior data; otherwise, the historical number of requested pages is taken as the historical normal behavior data. In general, crawlers usually request more pages than real users, so the number of requested pages can be used to distinguish crawler activities.

[0068] S1-4, analyze the historical suspected abnormal behavior data, the historical suspected abnormal request frequency, and the historical suspected abnormal access time period to obtain a comprehensive correlation analysis result.

[0069] Use a machine learning algorithm to analyze the correlation between crawler activities and the historical suspected abnormal behavior data, the historical suspected abnormal request frequency, and the historical suspected abnormal access time period. Input the request type of the historical request, which corresponds to the historical activity data (historical user behavior data, historical request frequency, and historical access time period), the historical suspected abnormal behavior data, the historical suspected abnormal request frequency, and the historical suspected abnormal access time period into the machine learning algorithm, and output the correlation between crawler activities and user behavior, the correlation between crawler activities and request frequency, and the correlation between crawler activities and access time period, i.e. comprehensive correlation analysis result.

[0070] S1-5, based on the comprehensive correlation analysis results, the frequency modulation parameters in the last round of requests are dynamically adjusted through an adaptive adjustment algorithm, and the corresponding process is:

[0071] The correlation between crawler activity and request frequency is that the higher the request frequency, the more active the crawler activity, so the modulation amplitude A and modulation frequency ω need to be adjusted ， Increase the fluctuation of request interval and request interval changes faster, prevent the crawler from sending requests at a fixed frequency. In addition, the pseudo-signal injection frequency can also be increased to limit high-frequency crawlers.

[0072] The correlation between crawler activity and user behavior is that fixed time intervals, abnormal request headers, or fixed paths are highly correlated with crawlers. Adjusting the modulation phase φ helps generate complex pseudo-signals and increases the difficulty of crawler analysis.

[0073] The correlation between crawler activity and access time period is that crawlers are more active during high-load periods. When the access time period is at peak, reduce the base request interval C0, shorten the request time interval of normal users, and at the same time, randomly assign new values to the modulation phase φ and update it, ensuring that the request time intervals of different users have large differences at the same time, avoiding crawlers from exploiting group rules for attacks.

[0074] S1-6, based on the current request activity data (IP request frequency), dynamically adjust the pseudo-signal generation parameters in the last round of requests; pseudo-signal generation parameters include pseudo-signal injection frequency and complexity.

[0075] When detecting that the IP request frequency of the current request is abnormal, increase the injection frequency and complexity of the pseudo-signal. Increasing the injection frequency of the pseudo-signal can more frequently interfere with the identification and analysis process of the crawler, making it more difficult for the crawler to distinguish between normal requests and pseudo-signals. For example, by randomly delaying and inserting pseudo-requests, simulate the browsing interval of normal users, increase the randomness between requests, and thus increase the injection frequency of the pseudo-signal.

[0076] Increasing the complexity of the pseudo-signal means increasing the dynamic page content of the pseudo-signal or changing the camouflage request header of the pseudo-signal.

[0077] In the increase of the dynamic page content of the pseudo-signal, the page content is dynamically generated or modified through JavaScript, or the page structure is changed according to user behavior (such as hiding elements, loading new content); since crawlers usually cannot handle dynamically loaded pages, the difficulty of crawler analysis is increased.

[0078] In the camouflage request header of the replacement pseudo-signal, the request headers such as User-Agent and Referer are periodically replaced, so that the access of different browsers and devices can be simulated; since the crawler often uses a fixed User-Agent (request header), by replacing the request header, the diversified access mode of normal users can be simulated, and the difficulty of crawler analysis is increased.

[0079] The application dynamically adjusts the frequency modulation parameter and the pseudo-signal generation parameter based on the comprehensive correlation analysis result in real time, increases the injection frequency or complexity of the pseudo-signal, further confuses the advanced crawler, and makes it more likely to expose the automated behavior.

[0080] S2, based on the frequency modulation parameter, and generating the allowed request time interval of the current request through a request interval nonlinear function;The frequency modulation parameter includes modulation amplitude, modulation frequency, modulation phase and basic request interval.

[0081] The S2 includes:

[0082] Determine the current timestamp of the client;Based on the current timestamp and the request interval nonlinear function, the allowed request time interval of the current request is generated. Wherein, the request interval nonlinear function is a formula corresponding to the nonlinear function f(t):

[0083] f(t)=A·sin(ω·t+φ)+C0;

[0084] sin(·) represents the sine function;A represents the modulation amplitude;φ represents the modulation phase, which is used to control the offset of the initial request interval;t represents the timestamp;ω represents the modulation frequency, which is used to determine the fluctuation speed of the allowed request time interval;C0 represents the basic request interval, which adopts the average value of the historical request time interval.

[0085] Based on the frequency modulation parameter, different allowed request time intervals can be generated at different time periods, improving the flexibility and robustness of the anti-crawler.

[0086] S3, based on the allowed request time interval, the current request is proposed through the client, and the actual request time interval of the previous request and the current request is calculated;

[0087] The S3 includes:

[0088] The client judges the allowed request time interval of the current request received and proposes the current request;

[0089] Determine whether the client sends the current request after the allowed request time interval;If yes, calculate the actual request time interval of the previous request and the current request;Otherwise, directly determine that the application type of the current request is illegal application, the applicant is crawler activity, reject the current request and start data protection, trigger the anti-crawler mechanism.

[0090] S4, generating a pseudo-signal based on the pseudo-signal generation parameter, randomly generating the pseudo-signal, judging whether the current request is responded, and obtaining a response result of the current request; the pseudo-signal is used to induce the current request to respond, and includes random data, a pseudo-HTTP header, pseudo-content, a fake URL parameter, and a fake HTTP status code. The response result is response or non-response.

[0091] The S4 includes:

[0092] S4-1, generating random data by using Random; generating a pseudo-HTTP header and pseudo-content; the pseudo-content includes invalid or fake JavaScript comments or pictures or other media resources.

[0093] S4-2, constructing a fake URL as a fake URL parameter; generating an error status code, such as 403, 404 or 500, as a fake HTTP status code.

[0094] S4-3, embedding the random data, the pseudo-HTTP header, the pseudo-content, the fake URL parameter, and the fake HTTP status code into a normal HTTP response of the current request, judging whether the client responds to the pseudo-signal, for example, whether the client attempts to access the address in the pseudo-signal; if yes, determining that the response result of the current request is response; otherwise, determining that the response result of the current request is non-response.

[0095] S5, determining a request type of the current request and executing a protection strategy based on an actual request time interval of the current request and the response result;

[0096] judging whether the actual request time interval of the current request satisfies any one of conditions that the request interval meets a nonlinear function and the response result is non-response; if yes, determining that the request type of the current request is a legal request, the applicant is a real user, and the current request is agreed; otherwise, determining that the request type of the current request is an illegal request, the applicant is a crawler activity, and the current request is rejected and data protection is started, triggering an anti-crawler mechanism, for example, banning the IP of the application or limiting access.

[0097] As shown in the figure, an anti-crawler system based on dynamic request frequency modulation and pseudo-signal interference includes: Figure 2 A request activity data collection module is configured to collect historical request activity data of a user and network-server data at a current timestamp.

[0098] A parameter setting module is configured to adjust frequency modulation parameters and pseudo-signal generation parameters based on the historical request activity data of the user and the network-server data at the current timestamp.

[0099]

[0100] ​A protection strategy determination module is configured to calculate an overall bandwidth occupancy based on network-server data at a current timestamp, and determine a protection strategy for a current request.

[0101] An allowed request time interval calculation module is configured to generate an allowed request time interval for the current request based on a frequency modulation parameter and through a request interval nonlinear function.

[0102] An actual request time interval calculation module is configured to calculate an actual request time interval of a previous request and the current request based on the allowed request time interval and through the client making the current request.

[0103] A pseudo-signal generation and response module is configured to randomly generate a pseudo-signal based on a pseudo-signal generation parameter, and determine whether the current request is responded, to obtain a response result of the current request.

[0104] A data protection module is configured to determine a request type of the current request and execute the protection strategy based on the actual request time interval and the response result of the current request.

[0105] To sum up, the application adjusts the request time interval in real time through the nonlinear function, enhances the dynamic nature of protection, induces the crawler response by injecting the pseudo-signal, actively identifies the automated behavior, combines the dual verification of the time interval and the pseudo-signal response, greatly improves the precision of the crawler detection, dynamically adjusts the protection strategy according to the network-server data, ensures the strong system adaptability and increases the protection effect, provides the efficient anti-crawler ability through the lightweight mathematical model and the pseudo-signal injection, and is suitable for the high concurrency scene.

[0106] The above is only a specific embodiment of the application, but the protection scope of the application is not limited to this, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the application, which should be covered in the protection scope of the application. Therefore, the protection scope of the application should be limited by the protection scope of the claims.

Claims

1. A method for preventing web crawling based on dynamic request frequency modulation and pseudo-signal interference, characterized in that, include: Based on the user's historical request activity data and the network-server data at the current timestamp, the frequency modulation parameters and pseudo-signal generation parameters are adjusted to determine the protection strategy; the historical request activity data includes historical user behavior data, historical request frequency, and historical page request count for different historical access time periods; the network-server data includes the user's network traffic and server load data; Based on frequency modulation parameters, the allowed request time interval for the current request is generated through a request interval nonlinear function; the frequency modulation parameters include modulation amplitude, modulation frequency, modulation phase, and basic request interval. Based on the allowed request time interval, the current request is made by the client, and the actual request time interval between the previous request and the current request is calculated. Based on the pseudo-signal generation parameters, a pseudo-signal is randomly generated to determine whether the current request should be responded to, and the response result of the current request is obtained. Based on the actual request time interval and response result of the current request, determine the request type of the current request and execute the protection policy; The adjusted frequency modulation parameters and pseudo-signal generation parameters include: Filter historical user behavior data and historical request frequency to determine historical suspected abnormal behavior data and historical suspected abnormal request frequency; use the historical access time periods corresponding to the historical abnormal behavior data and historical abnormal request frequency as historical suspected abnormal access time periods. By analyzing historical suspected abnormal behavior data, historical suspected abnormal request frequency, and historical suspected abnormal access time periods, a comprehensive correlation analysis result is obtained. Based on the comprehensive correlation analysis results, the frequency modulation parameters in the previous round of requests are dynamically adjusted using an adaptive adjustment algorithm; Based on the activity data of the current request, dynamically adjust the pseudo-signal generation parameters from the previous request; the pseudo-signal generation parameters include the injection frequency and complexity of the pseudo-signal; The determination of the protection strategy for the current request includes: Based on the user's network traffic, calculate the overall bandwidth utilization; based on the overall bandwidth utilization, determine whether the current request is in a peak period; if so, adjust the protection strategy of the previous request to obtain the initial protection strategy; otherwise, use the protection strategy of the previous request as the initial protection strategy. Based on server load data, calculate server resource utilization; based on resource utilization, adjust the initial protection strategy to obtain the protection strategy for the current request.

2. The anti-crawler method based on dynamic request frequency modulation and pseudo-signal interference according to claim 1, characterized in that, The historical user behavior data includes historical access logs, historical request time intervals, historical request header information, and historical access paths; historical access logs include historical IP addresses and historical request URL parameters; historical access paths include deep pages and resource requests; historical request frequency includes the number of user requests or the speed of requests.

3. The anti-crawler method based on dynamic request frequency modulation and pseudo-signal interference according to claim 2, characterized in that, The process of filtering historical user behavior data and historical request frequencies to determine historical suspected abnormal behavior data and historical suspected abnormal request frequencies includes: Identify whether the frequency of historical requests within different historical access periods exceeds the request threshold; if so, treat the historical request frequency as the historical suspected abnormal request frequency; otherwise, treat the historical request frequency as the historical normal request frequency. Identify whether the historical request time intervals are the same within different historical access periods; if so, treat the historical request time interval as historical suspected abnormal behavior data; otherwise, treat the historical request time interval as historical normal behavior data. Identify whether historical access logs within different historical access time periods meet any of the following conditions: the historical IP address duplication rate reaches a duplication rate threshold, the geographical distribution of historical IP addresses changes, or the historical request URL parameters are completely identical; if so, the historical access log is regarded as historical suspected abnormal behavior data; otherwise, the historical access log is regarded as historical normal behavior data. Identify whether the historical access paths within different historical access time periods meet the condition of fixed historical access paths; if so, treat the historical access path as historical suspected abnormal behavior data; otherwise, treat the historical access path as historical normal behavior data. Identify whether the historical request header information within different historical access time periods is abnormal or lacks browser identifiers; if so, treat the historical request header information as suspected abnormal behavior data; otherwise, treat the historical request header information as normal behavior data. Identify whether the number of historical application pages within different historical access time periods has reached the page count threshold; if so, treat the number of historical application pages as historical suspected abnormal behavior data; otherwise, treat the number of historical application pages as historical normal behavior data.

4. The anti-crawler method based on dynamic request frequency modulation and pseudo-signal interference according to claim 1, characterized in that, The step of calculating the actual request time interval between the previous request and the current request based on the allowed request time interval includes: The client determines the allowed time interval between receiving the current request and then submits the current request. Determine if the client sends the current request after the allowed request interval; if so, calculate the actual request interval between the previous request and the current request; otherwise, directly determine that the current request is an illegal request, the applicant is a web crawler, reject the current request, enable data protection, and trigger the anti-crawler mechanism.

5. The anti-crawler method based on dynamic request frequency modulation and pseudo-signal interference according to claim 1, characterized in that, The process of determining the request type of the current request and executing the protection strategy based on the actual request time interval and response result includes: Determine if the actual request time interval of the current request meets any one of the following conditions: the request interval is non-linear, or the response result is no response. If so, determine that the current request type is a legitimate request, the applicant is a real user, and agree to the current request. Otherwise, determine that the current request type is an illegitimate request, the applicant is a web crawler, reject the current request, enable data protection, and trigger the anti-crawler mechanism.

6. An anti-crawler system based on dynamic request frequency modulation and pseudo-signal interference, used to implement the anti-crawler method based on dynamic request frequency modulation and pseudo-signal interference as described in any one of claims 1 to 5, characterized in that, include: The request activity data collection module is used to collect users' historical request activity data and network-server data at the current timestamp; The parameter setting module is used to adjust frequency modulation parameters and pseudo-signal generation parameters based on the user's historical request activity data and network-server data at the current timestamp. The protection strategy determination module is used to calculate the overall bandwidth utilization based on the network-server data at the current timestamp and determine the protection strategy for the current request. The allowed request time interval calculation module is used to generate the allowed request time interval for the current request based on frequency modulation parameters and through a request interval nonlinear function. The actual request time interval calculation module is used to calculate the actual request time interval between the previous request and the current request based on the allowed request time interval when the client submits the current request. The pseudo-signal generation and response module is used to randomly generate pseudo-signals based on pseudo-signal generation parameters, determine whether the current request should be responded to, and obtain the response result of the current request. The data protection module is used to determine the request type of the current request and execute protection policies based on the actual request time interval and response result of the current request.

Images