A website self-adaptive anti-crawling method and system based on user behavior analysis

Abstract

The application discloses a website self-adaptive anti-crawling method and system based on user behavior analysis. The method comprises the following steps: collecting behavior data of each access request in real time; calculating the user behavior data through multi-dimensional feature weighting to output a comprehensive risk score of the access subject; triggering the corresponding level of disposal strategy according to the risk level interval where the comprehensive risk score is located, and the disposal measures of different intensities are corresponding to each level and gradually progress from non-susceptible verification to access restriction; establishing a normal value range of the behavior data based on historical normal traffic, and regularly inputting misjudgment information for analysis, automatically adjusting the feature weight, risk level threshold and disposal strategy parameters of the risk scoring model in combination with the access volume and time law to realize self-optimization; and moderately increasing the risk level if the access IP appears in the security alarm log. The application can effectively improve the anti-crawling identification accuracy and defense flexibility.

Description

Technical Field

[0001] This invention belongs to the field of network security and information protection technology, specifically relating to a website adaptive anti-crawling method and system based on user behavior analysis. Background Technology

[0002] With the rapid development of internet technology and the increasing prominence of data value, various websites and web applications face increasingly severe threats from web crawlers and data hijacking. Malicious web crawlers not only consume large amounts of server computing resources and network bandwidth, causing access delays or even service interruptions for legitimate users, but may also lead to the leakage of sensitive business data. Therefore, how to efficiently and accurately identify and block malicious web crawlers has become an important issue in the field of network security protection.

[0003] Currently, existing website anti-scraping protection methods mainly rely on the following traditional technical solutions: Protection mechanisms based on static rules and blacklists / whitelists: Traditional systems typically intercept access by configuring IP blacklists, restricting access to specified User-Agent strings, or setting fixed thresholds for the frequency of single IP access.

[0004] A mandatory verification mechanism based on CAPTCHA: When suspected abnormal access is detected, a verification interaction such as a graphic CAPTCHA or a sliding puzzle will be displayed directly to forcibly interrupt the user's continuous access.

[0005] A single-dimensional defense strategy based on fixed thresholds: Set a static access rate limit for a specific page or API interface, and trigger IP blocking or connection blocking once the threshold is exceeded.

[0006] However, the aforementioned existing technologies have the following obvious defects in practical applications and the reasons for these defects are as follows: Lack of ability to identify anthropomorphic / advanced crawlers: Existing static rules mainly rely on single and fixed features for matching. Because new crawlers widely use distributed proxy IP pools, automated headless browsers (such as Puppeteer and Selenium), and simulated human click delays, they are highly concealed in single-dimensional indicators, making it difficult for traditional mechanisms based on static rules and single-dimensional thresholds to effectively identify such anthropomorphic crawlers.

[0007] Poor user experience and high false positive rate: Existing protection mechanisms lack flexible, tiered handling strategies. Their defense logic is too crude (often directly blocking or displaying CAPTCHAs), and static thresholds cannot dynamically adapt to normal traffic fluctuations during peak periods. This makes them prone to misjudging legitimate users as web crawlers during traffic surges, resulting in frequent CAPTCHA pop-ups or even direct access blocking, severely damaging the user experience for legitimate users.

[0008] Outdated strategies and high maintenance costs: Existing systems rely heavily on static configuration and post-event maintenance of rules and thresholds based on human experience. Since web crawler developers constantly change attack strategies and behavior patterns, and traditional systems lack automatic adjustment mechanisms based on actual interception effects and false positive feedback, protection strategy updates are delayed. Furthermore, complete reliance on frequent manual adjustments of rule parameters by operations and maintenance personnel significantly increases system maintenance costs and response latency. Summary of the Invention

[0009] The following provides a brief overview of one or more aspects to offer a basic understanding of them. This overview is not an exhaustive summary of all conceived aspects, nor is it intended to identify key or decisive elements of all aspects, nor to define the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form to prepare for the more detailed descriptions that follow.

[0010] The purpose of this invention is to address the shortcomings of existing technologies, such as low recognition rate, high false positive rate, damage to user experience, and high policy maintenance cost of anthropomorphic crawlers due to static rules, single feature dimensions, and lack of dynamic adaptive mechanisms. This invention provides a website adaptive anti-crawling method and system based on user behavior analysis, which can accurately identify complex anthropomorphic crawlers, reduce the false positive rate of normal traffic, take into account user access experience, and reduce policy maintenance costs.

[0011] The technical solution of this invention is as follows: This invention discloses a website adaptive anti-crawling method based on user behavior analysis, comprising the following steps: Collect behavioral data for each access request in real time; Based on user behavior data, a comprehensive risk score for the accessing entity is output through multi-dimensional feature weighted calculation. Based on the risk level range where the comprehensive risk score falls, the corresponding level of response strategy will be triggered.

[0012] According to an embodiment of the website adaptive anti-crawling method based on user behavior analysis of the present invention, the behavior data includes: request attribute data, access time sequence data, interaction behavior data, and associated log data.

[0013] According to an embodiment of the website adaptive anti-crawling method based on user behavior analysis of the present invention, the comprehensive risk score is calculated as follows: The overall risk score is calculated as follows: Request attribute risk score × weight W1 + access time sequence risk score S2 × weight W2 + interaction behavior risk score S3 × weight W3 + historical alarm association risk score S4 × weight W4. The values ​​of the request attribute risk score, access time sequence risk score, interaction behavior risk score, and historical alarm association risk score range from 1 to 100, and the sum of all weights W1 to W4 is 1.

[0014] According to an embodiment of the website adaptive anti-crawling method based on user behavior analysis of the present invention, the weights W1, W2, W3, and W4 are 0.3, 0.25, and 0.15, respectively.

[0015] According to an embodiment of the website adaptive anti-crawling method based on user behavior analysis of the present invention, the risk level includes at least three levels: low risk, medium risk, and high risk, with each level corresponding to different intensity of handling measures, progressing step by step from seamless verification to access restriction.

[0016] According to an embodiment of the website adaptive anti-crawling method based on user behavior analysis of the present invention, if the comprehensive risk score is greater than or equal to 0 and less than 30, it corresponds to low risk, and the handling measure is to allow access and continue to observe without any user-perceived intervention measures; if the comprehensive risk score is greater than or equal to 30 and less than 60, it corresponds to medium risk, and the handling measure is to trigger a restriction on access frequency or appropriately reduce the response rate, with the user basically unaware; if the comprehensive risk score is greater than or equal to 60, it corresponds to high risk, and strong intervention measures, including temporary blocking, are triggered.

[0017] According to an embodiment of the website adaptive anti-crawling method based on user behavior analysis of the present invention, the method further includes a strategy adaptive adjustment step: Establish a normal value range for behavioral data based on historical normal traffic; By periodically inputting and analyzing misjudgment information, and combining it with access volume and time patterns, the system automatically adjusts the weights, risk level thresholds, and handling strategy parameters in the comprehensive risk score calculation, thereby achieving self-optimization of the anti-scraping strategy.

[0018] According to an embodiment of the website adaptive anti-crawling method based on user behavior analysis of the present invention, the method further includes a multi-log linkage step: If a user's IP address appears in relevant logs, including security alerts, the risk level is increased.

[0019] According to an embodiment of the website adaptive anti-crawling method based on user behavior analysis of the present invention, the method further includes: It provides a visual operations management backend, allowing operations personnel to view risk status, adjust strategy parameters, configure blacklists and whitelists, and define custom rules, enabling collaborative operation between manual intervention and automatic adjustment.

[0020] This invention also discloses a website adaptive anti-crawling system based on user behavior analysis, comprising: The behavior data acquisition module is used to collect behavior data for each access request in real time. The risk scoring module is used to calculate and output a comprehensive risk score for the access subject based on user behavior data and through multi-dimensional feature weighting. The strategy triggering module is used to trigger corresponding level of handling strategies based on the risk level range in which the comprehensive risk score falls.

[0021] The present invention also discloses a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the website adaptive anti-crawling method based on user behavior analysis as described above.

[0022] The present invention also discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the website adaptive anti-crawling method based on user behavior analysis as described above.

[0023] The present invention also discloses a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the website adaptive anti-crawling method based on user behavior analysis as described above.

[0024] Compared with the prior art, the present invention has the following advantages: 1. Based on multi-dimensional behavioral feature weighted scoring, the accuracy of recognition is significantly improved and the false positive rate is reduced. This invention collects multi-dimensional behavioral data in real time, including request attribute data, access time sequence data, interaction behavior data, and related log data. It then performs a weighted calculation based on preset weights (e.g., request attribute risk score weight 0.3, access time sequence risk score weight 0.3, interaction behavior risk score weight 0.25, and historical alarm association risk score weight 0.15) to output a comprehensive risk score. Simultaneously, it integrates multiple log linkages, increasing the risk level when a user's IP appears in security alarm-related logs. These technical features enable the system to comprehensively assess risk based on multi-dimensional user behavior profiles, including access frequency, path characteristics, interaction behavior, and time sequence patterns. This overcomes the limitations of traditional single-dimensional detection, accurately distinguishing between normal users and anthropomorphic crawlers, significantly reducing the probability of false blocking and misjudgment, and avoiding impact on normal business access.

[0025] 2. Implement a tiered response strategy that balances security defense with a user-friendly experience.

[0026] This invention categorizes risks into low risk (0-30 points), medium risk (30-60 points), and high risk based on a comprehensive risk score. The system is divided into three risk levels: low-risk (allowed access with continuous monitoring), medium-risk (access frequency restricted or speed reduced), and high-risk (temporary ban). This progressively escalating approach, from seamless verification to access restrictions, ensures a user-friendly experience with minimal disruption during low- and medium-risk scenarios. Simultaneously, the systematic defense effectively blocks high-risk crawlers, addressing common pain points in the industry such as rigid anti-crawling strategies and outdated defenses.

[0027] 3. By combining adaptive adjustment with visual collaboration, the operational costs of strategies are reduced and their scalability is enhanced.

[0028] This invention establishes a normal value range based on historical normal traffic and periodically inputs and analyzes false positive information. It automatically adjusts calculation weights, risk level thresholds, and handling strategy parameters based on access volume and time patterns, achieving self-optimization of anti-scraping strategies. Simultaneously, it provides a visual operation management backend, allowing operators to view risk status, adjust strategy parameters, configure blacklists and whitelists, and customize rules. The collaborative operation of adaptive strategy adjustment and manual intervention gives the protection strategy strong flexibility, autonomy, and controllability, good scalability, and significantly reduces manual optimization and maintenance costs.

[0029] 4. It has extremely high engineering implementation and practical value.

[0030] This invention can be deployed at the access layer and takes effect without requiring large-scale modifications to existing business systems. It can quickly adapt to various enterprise website architectures and has high practical value. Attached Figure Description

[0031] The above-described features and advantages of the present invention will be better understood after reading the following detailed description of embodiments of the present disclosure in conjunction with the accompanying drawings. In the drawings, components are not necessarily drawn to scale, and components having similar related properties or features may have the same or similar reference numerals.

[0032] Figure 1 The flowchart of an embodiment of the website adaptive anti-crawling method based on user behavior analysis of the present invention is shown.

[0033] Figure 2 The principle of an embodiment of the website adaptive anti-crawling system based on user behavior analysis of the present invention is shown. Detailed Implementation

[0034] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. It should be noted that the aspects described below with reference to the accompanying drawings and specific embodiments are merely exemplary and should not be construed as limiting the scope of protection of the present invention in any way.

[0035] Figure 1 The flowchart illustrates an embodiment of the website adaptive anti-scraping method based on user behavior analysis of the present invention. See also Figure 1 The method in this embodiment includes the following steps.

[0036] First, collect behavioral data for each access request in real time.

[0037] Typically, data collection probes are deployed at the website access layer (such as Nginx reverse proxy, API gateway, or WAF device) to collect full data for every HTTP / HTTPS request.

[0038] Behavioral data includes, for example: Request attribute data: IP address, User-Agent, Referer, Cookie, request method, request URL, request time, response status code, request body size, whether there are static resources, etc. Access time-series data: number of requests per unit time, request interval time, request time distribution, etc.; Interaction behavior data: page dwell time, access order, etc.

[0039] Related log data: Whether it appears in the security alert log.

[0040] Then, based on user behavior data, a comprehensive risk score for the accessing entity is output through multi-dimensional feature weighting calculation.

[0041] The comprehensive risk score R is calculated as follows: Comprehensive Risk Score R = Request Attribute Risk Score × Weight W1 + Access Timing Risk Score S2 × Weight W2 + Interaction Behavior Risk Score S3 × Weight W3 + Historical Alarm Correlation Risk Score S4 × Weight W4. The values ​​of Request Attribute Risk Score, Access Timing Risk Score, Interaction Behavior Risk Score, and Historical Alarm Correlation Risk Score range from 1 to 100. The sum of all weights W1 to W4 is 1, and the default values ​​are: weight W1 is 0.3, weight W2 is 0.3, weight W3 is 0.25, and weight W4 is 0.15.

[0042] As can be seen from the above formula, if the User Agent (UA) is abnormal, request headers are missing, there are a large number of accesses to dynamic interfaces, and very few accesses to static resources, the request attribute risk score will increase; if the request frequency exceeds the baseline, the request intervals are uniform and fixed, and requests are initiated in batches during concentrated periods, the access timing risk score will increase; if the page stay is extremely short and the access path is chaotic and has no normal browsing order, the interaction behavior risk score will increase; if there are historical security alarm records, the high risk score associated with historical alarms will be used; if there are no alarm records, a score of 0 will be recorded. All weight values ​​can be dynamically adjusted in the operations backend.

[0043] In addition, a historical risk attenuation coefficient K (with a value range of 0.7 to 1.0) is introduced. The impact of historical risks is automatically reduced when there are no abnormalities in the long term, and the final risk value is calculated as follows: final risk value Rfinal = R×K.

[0044] Finally, based on the risk level range in which the comprehensive risk score falls, the corresponding level of response strategy is triggered.

[0045] In embodiments of the present invention, the risk level includes at least three levels: low risk, medium risk, and high risk. Each level corresponds to different levels of handling measures, progressing step by step from seamless verification to access restrictions.

[0046] More specifically, a comprehensive risk score greater than or equal to 0 and less than 30 corresponds to low risk, and the handling measure is to allow access and continue to observe, without any intervention measures perceived by the user; a comprehensive risk score greater than or equal to 30 and less than 60 corresponds to medium risk, and the handling measure is to trigger restrictions on access frequency or appropriately reduce the response rate, with the user basically unaware of the intervention; a comprehensive risk score greater than or equal to 60 corresponds to high risk, triggering strong intervention measures including temporary bans.

[0047] Preferably, the method in this embodiment further includes a strategy adaptive adjustment step: Establish a normal value range for behavioral data based on historical normal traffic. In this step, historical normal traffic data within a certain time window is used to statistically analyze the distribution ranges of various indicators such as request attributes, access sequence, and interaction behavior, thus constructing a baseline for normal behavioral values. The greater the deviation of the access subject's indicators from the normal value baseline, the higher the risk sub-score for the corresponding dimension.

[0048] Regularly inputting and analyzing misjudgment information, and combining it with traffic volume and time patterns, automatically adjusts the weights, risk level thresholds, and handling strategy parameters in the comprehensive risk score calculation, achieving self-optimization of the anti-scraping strategy. Specifically, it continuously collects misjudgment event data, comprehensively considers the fluctuations in overall website traffic, and time-based traffic patterns such as daily / holiday periods, and periodically performs parameter optimization: ① Dynamically adjusts the weights of W1, W2, W3, and W4 in the risk assessment formula; ② Adjusts the score range thresholds corresponding to low / medium / high risks; ③ Synchronously updates the handling strategy parameters such as traffic limiting, verification, and blocking for each risk level. Relying on the above closed-loop feedback mechanism, the anti-scraping strategy achieves continuous self-optimization.

[0049] Preferably, the method in this embodiment further includes a multi-log linkage step: If a user's IP address appears in relevant logs, including security alerts, the risk level is increased.

[0050] Preferably, the method in this embodiment further includes: providing a visual operation management backend, which supports operators to view risk status, adjust strategy parameters, configure blacklists and whitelists, and define custom rules, so as to realize the collaborative operation of manual intervention and automatic adjustment.

[0051] Figure 2 The present invention illustrates a website adaptive anti-crawling system based on user behavior analysis, the system comprising the following modules.

[0052] The behavior data acquisition module is used to collect behavior data for each access request in real time.

[0053] Typically, data collection probes are deployed at the website access layer (such as Nginx reverse proxy, API gateway, or WAF device) to collect full data for every HTTP / HTTPS request.

[0054] Behavioral data includes, for example: Request attribute data: IP address, User-Agent, Referer, Cookie, request method, request URL, request time, response status code, request body size, whether there are static resources, etc. Access time-series data: number of requests per unit time, request interval time, request time distribution, etc.; Interaction behavior data: page dwell time, access order, etc.

[0055] Related log data: Whether it appears in the security alert log.

[0056] The risk scoring module is used to calculate and output a comprehensive risk score for the access subject based on user behavior data and through multi-dimensional feature weighting.

[0057] The comprehensive risk score R is calculated as follows: Comprehensive Risk Score R = Request Attribute Risk Score × Weight W1 + Access Timing Risk Score S2 × Weight W2 + Interaction Behavior Risk Score S3 × Weight W3 + Historical Alarm Correlation Risk Score S4 × Weight W4. The values ​​of Request Attribute Risk Score, Access Timing Risk Score, Interaction Behavior Risk Score, and Historical Alarm Correlation Risk Score range from 1 to 100. The sum of all weights W1 to W4 is 1, and the default values ​​are: weight W1 is 0.3, weight W2 is 0.3, weight W3 is 0.25, and weight W4 is 0.15.

[0058] As can be seen from the above formula, if the User Agent (UA) is abnormal, request headers are missing, there are a large number of accesses to dynamic interfaces, and very few accesses to static resources, the request attribute risk score will increase; if the request frequency exceeds the baseline, the request intervals are uniform and fixed, and requests are initiated in batches during concentrated periods, the access timing risk score will increase; if the page stay is extremely short and the access path is chaotic and has no normal browsing order, the interaction behavior risk score will increase; if there are historical security alarm records, the high risk score associated with historical alarms will be used; if there are no alarm records, a score of 0 will be recorded. All weight values ​​can be dynamically adjusted in the operations backend.

[0059] In addition, a historical risk attenuation coefficient K (with a value range of 0.7 to 1.0) is introduced. The impact of historical risks is automatically reduced when there are no abnormalities in the long term, and the final risk value is calculated as follows: final risk value Rfinal = R×K.

[0060] The strategy triggering module is used to trigger corresponding level of handling strategies based on the risk level range in which the comprehensive risk score falls.

[0061] In embodiments of the present invention, the risk level includes at least three levels: low risk, medium risk, and high risk. Each level corresponds to different levels of handling measures, progressing step by step from seamless verification to access restrictions.

[0062] More specifically, a comprehensive risk score greater than or equal to 0 and less than 30 corresponds to low risk, and the handling measure is to allow access and continue to observe, without any intervention measures perceived by the user; a comprehensive risk score greater than or equal to 30 and less than 60 corresponds to medium risk, and the handling measure is to trigger restrictions on access frequency or appropriately reduce the response rate, with the user basically unaware of the intervention; a comprehensive risk score greater than or equal to 60 corresponds to high risk, triggering strong intervention measures including temporary bans.

[0063] Preferably, the system in this embodiment also includes a strategy adaptive adjustment function: Establish normal value ranges for behavioral data based on historical normal traffic. Using historical normal traffic data within a time window, statistically analyze the distribution ranges of various indicators such as request attributes, access timing, and interaction behavior to construct a baseline for normal behavioral values. The greater the deviation of the access subject's indicators from the normal value baseline, the higher the corresponding risk sub-score.

[0064] Regularly inputting and analyzing misjudgment information, and combining it with traffic volume and time patterns, automatically adjusts the weights, risk level thresholds, and handling strategy parameters in the comprehensive risk score calculation, achieving self-optimization of the anti-scraping strategy. Specifically, it continuously collects misjudgment event data, comprehensively considers the fluctuations in overall website traffic, and time-based traffic patterns such as daily / holiday periods, and periodically performs parameter optimization: ① Dynamically adjusts the weights of W1, W2, W3, and W4 in the risk assessment formula; ② Adjusts the score range thresholds corresponding to low / medium / high risks; ③ Synchronously updates the handling strategy parameters such as traffic limiting, verification, and blocking for each risk level. Relying on the above closed-loop feedback mechanism, the anti-scraping strategy achieves continuous self-optimization.

[0065] Preferably, the system in this embodiment also includes a multi-log linkage function: If a user's IP address appears in relevant logs, including security alerts, the risk level is increased.

[0066] Preferably, the system in this embodiment also includes the following functions: providing a visual operation management backend, supporting operators to view risk status, adjust strategy parameters, configure blacklists and whitelists, and define custom rules, so as to realize the collaborative operation of manual intervention and automatic adjustment.

[0067] Furthermore, this invention also discloses a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements... Figure 1 The steps of an embodiment of a website adaptive anti-crawling method based on user behavior analysis are shown below.

[0068] Furthermore, this invention also discloses a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements as follows: Figure 1 The steps of an embodiment of a website adaptive anti-crawling method based on user behavior analysis are shown below.

[0069] Furthermore, this invention also discloses a computer program product, including a computer program / instructions. When the computer program / instructions are executed by a processor, they implement... Figure 1 The steps of an embodiment of a website adaptive anti-crawling method based on user behavior analysis are shown below.

[0070] Although the methods described above are illustrated and depicted as a series of actions for the sake of simplicity, it should be understood and appreciated that these methods are not limited by the order of the actions, as some actions may occur in a different order and / or concurrently with other actions from the illustrations and descriptions herein or not illustrated and described herein but which may be understood by those skilled in the art, according to one or more embodiments.

[0071] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps are described above in a generalized manner in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the invention.

[0072] The various illustrative logic blocks, modules, and circuits described in conjunction with the embodiments disclosed herein can be implemented or performed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but in alternatives, it may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration.

[0073] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of both. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor such that the processor can read and write information to / from the storage medium. In an alternative, the storage medium may be integrated into the processor. The processor and storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and storage medium may reside as discrete components in the user terminal.

[0074] In one or more exemplary embodiments, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functionality may be stored or transmitted as one or more instructions or code on or through a computer-readable medium. A computer-readable medium includes both computer storage media and communication media, encompassing any medium that facilitates the transfer of a computer program from one location to another. A storage medium may be any available medium accessible to a computer. By way of example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and is accessible to a computer. Any connection is also legitimately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of a medium. As used in this article, disk and disc include compact discs (CDs), laser discs, optical discs, digital multi-purpose discs (DVDs), floppy disks, and Blu-ray discs. Disks typically reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of these should also be included within the scope of computer-readable media.

[0075] The prior description of this disclosure is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not intended to be limited to the examples and designs described herein, but should be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A website adaptive anti-crawling method based on user behavior analysis, characterized in that, Includes the following steps: Collect behavioral data for each access request in real time; Based on user behavior data, a comprehensive risk score for the accessing entity is output through multi-dimensional feature weighted calculation. Based on the risk level range where the comprehensive risk score falls, the corresponding level of response strategy will be triggered.

2. The website adaptive anti-crawling method based on user behavior analysis according to claim 1, characterized in that, The behavioral data includes: request attribute data, access time sequence data, interaction behavior data, and associated log data.

3. The website adaptive anti-crawling method based on user behavior analysis according to claim 2, characterized in that, The overall risk score is calculated as follows: The overall risk score is calculated as follows: Request attribute risk score × weight W1 + access time sequence risk score S2 × weight W2 + interaction behavior risk score S3 × weight W3 + historical alarm association risk score S4 × weight W4. The values ​​of the request attribute risk score, access time sequence risk score, interaction behavior risk score, and historical alarm association risk score range from 1 to 100, and the sum of all weights W1 to W4 is 1.

4. The website adaptive anti-crawling method based on user behavior analysis according to claim 3, characterized in that, The weights W1, W2, W3, and W4 are 0.3, 0.25, and 0.15 respectively.

5. The website adaptive anti-crawling method based on user behavior analysis according to claim 3, characterized in that, The risk levels include at least three levels: low risk, medium risk, and high risk. Each level corresponds to different levels of handling measures, progressing from seamless verification to access restrictions.

6. The website adaptive anti-crawling method based on user behavior analysis according to claim 5, characterized in that, If the overall risk score is greater than or equal to 0 and less than 30, it corresponds to low risk. The handling measure is to allow access and continue to observe, without any intervention measures that are perceived by the user. If the overall risk score is greater than or equal to 30 and less than 60, it corresponds to medium risk. The handling measure is to trigger restrictions on access frequency or appropriately reduce the response rate, with the user basically unaware of the intervention. If the overall risk score is greater than or equal to 60, it corresponds to high risk, triggering strong intervention measures, including temporary bans.

7. The website adaptive anti-crawling method based on user behavior analysis according to claim 1, characterized in that, The method also includes a policy adaptive adjustment step: Establish a normal value range for behavioral data based on historical normal traffic; By periodically inputting and analyzing misjudgment information, and combining it with access volume and time patterns, the system automatically adjusts the weights, risk level thresholds, and handling strategy parameters in the comprehensive risk score calculation, thereby achieving self-optimization of the anti-scraping strategy.

8. The website adaptive anti-crawling method based on user behavior analysis according to claim 1, characterized in that, The method also includes a multi-log linkage step: If a user's IP address appears in relevant logs, including security alerts, the risk level is increased.

9. The website adaptive anti-crawling method based on user behavior analysis according to claim 1, characterized in that, The method also includes: It provides a visual operations management backend, allowing operations personnel to view risk status, adjust strategy parameters, configure blacklists and whitelists, and define custom rules, enabling collaborative operation between manual intervention and automatic adjustment.

10. A website adaptive anti-crawling system based on user behavior analysis, characterized in that, include: The behavior data acquisition module is used to collect behavior data for each access request in real time. The risk scoring module is used to calculate and output a comprehensive risk score for the access subject based on user behavior data and through multi-dimensional feature weighting. The strategy triggering module is used to trigger corresponding level of handling strategies based on the risk level range in which the comprehensive risk score falls.

11. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the website adaptive anti-crawling method based on user behavior analysis as described in any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the website adaptive anti-crawling method based on user behavior analysis as described in any one of claims 1 to 9.

13. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the website adaptive anti-crawling method based on user behavior analysis as described in any one of claims 1 to 9.

Images