A method for knowledge base anti-crawler

Abstract

A knowledge base anti-crawler method, comprising the following steps: step 1, checking the regular request header to identify the crawler tool when the server receives the request; step 2, performing API reinforcement to limit the IP amount; step 3, performing request frequency limitation to control the access frequency of the day and the recent time; step 4, performing dynamic protection to identify malicious behavior through a verification code; the above steps are carried out in the above order.

Description

Technical Field

[0001] This invention relates to knowledge base security issues, and in particular to a method for preventing web scraping in knowledge bases. Background Technology

[0002] Knowledge bases face risks such as data leakage and resource abuse from malicious crawlers, and need to balance security and user experience through anti-crawler technology. Summary of the Invention

[0003] To address the aforementioned technical issues, a method for anti-crawling of knowledge bases is proposed, comprising the following steps: Step 1, when the server receives a request, it verifies the regular request header to identify crawling tools; Step 2: Perform API hardening to limit the number of IPs; Step 3: Implement request frequency limits to control the frequency of accesses within the current day and at recent times; Step 4: Implement dynamic protection by identifying malicious behavior through CAPTCHA. The above steps are carried out in the order described above.

[0004] Preferably, in step 1, at least one of the User-Agent, referer, and Content-Type in the request header is validated.

[0005] Preferably, in step 2, it is checked whether the IP address is in a preset whitelist.

[0006] Preferably, in step 3, the user's total daily access count is checked and compared with a preset daily threshold. If it exceeds the daily threshold, a verification code is popped up for verification. Otherwise, the recent access frequency is checked and compared with a preset recent access frequency threshold. If it exceeds the preset recent access frequency threshold, a verification code is popped up for verification.

[0007] Preferably, in step 4, the number of times the verification code pops up is detected and compared with the threshold number of times the verification code pops up. If the number of times the verification code pops up exceeds the threshold number, the user cannot access the site again for the day and is added to the blacklist.

[0008] Preferably, the threshold for the number of times the verification code pops up is greater than or equal to 5.

[0009] Preferably, the preset recent access frequency threshold is selected as greater than or equal to 5 times / 2 seconds.

[0010] Preferably, the blacklist in step 4 is automatically unblocked after 24 hours.

[0011] This enables the construction of a layered defense system that combines real-time user access habits to ensure data security while maintaining efficient access. Core measures include: request frequency limiting (such as IP rate limiting), dynamic protection (behavioral analysis, CAPTCHA), and API hardening (IP verification), while optimizing access for legitimate users (such as whitelisting, CDN caching). Attached Figure Description

[0012] Figure 1 A schematic flowchart of the present invention is shown; Figure 2 and Figure 3 The application effects of the present invention are shown. Detailed Implementation

[0013] like Figure 1 As shown, a method for anti-crawling of knowledge bases includes the following steps: Step 1, when the server receives a request, it verifies the regular request header to identify crawling tools; Step 2: Perform API hardening to limit the number of IPs; Step 3: Implement request frequency limits to control the frequency of accesses within the current day and at recent times; Step 4: Implement dynamic protection by identifying malicious behavior through CAPTCHA. The above steps are carried out in the order described above.

[0014] Preferably, in step 1, at least one of the User-Agent, referer, and Content-Type in the request header is validated.

[0015] Preferably, in step 2, it is checked whether the IP address is in a preset whitelist.

[0016] Preferably, in step 3, the user's total daily access count is checked and compared with a preset daily threshold. If it exceeds the daily threshold, a verification code is popped up for verification. Otherwise, the recent access frequency is checked and compared with a preset recent access frequency threshold. If it exceeds the preset recent access frequency threshold, a verification code is popped up for verification.

[0017] Preferably, in step 4, the number of times the verification code pops up is detected and compared with the threshold number of times the verification code pops up. If the number of times the verification code pops up exceeds the threshold number, the user cannot access the site again for the day and is added to the blacklist.

[0018] Preferably, the threshold for the number of times the verification code pops up is greater than or equal to 5.

[0019] Preferably, the preset recent access frequency threshold is selected as greater than or equal to 5 times / 2 seconds.

[0020] Preferably, the blacklist in step 4 is automatically unblocked after 24 hours.

[0021] This enables the construction of a layered defense system that combines real-time user access habits to ensure data security while maintaining efficient access. Core measures include: request frequency limiting (such as IP rate limiting), dynamic protection (behavioral analysis, CAPTCHA), and API hardening (IP verification), while optimizing access for legitimate users (such as whitelisting, CDN caching).

[0022] exist Figure 2 and Figure 3 The interface shows the technical effects achieved by applying the technical solution of this invention, wherein... Figure 2 A pop-up window appeared requesting a verification code due to frequent operations. Figure 3 In the middle, you can see the user interface when using it normally without triggering the anti-crawler mechanism.

[0023] Preferred embodiments of the present invention have been exemplified above with reference to the accompanying drawings. However, those skilled in the art should understand that, without departing from the scope of protection of the appended claims, they can form new technical solutions by adding features, combining features, etc.

Claims

1. A method for preventing web scraping in knowledge bases, characterized in that, The steps include: Step 1, when the server receives a request, it verifies the regular request headers to identify the crawler tool; Step 2: Perform API hardening to limit the number of IPs; Step 3: Implement request frequency limits to control the frequency of accesses within the current day and at recent times; Step 4: Implement dynamic protection by identifying malicious behavior through CAPTCHA. The above steps are carried out in the order described above.

2. The method for anti-crawling of knowledge bases according to claim 1, characterized in that, In step 1, at least one of the User-Agent, referer, and Content-Type in the request header is validated.

3. The method for anti-crawling of knowledge bases according to claim 2, characterized in that, In step 2, check if the IP address is in the preset whitelist.

4. The method for anti-crawling of knowledge bases according to claim 3, characterized in that, In step 3, the user's total daily access count is checked and compared with a preset daily threshold. If it exceeds the daily threshold, a verification code is popped up for verification. Otherwise, the recent access frequency is checked and compared with a preset recent access frequency threshold. If it exceeds the preset recent access frequency threshold, a verification code is popped up for verification.

5. The method for anti-crawling of knowledge bases according to claim 4, characterized in that, In step 4, the number of times the verification code pops up is detected and compared with the threshold number of times the verification code pops up. If the number of times the verification code pops up exceeds the threshold number, the user cannot access the site again for the day and is added to the blacklist.

6. The method for anti-crawling of knowledge bases according to claim 5, characterized in that, The threshold for the number of times the verification code pops up is set to be greater than or equal to 5.

7. The method for anti-crawling of knowledge bases according to claim 6, characterized in that, The preset recent access frequency threshold is set to greater than or equal to 5 times / 2 seconds.

8. The method for anti-crawling of knowledge bases according to claim 7, characterized in that, The blacklist from step 4 will be automatically unblocked after 24 hours.

Images