An intelligent anti-crawler interception system

By building a multi-dimensional enterprise-specific defense framework and virtual web page monitoring technology, the problem that existing anti-crawler systems cannot effectively identify and intercept malicious crawlers is solved, and accurate identification and interception of malicious crawlers is achieved, and the system's defense capabilities are improved.

CN115550051BActive Publication Date: 2025-02-11WUHAN JIYI NETWORK TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202211261820.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-10-14
Publication Date
2025-02-11
Estimated Expiration
2042-10-14

AI Technical Summary

Technical Problem

Existing anti-crawler methods cannot effectively intercept crawler requests because malicious crawlers can pretend to be real users by imitating browser request headers.

Method used

Design an intelligent anti-crawler interception system to build a multi-dimensional enterprise-specific defense framework through a virtual defense platform, combine verification code verification and plug-in monitoring, identify and mark abnormal access behaviors, and conduct real-time monitoring and interception through virtual web pages.

Benefits of technology

It realizes accurate identification and interception of malicious crawlers, reduces the risk of accidentally injuring normal users, and improves the identification efficiency and defense capabilities of the anti-crawler system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115550051B_ABST
    Figure CN115550051B_ABST
Patent Text Reader

Abstract

The present invention discloses an intelligent anti-crawler interception system, including a virtual defense platform, wherein the virtual defense platform includes a defense construction unit, an abnormal sharing unit, a front-end processing unit and a virtual construction unit, and the present invention relates to the field of anti-crawler interception technology. The intelligent anti-crawler interception system directly performs deployment verification on the access data of registered users by constructing a multi-dimensional enterprise-specific defense framework, marks the registered users who meet the abnormal standard for defense abnormality, and uses the form of plug-ins to record the real access data of registered users, compares it with the verification access data of the exclusive defense framework, marks the registered users with inconsistent results for plug-in abnormality, realizes the identification of registered users with risks, cooperates with the setting of virtual web pages, and realizes the accurate determination and interception of malicious crawlers by monitoring the downloading of virtual web page materials.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of anti-crawler interception, and in particular to an intelligent anti-crawler interception system. Background Art

[0002] With the rapid development of the Internet, the impact of web crawlers has increased. Malicious crawlers have generated a large number of fake users, seriously increasing the operating costs of enterprises. In addition, enterprise resources have been taken away in batches, resulting in the loss of competitiveness and data security cannot be guaranteed.

[0003] At present, the more commonly used anti-crawler means are usually: HTTP request header analysis and interception. However, the information of the HTTP request header is encapsulated by the browser, but it can be easily simulated by tools. At the same time, multiple sets of browser information can be preset, which are randomly used and dynamically changed during access. The preset request header information uses highly simulated data. The server cannot distinguish whether it is a real access or a web crawler access, and thus cannot effectively intercept crawler requests. Summary of the invention

[0004] 1. Technical issues to be solved

[0005] In view of the deficiencies in the prior art, the present invention provides an intelligent anti-crawler interception system, which solves the problem that conventional anti-crawler means cannot effectively intercept crawler requests.

[0006] (II) Technical solution

[0007] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: an intelligent anti-crawler interception system, including a virtual defense platform, the virtual defense platform including a defense construction unit, an abnormal sharing unit, a front-end processing unit and a virtual construction unit, the defense construction unit is used to build a multi-dimensional black and white list based on the registered user's IP, UA parameters, region and path, set abnormal scenarios, and generate an enterprise-specific defense framework in combination with verification code verification, verify the access data generated by the registered user according to the enterprise-specific defense framework, and mark the registered users who meet the abnormal standards for defense abnormalities, the defense construction unit is connected to the abnormal sharing unit, and the abnormal sharing unit is used to record the determined malicious crawler information as a shared library for different enterprises, the defense construction unit is connected to the front-end processing unit, and the front-end processing unit is used to cooperate with the browser to use the plug-in as a registered user to access The prerequisite for the enterprise website is that when a registered user uses a plug-in to access the website, the access data of the registered user is recorded as standard data, and compared with the access data generated by the registered user verified by the enterprise's exclusive defense framework. The registered user with inconsistent comparison results is marked as a plug-in exception. The front-end processing unit is connected to the virtual construction unit. The virtual construction unit is used to build a false virtual web page according to the normal web page layout of the enterprise, and the web page browsed by the registered user with a defense exception mark and a plug-in exception mark is jumped to the virtual web page. The browsing and downloading of virtual web page materials within a limited time are monitored in real time. When the virtual web page material download occurs within the limited time, the corresponding registered user is locked, marked as a malicious crawler, and the exclusive defense framework intercepts the malicious crawler. The virtual construction unit is connected to the abnormal sharing unit to input the registered user information into the shared library.

[0008] By adopting the above technical solution, a multi-dimensional enterprise-specific defense framework is constructed, the access data of registered users is directly verified, and registered users who meet the abnormal standards are marked as defense anomalies. In addition, the real access data of registered users is recorded in the form of plug-ins. By comparing it with the verified access data of the exclusive defense framework, registered users with inconsistent results are marked as plug-in anomalies, thereby identifying registered users at risk. In conjunction with the setting of virtual web pages, the download of virtual web page materials can be monitored to accurately determine and intercept malicious crawlers.

[0009] The present invention is further configured as follows: the defense construction unit includes a multidimensional framework construction module, a standard content filling module, an enterprise adaptation and adjustment module and an enterprise defense module, the standard content filling module and the enterprise adaptation and adjustment module are both connected to the multidimensional framework construction module, and the multidimensional framework construction module is connected to the enterprise defense module.

[0010] The present invention is further configured as follows: the multi-dimensional framework construction module is used to construct a multi-dimensional black and white list based on the IP, UA parameters, region and access path of the registered user, and set the sending order of the access data generated when the registered user browses the webpage as a standard scenario. When a situation different from the standard scenario occurs again, it is marked as an abnormal scenario as a defense framework;

[0011] The standard content filling module is used to directly fill the malicious list in the abnormal sharing unit into the defense framework;

[0012] The enterprise adaptation and adjustment module is used to fill the enterprise needs into the defense framework according to multi-dimensional classification, and generate an enterprise-specific defense framework in combination with verification code verification, wherein the form of verification code includes but is not limited to intelligent combination verification, sliding puzzle verification, text selection verification, word order selection verification, and spatial reasoning verification. The verification code verification result feedback of the registered user is used to perform human-machine identification, that is, the registered user who fails the verification code is directly intercepted;

[0013] The enterprise defense module is used to verify the access data generated by the registered user according to the enterprise's exclusive defense framework, and when a situation that meets the abnormal standards set by the exclusive defense framework is detected, the registered user is marked as a defense abnormality.

[0014] By adopting the above technical solution, verification codes are used for human-machine identification interception, and preliminary intelligent interception is achieved. At the same time, the access data generated by registered users when browsing web pages is sent in a sequence set as a standard scenario, so as to achieve rapid judgment of abnormal scenarios and improve the recognition efficiency of malicious crawlers.

[0015] The present invention is further configured as follows: the abnormal sharing unit includes a malicious list library and a regulation and identification module, and the malicious list library is connected to the regulation and identification module.

[0016] The present invention is further configured as follows: the malicious list library is used to record the determined malicious crawler information as a shared library for different enterprises;

[0017] The control identification module is used to compare with the shared library when the enterprise receives a registered user visit, and mark the registered users screened out by the comparison as malicious crawlers.

[0018] By adopting the above technical solution, the shared library is used as a shared resource to identify malicious crawlers built into the shared library, further reducing the living space of malicious crawlers and providing protection for the normal operation of the enterprise.

[0019] The present invention is further configured as follows: the front-end processing unit includes a cooperation signing module, a browsing front-end module, a backup comparison module and a locking module, the cooperation signing module is connected to the browsing front-end module, the browsing front-end module is connected to the backup comparison module, and the backup comparison module is connected to the locking module.

[0020] The present invention is further configured as follows: the cooperation signing module is used to cooperate with different browsers in plug-in embedding, and after signing the cooperation contract, the plug-in is used as a precondition for registered users to visit the enterprise website;

[0021] The browsing front-end module is used to set the plug-in function, and when the registered user visits the website, the IP and UA parameters of the registered user are recorded as standard data;

[0022] The backup comparison module is used to compare the standard data with the IP and UA parameters of the registered user actually recorded in the defense construction unit;

[0023] The locking module is used to mark the registered user as having a plug-in exception when there is a mismatch in the UA parameter comparison.

[0024] By adopting the above technical solution, front-end monitoring is carried out in cooperation with the browser to ensure the accuracy and effectiveness of the data generated when registered users visit the website, thereby effectively avoiding the simulation replacement of UA parameters and further improving the efficiency of identifying malicious crawlers.

[0025] The present invention is further configured as follows: the virtual construction unit includes a web page virtual module, a download record module and a malicious marking module;

[0026] The webpage virtualization module is used to construct a false virtual webpage according to the normal webpage layout of the enterprise, and redirect the webpage browsed by the registered user with the defense abnormality mark and the plug-in abnormality mark to the virtual webpage;

[0027] The download record module is used to monitor the browsing and downloading of virtual web page materials within a limited time in real time;

[0028] The malicious marking module is used to lock the corresponding registered user when virtual web page data download occurs within a limited time. After marking it as a malicious crawler, the exclusive defense framework intercepts the malicious crawler and inputs the registered user information represented by the malicious crawler into the shared library. When there is no web page data download within the limited time, the registered users with defense anomaly marks who have not been intercepted are stored in the risk library, and a cooperative browser link is sent to the registered user. When the cooperative browser is not used after the limited time, the registered user is directly intercepted.

[0029] By adopting the above technical solution and utilizing the design of virtual web pages, virtual web page data downloading monitoring is performed on registered users who are at risk of malicious crawlers, thereby accurately identifying malicious crawlers and effectively avoiding accidental harm to normal registered users.

[0030] (III) Beneficial effects

[0031] The present invention provides an intelligent anti-crawler interception system. It has the following beneficial effects:

[0032] (1) The intelligent anti-crawler interception system builds a multi-dimensional enterprise-specific defense framework to directly verify the access data of registered users, mark registered users who meet the abnormal standards as defense anomalies, and use plug-ins to record the real access data of registered users. By comparing it with the verified access data of the exclusive defense framework, the registered users with inconsistent results are marked as plug-in anomalies, thereby identifying registered users with risks. In conjunction with the setting of virtual web pages, the system monitors the download of virtual web page data to accurately determine and intercept malicious crawlers.

[0033] (2) The intelligent anti-crawler interception system uses verification codes to perform human-machine identification interception, while achieving preliminary intelligent interception. At the same time, it sets the order of sending access data generated by registered users when browsing web pages as a standard scenario, realizes rapid judgment of abnormal scenarios, and improves the efficiency of identifying malicious crawlers.

[0034] (3) This intelligent anti-crawler interception system uses shared libraries as shared resources to identify malicious crawlers built into shared libraries, further reducing the living space of malicious crawlers and providing protection for the normal operation of the enterprise.

[0035] (4) The intelligent anti-crawler interception system conducts front-end monitoring by cooperating with the browser to ensure the accuracy and effectiveness of the data generated when registered users visit the website, thereby effectively avoiding the simulation replacement of UA parameters and further improving the efficiency of identifying malicious crawlers.

[0036] (5) The intelligent anti-crawler interception system utilizes the design of virtual web pages to monitor the virtual web page data downloads of registered users who are at risk of malicious crawlers, thereby accurately identifying malicious crawlers and effectively avoiding accidental harm to normal registered users. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a system principle block diagram of the present invention;

[0038] Figure 2 It is a system principle block diagram of the defense construction unit of the present invention;

[0039] Figure 3It is a system principle block diagram of the abnormal sharing unit of the present invention;

[0040] Figure 4 It is a system principle block diagram of the front-end processing unit of the present invention;

[0041] Figure 5 This is a system principle block diagram of the virtual building unit of the present invention.

[0042] In the figure, 1. Virtual defense platform; 2. Defense construction unit; 3. Abnormal sharing unit; 4. Front-end processing unit; 5. Virtual construction unit; 6. Multi-dimensional framework construction module; 7. Standard content filling module; 8. Enterprise adaptation and adjustment module; 9. Enterprise defense module; 10. Malicious list library; 11. Regulation and identification module; 12. Cooperation signing module; 13. Browsing pre-module; 14. Backup comparison module; 15. Locking module; 16. Web page virtual module; 17. Download record module; 18. Malicious marking module. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] See also Figure 1-5 The embodiment of the present invention provides a technical solution: an intelligent anti-crawler interception system, as shown in the attached Figure 1 As shown, it includes a virtual defense platform 1, which includes a defense construction unit 2, an abnormality sharing unit 3, a front-end processing unit 4 and a virtual construction unit 5.

[0045] As a preferred solution, the defense construction unit 2 is used to build a multi-dimensional black and white list based on the registered user's IP, UA parameters, region and path, set abnormal scenarios, generate an enterprise-specific defense framework in combination with verification code verification, verify the access data generated by the registered user according to the enterprise-specific defense framework, and mark the registered users who meet the abnormal standards for defense abnormality. Specifically, as shown in the attached Figure 2 As shown, the defense construction unit 2 includes a multi-dimensional framework construction module 6, a standard content filling module 7, an enterprise adaptation and adjustment module 8 and an enterprise defense module 9. The multi-dimensional framework construction module 6 is used to build a multi-dimensional black and white list based on the IP, UA parameters, region and access path of the registered user, and set the sending order of the access data generated when the registered user browses the web page as a standard scenario. When a situation different from the standard scenario occurs again, it is marked as an abnormal scenario as a defense framework;

[0046] The standard content filling module 7 is connected to the multi-dimensional framework building module 6, and the standard content filling module 7 is used to directly fill the malicious list in the abnormal sharing unit 3 into the defense framework;

[0047] In order to realize human-machine identification, the enterprise adaptation and adjustment module 8 is connected with the multi-dimensional framework construction module 6. The enterprise adaptation and adjustment module 8 is used to fill the enterprise needs into the defense framework according to the multi-dimensional classification, and generate an enterprise-specific defense framework in combination with the verification code verification. As a detailed description, the form of the verification code includes but is not limited to intelligent combination verification, sliding puzzle verification, text selection verification, word order selection verification, and spatial reasoning verification. The verification code verification result feedback of the registered user is used to perform human-machine identification, that is, the registered user who fails the verification code is directly intercepted;

[0048] The multi-dimensional framework construction module 6 is connected to the enterprise defense module 9. The enterprise defense module 9 is used to verify the access data generated by the registered users according to the enterprise-specific defense framework. When a situation that meets the abnormal standards set by the exclusive defense framework is detected, the registered user is marked as a defense abnormality.

[0049] As a preferred solution, the defense construction unit 2 is connected to the abnormal sharing unit 3. In order to further reduce the living space of malicious crawlers, the abnormal sharing unit 3 is used to record the determined malicious crawler information as a shared library of different enterprises. Specifically, as shown in the attached Figure 3 As shown, the abnormal sharing unit 3 includes a malicious list library 10 and a control identification module 11. The malicious list library 10 is used to record the determined malicious crawler information as a shared library for different enterprises;

[0050] The malicious list library 10 is connected to the control identification module 11. The control identification module 11 is used to compare with the shared library when the enterprise receives a registered user visit, and mark the registered users screened out by the comparison as malicious crawlers.

[0051] As a preferred solution, the defense construction unit 2 is connected to the front-end processing unit 4. The front-end processing unit 4 is used to cooperate with the browser and use the plug-in as a prerequisite for the registered user to visit the enterprise website. When the registered user uses the plug-in to visit the website, the registered user's access data is recorded and compared with the access data generated by the registered user verified by the enterprise's exclusive defense framework as standard data. The registered user with inconsistent comparison results is marked as plug-in abnormal. Specifically, as shown in the attached Figure 4 As shown, the front-end processing unit 4 includes a cooperation signing module 12, a browsing pre-module 13, a backup comparison module 14 and a locking module 15. The cooperation signing module 12 is used to cooperate with different browsers to embed plug-ins. After signing the cooperation contract, the plug-in is used as a pre-condition for registered users to visit the enterprise website;

[0052] The cooperation signing module 12 is connected to the browsing front-end module 13. The browsing front-end module 13 is used to set the plug-in function. When the registered user visits the website, the IP and UA parameters of the registered user are recorded as standard data;

[0053] The browsing pre-module 13 is connected to the backup comparison module 14, and the backup comparison module 14 is used to compare the standard data with the IP and UA parameters of the registered user actually recorded in the defense construction unit 2;

[0054] The backup comparison module 14 is connected to the locking module 15. The locking module 15 is used to mark the registered user's plug-in abnormality when there is an inconsistency in the UA parameter comparison.

[0055] As a preferred solution, the front-end processing unit 4 is connected to the virtual construction unit 5. The virtual construction unit 5 is used to build a false virtual web page according to the normal web page layout of the enterprise, and jump the web page browsed by the registered user with the defense exception mark and the plug-in exception mark to the virtual web page. The browsing and downloading of the virtual web page data within the limited time are monitored in real time. When the virtual web page data download occurs within the limited time, the corresponding registered user is locked and marked as a malicious crawler. The dedicated defense framework intercepts the malicious crawler. Specifically, as shown in the attached Figure 5 As shown, the virtual construction unit 5 includes a web page virtualization module 16, a download recording module 17 and a malicious marking module 18;

[0056] The webpage virtualization module 16 is used to construct a false virtual webpage according to the normal webpage layout of the enterprise, and redirect the webpage browsed by the registered user with the defense abnormality mark and the plug-in abnormality mark to the virtual webpage;

[0057] The download record module 17 is used to monitor the browsing and downloading of virtual web page data within a limited time in real time;

[0058] The malicious marking module 18 is used to lock the corresponding registered user when virtual web page data download occurs within a limited time. After marking it as a malicious crawler, the exclusive defense framework intercepts the malicious crawler and inputs the registered user information represented by the malicious crawler into the shared library.

[0059] As a preferred solution, in order to achieve effective updating of shared library resources, the virtual construction unit 5 is connected to the abnormal sharing unit 3 to input the registered user information into the shared library. As a detailed explanation, the malicious marking module 18 is used to store the registered users with defense abnormality marks who have not been intercepted in the risk library when there is no web page data download within a limited time, and send a cooperative browser link to the registered user. When the cooperative browser is not used after the limited time, the registered user is directly intercepted.

[0060] Human-machine identification interception and defense abnormality marking: Enterprise defense module 9 verifies the access data generated by registered users according to the enterprise-specific defense framework. When a situation different from the standard scenario occurs, a different verification code is randomly sent to the registered user. If the verification fails, the registered user is directly intercepted. After the verification passes, the registered user is marked as a defense abnormality;

[0061] Plugin abnormality marking: When a registered user visits a website, the browsing front-end module 13 records the registered user's IP and UA parameters as standard data, and compares them with the registered user's IP and UA parameters actually recorded in the enterprise defense module 9. If the UA parameter comparison is inconsistent, the locking module 15 marks the registered user's plugin abnormality;

[0062] Virtual verification: The webpage virtualization module 16 constructs a fake virtual webpage according to the normal webpage layout of the enterprise, and redirects the webpage browsed by the registered users with defense abnormality marks and plug-in abnormality marks to the virtual webpage. When the virtual webpage data download occurs within a limited time, the malicious marking module 18 locks the corresponding registered user, marks it as a malicious crawler, and the exclusive defense framework intercepts the malicious crawler. At the same time, the registered user information represented by the malicious crawler is input into the shared library, and the registered user with the plug-in abnormality mark continues to browse the virtual webpage;

[0063] Correction interception of registered users: When there is no virtual web page data download within the specified time, the malicious marking module 18 stores the registered users with defense anomaly marks who have not been intercepted in the risk library, and sends the cooperative browser link to the registered user as a normal browsing channel. When the cooperative browser is not used after the specified time, the registered user is directly intercepted. Specifically, the registered user with the defense anomaly mark is directly compared with the registered users stored in the risk library. If there is an existing registered user in the risk library, the registered user is directly intercepted.

[0064] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An intelligent anti-crawler interception system, comprising a virtual defense platform (1), characterized in that: The virtual defense platform (1) comprises a defense construction unit (2), an abnormality sharing unit (3), a front-end processing unit (4) and a virtual construction unit (5). The defense construction unit (2) is used to construct a multi-dimensional black and white list based on the registered user's IP, UA parameters, region and path, set abnormal scenarios, generate an enterprise-specific defense framework in combination with verification code verification, verify the access data generated by the registered user according to the enterprise-specific defense framework, and mark the registered user who meets the abnormal standard for defense abnormality. The defense construction unit (2) is connected to the abnormality sharing unit (3). The abnormality sharing unit (3) is used to record the determined malicious crawler information as a shared library of different enterprises. The defense construction unit (2) is connected to the front-end processing unit (4). The front-end processing unit (4) is used to cooperate with the browser and use the plug-in as a precondition for the registered user to access the enterprise website. When a registered user uses a plug-in to access a website, the access data of the registered user is recorded as standard data and compared with the access data generated by the registered user verified by the enterprise's exclusive defense framework. The registered user with inconsistent comparison results is marked as plug-in abnormal. The front-end processing unit (4) is connected to the virtual construction unit (5). The virtual construction unit (5) is used to build a false virtual web page according to the normal web page layout of the enterprise, jump the web page browsed by the registered user with the defense abnormality mark and the plug-in abnormality mark to the virtual web page, and monitor the browsing and downloading of the virtual web page data within a limited time in real time. When the virtual web page data download occurs within the limited time, the corresponding registered user is locked and marked as a malicious crawler. The exclusive defense framework intercepts the malicious crawler. The virtual construction unit (5) is connected to the abnormal sharing unit (3) to input the registered user information into the shared library; The defense construction unit (2) comprises a multi-dimensional framework construction module (6), a standard content filling module (7), an enterprise adaptation and adjustment module (8) and an enterprise defense module (9), wherein the standard content filling module (7) and the enterprise adaptation and adjustment module (8) are both connected to the multi-dimensional framework construction module (6), and the multi-dimensional framework construction module (6) is connected to the enterprise defense module (9); The multi-dimensional framework construction module (6) is used to construct a multi-dimensional black and white list based on the registered user's IP, UA parameters, region and access path, and set the sending order of the access data generated when the registered user browses the web page as a standard scenario. When a situation different from the standard scenario occurs, it is marked as an abnormal scenario as a defense framework; The standard content filling module (7) is used to directly fill the malicious list in the abnormal sharing unit (3) into the defense framework; The enterprise adaptation and adjustment module (8) is used to fill the enterprise requirements into the defense framework according to multi-dimensional classification, and generate an enterprise-specific defense framework in combination with verification code verification; The enterprise defense module (9) is used to verify the access data generated by the registered user according to the enterprise's exclusive defense framework, and when a situation that meets the abnormal standards set by the exclusive defense framework is detected, the registered user is marked as a defense abnormality.

2. The intelligent anti-crawler interception system according to claim 1, characterized in that: The abnormality sharing unit (3) comprises a malicious list library (10) and a control identification module (11), and the malicious list library (10) is connected to the control identification module (11).

3. The intelligent anti-crawler interception system according to claim 2, characterized in that: The malicious list library (10) is used to record the determined malicious crawler information as a shared library for different enterprises; The control identification module (11) is used to compare with the shared library when the enterprise receives a registered user's visit, and mark the registered user screened out by the comparison as a malicious crawler.

4. The intelligent anti-crawler interception system according to claim 3, characterized in that: The front-end processing unit (4) comprises a cooperation signing module (12), a browsing front-end module (13), a backup comparison module (14) and a locking module (15); the cooperation signing module (12) is connected to the browsing front-end module (13); the browsing front-end module (13) is connected to the backup comparison module (14); and the backup comparison module (14) is connected to the locking module (15).

5. The intelligent anti-crawler interception system according to claim 4, characterized in that: The cooperation signing module (12) is used to cooperate with different browsers in plug-in embedding. After signing the cooperation contract, the plug-in is used as a precondition for registered users to visit the enterprise website; The browsing front-end module (13) is used to set the plug-in function, and when the registered user visits the website, the IP and UA parameters of the registered user are recorded as standard data; The backup comparison module (14) is used to compare the standard data with the IP and UA parameters of the registered user actually recorded in the defense construction unit (2); The locking module (15) is used to mark the registered user as having a plug-in abnormality when a UA parameter comparison is inconsistent.

6. The intelligent anti-crawler interception system according to claim 5, characterized in that: The virtual construction unit (5) comprises a webpage virtualization module (16), a download recording module (17) and a malicious marking module (18); The webpage virtualization module (16) is used to construct a false virtual webpage according to the normal webpage layout of the enterprise, and redirect the webpage browsed by the registered user with the defense abnormality mark and the plug-in abnormality mark to the virtual webpage; The download recording module (17) is used to monitor in real time the browsing and downloading of virtual web page materials within a limited time; The malicious marking module (18) is used to lock the corresponding registered user when a virtual web page data download occurs within a limited time, mark it as a malicious crawler, and then intercept the malicious crawler using a dedicated defense framework. At the same time, the registered user information represented by the malicious crawler is input into a shared library. When no web page data download occurs within the limited time, the registered user with the defense abnormality mark that has not been intercepted is stored in a risk library, and a cooperative browser link is sent to the registered user. When the cooperative browser is not used after the limited time, the registered user is directly intercepted.

Citation Information

Patent Citations

  • CC attack defense method and CC attack defense system

    CN106789983A

  • Crawler identification and interception method and system based on service data

    CN106790105A