Intelligent crawler safe collection method and system based on dynamic feature recognition

Abstract

The invention discloses an intelligent crawler security acquisition method and system based on dynamic feature recognition. The system comprises a URL manager, a scheduler, an analyzer, a downloader and a data memory. The URL manager is used for adding and maintaining a to-be-crawled URL set and a crawled URL set to realize URL duplicate removal; the scheduler is used for coordinating and distributing downloading tasks; the analyzer is used for analyzing page types, structures and request response characteristics of a target website, automatically adjusting crawling parameters, optimizing page access frequency, automatically verifying verification codes, analyzing page depth and verifying crawling contents in a compliance manner; the downloader is used for simulating user behaviors to download pages; the data memory is used for storing data. The method has the advantages that dynamic adjustment of crawling parameters, self-adaption of access frequency, automatic processing of verification codes, anthropomorphic crawling process, controllable depth of crawled pages and compliance of crawled contents are completed by using a dynamic feature recognition method, and self-adaption, high efficiency and safety of a network data acquisition process are realized.

Description

Technical Field

[0001] The present invention relates to the field of data acquisition technology, and in particular to a method and system for secure data acquisition using an intelligent crawler based on dynamic feature recognition. Background Art

[0002] With the rapid growth of online information, web crawler technology has been widely used to collect website data. However, traditional web crawlers suffer from static strategies that are prone to failure. They are unable to adapt to dynamic anti-crawling mechanisms such as dynamic verification codes and page structure mutations. Moreover, they do not comply with the robots protocol or privacy protection standards, which can easily lead to being blocked by anti-crawling systems and cause legal disputes.

[0003] With the advancement of science and technology, automated website data crawling methods have emerged in the field of data collection technology. For example, Chinese patent CN118656537A discloses an automated website data crawling method based on the Internet of Things. This method primarily addresses the field of automated crawling data compression technology, but does not address issues such as crawling strategy failure, manual verification of dynamic CAPTCHAs, sudden changes in web page structure, and non-compliance with the Robots protocol. Therefore, how to effectively and dynamically identify website structure, automatically verify CAPTCHAs, adaptively adjust crawling strategies, and strictly adhere to Robots protocol specifications are pressing technical challenges. Summary of the Invention

[0004] In view of this, the present invention proposes an intelligent crawler security collection method and system based on dynamic feature recognition to solve the problems of failure of existing crawling strategies, manual verification of dynamic verification codes, sudden changes in web page structure, non-compliance with robots protocol, etc., and realizes the adaptability, efficiency and security of the network data collection process.

[0005] The technical solution of the present invention is achieved as follows: On the one hand, the present invention provides an intelligent crawler security collection system based on dynamic feature recognition, including a URL manager, a scheduler, a parser, a downloader and a data storage device; The URL manager is used to add and maintain the set of URLs to be crawled and the set of URLs that have been crawled, and to achieve URL deduplication through the link record table; The scheduler is used to coordinate and distribute download tasks; The parser includes a dynamic feature recognition module, a page parsing module, a security compliance module, and a page depth analysis module. The dynamic feature recognition module is used to analyze the page type, structure, and request response characteristics of the target website in real time, automatically adjust the proxy IP, dynamically generate the User-Agent, add a complete HTTP request header, dynamically render, optimize the page access time interval, and automatically verify the verification code; the page parsing module is used to extract the downloaded data and obtain structured data in real time; the security compliance module is used to encrypt and parse the data during the download process of the web page and comply with the robots.txt rules to limit the crawling scope; the page depth analysis module is used to analyze and record the depth status of the crawled pages to reduce the risk of triggering anti-crawling bans; The downloader includes a user behavior simulation module and a download module, wherein the user behavior simulation module is used to simulate normal user mouse behavior to reduce the risk of being detected by the anti-crawler mechanism; the download module is used to obtain the web page data pointed to by the URL; The data storage device is used to store the crawled content in a structured form in a database. On the other hand, the present invention provides a method for secure collection by an intelligent crawler based on dynamic feature recognition, comprising the following steps: S1. Set the seed URL address, target field name, running host and output data source information; S2. dispatching URLs to access target website pages to obtain data, and analyzing the target website's page type, structure, and request-response characteristics using page type analysis, page structure analysis, and request-response analysis; S3. If the response fails, the proxy IP is automatically adjusted based on the HTTP status code, the User-Agent is dynamically generated, and the complete HTTP request header is added. After dynamic rendering, the process proceeds to S2. If the response is successfully obtained, the process proceeds to S4. S4. Based on the response time and error rate, the exponential backoff algorithm formula is used to enumerate the optimal access time interval, and the probability of success of the access time interval is dynamically set using the formula to calculate the probability of adjusting the actual access time interval; S5. If a verification code appears on the target website, use image verification code recognition, sliding puzzle click text verification code recognition, or SMS verification code recognition to achieve automated verification; S6. The user behavior simulation module uses a machine learning algorithm to simulate the mouse sliding speed, movement trajectory, click interval duration, and page dwell time according to normal user mouse behavior, and generates anthropomorphic operations for the web page download module; S7. Encrypt and download the web page, complete page parsing, extract new URLs, add them to the URL manager, and compare them with the link record table to remove duplicate URLs; S8. Use the callback function to get the depth of the currently crawled page. If the current depth is greater than 5, stop crawling. If the current depth is less than or equal to 5, enter S9. S9. If a download exception occurs during the download process, proceed to S3; S10. If the download page is normal, enable the security compliance module to parse the page according to the robots.txt rules and remove crawling of sensitive fields; S11. Store security and compliance data in a pre-configured database.

[0006] The intelligent crawler security collection method and system based on dynamic feature recognition of the present invention has the following beneficial effects compared with the prior art: The proxy IP, User-Agent, and HTTP request header are automatically adjusted through page type analysis, page structure analysis, and request response analysis. The actual access frequency is dynamically adjusted using the exponential backoff algorithm and the access time interval success probability formula. Automatic verification is achieved using image verification code recognition, sliding puzzle point text verification code recognition, and SMS verification code recognition methods, avoiding manual adjustment of strategies during the crawling process. The machine learning algorithm is used to humanize the crawling process, analyze the depth of the crawled pages, and reduce the risk of triggering anti-crawling bans; The crawled data is encrypted and parsed in compliance with robots.txt rules, which improves data security and reduces the legal risks of data collection. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0008] Figure 1 This is a structural diagram of the intelligent crawler security collection system based on dynamic feature recognition of the present invention; Figure 2 This is a workflow diagram of the intelligent crawler security collection method and system based on dynamic feature recognition of the present invention; Figure 3 This is a workflow diagram of the machine learning algorithm of the present invention. DETAILED DESCRIPTION

[0009] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention. Example 1

[0010] like Figure 1 As shown, an intelligent crawler security collection system based on dynamic feature recognition is provided, including a URL manager, a scheduler, a parser, a downloader and a data storage device; Among them, the role of the URL manager is to add and maintain the set of URLs to be crawled and the set of URLs that have been crawled, and to achieve URL deduplication through the link record table; Among them, the role of the scheduler is to coordinate and allocate download tasks; The parser includes a dynamic feature recognition module, a page parsing module, a security compliance module, and a page depth analysis module. The dynamic feature recognition module analyzes the target website's page type, structure, and request response characteristics in real time, automatically adjusts the proxy IP, dynamically generates the User-Agent, adds a complete HTTP request header, dynamically renders, optimizes page access time intervals, and automatically verifies verification codes. The page parsing module extracts downloaded data and obtains structured data in real time. The security compliance module encrypts and parses data during web page downloads and complies with robots.txt rules to limit the crawling scope. The page depth analysis module analyzes and records the depth of crawled pages to reduce the risk of triggering anti-crawling bans. The downloader includes a user behavior simulation module and a download module. The user behavior simulation module simulates normal user mouse behavior to reduce the risk of being detected by anti-crawler mechanisms. The download module obtains the web page data pointed to by the URL. Among them, the role of the data storage is to store the crawled content in a structured form in the database.

[0011] This system uses a URL manager, scheduler, parser, downloader and data storage device to implement an intelligent crawler security collection system based on dynamic feature recognition. The system URL manager is used to add and maintain the set of URLs to be crawled and the set of URLs that have been crawled to achieve URL deduplication. The scheduler is used to coordinate and allocate download tasks; the parser is used to analyze the page type, structure, and request response characteristics of the target website, automatically adjust the crawling parameters, optimize the page access frequency, automatically verify the verification code, analyze the page depth, and verify the compliance of the crawled content. The downloader is used to simulate user behavior to download pages, and the data storage device is used to store data. By using dynamic feature recognition methods, dynamic adjustment of crawling parameters, adaptive access frequency, automatic processing of verification codes, humanization of the crawling process, controllable crawling page depth, and compliance of crawled content are completed. The adaptability, efficiency and security of the network data collection process are achieved. Through design, simulation and verification, modular products are formed, which can achieve rapid transplantation between different platforms and accelerate the product development process. Example 2

[0012] like Figure 2 As shown, a method for secure collection of data by an intelligent crawler based on dynamic feature recognition is provided, which adopts the intelligent crawler secure collection system based on dynamic feature recognition as described in Example 1, and specifically includes the following steps: S1. Set the seed URL address, target field name, running host and output data source information, S2. dispatching URLs to access target website pages to obtain data, and analyzing the target website's page type, structure, and request-response characteristics using page type analysis, page structure analysis, and request-response analysis. S3. If the response fails, the proxy IP is automatically adjusted based on the HTTP status code, the User-Agent is dynamically generated, and the complete HTTP request header is added. After dynamic rendering, the process proceeds to S2. If the response is successfully obtained, the process proceeds to S4. S4. Based on the response time and error rate, an exponential backoff algorithm formula is used to enumerate the optimal access time interval after three consecutive correct responses. The access time interval success probability formula is used to dynamically set the probability calculation to adjust the actual access time interval. S5. If a verification code appears on the page, use image verification code recognition, sliding puzzle click text verification code recognition, or SMS verification code recognition to achieve automated verification; S6. The user behavior simulation module uses a machine learning algorithm to simulate the mouse sliding speed, movement trajectory, click interval duration, and page dwell time according to normal user mouse behavior, and generates anthropomorphic operations for the web page download module; S7. Encrypt and download the web page, complete page parsing, extract new URLs, add them to the URL manager, and compare them with the link record table to remove duplicate URLs; S8. Use the callback function to get the depth of the currently crawled page. If the current depth is greater than 5, stop crawling. If the current depth is less than or equal to 5, enter S9. S9. If a download exception occurs during the download process, proceed to S3; S10. If the download page is normal, enable the security compliance module to parse the page according to the robots.txt rules and remove crawling of sensitive fields; S11. Store security and compliance data in a pre-configured database.

[0013] Specifically, in step S2, the page analysis method is to obtain the HTML response data of the web page by sending a GET request using the requests library. If the returned HTML data is complete and there are no dynamically generated placeholders, then it can be determined that this is a static page; if the HTML data contains placeholders or dynamic content needs to be generated, then it can be determined that this is a dynamic page.

[0014] Specifically, in step S2, the page structure analysis method includes determining the storage location of the target data by analyzing the HTML tag hierarchy, CSS selectors, or DOM tree structure, determining whether headless browser crawling is required by detecting whether there is dynamic data rendered by JavaScript on the page, and providing path planning for automatic crawling by analyzing page jump rules and form submission mechanisms. Specifically, in step S2, the request response analysis method is to analyze the HTTP status code of the page response, the length of the response content, and verify whether the page is redirected based on the page response data; Specifically, in step S3, response failure means that the HTTP status code of the page response is any one of 300, 301, 302, ‌400, ‌401, ‌403, ‌404, ‌405, ‌406, ‌408, ‌410, ‌413, and ‌414.

[0015] Specifically, in step S3, automatically adjusting the proxy IP refers to selecting an IP replacement in the IP proxy pool that has the same country and language as the target seed URL.

[0016] Specifically, in step S3, dynamically generating the User-Agent refers to using the fake_useragent library to randomly generate the User-Agent of the MSIE, Chrome, or Firefox browser.

[0017] Specifically, in step S3, adding a complete HTTP request header includes adding User-Agent, Accept, Accept-Language, Accept-Encoding, Referer, Connection, and Cookie request header fields.

[0018] Specifically, in step S4, the exponential backoff algorithm formula is: , where t min To set the minimum access time interval, unit S; t max is the maximum access time interval, unit is S; t is the access time interval, unit is S.

[0019] Specifically, in step S4, the access time interval success probability formula is: , where t is the access time interval, unit S; t opt is the optimal access time interval, unit is S; P is the access success probability, dimensionless number.

[0020] Specifically, in step S4, dynamically setting the probability refers to setting the success probability of the access time interval to a random probability between [0.8, 1].

[0021] Specifically, in step S5, the image verification code recognition method refers to using OCR or calling a third-party image verification platform API for recognition.

[0022] Specifically, in step S5, the sliding puzzle point selection text recognition method refers to using a convolutional neural network to train an image recognition model for recognition.

[0023] Specifically, in step S5, the SMS verification code identification method refers to using a third-party SMS verification platform API for identification.

[0024] Specifically, in step S6, the machine learning algorithm specifically includes the following steps: L1. Collect download event sequence data of the entire process of real user mouse click timestamps, coordinates, sliding speed, motion trajectory, keystroke behavior, and click interval duration; L2: Record the user device’s CPU usage, memory usage, and network latency through the tracking system; L3. Clean the download event sequence data, filter abnormal mouse click behaviors, and eliminate machine behavior interference; L4. Mark the mouse click intervals according to the age groups of the mouse operators; L5. Obtain the mean, variance, and correlation of mouse click intervals based on temporal features. Obtain the mouse click coordinate distribution density and movement trajectory curvature based on spatial features. Obtain the user's operation history preferences based on contextual features. L6. Use GMM model or HMM model to generate mouse click interval sequence; L7, Generate a nonlinear random path based on the mouse click coordinate distribution density range, the movement trajectory curvature range, and the mouse click interval sequence; L8, using LSTM or Transformer architecture to predict the dynamic changes of user mouse click intervals; L9, using Q-learning to simulate the user's decision-making process and dynamically adjust the click strategy; L10, randomly inject ±5%-15% disturbance into the model output to complete anthropomorphic mouse operation; Specifically, in step S7, web page encryption refers to using the TLS 1.3 protocol to encrypt the web page during downloading to prevent tampering.

[0025] Specifically, in step S7, page parsing is to extract data from the downloaded data using a DOM tree algorithm or a tag path positioning algorithm to obtain structured data in real time.

[0026] Specifically, in step S7, URL deduplication refers to removing duplicate downloaded URL webpage links using one of a database, a memory collection, an MD5 hash or a Bloom filter.

[0027] Among them, the requests library, GMM model, HMM model, LSTM architecture, Transformer architecture, Q-learning simulation of user operation decisions, DOM tree algorithm, and label path positioning algorithm are all compiled and passed under the Windows operating system using the standard Python language. The third-party image verification platform uses Super Eagle, and the third-party SMS verification platform uses Twilio.

[0028] like Figure 2 As shown in the figure, the seed URL address, target field name, running host, output data source information are set, URL access page is scheduled to obtain data, proxy IP is automatically adjusted, User-Agent is dynamically generated and a complete HTTP request header is added, dynamic rendering and dynamic setting of probability calculation are performed to adjust the actual access time interval, verification code verification is automated, the download process is humanized, pages are encrypted, downloaded and parsed, URL deduplication is performed, crawling page depth is detected, sensitive fields are removed, and structured data is stored to complete the entire process of secure collection by intelligent crawlers.

[0029] Specifically, the intelligent crawler security collection method and system based on dynamic feature recognition provided by the present invention are compiled using Python on the Windows operating system, adopt a modular design method, and use a URL manager, a scheduler, a parser, a downloader and a data storage device to implement an intelligent crawler security collection system based on dynamic feature recognition. In this system, the URL manager is used to add and maintain a set of URLs to be crawled and a set of URLs that have been crawled to achieve URL deduplication; the scheduler is used to coordinate and allocate download tasks; the parser is used to analyze the page type, structure, and request response characteristics of the target website, automatically adjust the crawling parameters, optimize the page access frequency, automatically verify the verification code, analyze the page depth, and verify the compliance of the crawled content. The downloader is used to simulate user behavior to download pages, and the data storage device is used to store data. By using the dynamic feature recognition method, dynamic adjustment of crawling parameters, adaptive access frequency, automatic processing of verification codes, humanization of the crawling process, controllable crawling page depth, and compliance of crawled content are achieved, thereby achieving the adaptability, efficiency and security of the network data collection process.

[0030] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An intelligent crawler-based secure data collection method based on dynamic feature recognition, characterized by: The steps include: S1. Set the seed URL address, target field name, running host and output data source information; S2. dispatching URLs to access target website pages to obtain data, and analyzing the target website's page type, structure, and request-response characteristics using page type analysis, page structure analysis, and request-response analysis; S3. If the response fails, the proxy IP is automatically adjusted based on the HTTP status code, the User-Agent is dynamically generated, and the complete HTTP request header is added. After dynamic rendering, the process proceeds to S2. If the response is successfully obtained, the process proceeds to S4. S4. Based on the response time and error rate, the exponential backoff algorithm formula is used to enumerate the optimal access time interval, and the probability of success of the access time interval is dynamically set using the formula to calculate the probability of adjusting the actual access time interval; S5. If a verification code appears on the target website, use image verification code recognition, sliding puzzle and clicking text verification code recognition, or SMS verification code recognition to achieve automated verification; S6. The user behavior simulation module uses a machine learning algorithm to simulate the mouse sliding speed, movement trajectory, click interval duration, and page dwell time according to normal user mouse behavior, and generates anthropomorphic operations for the web page download module; S7. Encrypt and download the web page, complete page parsing, extract new URLs, add them to the URL manager, and compare them with the link record table to remove duplicate URLs; S8. Use the callback function to obtain the depth of the currently crawled page. If the current depth is greater than 5, stop crawling. If the current depth is less than or equal to 5, enter S9. S9. If a download exception occurs during the download process, proceed to S3; S10. If the download page is normal, enable the security compliance module to parse the page according to the robots.txt rules and remove crawling of sensitive fields; S11. Store the security and compliance data in a pre-configured database.

2. The intelligent crawler security collection method based on dynamic feature recognition according to claim 1, characterized in that: In the step S2: The page analysis method is to obtain the HTML response data of the web page by sending a GET request using the requests library. If the returned HTML data is complete and does not contain dynamically generated placeholders, it can be determined that this is a static page; if the HTML data contains placeholders or needs to generate dynamic content, it can be determined that this is a dynamic page. The page structure analysis method includes determining the storage location of the target data by analyzing the HTML tag hierarchy, CSS selectors, or DOM tree structure of the web page, determining whether headless browser crawling is necessary by detecting whether there is dynamic data rendered by JavaScript on the page, and providing path planning for automatic crawling by analyzing page jump rules and form submission mechanisms; The request response analysis method is to analyze the HTTP status code of the page response, the response content length and verify whether the page is redirected based on the page response data.

3. The intelligent crawler security collection method based on dynamic feature recognition according to claim 1 is characterized in that: In the step S3: The response failure means that the HTTP status code of the page response is any one of 300, 301, 302, ‌400, ‌401, ‌403, ‌404, ‌405, ‌406, ‌408, ‌410, ‌413, ‌414; The automatic adjustment of proxy IP refers to selecting an IP replacement in the IP proxy pool with the same country and language as the target seed URL; Dynamically generating User-Agent refers to using fake_useragent library to randomly generate User-Agent of MSIE, Chrome or Firefox browser; Adding a complete HTTP request header includes adding User-Agent, Accept, Accept-Language, Accept-Encoding, Referer, Connection, and Cookie request header fields.

4. The intelligent crawler security collection method based on dynamic feature recognition according to claim 1, characterized in that: In the step S4: The exponential backoff algorithm formula is: , where t min To set the minimum access time interval, unit S; t max is the maximum access time interval, in units of S; t is the access time interval, in units of S; The access time interval success probability formula is: , where t is the access time interval, unit S; t opt is the optimal access time interval, unit S; P is the access success probability, dimensionless number; The dynamic setting probability refers to the random probability of setting the success probability of the access time interval between [0.8, 1].

5. The intelligent crawler security collection method based on dynamic feature recognition according to claim 1 is characterized in that: In the step S5: The image verification code recognition method refers to using OCR or calling a third-party image verification platform API for recognition; The sliding puzzle point selection text recognition method refers to using a convolutional neural network to train an image recognition model for recognition; The SMS verification code recognition method refers to the use of a third-party SMS verification platform API for recognition.

6. The intelligent crawler security collection method based on dynamic feature recognition according to claim 1, characterized in that: In the step S6: The machine learning algorithm specifically includes the following steps: L1. Collect download event sequence data of the entire process of real user mouse click timestamps, coordinates, sliding speed, motion trajectory, keystroke behavior, and click interval duration; L2: Record the user device’s CPU usage, memory usage, and network latency through the tracking system; L3. Clean the download event sequence data, filter abnormal mouse click behaviors, and eliminate machine behavior interference; L4. Mark the mouse click intervals according to the age groups of the mouse operators; L5. Obtain the mean, variance, and correlation of mouse click intervals based on temporal features. Obtain the mouse click coordinate distribution density and movement trajectory curvature based on spatial features. Obtain the user's operation history preferences based on contextual features. L6. Use GMM model or HMM model to generate mouse click interval sequence; L7, Generate a nonlinear random path based on the mouse click coordinate distribution density range, the movement trajectory curvature range, and the mouse click interval sequence; L8, using LSTM or Transformer architecture to predict the dynamic changes of user mouse click intervals; L9, using Q-learning to simulate the user's decision-making process and dynamically adjust the click strategy; L10. Randomly inject ±5%-15% disturbance into the model output to complete anthropomorphic mouse operation.

7. The intelligent crawler security collection method based on dynamic feature recognition according to claim 1, characterized in that: In step S7: The webpage encryption mentioned above refers to the use of TLS 1.3 protocol to encrypt the webpage during downloading to prevent tampering; Page parsing refers to extracting data from downloaded data using a DOM tree algorithm or a tag path positioning algorithm to obtain structured data in real time; The URL deduplication refers to removing duplicate downloaded URL webpage links using one of a database, a memory collection, an MD5 hash or a Bloom filter.

8. A system using the intelligent crawler secure collection method based on dynamic feature recognition according to any one of claims 1 to 7, characterized in that: Includes URL manager, dispatcher, parser, downloader and data storage; The URL manager is used to add and maintain the set of URLs to be crawled and the set of URLs that have been crawled, and to achieve URL deduplication through the link record table; The scheduler is used to coordinate and distribute download tasks; The parser includes a dynamic feature recognition module, a page parsing module, a security compliance module, and a page depth analysis module. The dynamic feature recognition module is used to analyze the page type, structure, and request response characteristics of the target website in real time, automatically adjust the proxy IP, dynamically generate the User-Agent, add a complete HTTP request header, dynamically render, optimize the page access time interval, and automatically verify the verification code; the page parsing module is used to extract the downloaded data and obtain structured data in real time; the security compliance module is used to encrypt and parse the data during the download process of the web page and comply with the robots.txt rules to limit the crawling scope; the page depth analysis module is used to analyze and record the depth status of the crawled pages to reduce the risk of triggering anti-crawling bans; The downloader includes a user behavior simulation module and a download module, wherein the user behavior simulation module is used to simulate normal user mouse behavior to reduce the risk of being detected by the anti-crawler mechanism; the download module is used to obtain the web page data pointed to by the URL; The data storage device is used to store the crawled content in a structured form in a database.

Images