Intelligent crawler security collection method and system based on dynamic feature recognition

Abstract

The application discloses a kind of based on dynamic feature identification intelligent crawler safety collection method and system, including URL manager, dispatcher, resolver, downloader and data storage.Will be added and maintained to be crawled URL set and the URL set of having been crawled by URL manager, realize URL deduplication;Dispatcher is used for coordinating and allocating download task;Resolver is used for analyzing the page type of target website, structure, request response feature, automatically adjusting crawling parameter, optimizing page access frequency, automatic verification Captcha, analyzing page depth, compliance verification crawling content;Downloader is used for simulating user behavior downloading page;Data storage is used for storing data.The application has the advantages that: using dynamic feature identification method completes the dynamic adjustment of crawling parameter, access frequency self-adaptation, Captcha automatic processing, crawling process personification, crawling page depth controllable, crawling content compliance, realizes the adaptability, high efficiency and safety of network data acquisition process.

Description

Technical Field

[0001] This invention relates to the field of data acquisition technology, and in particular to a secure data acquisition method and system based on dynamic feature recognition for intelligent web crawlers. Background Technology

[0002] With the rapid growth of online information, web crawler technology has been widely used in the field of website data collection. However, traditional web crawlers suffer from several drawbacks: static strategies are prone to failure, they cannot adapt to dynamic anti-crawling mechanisms such as dynamic CAPTCHAs and sudden changes in page structure, and they do not comply with the robots.txt protocol or privacy protection standards, making them highly susceptible to being blocked by anti-crawling systems and causing numerous problems such as legal disputes.

[0003] With the development of science and technology, automated website data crawling methods have emerged in the field of data acquisition technology. For example, Chinese patent CN118656537A discloses an automated website data crawling method based on the Internet of Things, mainly targeting the field of automated crawling data compression technology, and does not address issues such as crawling strategy failure, manual verification of dynamic CAPTCHAs, sudden changes in webpage structure, and non-compliance with the robots.txt protocol. Therefore, how to effectively and dynamically identify website structure, automatically verify CAPTCHAs, adaptively adjust crawling strategies, and strictly adhere to the robots.txt protocol specifications are urgent technical problems that need to be solved. Summary of the Invention

[0004] In view of this, the present invention proposes an intelligent web crawler security collection method and system based on dynamic feature recognition to solve problems such as the failure of existing crawling strategies, manual verification of dynamic CAPTCHAs, sudden changes in web page structure, and non-compliance with the robots protocol, thereby achieving adaptability, efficiency and security in the network data collection process.

[0005] The technical solution of this invention is implemented as follows:

[0006] On the one hand, the present invention provides an intelligent web crawler security data collection system based on dynamic feature recognition, including a URL manager, scheduler, parser, downloader and data storage device;

[0007] The URL manager is used to add and maintain the set of URLs to be crawled and the set of URLs already crawled, and to remove duplicate URLs through the link record table;

[0008] The scheduler is used to coordinate and allocate download tasks;

[0009] The parser includes a dynamic feature recognition module, a page parsing module, a security compliance module, and a page depth analysis module. The dynamic feature recognition module analyzes the target website's page type, structure, and request / response characteristics in real time, automatically adjusting proxy IPs, dynamically generating User-Agents, adding complete HTTP request headers, dynamically rendering, optimizing page access time intervals, and automating CAPTCHA verification. The page parsing module extracts data from the downloaded data to obtain structured data in real time. The security compliance module encrypts and parses data during webpage downloads and adheres to robots.txt rules to limit the crawling scope. The page depth analysis module analyzes and records the depth status of crawled pages, reducing the risk of triggering anti-crawling blocking.

[0010] The downloader includes a user behavior simulation module and a download module. The user behavior simulation module is used to simulate normal user mouse behavior to reduce the risk of being detected by anti-crawler mechanisms. The download module is used to obtain the webpage data pointed to by the URL.

[0011] The data storage device is used to store the crawled content in a structured format in the database.

[0012] On the other hand, the present invention provides a secure data collection method for intelligent web crawlers based on dynamic feature recognition, comprising the following steps:

[0013] S1. Set the seed URL address, target field name, running host, and output data source information;

[0014] S2. Dispatch the URL to access the target website page to obtain data, and use page type analysis, page structure analysis and request-response analysis to analyze the page type, structure and request-response characteristics of the target website;

[0015] S3. If the response fails to be obtained, the proxy IP is automatically adjusted according to the HTTP status code, the User-Agent is dynamically generated, and the complete HTTP request header is added. After dynamic rendering, proceed to S2; if the response is obtained successfully, proceed to S4.

[0016] S4. Based on the response time and error rate, the optimal access time interval is determined by enumerating the exponential backoff algorithm formula, and the actual access time interval is adjusted by dynamically setting the probability calculation using the access time interval success probability formula.

[0017] S5. If a verification code appears on the target website page, use image verification code recognition method, sliding puzzle text verification code recognition method, or SMS verification code recognition method to achieve automated verification;

[0018] S6. The simulated user behavior module simulates mouse movement speed, trajectory, click interval, and page dwell time using machine learning algorithms, based on normal user mouse usage behavior, and generates human-like operations for the webpage download module.

[0019] S7. Encrypt and download the webpage, complete page parsing, extract the new URL, add it to the URL manager, and compare it with the link record table to remove duplicate URLs;

[0020] S8. Use a callback function to get the current page depth being crawled. If the current depth is greater than 5, stop crawling. If the current depth is less than or equal to 5, proceed to S9.

[0021] S9. If a download error occurs during the download process, proceed to S3;

[0022] S10. If the download page is normal, enable the security compliance module to parse the page according to the robots.txt rules and remove the crawling of sensitive fields;

[0023] S11. Store secure and compliant data in a pre-configured database.

[0024] The intelligent web crawler secure data collection method and system based on dynamic feature recognition of this invention has the following advantages over existing technologies:

[0025] The proxy IP, User-Agent, and HTTP request headers are automatically adjusted using page type analysis, page structure analysis, and request-response analysis. The actual access frequency is dynamically adjusted using the exponential backoff algorithm formula and the access time interval success probability formula. Automated verification is achieved by using image verification code recognition, sliding puzzle text verification code recognition, and SMS verification code recognition methods, thus avoiding manual adjustment of strategies during the crawling process.

[0026] Machine learning algorithms were used to simulate human-like operations during the crawling process, and the depth and state of the crawled pages were analyzed, reducing the risk of triggering anti-crawling blocking.

[0027] Data crawling is encrypted and parsed in accordance with robots.txt rules, which improves data security and reduces the legal risks of data collection. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a structural diagram of the intelligent crawler security data collection system based on dynamic feature recognition according to the present invention;

[0030] Figure 2 This is a flowchart illustrating the workflow of the intelligent web crawler secure data collection method and system based on dynamic feature recognition of the present invention.

[0031] Figure 3 This is a flowchart of the machine learning algorithm of the present invention. Detailed Implementation

[0032] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention. Example 1

[0033] like Figure 1 As shown, an intelligent web crawler security data collection system based on dynamic feature recognition is provided, including a URL manager, scheduler, parser, downloader, and data storage.

[0034] The URL manager is responsible for adding and maintaining the set of URLs to be crawled and the set of URLs already crawled, and for deduplicating URLs through the link record table.

[0035] The scheduler's role is to coordinate and allocate download tasks;

[0036] The parser comprises a dynamic feature recognition module, a page parsing module, a security and compliance module, and a page depth analysis module. The dynamic feature recognition module analyzes the target website's page type, structure, and request / response characteristics in real time, automatically adjusting proxy IPs, dynamically generating User-Agents, adding complete HTTP request headers, dynamically rendering, optimizing page access time intervals, and automating CAPTCHA verification. The page parsing module extracts data from the downloaded data to obtain structured data in real time. The security and compliance module encrypts and parses data during webpage downloads, adhering to robots.txt rules and limiting the crawling scope. The page depth analysis module analyzes and records the depth status of crawled pages, reducing the risk of triggering anti-crawling blocking.

[0037] The downloader includes a user behavior simulation module and a download module. The user behavior simulation module simulates normal user mouse behavior to reduce the risk of being detected by anti-scraping mechanisms. The download module obtains the webpage data pointed to by the URL.

[0038] The data storage device is used to store the crawled content in a structured format in the database.

[0039] This system employs a URL manager, scheduler, parser, downloader, and data storage to implement an intelligent web crawler security data collection system based on dynamic feature recognition. The URL manager adds and maintains the sets of URLs to be crawled and those already crawled, enabling URL deduplication. The scheduler coordinates and allocates download tasks. The parser analyzes the target website's page type, structure, and request-response characteristics, automatically adjusting crawling parameters, optimizing page access frequency, automating CAPTCHA verification, analyzing page depth, and verifying compliant crawled content. The downloader simulates user behavior to download pages, and the data storage stores data. By using dynamic feature recognition, the system achieves dynamic adjustment of crawling parameters, adaptive access frequency, automated CAPTCHA processing, a human-like crawling process, controllable page depth, and compliant crawled content, realizing the adaptability, efficiency, and security of the web data collection process. Through design, simulation, and verification, a modular product is formed, enabling rapid portability across different platforms and accelerating the product development process. Example 2

[0040] like Figure 2 As shown, a secure data collection method for intelligent web crawlers based on dynamic feature recognition is provided. This method employs the secure data collection system for intelligent web crawlers based on dynamic feature recognition as described in Example 1, and specifically includes the following steps:

[0041] S1. Set the seed URL address, target field name, running host, and output data source information.

[0042] S2. Dispatch the URL to access the target website's pages to obtain data, and use page type analysis, page structure analysis, and request-response analysis to analyze the target website's page type, structure, and request-response characteristics;

[0043] S3. If the response fails to be obtained, the proxy IP is automatically adjusted according to the HTTP status code, the User-Agent is dynamically generated, and the complete HTTP request header is added. After dynamic rendering, proceed to S2; if the response is obtained successfully, proceed to S4.

[0044] S4. Based on the response time and error rate, the exponential backoff algorithm formula is used for enumeration. The optimal access time interval is determined after three consecutive successful responses. The probability calculation is dynamically set and adjusted using the access time interval success probability formula.

[0045] S5. If a verification code appears on the page, use image verification code recognition method, sliding puzzle text verification code recognition method or SMS verification code recognition method to achieve automated verification;

[0046] S6. The simulated user behavior module simulates mouse movement speed, trajectory, click interval, and page dwell time using machine learning algorithms, based on normal user mouse usage behavior, and generates human-like operations for the webpage download module.

[0047] S7. Encrypt and download the webpage, complete page parsing, extract the new URL, add it to the URL manager, and compare it with the link record table to remove duplicate URLs;

[0048] S8. Use a callback function to get the current page depth being crawled. If the current depth is greater than 5, stop crawling. If the current depth is less than or equal to 5, proceed to S9.

[0049] S9. If a download error occurs during the download process, proceed to S3;

[0050] S10. If the download page is normal, enable the security compliance module to parse the page according to the robots.txt rules and remove the crawling of sensitive fields;

[0051] S11. Store secure and compliant data in a pre-configured database.

[0052] Specifically, in step S2, the page analysis method uses the requests library to send a GET request to obtain the HTML response data of the webpage. If the returned HTML data is complete and does not contain dynamically generated placeholders, then it can be determined that this is a static page; if the HTML data contains placeholders or requires dynamic content to be generated, then it can be determined that this is a dynamic page.

[0053] Specifically, in step S2, the page structure analysis method includes determining the storage location of the target data by analyzing HTML tag levels, CSS selectors, or the DOM tree structure; determining whether to use a headless browser for crawling by detecting whether there is dynamic data rendered by JavaScript on the page; and providing path planning for automatic crawling by analyzing page redirection rules and form submission mechanisms.

[0054] Specifically, in step S2, the request-response analysis method analyzes the HTTP status code, response content length, and verifies whether the page is redirected based on the page response data.

[0055] Specifically, in step S3, a response failure refers to any one of the following HTTP status codes in the page response: 300, 301, 302, 400, 401, 403, 404, 405, 406, 408, 410, 413, and 414.

[0056] Specifically, in step S3, automatically adjusting the proxy IP means selecting an IP from the IP proxy pool that is in the same country and language as the target seed URL and replacing it.

[0057] Specifically, in step S3, dynamically generating User-Agent means using the fake_useragent library to randomly generate User-Agent for MSIE, Chrome, or Firefox browsers.

[0058] Specifically, in step S3, adding complete HTTP request headers includes adding User-Agent, Accept, Accept-Language, Accept-Encoding, Referer, Connection, and Cookie request header fields.

[0059] Specifically, in step S4, the formula for the exponential backoff algorithm is as follows: , where t min To set the minimum access time interval, the unit is seconds (s); t max The maximum access time interval is set in seconds (S); t is the access time interval in seconds.

[0060] Specifically, in step S4, the formula for the success probability of the access time interval is: Where t is the access time interval, in seconds (s); t opt is the optimal access time interval, in seconds; P is the probability of successful access, a dimensionless number.

[0061] Specifically, in step S4, the dynamic setting probability refers to the random probability of setting the access time interval success probability between [0.8, 1].

[0062] Specifically, in step S5, the image verification code recognition method refers to using OCR or calling a third-party image verification platform API for recognition.

[0063] Specifically, in step S5, the sliding puzzle point selection text recognition method refers to using a convolutional neural network to train an image recognition model for recognition.

[0064] Specifically, in step S5, the SMS verification code recognition method refers to using a third-party SMS verification platform API for recognition.

[0065] Specifically, in step S6, the machine learning algorithm includes the following steps:

[0066] L1. Collect download event sequence data of the entire process of real user mouse click timestamps, coordinates, sliding speed, movement trajectory, key behavior and click interval duration;

[0067] L2. Record user device CPU usage, memory usage, and network latency through a data tracking system;

[0068] L3. Clean the downloaded event sequence data, filter out abnormal mouse click behavior, and eliminate machine behavior interference;

[0069] L4. Mark the mouse click interval according to the different age groups of the mouse operators;

[0070] L5. Obtain the mean, variance, and correlation of mouse click intervals based on temporal features; obtain the distribution density of mouse click coordinates and curvature of movement trajectory based on spatial features; and obtain user operation history preferences based on contextual features.

[0071] L6. Generate mouse click interval sequences using GMM or HMM models;

[0072] L7. Generate a nonlinear random path based on the range of mouse click coordinate distribution density, the range of movement trajectory curvature, and the mouse click interval sequence;

[0073] L8. Use LSTM or Transformer architecture to predict the dynamic changes in the interval between user mouse clicks;

[0074] L9. Employ Q-learning to simulate the user's decision-making process and dynamically adjust the click strategy;

[0075] L10. Randomly inject ±5%-15% perturbation into the model output to achieve human-like mouse operation;

[0076] Specifically, in step S7, webpage encryption refers to encrypting the webpage during the download process using the TLS 1.3 protocol to prevent tampering.

[0077] Specifically, in step S7, page parsing involves using the DOM tree algorithm or tag path positioning algorithm to extract data from the downloaded data and obtain structured data in real time.

[0078] Specifically, in step S7, URL deduplication refers to removing duplicate downloaded URL webpage links using one of the following: database, in-memory collection, MD5 hash, or Bloom filter.

[0079] Among them, the requests library, GMM model, HMM model, LSTM architecture, Transformer architecture, Q-learning to simulate user operation decisions, DOM tree algorithm, and label path localization algorithm were all compiled and passed under the Windows operating system using standard Python language. The third-party image verification platform used SuperEagle, and the third-party SMS verification platform used Twilio.

[0080] like Figure 2 As shown, the entire process of intelligent web crawler secure data collection is completed by setting the seed URL address, target field name, running host, and output data source information, scheduling the URL to access the page to obtain data, automatically adjusting the proxy IP, dynamically generating the User-Agent and adding complete HTTP request headers, dynamically rendering and setting probability calculations to adjust the actual access time interval, automating CAPTCHA verification, humanizing the download process, encrypting, downloading and parsing the page, deduplicating URLs, detecting the depth of the crawled page, removing sensitive fields, and storing structured data.

[0081] Specifically, the intelligent web crawler security data collection method and system based on dynamic feature recognition provided by this invention is compiled using Python on a Windows operating system. It adopts a modular design approach and utilizes a URL manager, scheduler, parser, downloader, and data storage to implement the intelligent web crawler security data collection system based on dynamic feature recognition. In this system, the URL manager is used to add and maintain the set of URLs to be crawled and the set of already crawled URLs, achieving URL deduplication; the scheduler is used to coordinate and allocate download tasks; the parser is used to analyze the page type, structure, and request-response characteristics of the target website, automatically adjust crawling parameters, optimize page access frequency, automatically verify CAPTCHAs, analyze page depth, and verify crawled content in compliance with regulations; the downloader is used to simulate user behavior to download pages; and the data storage is used to store data. By using the dynamic feature recognition method, it achieves dynamic adjustment of crawling parameters, adaptive access frequency, automated CAPTCHA processing, human-like crawling process, controllable crawling page depth, and compliant crawled content, realizing the adaptability, efficiency, and security of the network data collection process.

[0082] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A secure data collection method for intelligent web crawlers based on dynamic feature recognition, characterized in that: Includes the following steps: S1. Set the seed URL address, target field name, running host, and output data source information; S2. Dispatch the URL to access the target website page to obtain data, and use page type analysis, page structure analysis and request-response analysis to analyze the page type, structure and request-response characteristics of the target website; S3. If the response fails to be obtained, the proxy IP is automatically adjusted according to the HTTP status code, the User-Agent is dynamically generated, and the complete HTTP request header is added. After dynamic rendering, proceed to S2; if the response is obtained successfully, proceed to S4. S4. Based on the response time and error rate, the optimal access time interval is determined by enumerating the exponential backoff algorithm formula, and the actual access time interval is adjusted by dynamically setting the probability calculation using the access time interval success probability formula. S5. If a verification code appears on the target website page, use image verification code recognition method, sliding puzzle text verification code recognition method, or SMS verification code recognition method to achieve automated verification; S6. The simulated user behavior module simulates mouse movement speed, trajectory, click interval, and page dwell time using machine learning algorithms, based on normal user mouse usage behavior, and generates human-like operations for the webpage download module. S7. Encrypt and download the webpage, complete page parsing, extract the new URL, add it to the URL manager, and compare it with the link record table to remove duplicate URLs; S8. Use a callback function to get the current page depth being crawled. If the current depth is greater than 5, stop crawling. If the current depth is less than or equal to 5, proceed to S9. S9. If a download error occurs during the download process, proceed to S3; S10. If the download page is normal, enable the security compliance module to parse the page according to the robots.txt rules and remove the crawling of sensitive fields; S11. Store secure and compliant data in a pre-configured database.

2. The intelligent web crawler secure data collection method based on dynamic feature recognition as described in claim 1, characterized in that, In step S2: The page analysis method described above uses the requests library to send GET requests to obtain the HTML response data of a webpage. If the returned HTML data is complete and does not contain dynamically generated placeholders, then it can be determined that this is a static page; if the HTML data contains placeholders or requires the generation of dynamic content, then it can be determined that this is a dynamic page. The page structure analysis method includes determining the storage location of target data by analyzing the HTML tag hierarchy, CSS selectors, or DOM tree structure of the webpage; determining whether to use a headless browser for crawling by detecting whether there is dynamic data rendered by JavaScript on the page; and providing path planning for automatic crawling by analyzing page redirection rules and form submission mechanisms. The request-response analysis method analyzes the HTTP status code, response content length, and verifies whether the page has been redirected based on the page response data.

3. The intelligent web crawler secure data collection method based on dynamic feature recognition as described in claim 1, characterized in that, In step S3: The failure to respond refers to any one of the following HTTP status codes in the page response: 300, 301, 302, 400, 401, 403, 404, 405, 406, 408, 410, 413, and 414. The automatic adjustment of proxy IP refers to selecting and replacing IPs from the IP proxy pool that are in the same country and language as the target seed URL; The dynamically generated User-Agent refers to the use of the fake_useragent library to randomly generate a User-Agent for MSIE, Chrome, or Firefox browsers; Adding complete HTTP request headers includes adding User-Agent, Accept, Accept-Language, Accept-Encoding, Referer, Connection, and Cookie request header fields.

4. The intelligent web crawler secure data collection method based on dynamic feature recognition as described in claim 1, characterized in that, In step S4: The formula for the exponential backoff algorithm is: , where t min To set the minimum access time interval, the unit is seconds (s); t max To set the maximum access time interval, in seconds (s); t is the access time interval, in seconds (s). The formula for the success probability of the access time interval is: Where t is the access time interval, in seconds (s); t opt The optimal access time interval is in seconds (S); P is the probability of successful access, a dimensionless number. The dynamic setting probability refers to the random probability that the success rate of setting the access time interval is between [0.8, 1].

5. The intelligent web crawler secure data collection method based on dynamic feature recognition as described in claim 1, characterized in that, In step S5: The image verification code recognition method refers to using OCR or calling the API of a third-party image verification platform for recognition. The sliding puzzle point selection text recognition method refers to using a convolutional neural network to train an image recognition model for recognition. The SMS verification code recognition method refers to using a third-party SMS verification platform API for recognition.

6. The intelligent web crawler secure data collection method based on dynamic feature recognition as described in claim 1, characterized in that, In step S6: The machine learning algorithm specifically includes the following steps: L1. Collect download event sequence data of the entire process of real user mouse click timestamps, coordinates, sliding speed, movement trajectory, key behavior and click interval duration; L2. Record user device CPU usage, memory usage, and network latency through a data tracking system; L3. Clean the downloaded event sequence data, filter out abnormal mouse click behavior, and eliminate machine behavior interference; L4. Mark the mouse click interval according to the different age groups of the mouse operators; L5. Obtain the mean, variance, and correlation of mouse click intervals based on temporal features; obtain the distribution density of mouse click coordinates and curvature of movement trajectory based on spatial features; and obtain user operation history preferences based on contextual features. L6. Generate mouse click interval sequences using GMM or HMM models; L7. Generate a nonlinear random path based on the range of mouse click coordinate distribution density, the range of movement trajectory curvature, and the mouse click interval sequence; L8. Use LSTM or Transformer architecture to predict the dynamic changes in the interval between user mouse clicks; L9. Employ Q-learning to simulate the user's decision-making process and dynamically adjust the click strategy; L10. Randomly inject ±5%-15% perturbation into the model output to achieve human-like mouse operation.

7. The intelligent web crawler secure data collection method based on dynamic feature recognition as described in claim 1, characterized in that, In step S7: The webpage encryption refers to encrypting the webpage during the download process using the TLS 1.3 protocol to prevent tampering. The page parsing refers to extracting data from the downloaded data using DOM tree algorithms or tag path positioning algorithms, and obtaining structured data in real time; URL deduplication refers to removing duplicate downloaded URL webpage links using one of the following methods: database, in-memory collection, MD5 hash, or Bloom filter.

8. A system using the intelligent crawler secure data collection method based on dynamic feature recognition as described in any one of claims 1-7, characterized in that: It includes a URL manager, scheduler, parser, downloader, and data storage; The URL manager is used to add and maintain the set of URLs to be crawled and the set of URLs already crawled, and to remove duplicate URLs through the link record table; The scheduler is used to coordinate and allocate download tasks; The parser includes a dynamic feature recognition module, a page parsing module, a security compliance module, and a page depth analysis module. The dynamic feature recognition module analyzes the target website's page type, structure, and request / response characteristics in real time, automatically adjusting proxy IPs, dynamically generating User-Agents, adding complete HTTP request headers, dynamically rendering, optimizing page access time intervals, and automating CAPTCHA verification. The page parsing module extracts data from the downloaded data to obtain structured data in real time. The security compliance module encrypts and parses data during webpage downloads and adheres to robots.txt rules to limit the crawling scope. The page depth analysis module analyzes and records the depth status of crawled pages, reducing the risk of triggering anti-crawling blocking. The downloader includes a user behavior simulation module and a download module. The user behavior simulation module is used to simulate normal user mouse behavior to reduce the risk of being detected by anti-crawler mechanisms. The download module is used to obtain the webpage data pointed to by the URL. The data storage device is used to store the crawled content in a structured form in the database.

Images