Intelligent crawler generation method and system based on large language model and mcp protocol

Abstract

The application discloses a method and system for generating intelligent crawlers based on a large language model and an MCP protocol, and belongs to the technical field of network data collection. The method solves the problem that the prior art cannot effectively integrate an LLM and a browser interaction protocol, resulting in the inability to fully exert the capability of the LLM in the dynamic webpage analysis and anti-crawling strategy generation links. The method comprises the following steps: analyzing the collection requirement based on a large language model and generating a standardized requirement description document; realizing the interaction between the large language model and the browser based on an MCP protocol; and performing complete DOM tree structure analysis and processing on a page through a crawler script generation system, and performing quality checking and intelligent repairing on the generated crawler script. In the application, the complete DOM tree and dynamic data after browser rendering are obtained through the MCP protocol, and the large language model can be called to automatically analyze element positioning strategies, so that the collection script can be adaptively generated, and the efficiency, intelligence and automation of webpage data collection are ensured.

Description

Technical Field

[0001] This invention belongs to the field of network data acquisition technology, specifically relating to a method and system for generating intelligent web crawlers based on a large language model and the MCP protocol. Background Technology

[0002] With the rapid development of the internet, web pages have become the core carrier of global information dissemination, and their data scale is growing exponentially. Web page data parsing technology, as a core means of extracting valuable information from unstructured web pages, is widely used in market research, business analysis, and public opinion monitoring. Traditional web page data collection mainly relies on manually written rules (such as XPath and CSS selectors) or static parsing methods based on regular expressions, using web crawling frameworks (such as Scrapy and BeautifulSoup) to crawl data. However, with the evolution of web page technologies (such as dynamic rendering and upgraded anti-crawling mechanisms) and the increase in data diversity, traditional methods have gradually revealed the following shortcomings:

[0003] 1) High dependence on webpage structure and high maintenance costs: Traditional web crawlers rely on fixed HTML tag paths (such as XPath / CSS selectors) to locate data. When the webpage structure undergoes minor changes (such as tag name adjustments or hierarchical structure adjustments), the original rules become invalid, requiring manual re-analysis of the DOM tree and modification of the rules. Statistics show that in websites that frequently redesign, rule maintenance costs account for more than 60% of the total cost of data collection projects, leading to delays in data collection task response.

[0004] 2) Insufficient dynamic content processing capabilities: Modern web pages widely adopt JavaScript dynamic rendering technologies (such as AJAX asynchronous loading and WebSocket real-time updates). Traditional web crawlers only parse the initial HTML content and cannot capture dynamically loaded data. For example, scrolling lists triggered by user interaction and key fields hidden in pop-ups will be missed, resulting in compromised data integrity.

[0005] 3) Weak anti-scraping mechanisms: Websites commonly deploy anti-scraping strategies (such as User-Agent verification, dynamic token authentication, and CAPTCHA blocking), and traditional web crawlers lack adaptive mechanisms. Once a website upgrades its anti-scraping measures, the crawler is easily blocked or unable to obtain data, requiring manual intervention to adjust the request strategy, which is inefficient.

[0006] 4) Difficulty in integrating multi-source data: Data formats vary significantly across different websites (e.g., structured tables, semi-structured JSON, unstructured text), requiring manual format conversion and cleaning using traditional methods. For example, integrating structured product data from e-commerce platforms with unstructured comments from forums requires considerable effort to address data heterogeneity issues.

[0007] 5) Low level of automation and reliance on manual intervention: From analyzing data collection needs to generating the final collection script, the entire process heavily relies on manual intervention. The limitations of manual operation become increasingly apparent when faced with large-scale and diverse collection tasks. For example, if a company needs to collect data from hundreds of different types of websites simultaneously for market trend analysis, manually analyzing the structure of each website and writing collection rules is not only extremely time-consuming but also makes it difficult to guarantee the consistency and accuracy of the rules, failing to respond promptly to the data demands of dynamic market changes.

[0008] With the rise of Large Language Models (LLMs), their semantic understanding and reasoning capabilities have brought new opportunities to web crawling technology. However, existing technologies cannot effectively integrate LLMs with browser interaction protocols (such as MCP), resulting in the inability to fully utilize the capabilities of LLMs in dynamic webpage parsing and anti-crawling strategy generation. To address these issues, we propose an intelligent web crawler generation method and system based on Large Language Models and the MCP protocol. Summary of the Invention

[0009] The purpose of this invention is to address the shortcomings of existing technologies by providing an intelligent web crawler generation method and system based on the Large Language Model and MCP protocol. This solves the problem that existing technologies cannot effectively integrate LLM with browser interaction protocols, resulting in the inability to fully utilize the capabilities of LLM in dynamic webpage parsing and anti-crawling strategy generation.

[0010] This invention is implemented as follows: an intelligent web crawler generation method based on a large language model and the MCP protocol, wherein the intelligent web crawler generation method based on a large language model and the MCP protocol includes:

[0011] S10, the user inputs the collection requirements through natural language, and the collection requirements are parsed based on the large language model to generate a standardized requirement description document;

[0012] S20, based on the MCP protocol, enables the interaction between the large language model and the browser, drives the browser to complete the loading of target web page content and dynamic data capture, and outputs basic page information and complete DOM tree structure;

[0013] S30 loads basic page information and the complete DOM tree structure, and uses the crawler script generation system to parse and process the complete DOM tree structure of the page.

[0014] S40 combines a large language model with multi-dimensional parsing results to generate adaptive crawler scripts through template-driven generation.

[0015] S50 performs quality verification and intelligent repair on the generated crawler scripts, forming executable and customized data collection scripts;

[0016] S60 crawls multi-source data based on data acquisition scripts, preprocesses the multi-source data, and stores the preprocessed multi-source data in the target database, forming a traceable data processing chain.

[0017] Preferably, the method for parsing the collection requirements based on a large language model and generating a standardized requirement description document includes:

[0018] S101, the user interacts with the large language model to determine the collection requirements, and uses natural language processing technology to parse the unstructured collection requirements input by the user;

[0019] S102, based on the built-in prompt word template, performs in-depth semantic analysis of user needs, extracts core keywords, data types and structured elements of the collection target;

[0020] S103: When the user's input collection request contains ambiguous or unclear content, a multi-turn dialogue process is triggered to intelligently complete the key information.

[0021] S104, based on the requirement parser, decomposes natural language requirements into semantic units, maps the parsed semantic units to predefined technical specification templates, and automatically generates a standardized requirement description document containing website address, data model, constraints, and collection frequency.

[0022] Preferably, the method for implementing interaction between a large language model and a browser based on the MCP protocol includes:

[0023] S201, the module based on the MCP protocol calls the BrowserTools MCP service through Anthropic's Model Context Protocol MCP to establish a communication channel with the Chrome browser;

[0024] S202, the browser page loading module sends navigation instructions to the BrowserTools MCP service through the MCP protocol, drives the Chrome browser to open the target website page, and monitors the page loading status;

[0025] S203 After the page is successfully loaded, the context connection establishment module establishes a persistent context connection with the browser page through the MCP protocol, obtains the basic information of the page and the complete DOM tree structure, and stores it in memory. The basic information of the page includes the URL, document type, and encoding format.

[0026] S204 listens for JavaScript events on the page via the MCP protocol, captures and parses dynamically loaded data in real time, and updates the page state in memory.

[0027] Preferably, the method for parsing and processing the complete DOM tree structure of a page using a web crawler script generation system includes:

[0028] S301, based on the DOM tree parsing module, performs hierarchical traversal of the DOM structure, generates XPath / CSS selectors based on element tags, class names, ID attributes and semantic understanding, and determines the element positioning strategy;

[0029] S302 uses an OCR screenshot recognition module to obtain webpage screenshots and combines OCR technology to recognize button text and embedded text in images, and cross-validates with DOM structure to enhance positioning accuracy;

[0030] S303, the API data parsing module parses and processes HTTP requests and responses, matches data interfaces according to standardized requirements, and establishes semantic associations between interface response content and required data fields;

[0031] S304, the authentication parameter parsing module extracts and parses the website's encrypted JS file, restores the encryption algorithm process through a large language model, and generates runnable equivalent encryption / decryption code.

[0032] Preferably, the method for quality verification and intelligent repair of the generated crawler script includes:

[0033] S401, the large language model calls script templates based on the semantics and multi-dimensional parsing results of the large model, and selects the appropriate script template through the template-driven generation method. Based on the appropriate script template, it automatically generates a crawler script containing data extraction, anti-crawling strategies and exception handling logic.

[0034] S402 performs syntax validity checks and small-scale gray-scale tests on the generated crawler script, and monitors the script execution results and data integrity indicators.

[0035] S403 If the crawler script fails the verification, a two-level repair mechanism combining rule-driven and LLM-assisted repair is adopted to intelligently repair the crawler script. When intelligently repairing the crawler script, common errors are automatically repaired based on predefined rules, and complex problems are repaired by generating repair solutions through in-depth analysis of large models. If the standards are not met, the parsing results are re-analyzed and the script strategy is adjusted.

[0036] Preferably, the method for preprocessing multi-source data includes:

[0037] S501 loads multi-source data and combines regular expressions and a rule engine to remove invalid data.

[0038] S502 utilizes a large language model to perform semantic standardization processing on multi-source data, which includes unifying time formats and parsing fuzzy text.

[0039] S503 uses data mapping technology to convert heterogeneous data into a unified format and outputs standardized datasets that meet business needs.

[0040] Preferably, when storing the preprocessed multi-source data into the target database, structured data is stored in PostgreSQL, MySQL, or MongoDB databases, and data consistency and query performance are ensured through database transactions and indexes; unstructured data is stored in S3, MinIO, or OSS databases, and metadata indexes such as file type, creation time, and data tags are established.

[0041] On the other hand, the present invention also provides an intelligent web crawler generation system based on a large language model and the MCP protocol, wherein the intelligent web crawler generation system based on a large language model and the MCP protocol includes:

[0042] The requirement description module allows users to input requirements via natural language. The requirement description module then parses the collected requirements based on a large language model and generates a standardized requirement description document.

[0043] The MCP protocol interaction system is based on the MCP protocol to realize the interaction between the large language model and the browser, drive the browser to complete the loading of target web page content and dynamic data capture, and output basic page information and complete DOM tree structure.

[0044] The web crawler script generation system is used to load basic page information and the complete DOM tree structure, and to parse and process the complete DOM tree structure of the page.

[0045] The script verification system combines a large language model with multi-dimensional parsing results and generates adaptive crawler scripts through template-driven generation. It performs quality verification and intelligent repair on the generated crawler scripts to form executable customized data collection scripts. The script verification system includes a script generation module and a script quality verification module.

[0046] The data storage module crawls multi-source data based on data acquisition scripts, preprocesses the multi-source data, and stores the preprocessed multi-source data into the target database, forming a traceable data processing chain.

[0047] Preferably, the MCP protocol interaction system includes:

[0048] The MCP protocol initialization module calls the BrowserTools MCP service through Anthropic's Model Context Protocol (MCP) to establish a communication channel with the Chrome browser.

[0049] The browser page loading module sends navigation instructions to the BrowserTools MCP service via the MCP protocol, drives the Chrome browser to open the target website page, and monitors the page loading status.

[0050] The context connection establishment module establishes a persistent context connection with the browser page through the MCP protocol after the page is successfully loaded, obtains the basic information and complete DOM tree structure of the page, and stores them in memory. It also listens for JavaScript events in the page through the MCP protocol, captures and parses dynamically loaded data in real time, and updates the page state in memory.

[0051] Preferably, the crawler script generation system includes:

[0052] The DOM tree parsing module is used to traverse the DOM structure hierarchically, generate XPath / CSS selectors based on element tags, class names, ID attributes and semantic understanding, and determine the element positioning strategy.

[0053] The OCR screenshot recognition module is used to obtain webpage screenshots and combine OCR technology to recognize button text and text embedded in images, and cross-validate with DOM structure to enhance positioning accuracy.

[0054] The API data parsing module is used to parse and process HTTP requests and responses, match data interfaces according to standardized requirements, and establish semantic associations between interface response content and required data fields.

[0055] The authentication parameter parsing module is used to extract and parse encrypted JS files from the website, reconstruct the encryption algorithm process through a large language model, and generate runnable equivalent encryption / decryption code.

[0056] Compared with the prior art, the embodiments of this application have the following main advantages:

[0057] In this embodiment of the invention, the complete DOM tree and dynamic data rendered by the browser are obtained through the MCP protocol, and the large language model can be called to automatically analyze the element positioning strategy, thereby adaptively generating the collection script. Combined with visual feature analysis, multimodal data utilization is realized, which ensures the efficiency, intelligence and automation of web page data collection. It solves the core problems of poor dynamic adaptability and weak anti-crawling resistance in existing crawling technology, and is conducive to realizing large-scale and automated data collection and crawling.

[0058] This invention combines a large language model with the MCP protocol, leveraging the semantic understanding capabilities of the large language model to achieve end-to-end automation from requirement understanding and script generation to execution monitoring. It can adapt to constantly changing webpage environments, significantly improving the intelligence and automation level of webpage data collection. It boasts core advantages such as automated parsing and script generation, and automatic analysis of DOM structure, JavaScript logic, and visual features by the large model. By replacing manual rule writing, it can shorten the development cycle and supports batch generation of crawler scripts adapted to different website structures. The time complexity of large-scale tasks is significantly better than traditional solutions. Furthermore, this invention also offers the advantages of dynamic adaptation and low-cost maintenance. By automatically identifying changes in webpage structure and adjusting collection rules in real time, it reduces manual intervention, resulting in low labor costs throughout the process. It is relatively simple and efficient, suitable for high-frequency iterative internet data collection scenarios.

[0059] In this embodiment of the invention, script generation is based on intelligent parsing driven by a large model. By automatically analyzing the DOM structure, JavaScript logic, and visual features of a webpage through a large language model, crawler scripts adapted to different websites are generated, thereby replacing the traditional process of manually writing rules, significantly improving development efficiency and dynamic adaptability. At the same time, the MCP protocol is used to establish a contextual connection with the browser page, thereby obtaining the complete DOM tree and dynamically loaded data, solving the problem of insufficient collection of dynamically loaded data by traditional crawlers. Attached Figure Description

[0060] Figure 1 This is a schematic diagram illustrating the implementation process of the intelligent crawler generation method based on a large language model and the MCP protocol provided by this invention.

[0061] Figure 2 The diagram illustrates the process of implementing quality verification and intelligent repair methods for the generated crawler script.

[0062] Figure 3 This is a schematic diagram of the intelligent crawler generation system based on a large language model and the MCP protocol provided by the present invention.

[0063] In the diagram: 100 - Requirements Description Module, 200 - MCP Protocol Interaction System, 210 - MCP Protocol Initialization Module, 220 - Browser Page Loading Module, 230 - Context Connection Establishment Module, 300 - Web Crawler Script Generation System, 310 - DOM Tree Parsing Module, 320 - OCR Screenshot Recognition Module, 330 - API Data Parsing Module, 340 - Authentication Parameter Parsing Module, 400 - Script Verification System, 410 - Script Generation Module, 420 - Script Quality Verification Module, 500 - Data Storage Module. Detailed Implementation

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0065] Existing technologies cannot effectively integrate LLM with browser interaction protocols (such as MCP), resulting in an inability to fully leverage the capabilities of LLM in dynamic webpage parsing and anti-crawling strategy generation. To address these issues, we propose an intelligent web crawler generation method and system based on a large language model and the MCP protocol. In short, the method first parses the data collection requirements using a large language model and generates a standardized requirement description document. Then, it implements interaction between the large language model and the browser using the MCP protocol. Next, a web crawler script generation system parses and processes 300 pairs of complete DOM tree structures of the page. Adaptive web crawler scripts are generated through template-driven processing. The generated web crawler scripts undergo quality verification and intelligent repair to form an executable, customized data collection script. Multi-source data is preprocessed, and finally, the preprocessed multi-source data is stored in the target database, forming a traceable data processing chain. In this embodiment of the invention, the complete DOM tree and dynamic data rendered by the browser are obtained through the MCP protocol, and the large language model can be called to automatically analyze the element positioning strategy, thereby adaptively generating the collection script. Combined with visual feature analysis, multimodal data utilization is realized, which ensures the efficiency, intelligence and automation of web page data collection. It solves the core problems of poor dynamic adaptability and weak anti-crawling resistance in existing crawling technology, and is conducive to realizing large-scale and automated data collection and crawling.

[0066] It should be noted that the relevant terms and definitions in this embodiment are as follows:

[0067] Large Language Models (LLMs) are language models constructed from deep neural networks containing hundreds of billions of parameters.

[0068] The Model Context Protocol (MCP) is a new standard open-sourced by Anthropic on November 25, 2024. MCP is an open standard designed to connect AI assistants with the systems containing their data. The MCP protocol includes content repositories, business tools, and development environments. Its goal is to help cutting-edge models generate better and more relevant responses. MCP can be seen as a "USB-C port" for AI applications. Just as USB-C provides a standardized way to connect devices to various peripherals, MCP provides a standardized method for AI models to connect to different data sources and tools.

[0069] DOM Tree: The DOM is the standard way to access and manipulate HTML documents; it represents HTML documents as a tree structure.

[0070] OCR technology: OCR stands for Optical Character Recognition. It uses optical and computer technologies to read text printed or written on paper and convert it into a format that computers can accept and humans can understand.

[0071] Browser Interface: By capturing network packets in the browser's network module, you can see related requests, including information such as APIs, JavaScript, CSS, images, media, and text. API information includes name, method, status, and time.

[0072] Gray-scale testing: Gray-scale testing refers to the process of verifying the stability and compatibility of a new version on a small scale before its official release, thereby reducing the risks associated with a full rollout of the new version.

[0073] BrowserTools: BrowserTools MCP is a browser plugin and local service suite based on the MCP (Model Context Protocol). Its functions include browser monitoring and interaction, enabling screenshotting, log analysis, and network activity tracking. Its purpose is to provide visual debugging of web applications via a Chrome extension.

[0074] Example 1

[0075] This invention provides an intelligent web crawler generation method based on a large language model and the MCP protocol. Figure 1 The diagram illustrates the implementation flow of an intelligent web crawler generation method based on a large language model and the MCP protocol. Specifically, the intelligent web crawler generation method based on a large language model and the MCP protocol includes:

[0076] S10, the user inputs the collection requirements through natural language, and the collection requirements are parsed based on the large language model to generate a standardized requirement description document;

[0077] S20, based on the MCP protocol, enables the interaction between the large language model and the browser, drives the browser to complete the loading of target web page content and dynamic data capture, and outputs basic page information and complete DOM tree structure;

[0078] S30 loads basic page information and the complete DOM tree structure, and generates a system 300 parsing and processing of the complete DOM tree structure of the page through the crawler script;

[0079] S40 combines a large language model with multi-dimensional parsing results to generate adaptive crawler scripts through template-driven generation.

[0080] S50 performs quality verification and intelligent repair on the generated crawler scripts, forming executable and customized data collection scripts;

[0081] S60 crawls multi-source data based on data acquisition scripts, preprocesses the multi-source data, and stores the preprocessed multi-source data in the target database, forming a traceable data processing chain.

[0082] In this embodiment of the invention, the complete DOM tree and dynamic data rendered by the browser are obtained through the MCP protocol, and the large language model can be called to automatically analyze the element positioning strategy, thereby adaptively generating the collection script. Combined with visual feature analysis, multimodal data utilization is realized, which ensures the efficiency, intelligence and automation of web page data collection. It solves the core problems of poor dynamic adaptability and weak anti-crawling resistance in existing crawling technology, and is conducive to realizing large-scale and automated data collection and crawling.

[0083] This invention provides a method for parsing the data collection requirements based on a large language model and generating a standardized requirement description document. The method specifically includes:

[0084] S101, the user interacts with the large language model to determine the collection requirements, and uses natural language processing technology to parse the unstructured collection requirements input by the user;

[0085] It should be noted that the large language model can be DeepSeek or Claude 3.7. The user request acquisition refers to users submitting their requests via text boxes using colloquial or professional language, such as inputting "Collect product information and release dates related to a certain medical device from a certain e-commerce platform within the past week." By calling the large language model and then leveraging natural language processing techniques, semantic understanding of multilingual mixed inputs can be processed.

[0086] S102, based on the built-in prompt word template, performs in-depth semantic analysis of user needs, extracts core keywords, data types and structured elements of the collection target;

[0087] S103 combines the reasoning capabilities of a large language model to complete fuzzy requirements. When the user's input collection requirements contain fuzzy or ambiguous content, a multi-turn dialogue process is triggered to intelligently complete key information. For example, confirming the specific start and end dates of "the most recent week" or the specific category of the equipment, key entities and constraints are extracted from the natural language parsing results, and the large model reasoning reduces the deviation in understanding the requirements, so that non-technical users can define complex collection requirements without writing code.

[0088] S104, based on the requirement parser, decomposes natural language requirements into semantic units, maps the parsed semantic units to predefined technical specification templates, and automatically generates a standardized requirement description document containing website address, data model, constraints, and collection frequency.

[0089] In this embodiment, user requirements are standardized, aiming to transform natural language requirements into executable technical specifications. A requirement parser module decomposes these requirements into semantic units, mapping them to predefined technical specification templates. This automatically generates a standardized requirement description document containing website addresses, data models, constraints, and data collection frequencies. This achieves precise conversion from natural language requirements to structured instructions, providing a clear basis for subsequent crawler script generation and data collection.

[0090] This invention provides a method for implementing interaction between a large language model and a browser based on the MCP protocol. Specifically, this method includes:

[0091] S201, The initialization module 210 based on the MCP protocol calls the BrowserTools MCP service through Anthropic's Model Context Protocol MCP to establish a communication channel with the Chrome browser;

[0092] It should be noted that the MCP protocol initialization module 210 is a large language model that calls the BrowserTools MCP service through Anthropic's Model Context Protocol MCP and establishes a communication channel with the Chrome browser, thereby ensuring reliable transmission of instructions and data.

[0093] S202, the browser page loading module 220 sends navigation instructions to the BrowserTools MCP service through the MCP protocol, drives the Chrome browser to open the target website page, and monitors the page loading status;

[0094] S203, after the page is successfully loaded, the context connection establishment module 230 establishes a persistent context connection with the browser page through the MCP protocol, obtains the basic information of the page and the complete DOM tree structure, and stores it in memory. The basic information of the page includes the URL, document type, and encoding format.

[0095] S204: JavaScript events on the page are monitored via the MCP protocol. Dynamically loaded data is captured and parsed in real time, and the page state in memory is updated. It should be noted that JavaScript events on the page include, but are not limited to, AJAX requests and WebSocket communication. In this embodiment, the page state in memory is updated by capturing and parsing dynamically loaded data in real time, and then provided for use by the large language model.

[0096] In this embodiment of the invention, script generation is based on intelligent parsing driven by a large model. By automatically analyzing the DOM structure, JavaScript logic, and visual features of a webpage through a large language model, crawler scripts adapted to different websites are generated, thereby replacing the traditional process of manually writing rules, significantly improving development efficiency and dynamic adaptability. At the same time, the MCP protocol is used to establish a contextual connection with the browser page, thereby obtaining the complete DOM tree and dynamically loaded data, solving the problem of insufficient collection of dynamically loaded data by traditional crawlers.

[0097] This invention provides a method for parsing and processing the complete DOM tree structure of a page using a web crawler script generation system 300. The method specifically includes:

[0098] S301, based on the DOM tree parsing module 310, performs hierarchical traversal of the DOM structure, generates XPath / CSS selectors based on element tags, class names, ID attributes and semantic understanding, and determines the element positioning strategy;

[0099] It should be noted that the DOM tree parsing module 310 is used to perform hierarchical traversal of the obtained complete DOM structure, identify HTML element nodes, and generate XPath and CSS selectors based on the element tag name, class name, ID and other attributes, combined with the predefined rule library, and analyze semantic tags by utilizing the semantic understanding capabilities of the large model, thereby determining the element positioning strategy.

[0100] S302 uses the OCR screenshot recognition module 320 to obtain webpage screenshots and combines OCR technology to recognize button text and embedded text in images, and cross-validates with DOM structure to enhance positioning accuracy;

[0101] In this embodiment of the invention, the OCR screenshot recognition module 320 is used to acquire webpage screenshots and combine OCR technology to recognize visual text such as button text and text embedded in images; the visual text is cross-validated with the DOM structure to enhance the accuracy of element positioning.

[0102] S303, API data parsing module 330 parses and processes HTTP requests and responses, matches data interfaces according to standardized requirements, and establishes semantic associations between interface response content and required data fields;

[0103] The API data parsing module 330 is used to obtain HTTP request and response information, determine the data source interface according to standardized requirements, and semantically match the identified interface response content with the data fields in the standardized requirements, thereby establishing the association between the API interface and the data requirements.

[0104] S304, the authentication parameter parsing module 340 extracts and parses the website's encrypted JS file, restores the encryption algorithm process through a large language model, and generates runnable equivalent encryption / decryption code.

[0105] In this embodiment, the authentication parameter parsing module 340 is a core component for dealing with complex encryption mechanisms on websites. It analyzes JS code using a large model to reconstruct the encryption algorithm process. It extracts and parses encryption-related JS files from the browser environment, analyzes the JS code structure, extracts key encryption functions, uses LLM to understand the encryption logic, simulates the encryption and decryption process, generates an equivalent implementation without changing the algorithm logic, returns directly executable code, and verifies the correctness of the algorithm reconstruction. It can be used to automatically identify changes in the website's encryption mechanism and update the reconstruction algorithm.

[0106] In this embodiment of the invention, when the crawler script generation system parses and processes the complete DOM tree structure of the page, it combines large model semantic understanding and OCR technology to cross-validate the DOM structure and visual cues (such as text-image association and button text), thereby improving the accuracy of element positioning and the ability to extract multimodal data.

[0107] This invention provides a method for quality verification and intelligent repair of generated crawler scripts. Figure 2 This diagram illustrates the implementation process of a method for quality verification and intelligent repair of generated crawler scripts. The method specifically includes:

[0108] S401, the large language model calls script templates based on the semantics and multi-dimensional parsing results of the large model, and selects the appropriate script template through the template-driven generation method. Based on the appropriate script template, it automatically generates a crawler script containing data extraction, anti-crawling strategies and exception handling logic.

[0109] S402 performs syntax validity checks and small-scale gray-scale tests on the generated crawler script, and monitors the script execution results and data integrity indicators.

[0110] S403 If the crawler script fails the verification, a two-level repair mechanism combining rule-driven and LLM-assisted repair is adopted to intelligently repair the crawler script. When intelligently repairing the crawler script, common errors are automatically repaired based on predefined rules, and complex problems are repaired by generating repair solutions through in-depth analysis of large models. If the standards are not met, the parsing results are re-analyzed and the script strategy is adjusted.

[0111] In this embodiment, quality verification and intelligent repair of the generated crawler script are implemented through a script generation module 410 and a script quality verification module 420. The script generation module 410 automatically adapts to the data structure of different websites and automatically selects script templates by utilizing the semantic understanding capabilities of a large language model and DOM structure or API interfaces. If multi-level nested tags are detected, a script containing recursive parsing logic is generated; if it is a simple list page, a batch extraction loop structure is generated. Highly adaptable customized crawling scripts are generated, covering data extraction, dynamic interaction, and anti-crawling response scenarios. When executing anti-crawling response scenarios, an intelligent anti-crawling strategy is generated: by analyzing the website's anti-crawling mechanisms (such as parameter encryption and CAPTCHA) through a large speech model, the encryption algorithm is automatically inferred and restored, thereby dynamically adjusting the request strategy (such as random UA and IP pool rotation) to improve the success rate of data collection.

[0112] The script quality verification module 420 is used to perform multi-level quality verification on the generated scripts. When performing multi-level quality verification on the generated scripts, it first calls the syntax parser of the corresponding language (such as the ast module of Python) to check the syntax legality, and then conducts a usability gray-scale test on a small range (such as 1% of the target URLs) to monitor the script execution results, data integrity and other indicators. If the verification finds problems, the crawler script is determined to fail the verification. A two-level repair mechanism of rule-driven and LLM-assisted intelligent repair is adopted. That is, common error types are automatically repaired based on predefined repair rules. For complex problems, a large model is called to perform in-depth analysis and generate targeted repair solutions. If the indicators still do not meet the standards, the large model is triggered to re-analyze the parsing results and automatically adjust the script generation strategy, forming a closed-loop system from script generation to quality verification to intelligent repair, which significantly improves the reliability and development efficiency of crawler scripts.

[0113] In this embodiment of the invention, the method for preprocessing multi-source data includes:

[0114] S501 loads multi-source data and combines regular expressions and a rule engine to remove invalid data.

[0115] S502 utilizes a large language model to perform semantic standardization processing on multi-source data, which includes unifying time formats and parsing fuzzy text.

[0116] S503 uses data mapping technology to convert heterogeneous data into a unified format, outputting standardized datasets that meet business needs, providing a high-quality data foundation for subsequent analysis and applications.

[0117] In this embodiment of the invention, when storing the preprocessed multi-source data into the target database, the structured data is stored in a PostgreSQL, MySQL, or MongoDB database, and data consistency and query performance are ensured through database transactions and indexes; the unstructured data is stored in an S3, MinIO, or OSS database, and metadata indexes such as file type, creation time, and data tags are established.

[0118] It should be noted that, for structured data, this embodiment of the invention supports storage in relational databases such as PostgreSQL and MySQL, or document databases such as MongoDB, utilizing database transactions and indexing mechanisms to ensure data consistency and query performance. For unstructured data, object storage such as S3, MinIO, or OSS databases is used for storage, and a metadata indexing system is used to establish index information such as file type, creation time, and data tags to improve retrieval efficiency. At the data update level, incremental data entry is implemented based on timestamps or ID fields, and duplicate writes are avoided through hash verification and version comparison. At the same time, the system records processing information such as data source URL, crawling time, script version number, and cleaning rules, forming a traceable data processing chain. This provides complete evidence for data auditing, problem investigation, and version backtracking, ensuring the standardization and reliability of data management.

[0119] Example 2

[0120] This invention provides an intelligent web crawler generation method based on a large language model and the MCP protocol. Specifically, this invention takes the collection of bidding information from three platforms as an example, assuming they are a resource trading platform (Province A), a resource trading platform (Province B), and a resource trading platform (Province C). For example, when generating a standardized bidding project dataset, the intelligent web crawler generation method based on a large language model and the MCP protocol specifically includes:

[0121] S10, the user inputs the collection requirements through natural language, and the collection requirements are parsed based on the large language model to generate a standardized requirement description document;

[0122] The user input includes information such as "collecting bidding information for medical device projects on the above-mentioned resource trading platform within the past three months, including project name, bidding unit, bid opening time, budget amount, and attachments".

[0123] When parsing the data collection requirements based on the Large Language Model (LLM) and generating standardized requirement description documents, the LLM requirement parsing involves a deep analysis of the user-input requirements. It identifies key information such as the time frame (last 3 months), project type (medical devices), and the data fields to be collected (project name, bidding entity, bid opening time, budget amount, attachments). For requirements involving three platforms, it is determined that subsequent data collection operations should be performed separately for each of the three platforms.

[0124] When generating standardized requirements description documents: integrate the requirements parsed from the large language model into a standardized document. Clearly define the requirements for selecting bidding projects from three resource trading platforms based on specific time and project type, and standardize the requirements for obtaining specified data fields. At the same time, clearly define the data format (e.g., the date format for bid opening time), precision, etc.

[0125] S20, based on the MCP protocol, enables the interaction between the large language model and the browser, drives the browser to complete the loading of target web page content and dynamic data capture, and outputs basic page information and complete DOM tree structure;

[0126] In this embodiment of the invention, when implementing the interaction between a large language model and a browser based on the MCP protocol, for a resource trading platform (Province A), the browser is launched to access the platform's website via the MCP protocol. During page loading, the status is monitored, a context connection is established, basic page information and the complete DOM tree structure are obtained, JavaScript events on the page are listened to, and dynamically loaded data is captured in real time. Similarly, MCP protocol browser interaction operations are performed on resource trading platforms (Province B) and (Province C) to establish connections with the corresponding platform pages and obtain relevant information.

[0127] S30 loads basic page information and the complete DOM tree structure, and generates a system 300 parsing and processing of the complete DOM tree structure of the page through the crawler script;

[0128] In this process, the web crawler script generates 300 pairs of complete DOM tree structures for each page. It then parses the DOM / API data, specifically by hierarchically traversing the DOM tree obtained from a resource trading platform (Province A) to generate XPath and CSS selectors to locate the bidding project information area. HTTP requests are analyzed to find relevant API interfaces (if any), and the interface responses are semantically matched with the required fields to determine the information source. Similarly, DOM / API data parsing is performed on resource trading platforms (Province B) and (Province C) respectively, and the data extraction method is determined based on the page structure and data transmission characteristics of each platform.

[0129] S40 combines a large language model with multi-dimensional parsing results to generate adaptive crawler scripts through template-driven generation.

[0130] When generating adaptable crawler scripts using template-driven methods, based on the data parsing results of a resource trading platform (Province A) and considering its page structure characteristics, an adaptable crawler script is generated to extract information such as project names and bidding units from the platform. For a resource trading platform (Province B) and a resource trading platform (Province C), corresponding crawler scripts are generated according to their respective data parsing results. The scripts cover the logic of extracting target information from pages with different structures.

[0131] S50 performs quality verification and intelligent repair on the generated crawler scripts, forming executable and customized data collection scripts;

[0132] When performing quality verification and intelligent repair on the generated crawler scripts, the crawler scripts generated from a resource trading platform (Province A) are first subjected to syntax checks, and then tested on a small scale. If problems are found, they are repaired according to rules-driven mechanisms combined with LLM assistance. If the indicators still do not meet the standards, the script generation strategy is re-analyzed and adjusted. The same process is then used to perform quality verification and repair optimization on the crawler scripts generated from the resource trading platform (Province B) and the resource trading platform (Province C).

[0133] S60 crawls multi-source data based on data acquisition scripts, preprocesses the multi-source data, and stores the preprocessed multi-source data in the target database, forming a traceable data processing chain.

[0134] In this embodiment, during multi-source data preprocessing, data collected from three platforms is integrated. A large language model is used to unify data semantics, such as standardizing the bid opening time format; regular expressions and rule engines are used to remove invalid data; and differences between structured and unstructured data from different platforms are addressed through data mapping to unify the format, forming a standardized bidding project dataset. When storing the preprocessed multi-source data in the target database, the cleaned and preprocessed bidding project data can be stored in the database. Considering the structured nature of the data, a relational database (such as MySQL) is selected for storage. Database transactions are used to ensure data consistency, and indexes are created for frequently used query fields to improve query performance.

[0135] Example 3

[0136] This invention provides an intelligent web crawler generation system based on a large language model and the MCP protocol. Figure 3 A schematic diagram of an intelligent web crawler generation system based on a large language model and the MCP protocol is shown. This intelligent web crawler generation system specifically includes:

[0137] The requirement description module 100 allows users to input requirements via natural language. The requirement description module 100 then parses the input requirements based on a large language model and generates a standardized requirement description document.

[0138] The MCP protocol interaction system 200 is based on the MCP protocol to realize the interaction between the large language model and the browser, drive the browser to complete the loading of target web page content and dynamic data capture, and output basic page information and complete DOM tree structure.

[0139] The MCP protocol interaction system 200 includes:

[0140] MCP protocol initialization module 210 calls BrowserTools MCP service through Anthropic's Model Context Protocol MCP to establish a communication channel with Chrome browser.

[0141] The browser page loading module 220 sends navigation instructions to the BrowserToolsMCP service via the MCP protocol, drives the Chrome browser to open the target website page, and monitors the page loading status.

[0142] After the page is successfully loaded, the context connection establishment module 230 establishes a persistent context connection with the browser page through the MCP protocol, obtains the basic information and complete DOM tree structure of the page, and stores them in memory. It also listens for JavaScript events in the page through the MCP protocol, captures and parses dynamically loaded data in real time, and updates the page state in memory.

[0143] The crawler script generation system 300 is used to load basic page information and the complete DOM tree structure, and to parse and process the complete DOM tree structure of the page.

[0144] It should be noted that the crawler script generation system 300 includes:

[0145] DOM tree parsing module 310 is used to traverse the DOM structure hierarchically, generate XPath / CSS selectors based on element tags, class names, ID attributes and semantic understanding, and determine the element positioning strategy.

[0146] The OCR screenshot recognition module 320 is used to acquire webpage screenshots and combine OCR technology to recognize button text and embedded text in images, and cross-validate with DOM structure to enhance positioning accuracy.

[0147] API data parsing module 330 is used to parse and process HTTP requests and responses, match data interfaces according to standardized requirements, and establish semantic associations between interface response content and required data fields.

[0148] The authentication parameter parsing module 340 is used to extract and parse the encrypted JS file of the website, restore the encryption algorithm process through a large language model, and generate runnable equivalent encryption / decryption code.

[0149] The script verification system 400 is used to combine a large language model with multi-dimensional parsing results and generate an adaptive crawler script through template-driven generation. The generated crawler script is then subjected to quality verification and intelligent repair to form an executable customized data collection script. The script verification system 400 includes a script generation module 410 and a script quality verification module 420.

[0150] The data storage module 500 crawls multi-source data based on the data acquisition script, preprocesses the multi-source data, and stores the preprocessed multi-source data into the target database, forming a traceable data processing chain.

[0151] In summary, this invention provides an intelligent web crawler generation method and system based on a large language model and the MCP protocol. In the embodiments of this invention, the complete DOM tree and dynamic data rendered by the browser are obtained through the MCP protocol, and the large language model can be called to automatically analyze element positioning strategies, thereby adaptively generating collection scripts. Combined with visual feature analysis, multimodal data utilization is realized, ensuring the high efficiency, intelligence and automation of web page data collection. This solves the core problems of poor dynamic adaptability and weak anti-crawling resistance in existing web crawler technologies, and is conducive to realizing large-scale, automated data collection and crawling.

[0152] This invention combines a large language model with the MCP protocol, leveraging the semantic understanding capabilities of the large language model to achieve end-to-end automation from requirement understanding and script generation to execution monitoring. It can adapt to constantly changing webpage environments, significantly improving the intelligence and automation level of webpage data collection. It boasts core advantages such as automated parsing and script generation, and automatic analysis of DOM structure, JavaScript logic, and visual features by the large model. By replacing manual rule writing, it can shorten the development cycle and supports batch generation of crawler scripts adapted to different website structures. The time complexity of large-scale tasks is significantly better than traditional solutions. Furthermore, this invention also offers the advantages of dynamic adaptation and low-cost maintenance. By automatically identifying changes in webpage structure and adjusting collection rules in real time, it reduces manual intervention, resulting in low labor costs throughout the process. It is relatively simple and efficient, suitable for high-frequency iterative internet data collection scenarios.

[0153] It should be noted that, for the sake of simplicity, the foregoing embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to the present invention. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0154] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on these embodiments, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art can still combine, add, delete, or otherwise adjust the features of the various embodiments of the present invention according to the circumstances without conflict or creative effort, thereby obtaining different technical solutions that do not fundamentally depart from the concept of the present invention. These technical solutions also fall within the scope of protection of the present invention.

Claims

1. A method for generating intelligent web crawlers based on a large language model and the MCP protocol, characterized in that, The method includes: S10, the user inputs the collection requirements through natural language, and the collection requirements are parsed based on the large language model to generate a standardized requirement description document; S20, based on the MCP protocol, enables the interaction between the large language model and the browser, drives the browser to complete the loading of target web page content and dynamic data capture, and outputs basic page information and complete DOM tree structure; S30 loads basic page information and the complete DOM tree structure, and uses the crawler script generation system to parse and process the complete DOM tree structure of the page. S40 combines a large language model with multi-dimensional parsing results to generate adaptive crawler scripts through template-driven generation. S50 performs quality verification and intelligent repair on the generated crawler scripts, forming executable and customized data collection scripts; S60 crawls multi-source data based on data acquisition scripts, preprocesses the multi-source data, and stores the preprocessed multi-source data in the target database to form a traceable data processing chain; The method for parsing and processing the complete DOM tree structure of a page using a web crawler script generation system includes: S301, based on the DOM tree parsing module, performs hierarchical traversal of the DOM structure, generates XPath / CSS selectors based on element tags, class names, ID attributes and semantic understanding, and determines the element positioning strategy; S302 uses an OCR screenshot recognition module to obtain webpage screenshots and combines OCR technology to recognize button text and embedded text in images, and cross-validates with DOM structure to enhance positioning accuracy; S303, the API data parsing module parses and processes HTTP requests and responses, matches data interfaces according to standardized requirements, and establishes semantic associations between interface response content and required data fields; S304, the authentication parameter parsing module extracts and parses the website's encrypted JS file, restores the encryption algorithm process through a large language model, and generates runnable equivalent encryption / decryption code; The method for quality verification and intelligent repair of the generated crawler script includes: S401, the large language model calls script templates based on the semantics and multi-dimensional parsing results of the large model, and selects the appropriate script template through the template-driven generation method. Based on the appropriate script template, it automatically generates a crawler script containing data extraction, anti-crawling strategies and exception handling logic. S402 performs syntax validity checks and small-scale gray-scale tests on the generated crawler script, and monitors the script execution results and data integrity indicators. S403 If the crawler script fails the verification, a two-level repair mechanism combining rule-driven and LLM-assisted repair is adopted to intelligently repair the crawler script. When intelligently repairing the crawler script, common errors are automatically repaired based on predefined rules, and complex problems are repaired by generating repair solutions through in-depth analysis of large models. If the standards are not met, the parsing results are re-analyzed and the script strategy is adjusted.

2. The intelligent web crawler generation method based on large language model and MCP protocol as described in claim 1, characterized in that: The method for parsing the collection requirements based on a large language model and generating a standardized requirement description document includes: S101, the user interacts with the large language model to determine the collection requirements, and uses natural language processing technology to parse the unstructured collection requirements input by the user; S102, based on the built-in prompt word template, performs in-depth semantic analysis of user needs, extracts core keywords, data types and structured elements of the collection target; S103: When the user's input collection request contains ambiguous or unclear content, a multi-turn dialogue process is triggered to intelligently complete the key information. S104, based on the requirement parser, decomposes natural language requirements into semantic units, maps the parsed semantic units to predefined technical specification templates, and automatically generates a standardized requirement description document containing website address, data model, constraints, and collection frequency.

3. The intelligent web crawler generation method based on large language model and MCP protocol as described in claim 1, characterized in that: The method for implementing interaction between a large language model and a browser based on the MCP protocol includes: S201, the module based on the MCP protocol calls the BrowserTools MCP service through Anthropic's Model Context Protocol MCP to establish a communication channel with the Chrome browser; S202, the browser page loading module sends navigation instructions to the BrowserTools MCP service through the MCP protocol, drives the Chrome browser to open the target website page, and monitors the page loading status; S203 After the page is successfully loaded, the context connection establishment module establishes a persistent context connection with the browser page through the MCP protocol, obtains the basic information of the page and the complete DOM tree structure, and stores it in memory. The basic information of the page includes the URL, document type, and encoding format. S204 listens for JavaScript events on the page via the MCP protocol, captures and parses dynamically loaded data in real time, and updates the page state in memory.

4. The intelligent web crawler generation method based on large language model and MCP protocol as described in claim 3, characterized in that: The method for preprocessing multi-source data includes: S501 loads multi-source data and combines regular expressions and a rule engine to remove invalid data. S502 utilizes a large language model to perform semantic standardization processing on multi-source data, which includes unifying time formats and parsing fuzzy text. S503 uses data mapping technology to convert heterogeneous data into a unified format and outputs standardized datasets that meet business needs.

5. The intelligent crawler generation method based on large language model and MCP protocol as described in claim 4, characterized in that: When storing the preprocessed multi-source data into the target database, structured data is stored in PostgreSQL, MySQL, or MongoDB databases, and data consistency and query performance are ensured through database transactions and indexes; unstructured data is stored in S3, MinIO, or OSS databases, and metadata indexes for file type, creation time, and data tags are established.

6. An intelligent web crawler generation system based on a large language model and the MCP protocol, used to implement the intelligent web crawler generation method based on a large language model and the MCP protocol as described in any one of claims 1-5, characterized in that: The intelligent web crawler generation system based on large language models and the MCP protocol includes: The requirement description module allows users to input requirements via natural language. The requirement description module then parses the collected requirements based on a large language model and generates a standardized requirement description document. The MCP protocol interaction system is based on the MCP protocol to realize the interaction between the large language model and the browser, drive the browser to complete the loading of target web page content and dynamic data capture, and output basic page information and complete DOM tree structure. The web crawler script generation system is used to load basic page information and the complete DOM tree structure, and to parse and process the complete DOM tree structure of the page. The script verification system combines a large language model with multi-dimensional parsing results and generates adaptive crawler scripts through template-driven generation. It performs quality verification and intelligent repair on the generated crawler scripts to form executable customized data collection scripts. The script verification system includes a script generation module and a script quality verification module. The data storage module crawls multi-source data based on data acquisition scripts, preprocesses the multi-source data, and stores the preprocessed multi-source data into the target database, forming a traceable data processing chain.

7. The intelligent web crawler generation system based on large language model and MCP protocol as described in claim 6, characterized in that: The MCP protocol interaction system includes: The MCP protocol initialization module calls the BrowserTools MCP service through Anthropic's Model Context Protocol (MCP) to establish a communication channel with the Chrome browser. The browser page loading module sends navigation instructions to the BrowserTools MCP service via the MCP protocol, drives the Chrome browser to open the target website page, and monitors the page loading status. The context connection establishment module establishes a persistent context connection with the browser page through the MCP protocol after the page is successfully loaded, obtains the basic information and complete DOM tree structure of the page, and stores them in memory. It also listens for JavaScript events in the page through the MCP protocol, captures and parses dynamically loaded data in real time, and updates the page state in memory.

8. The intelligent web crawler generation system based on large language model and MCP protocol as described in claim 7, characterized in that: The crawler script generation system includes: The DOM tree parsing module is used to traverse the DOM structure hierarchically, generate XPath / CSS selectors based on element tags, class names, ID attributes and semantic understanding, and determine the element positioning strategy. The OCR screenshot recognition module is used to obtain webpage screenshots and combine OCR technology to recognize button text and text embedded in images, and cross-validate with DOM structure to enhance positioning accuracy. The API data parsing module is used to parse and process HTTP requests and responses, match data interfaces according to standardized requirements, and establish semantic associations between interface response content and required data fields. The authentication parameter parsing module is used to extract and parse encrypted JS files from the website, reconstruct the encryption algorithm process through a large language model, and generate runnable equivalent encryption / decryption code.

Images