Intelligent crawler generation method and system based on large language model and MCP protocol

Abstract

The invention discloses an intelligent crawler generation method and system based on a large language model and an MCP protocol, belongs to the technical field of network data collection, and solves the problem that the capability of LLM in dynamic webpage analysis and anti-crawling strategy generation links cannot be fully exerted due to the fact that LLM and browser interaction protocols cannot be effectively integrated in the prior art. The method comprises the steps of analyzing an acquisition demand based on a large language model and generating a standardized demand description document, realizing interaction between the large language model and a browser based on an MCP protocol, analyzing a page complete DOM tree structure through a crawler script generation system, and performing quality verification and intelligent repair on a generated crawler script. According to the method, the complete DOM tree and the dynamic data rendered by the browser are obtained through the MCP, and the large language model can be called to automatically analyze the element positioning strategy, so that the collection script is adaptively generated, and high efficiency, intelligence and automation of webpage data collection are ensured.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of network data collection, and particularly relates to an intelligent crawler generation method and system based on a large language model and an MCP protocol. BACKGROUND

[0002] With the rapid development of the Internet, web pages have become the core carrier of global information dissemination, and the data scale thereof grows exponentially. Web page data analysis technology, as a core means for extracting valuable information from unstructured web pages, is widely used in market research, business analysis, public opinion monitoring and other fields. Traditional web page data collection mainly relies on manually written rules (such as XPath, CSS selector) or static parsing methods based on regular expressions, and data crawling is achieved through a crawler framework (such as Scrapy, BeautifulSoup). However, with the evolution of web page technology (such as dynamic rendering, upgraded anti-crawling mechanisms) and the increase in data diversity, traditional methods gradually reveal the following defects:

[0003] 1) Strong dependence on web page structure, high maintenance cost: Traditional crawlers rely on fixed HTML tag paths (such as XPath / CSS selector) to locate data. When the web page structure changes slightly (such as label name adjustment, hierarchical structure adjustment), the original rules become invalid, and manual analysis of the DOM tree and modification of the rules are required. According to statistics, in websites that frequently update, the cost of rule maintenance accounts for more than 60% of the total cost of data collection projects, resulting in delayed response to data collection tasks.

[0004] 2) Insufficient dynamic content processing capability: Modern web pages widely use JavaScript dynamic rendering technology (such as AJAX asynchronous loading, WebSocket real-time updating), and traditional crawlers can only parse the initial HTML content, failing to capture dynamically loaded data. For example, user interaction triggered scroll loading lists and key fields hidden in pop-up windows are missed, resulting in compromised data integrity.

[0005] 3) Weakness in anti-crawling mechanism response: Websites commonly deploy anti-crawling strategies (such as User-Agent verification, dynamic Token authentication, and CAPTCHA interception), and traditional crawlers lack adaptive mechanisms. Once the website upgrades its anti-crawling measures, the crawler is easily blocked or cannot obtain data, requiring manual intervention to adjust the request strategy, which is inefficient.

[0006] 4) Difficulty in multi-source data fusion: Different websites have significant data format differences (such as structured tables, semi-structured JSON, and unstructured text), and traditional methods require manual format conversion and cleaning. For example, when integrating structured product data from e-commerce platforms and unstructured comments from forums, a lot of effort is required to handle data heterogeneity issues.

[0007] 5) Low degree of intelligence, relying on manual intervention: From the analysis of data collection needs to the generation of the final collection script, the entire process relies heavily on manual intervention. In the face of large-scale and diversified collection tasks, the limitations of manual operation become increasingly apparent. For example, an enterprise needs to collect data from hundreds of different types of websites simultaneously for market trend analysis. Manually analyzing the structure of each website and writing collection rules not only takes a very long time, but also makes it difficult to ensure the consistency and accuracy of the rules, and it is impossible to respond to changes in market dynamics in a timely manner.

[0008] With the rise of large language models (LLM), their semantic understanding and reasoning capabilities have opened up new opportunities for crawler technology. However, existing technologies cannot effectively integrate LLM with browser interaction protocols such as MCP, resulting in a failure to fully leverage the capabilities of LLM in dynamic web page analysis and anti-crawling strategy generation. To address the above problems, we propose an intelligent crawler generation method and system based on large language models and MCP protocols. SUMMARY

[0009] The present application aims to address the shortcomings of existing technologies by providing an intelligent crawler generation method and system based on large language models and MCP protocols, which solves the problem of existing technologies failing to effectively integrate LLM with browser interaction protocols, resulting in a failure to fully leverage the capabilities of LLM in dynamic web page analysis and anti-crawling strategy generation.

[0010] The present application is implemented as follows: an intelligent crawler generation method based on large language models and MCP protocols, which includes:

[0011] S10, a user inputs collection requirements through natural language, and a large language model is used to analyze the collection requirements and generate a standardized requirement description document;

[0012] S20, based on the MCP protocol, the interaction between the large language model and the browser is realized, and the browser is driven to complete the loading of target web page content and the capture of dynamic data, and the basic information of the page and the complete DOM tree structure are output;

[0013] S30, load the basic information of the page and the complete DOM tree structure, and use a crawler script generation system to analyze and process the complete DOM tree structure of the page;

[0014] S40, combine the large language model with the multi-dimensional analysis results, and generate adaptive crawler scripts through template-driven generation;

[0015] S50, quality verification and intelligent repair of the generated crawler scripts are performed to form executable customized data collection scripts;

[0016] S60, based on the data collection script crawling multi-source data, preprocessing multi-source data, storing the preprocessed multi-source data to the target database, forming a traceable data processing chain.

[0017] Preferably, the method for analyzing the collection requirement based on the large language model and generating a standardized requirement description document comprises:

[0018] S101, the user interacts with the large language model to determine the collection requirement, and the unstructured collection requirement input by the user is analyzed through natural language processing technology;

[0019] S102, the user requirement is deeply semantically analyzed according to the built-in prompt word template, and the core keyword, data type and collection target structured elements are extracted;

[0020] S103, when the collection requirement input by the user has ambiguous or ambiguous content, a multi-round dialogue process is triggered to intelligently complete the key information;

[0021] S104, based on the requirement parser, the natural language requirement is decomposed into semantic units, the parsed semantic units are mapped to the predefined technical specification template, and the standardized requirement description document containing the website address, data model, constraint condition and collection frequency is automatically generated.

[0022] Preferably, the method for realizing the interaction between the large language model and the browser based on the MCP protocol comprises:

[0023] S201, the MCP initialization module based on the model context protocol (MCP) of Anthropic calls the BrowserTools MCP service to establish a communication channel with the Chrome browser;

[0024] S202, the browser page loading module sends a navigation instruction to the BrowserTools MCP service through the MCP protocol, drives the Chrome browser to open the target website page, and monitors the page loading state;

[0025] S203, after the page is successfully loaded, the context connection establishment module establishes a persistent context connection with the browser page through the MCP protocol, obtains the basic information and complete DOM tree structure of the page, and stores them to the memory, wherein the basic information of the page includes URL, document type and encoding format;

[0026] S204, the JavaScript events in the page are monitored through the MCP protocol, and the dynamically loaded data is captured and analyzed in real time, and the page state in the memory is updated.

[0027] Preferably, the method for analyzing and processing the complete DOM tree structure of the page through the crawler script generation system comprises:

[0028] S301, the DOM tree parsing module is used for hierarchical traversal of the DOM structure, an XPath / CSS selector is generated based on an element tag, a class name, an ID attribute and semantic understanding, and an element positioning strategy is determined;

[0029] S302, a screenshot recognition module is used to obtain a web screenshot and recognize button text and picture-embedded text in combination with an OCR technology, and cross verification is performed with the DOM structure to enhance positioning accuracy;

[0030] S303, an API data parsing module is used for HTTP request and response parsing and processing, a data interface is matched according to a standardized requirement, and semantic association between interface response content and requirement data fields is established;

[0031] S304, an authentication parameter parsing module is used to extract and parse a website encrypted JS file, an encryption algorithm process is restored through a large language model, and executable equivalent encryption / decryption code is generated.

[0032] Preferably, the method for quality checking and intelligent repairing of the generated crawler script comprises:

[0033] S401, a large language model calls a script template based on a large model semantic and a multi-dimensional analysis result, selects an adaptive script template through a template-driven generation method, and automatically generates a crawler script containing data extraction, anti-crawling strategy and exception handling logic based on the adaptive script template;

[0034] S402, syntax legality detection and small-range gray test are performed on the generated crawler script, and script execution results and data integrity indicators are monitored;

[0035] S403, if the crawler script fails the checking, a rule-driven two-level repairing mechanism is used in combination with an LLM to intelligently repair the crawler script, wherein, when the crawler script is intelligently repaired, common errors are automatically repaired based on predefined rules, complex problems are analyzed in depth through a large model to generate a repairing scheme, and when the scheme does not meet the requirements, the analysis result is reanalyzed and the script strategy is adjusted.

[0036] Preferably, the method for preprocessing of multi-source data comprises:

[0037] S501, multi-source data is loaded, and invalid data is removed in combination with a regular expression and a rule engine;

[0038] S502, a large language model is used for semantic standardization processing of the multi-source data, wherein the semantic standardization processing comprises unified time format and fuzzy text analysis;

[0039] S503, heterogeneous data is converted into a unified format through a data mapping technology, and a standardized data set meeting a business requirement is output.

[0040] Preferably, when storing the pre-processed multi-source data to the target database, the structured data is stored to a PostgreSQL, MySQL or MongoDB database, and the data consistency and query performance are guaranteed through database transactions and indexes; the unstructured data is stored to an S3, MinIO or OSS database, and metadata indexes such as file type, creation time and data tag are established.

[0041] In another aspect, the present application also provides an intelligent crawler generation system based on a large language model and an MCP protocol, which comprises:

[0042] A demand description module, through which a user inputs and collects demands in natural language, and the demand description module analyzes the collected demands based on a large language model and generates a standardized demand description document;

[0043] An MCP protocol interaction system, which realizes the interaction between the large language model and the browser based on the MCP protocol, drives the browser to complete target webpage content loading and dynamic data capture, and outputs page basic information and complete DOM tree structure;

[0044] A crawler script generation system, which is used to load page basic information and complete DOM tree structure, and analyzes and processes the complete DOM tree structure of the page through the crawler script generation system;

[0045] A script verification system, which is used to combine the large language model and multi-dimensional analysis results, and generates adaptive crawler scripts through template driving, and performs quality verification and intelligent repair on the generated crawler scripts, forms executable customized data collection scripts, and comprises a script generation module and a script quality verification module;

[0046] A data storage module, which crawls multi-source data based on the data collection scripts, pre-processes the multi-source data, stores the pre-processed multi-source data to a target database, and forms a traceable data processing chain.

[0047] Preferably, the MCP protocol interaction system comprises:

[0048] An MCP protocol initialization module, which calls the BrowserTools MCP service through the model context protocol MCP of Anthropic, establishes a communication channel with the Chrome browser, and

[0049] A browser page loading module, which sends navigation instructions to the BrowserTools MCP service through the MCP protocol, drives the Chrome browser to open a target website page, and monitors the page loading state;

[0050] A context connection establishment module, after the page is successfully loaded, the context connection establishment module establishes a persistent context connection with the browser page through the MCP protocol, obtains the basic information and complete DOM tree structure of the page, and stores them to the memory, monitors the JavaScript events in the page through the MCP protocol, captures and analyzes the dynamically loaded data in real time, and updates the page state in the memory.

[0051] Preferably, the crawler script generation system comprises:

[0052] A DOM tree analysis module, the DOM tree analysis module is used for hierarchical traversal of the DOM structure, generation of XPath / CSS selectors based on element tags, class names, ID attributes and semantic understanding, and determination of element positioning strategies;

[0053] An OCR screenshot recognition module, the OCR screenshot recognition module is used for obtaining a web screenshot and recognizing button text and picture embedded text in combination with OCR technology, and cross-verification with the DOM structure to enhance positioning accuracy;

[0054] An API data analysis module, the API data analysis module is used for HTTP request and response analysis and processing, matching of data interfaces according to standardized requirements, establishment of semantic association between interface response content and requirement data fields, and determination of element positioning strategies;

[0055] An authentication parameter analysis module, the authentication parameter analysis module is used for extraction and analysis of website encrypted JS files, restoration of the encryption algorithm process through a large language model, and generation of executable equivalent encryption / decryption code.

[0056] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0057] In the embodiments of the present application, the complete DOM tree and dynamic data after browser rendering are obtained through the MCP protocol, and a large language model can be called to automatically analyze element positioning strategies, so that the collection script is adaptively generated, and multi-modal data utilization is realized in combination with visual feature analysis, thereby ensuring the efficiency, intelligence and automation of web data collection, solving the core problems of poor dynamic adaptability and weak anti-crawling resistance in existing crawler technologies, and being beneficial to realizing large-scale and automated data collection and crawling.

[0058] The application can adapt to the changing web environment by combining a large language model and an MCP protocol, and can realize end-to-end automation from requirement understanding, script generation to execution monitoring by using the semantic understanding ability of the large language model, significantly improving the intelligent and automated level of web data collection, and having the core advantages of automatic parsing and script generation, large model automatic analysis of DOM structure, JavaScript logic and visual features, etc., which can reduce the development cycle by replacing manual rule writing, and support batch generation of crawler scripts adapted to different website structures, and the time complexity of large-scale tasks is significantly better than that of traditional solutions. On the other hand, the application also has the advantages of dynamic adaptation and low-cost maintenance, which can automatically identify changes in web structure and adjust the collection rules in real time, reducing manual intervention, and the whole process has low labor cost and is relatively simple and efficient, and is suitable for high-frequency iteration of internet data collection scenarios.

[0059] In the embodiment of the application, the script generation is realized based on intelligent analysis of the large model, the large language model automatically analyzes the web DOM structure, JavaScript logic and visual features, thereby generating a crawler script adapted to different websites, and replacing the traditional manual rule writing process, significantly improving the development efficiency and dynamic adaptability, and at the same time, the MCP protocol is used to establish a context connection with the browser page, thereby obtaining a complete DOM tree and dynamic loading data, solving the problem of insufficient collection of dynamic loading data by traditional crawlers. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 is the implementation process schematic diagram of the intelligent crawler generation method based on the large language model and the MCP protocol provided by the application.

[0061] Figure 2 shows the implementation process schematic diagram of the quality verification and intelligent repair method for the generated crawler script.

[0062] Figure 3 is the structure schematic diagram of the intelligent crawler generation system based on the large language model and the MCP protocol provided by the application.

[0063] In the figure: 100 - requirement description module, 200 - MCP protocol interaction system, 210 - MCP protocol initialization module, 220 - browser page loading module, 230 - context connection establishment module, 300 - crawler script generation system, 310 - DOM tree analysis module, 320 - OCR screenshot identification module, 330 - API data analysis module, 340 - authentication parameter analysis module, 400 - script verification system, 410 - script generation module, 420 - script quality verification module, 500 - data storage module. DETAILED DESCRIPTION

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting of the application; the use herein of terms such as "comprise" and "comprising", or "have" and "having", or "include" and "including" and any variations thereof, is intended to cover a non-exclusive inclusion; the use herein of terms such as "first", "second" and the like is intended to distinguish between similar objects, rather than to describe a particular order.

[0065] The prior art cannot effectively integrate the LLM and the browser interaction protocol (such as MCP), resulting in the inability to fully exert the ability of the LLM in the links of dynamic web page analysis, anti-crawling strategy generation, etc. In view of the above problems, the present application provides an intelligent crawler generation method and system based on a large language model and a MCP protocol. In short, when the method is implemented, first, the collection demand is analyzed based on a large language model and a standardized demand description document is generated, the interaction between the large language model and the browser is realized based on the MCP protocol, then the page complete DOM tree structure is analyzed and processed by the crawler script generation system 300, the adaptive crawler script is generated by template driving, the generated crawler script is quality checked and intelligently repaired, the executable customized data collection script is formed, the multi-source data is preprocessed, finally the preprocessed multi-source data is stored in the target database, and the traceable data processing chain is formed. In the embodiment of the present application, the complete DOM tree and dynamic data rendered by the browser are obtained through the MCP protocol, and the large language model can be called to automatically analyze the element positioning strategy, so as to adaptively generate the collection script, and the multi-modal data utilization is realized in combination with the visual feature analysis, thereby ensuring the efficiency, intelligence and automation of web page data collection, solving the core problems of poor dynamic adaptability and weak anti-crawling ability in the existing crawler technology, and being beneficial to realizing large-scale and automated data collection and crawling.

[0066] It should be noted that the related terms and definitions in the present embodiment are as follows:

[0067] Large language model: Large Language Models (LLM) is a language model constructed by a deep neural network containing more than 100 billion parameters.

[0068] MCP protocol: Model Context Protocol (MCP) is a new standard open-sourced by Anthropic on November 25, 2024. MCP is an open standard designed to connect AI assistants with the systems where data resides, including content repositories, business tools, and development environments. Its goal is to help frontier models produce better, more relevant responses. MCP can be seen as the "USB-C port" for AI applications. Just as USB-C provides a standardized way to connect devices to various peripherals, MCP provides a standardized method for AI models to connect to different data sources and tools.

[0069] DOM tree: DOM is a standard method for accessing and manipulating HTML documents, which expresses HTML documents as a tree structure.

[0070] OCR technology: OCR stands for Optical Character Recognition, which is a technology that uses optical and computer techniques to read and convert printed or handwritten text on paper into a format that computers can accept and humans can understand.

[0071] Browser interface: In the browser's network module, you can see the relevant requests, including interface, js, css, image, media, text, etc. Interface information includes name, method, status, time, etc.

[0072] Gray test: Gray test refers to the process of verifying the stability and compatibility of a new version of a software product before it is officially released. This helps to reduce the risks associated with the full-scale launch of the new version.

[0073] BrowserTools: BrowserTools MCP is a browser plugin + local service combination kit based on MCP (Model Context Protocol) protocol. Its functions include browser monitoring and interaction toolkit, screenshot, log analysis, and network activity tracking. Its use is to visually debug web applications through Chrome extensions.

[0074] Example 1

[0075] The embodiments of the present application provide a method for generating an intelligent crawler based on a large language model and an MCP protocol, Figure 1 The method for generating an intelligent crawler based on a large language model and an MCP protocol is shown in the flowchart. The method specifically includes:

[0076] S10, the user collects requirements through natural language input, and a large language model is used to analyze the collected requirements and generate a standardized requirement description document;

[0077] S20, the large language model and the browser are interacted based on the MCP protocol, the browser is driven to complete target webpage content loading and dynamic data capture, and page basic information and a complete DOM tree structure are output;

[0078] S30, the page basic information and the complete DOM tree structure are loaded, and the complete DOM tree structure of the page is analyzed and processed by a crawler script generation system 300;

[0079] S40, the large language model and multi-dimensional analysis results are combined, and adaptive crawler scripts are generated through template driving;

[0080] S50, the generated crawler scripts are quality checked and intelligently repaired to form executable customized data collection scripts;

[0081] S60, multi-source data is crawled based on the data collection scripts, the multi-source data is preprocessed, the preprocessed multi-source data is stored in a target database, and a traceable data processing chain is formed.

[0082] In the embodiment of the application, the complete DOM tree and dynamic data rendered by the browser are obtained through the MCP protocol, and the large language model can be called to automatically analyze element positioning strategies, so that the collection scripts are adaptively generated, and multi-modal data utilization is realized by combining visual feature analysis, thereby ensuring the efficiency, intelligence and automation of webpage data collection, solving the core problems of poor dynamic adaptability and weak anti-crawling ability in existing crawler technology, and facilitating large-scale and automated data collection and crawling.

[0083] The embodiment of the application provides a method for analyzing collected requirements by a large language model and generating a standardized requirement description document, which specifically comprises:

[0084] S101, a user interacts with a large language model to determine collection requirements, and a natural language processing technology is used to analyze unstructured collection requirements input by the user;

[0085] It should be noted that the large language model can be DeepSeek or Claude 3.7 model, and the user requirement acquisition means that the user can submit requirements in colloquial or professional expressions through a text box, such as inputting "collect the information of certain instruments related to certain e-commerce in the past week and the release time". By calling the large language model, the semantic understanding of the mixed language input can be processed based on the natural language processing technology.

[0086] S102, according to the built-in prompt word template, the user demand is deeply semantically analyzed, and the core keyword, data type and collected structured element are extracted;

[0087] S103, the reasoning ability of the large language model is combined to realize fuzzy demand completion, when the user input collection demand has fuzzy or ambiguous content, a multi-round dialogue process is triggered, and the key information is intelligently completed; for example, confirming the specific start and end date of the "last week", the specific category range of the instrument, extracting the key entity and constraint condition from the natural language analysis result, reducing the demand understanding deviation through the large model reasoning, so that non-technical users can define complex collection requirements without writing code.

[0088] S104, based on the demand parser, the natural language demand is decomposed into semantic units, and the parsed semantic units are mapped to the predefined technical specification template, and a standardized demand description document containing website address, data model, constraint condition and collection frequency is automatically generated.

[0089] In the embodiment, the user demand is standardized described, the target is to convert the natural language demand of the user into executable technical specification, based on the demand parser module, the natural language demand is decomposed into semantic units, the parsed semantic units are mapped to the predefined technical specification template, and a standardized demand description document containing website address, data model, constraint condition and collection frequency is automatically generated. Therefore, accurate conversion from natural language demand to structured instruction is realized, and clear basis is provided for subsequent crawler script generation and data collection.

[0090] The embodiment of the application provides an interaction method between a large language model and a browser based on an MCP protocol.

[0091] S201, the MCP protocol initialization module 210 calls the BrowserTools MCP service through the model context protocol MCP of Anthropic, establishes a communication channel with the Chrome browser, and establishes a communication channel with the Chrome browser.

[0092] It should be noted that the MCP protocol initialization module 210 is a large language model that calls the BrowserTools MCP service through the model context protocol MCP of Anthropic, and establishes a communication channel with the Chrome browser, so as to ensure reliable transmission of instructions and data.

[0093] S202, the browser page loading module 220 sends a navigation instruction to the BrowserTools MCP service through the MCP protocol, drives the Chrome browser to open the target website page, and monitors the page loading state.

[0094] S203, after the page is successfully loaded, the context connection establishment module 230 establishes a persistent context connection with the browser page through the MCP protocol, obtains the basic information of the page and the complete DOM tree structure, and stores them to the memory, wherein the basic information of the page includes URL, document type, and encoding format;

[0095] S204, the JavaScript events in the page are monitored through the MCP protocol, and the dynamically loaded data is captured and parsed in real time, and the page state in the memory is updated, and it should be noted that the JavaScript events in the page include but are not limited to AJAX request and WebSocket communication, in the embodiment, the dynamically loaded data is captured and parsed in real time, and the page state in the memory is updated.

[0096] In the embodiment of the application, the script generation is based on the intelligent analysis of the large model driven, the web page DOM structure, JavaScript logic and visual features are automatically analyzed by the large language model, so as to generate a crawler script adapted to different websites, and then replace the traditional manual rule writing process, significantly improve the development efficiency and dynamic adaptability, and at the same time, the MCP protocol is used to establish a context connection with the browser page, so as to obtain the complete DOM tree and the dynamically loaded data, and solve the problem of insufficient data collection of the traditional crawler for dynamically loaded data.

[0097] The embodiment of the application provides a method for analyzing and processing a complete DOM tree structure of a page by a crawler script generation system 300, and the method specifically comprises the following steps:

[0098] S301, based on the DOM tree analysis module 310, the DOM structure is hierarchically traversed, the XPath / CSS selector is generated based on the element tag, class name, ID attribute and semantic understanding, and the element positioning strategy is determined;

[0099] It should be noted that the DOM tree analysis module 310 is used to hierarchically traverse the obtained complete DOM structure, identify the HTML element node, generate XPath, CSS selector according to the element tag name, class name, ID and other attributes, and analyze the semantic label by combining the semantic understanding ability of the large model, so as to determine the element positioning strategy.

[0100] S302, the OCR screenshot recognition module 320 is used to obtain a web screenshot and combine OCR technology to recognize button text and picture embedded text, and cross-verify with the DOM structure to enhance the positioning accuracy;

[0101] In the embodiment of the application, the OCR screenshot recognition module 320 is configured to acquire a webpage screenshot, recognize visual text such as button text and picture-embedded text by combining with an OCR technology, cross-verify the visual text with a DOM structure, and enhance the element positioning accuracy.

[0102] In S303, the API data analysis module 330 performs analysis and processing on HTTP requests and responses, matches data interfaces according to standardized requirements, establishes semantic association between interface response content and requirement data fields, and the like.

[0103] The API data analysis module 330 is configured to acquire HTTP request and response information, determine data source interfaces according to standardized requirements, perform semantic matching between recognized interface response content and data fields in the standardized requirements, and establish an association relationship between API interfaces and data requirements.

[0104] In S304, the authentication parameter analysis module 340 extracts and analyzes a website encrypted JS file, restores an encryption algorithm process by a large language model, and generates executable equivalent encryption / decryption code.

[0105] In the embodiment, the authentication parameter analysis module 340 is a core component for coping with complex encryption mechanisms of websites. The module restores an encryption algorithm process by analyzing JS code through a large model, extracts key encryption functions by analyzing JS code structures from encrypted JS files extracted from a browser environment, understands encryption logic by using a LLM, simulates encryption and decryption processes, generates equivalent implementations under the condition that algorithm logic is unchanged, returns executable code, and verifies the correctness of the restored algorithm. The module can be used to automatically identify changes in website encryption mechanisms and update the restored algorithm.

[0106] In the embodiment of the application, when the crawler script generation system 300 performs analysis and processing on a complete DOM tree structure of a page, the system combines semantic understanding of a large model with an OCR technology, cross-verifies DOM structures and visual clues (such as image-text association and button text), and further improves element positioning accuracy and multi-modal data extraction capability.

[0107] The embodiment of the application provides a quality checking and intelligent repairing method for the generated crawler script, Figure 2 A flowchart of a quality checking and intelligent repairing method for the generated crawler script is shown, and the quality checking and intelligent repairing method for the generated crawler script specifically includes the following steps.

[0108] In S401, a large language model calls a script template based on semantic understanding of a large model and multi-dimensional analysis results, selects an adaptive script template by a template-driven generation method, and automatically generates a crawler script containing data extraction, anti-crawling strategies, and abnormality handling logic based on the adaptive script template.

[0109] S402, syntax legality detection, small-scale gray test, monitoring script execution results, data integrity indicators are performed on the generated crawler script;

[0110] S403, if the crawler script fails the verification, a two-level repair mechanism assisted by rule driving and LLM is used for intelligent repair of the crawler script, wherein, when the crawler script is intelligently repaired, for common errors, the pre-defined rules are used for automatic repair, and for complex problems, a repair scheme is generated through deep analysis of the large model, and when the indicators do not meet the requirements, the analysis result is re-analyzed and the script strategy is adjusted.

[0111] In the embodiment, the quality verification and intelligent repair of the generated crawler script are implemented by the script generation module 410 and the script quality verification module 420, wherein the script generation module 410 automatically adapts the data structure of different websites by using the semantic understanding ability of the large language model and the DOM structure or API interface, and automatically selects a script template. If multiple levels of nested tags are detected, a script containing recursive analysis logic is generated; if it is a simple list page, a loop structure for batch extraction is generated. A customized crawling script with strong adaptability is generated, covering data extraction, dynamic interaction and anti-crawling scenarios, wherein when the anti-crawling scenario is executed, an intelligent anti-crawling strategy is generated: the website anti-crawling mechanism (such as parameter encryption, verification code) is analyzed by the large voice model, the encryption algorithm is automatically inferred and restored, the request strategy (such as random UA, IP pool rotation) is dynamically adjusted, and the success rate of collection is improved.

[0112] The script quality verification module 420 is used for multi-level quality verification of the generated script. When the multi-level quality verification of the generated script is performed, a syntax parser corresponding to the language (such as the ast module of Python) is first called for syntax legality detection, and then a small range (such as 1% of the target URL) is used for usability gray test, monitoring script execution results, data integrity and other indicators. If the verification finds problems, it is determined that the crawler script fails the verification, and a two-level repair mechanism assisted by rule driving and LLM is used for intelligent repair, that is, for common error types, pre-defined repair rules are used for automatic repair, and for complex problems, a large model is called to perform deep analysis and generate a targeted repair scheme. If the indicators still do not meet the requirements, the large model is triggered to re-analyze the analysis result and automatically adjust the script generation strategy, forming a closed-loop system from script generation to quality verification and then to intelligent repair, which significantly improves the reliability and development efficiency of the crawler script.

[0113] In the embodiment of the application, the method for preprocessing multi-source data comprises:

[0114] S501, load multi-source data, remove invalid data by combining regular expressions and rule engines;

[0115] In S502, the multi-source data is processed by a large language model for semantic standardization, wherein the semantic standardization includes unifying time formats and resolving ambiguous texts.

[0116] In S503, the heterogeneous data is converted into a unified format by a data mapping technology, and a standardized data set meeting business requirements is output, thereby providing a high-quality data basis for subsequent analysis and application.

[0117] In the embodiment of the application, when the preprocessed multi-source data is stored in the target database, the structured data is stored in a PostgreSQL, MySQL or MongoDB database, and the database transaction and index are used to ensure data consistency and query performance; the unstructured data is stored in an S3, MinIO or OSS database, and the metadata index of file type, creation time and data label is established.

[0118] It should be noted that for structured data, the embodiment of the application supports storage in a relational database such as PostgreSQL, MySQL or a document database such as MongoDB, and the database transaction and index mechanism are used to ensure data consistency and query performance; for unstructured data, object storage such as S3, MinIO or OSS database is used for storage, and the metadata index system is used to establish index information such as file type, creation time and data label, thereby improving retrieval efficiency. In terms of data update, incremental storage is realized based on timestamp or ID field, and hash check and version comparison are used to avoid repeated writing; at the same time, the system records processing information such as data source URL, crawling time, script version number and cleaning rules, thereby forming a traceable data processing chain, providing complete basis for data audit, problem troubleshooting and version backtracking, and ensuring the standardization and reliability of data management.

[0119] Embodiment 2

[0120] The embodiment of the application provides an intelligent crawler generation method based on a large language model and an MCP protocol, and specifically, the embodiment of the application takes the collection of bidding information of three platforms as an example, and assumes that the three platforms are a resource trading platform (A province), a resource trading platform (B province) and a resource trading platform (C province). Specifically, when generating a standardized bidding project data set, the intelligent crawler generation method based on the large language model and the MCP protocol includes the following steps.

[0121] In S10, a user inputs a collection requirement through natural language, and a large language model is used to analyze the collection requirement and generate a standardized requirement description document.

[0122] For example, the user inputs the following information: "collect the bidding project information of medical devices on the above-mentioned resource trading platforms within the last 3 months, including project name, bidding unit, bidding time, budget amount and attachments".

[0123] And based on the large language model to parse the collection requirements and generate standardized requirement description documents, the large language model (LLM) requirement analysis: the large language model deeply analyzes the user input requirements. Identify key information such as time range (3 months), project type (medical device), and data fields to be collected (project name, bidding unit, bid opening time, budget amount, attachments). For requirements involving three platforms, determine to perform subsequent collection operations on the three platforms respectively.

[0124] When generating a standardized requirement description document: integrate the requirements parsed by the large language model into a standardized document. Clearly define the requirements for filtering bid projects from three resource trading platforms according to specific time and project type, and standardize the requirements for obtaining specified data fields. At the same time, make clear provisions for the format of data (such as the date format of the bid opening time) and precision.

[0125] S20, based on the MCP protocol, the interaction between the large language model and the browser is realized, the browser is driven to complete the target webpage content loading and dynamic data capture, and the page basic information and complete DOM tree structure are output;

[0126] In the embodiment of the application, when the large language model and the browser interact based on the MCP protocol, for a resource trading platform (A province), the browser is started to access the platform website through MCP protocol and browser interaction. Monitor the state when the page is loaded, establish a context connection, obtain the page basic information and complete DOM tree structure, monitor the JavaScript event, capture the dynamic loading data in real time, and in the same way, MCP protocol browser interaction operation is carried out on a resource trading platform (B province) and a resource trading platform (C province) respectively, and the connection with the corresponding platform page is established and the related information is obtained.

[0127] S30, load the page basic information and complete DOM tree structure, and the crawler script generation system 300 analyzes and processes the complete DOM tree structure of the page;

[0128] Wherein, when the crawler script generation system 300 analyzes and processes the complete DOM tree structure of the page, the DOM / API data is parsed, that is, the DOM tree obtained by the resource trading platform (A province) is traversed in layers, and the XPath and CSS selector are generated to locate the bid project information area. Analyze the HTTP request, find the related API interface (if any), match the interface response with the requirement field semantics, determine the information source, and in the same way, carry out DOM / API data analysis on the resource trading platform (B province) and the resource trading platform (C province) respectively, and determine the data extraction method according to the page structure and data transmission characteristics of each platform.

[0129] S40, combined with the large language model and the multi-dimensional analysis result, the adaptive crawler script is generated through the template driving;

[0130] When the adaptive crawler script is generated through the template driving, according to the data analysis result of a resource trading platform (A province), the adaptive crawler script is generated considering the page structure characteristics, the project name, the bidding unit and other information are extracted from the platform, and the corresponding crawler scripts are generated according to the respective data analysis for a resource trading platform (B province) and a resource trading platform (C province), and the scripts cover the logic of extracting target information from different structure pages.

[0131] S50, quality checking and intelligent repairing are performed on the generated crawler script to form an executable customized data collection script;

[0132] In the quality checking and intelligent repairing of the generated crawler script, the crawler script generated from the resource trading platform (A province) is first subjected to syntax detection and then small range test. If there is a problem, the repairing is performed according to the rule driving combined with the LLM auxiliary mechanism, if the index is still not up to standard, the script generation strategy is reanalyzed and adjusted, and the quality checking and repairing optimization are performed on the crawler scripts generated from the resource trading platform (B province) and the resource trading platform (C province) respectively according to the same process.

[0133] S60, based on the data collection script, multi-source data is crawled, the multi-source data is preprocessed, the preprocessed multi-source data is stored in a target database, and a traceable data processing chain is formed.

[0134] In the present embodiment, when the multi-source data is preprocessed, the data collected from the three platforms is integrated. The large language model is used to unify the data semantics, such as standardizing the bid opening time format; the regular expression and the rule engine are used to remove invalid data; the structured and unstructured data difference processing is performed according to different platform structures, the data is mapped to unify the format, and the standardized bidding project data set is formed. When the preprocessed multi-source data is stored in the target database, the cleaned and preprocessed bidding project data can be stored in the database. Considering the data structure characteristics, a relational database (such as MySQL database) is selected for storage. With the help of database transaction, the data consistency is guaranteed, and the index is built for the commonly used query field to improve the query performance.

[0135] Embodiment 3

[0136] The embodiment of the present application provides an intelligent crawler generation system based on a large language model and an MCP protocol, Figure 3 The structure schematic diagram of the intelligent crawler generation system based on the large language model and the MCP protocol is shown, and the intelligent crawler generation system based on the large language model and the MCP protocol specifically comprises:

[0137] The demand description module 100 collects demands through natural language input, and the demand description module 100 analyzes the collected demands based on a large language model and generates a standardized demand description document;

[0138] The MCP protocol interaction system 200 implements the interaction between the large language model and the browser based on the MCP protocol, drives the browser to complete target webpage content loading and dynamic data capture, and outputs page basic information and complete DOM tree structure;

[0139] The MCP protocol interaction system 200 includes:

[0140] The MCP protocol initialization module 210 calls the BrowserTools MCP service through the Anthropic model context protocol MCP, establishes a communication channel with the Chrome browser, and drives the Chrome browser to open the target website page.

[0141] The browser page loading module 220 sends navigation instructions to the BrowserTools MCP service through the MCP protocol, drives the Chrome browser to open the target website page, and monitors the page loading state.

[0142] The context connection establishment module 230 establishes a persistent context connection with the browser page through the MCP protocol after the page is successfully loaded, obtains the basic information and complete DOM tree structure of the page, and stores them in the memory. The MCP protocol is used to monitor JavaScript events in the page, capture and analyze dynamically loaded data in real time, and update the page state in the memory.

[0143] The crawler script generation system 300 is used to load page basic information and complete DOM tree structure, and the crawler script generation system 300 is used to analyze and process the complete DOM tree structure of the page.

[0144] It should be noted that the crawler script generation system 300 includes:

[0145] The DOM tree analysis module 310 is used to hierarchically traverse the DOM structure, generate XPath / CSS selectors based on element tags, class names, ID attributes and semantic understanding, and determine element positioning strategies.

[0146] The OCR screenshot recognition module 320 is used to obtain a web screenshot and recognize button text and picture embedded text through OCR technology, and cross-verify with the DOM structure to enhance positioning accuracy.

[0147] The API data analysis module 330 is configured to analyze and process HTTP requests and responses, match data interfaces according to standardized requirements, and establish semantic association between interface response content and requirement data fields.

[0148] The authentication parameter analysis module 340 is configured to extract and analyze website encrypted JS files, restore the encryption algorithm process through a large language model, and generate executable encryption / decryption code.

[0149] The script verification system 400 is configured to combine a large language model and multi-dimensional analysis results, generate adaptive crawler scripts through a template, perform quality verification and intelligent repair on the generated crawler scripts, form executable customized data collection scripts, and the script verification system 400 comprises a script generation module 410 and a script quality verification module 420.

[0150] The data storage module 500 is configured to crawl multi-source data based on the data collection script, pre-process the multi-source data, store the pre-processed multi-source data into a target database, and form a traceable data processing chain.

[0151] In summary, the present application provides an intelligent crawler generation method and system based on a large language model and an MCP protocol.

[0152] The present application combines a large language model and an MCP protocol, uses the semantic understanding ability of the large language model to realize end-to-end automation from requirement understanding, script generation to execution monitoring, can adapt to changing web environments, significantly improves the intelligence and automation level of web data collection, and has the core advantages of automatic analysis of DOM structure, JavaScript logic and visual features, etc.

[0153] It should be noted that, for the foregoing embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the order of the described actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily necessary for the present application.

[0154] The above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the protection scope of the application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on these embodiments, all other embodiments obtained by those of ordinary skill in the art without creative labor belong to the scope of protection of the present application. Although the present application has been described in detail with reference to the above embodiments, those of ordinary skill in the art can still combine, add or delete or make other adjustments to the features of the embodiments of the present application according to the circumstances without conflict and without creative labor, so as to obtain different other technical solutions which do not deviate from the concept of the present application in essence. These technical solutions also belong to the scope of protection of the present application.

Claims

1. A method for generating an intelligent crawler based on a large language model and an MCP protocol, characterized in that, The method comprises: S10, a user collects requirements through natural language input, and a large language model is used to analyze the collected requirements and generate a standardized requirement description document; S20, a large language model is interacted with a browser based on an MCP protocol, the browser is driven to complete target webpage content loading and dynamic data capture, and page basic information and a complete DOM tree structure are output; S30, the page basic information and the complete DOM tree structure are loaded, and a page complete DOM tree structure is analyzed and processed by a crawler script generation system; S40, a suitable crawler script is generated by template driving based on a large language model and multi-dimensional analysis results; S50, the generated crawler script is quality checked and intelligently repaired to form an executable customized data collection script; S60, multi-source data is crawled based on the data collection script, the multi-source data is preprocessed, the preprocessed multi-source data is stored in a target database, and a traceable data processing chain is formed.

2. The method of claim 1, wherein the method further comprises: The method for analyzing the collected requirements based on the large language model and generating the standardized requirement description document comprises: S101, a user interacts with a large language model to determine collection requirements, and a non-structured collection requirement input by the user is analyzed through natural language processing technology; S102, a user requirement is deeply semantically analyzed according to a built-in prompt word template, core keywords, data types and collection target structured elements are extracted; S103, when the collection requirement input by the user contains ambiguous or ambiguous content, a multi-round dialogue process is triggered, and key information is intelligently completed; S104, a natural language requirement is decomposed into semantic units based on a requirement analyzer, the analyzed semantic units are mapped to a predefined technical specification template, and a standardized requirement description document containing a website address, a data model, a constraint condition and a collection frequency is automatically generated.

3. The method of claim 1, wherein the method further comprises: The method for realizing interaction between a large language model and a browser based on an MCP protocol comprises: S201, an MCP protocol initialization module calls a BrowserTools MCP service through an Anthropic model context protocol MCP, a communication channel with a Chrome browser is established, and a context connection establishment module is triggered; S202, a browser page loading module sends a navigation instruction to the BrowserTools MCP service through the MCP protocol, drives the Chrome browser to open a target website page, and monitors a page loading state; S203, after the page is successfully loaded, the context connection establishment module establishes a persistent context connection with the browser page through the MCP protocol, obtains page basic information and a complete DOM tree structure, and stores the information in a memory, wherein the page basic information includes a URL, a document type and an encoding format; S204, a JavaScript event in the page is monitored through the MCP protocol, dynamically loaded data is captured and analyzed in real time, and a page state in the memory is updated.

4. The method of claim 3, wherein the method further comprises: The method for analyzing and processing a complete DOM tree structure of a page by a crawler script generation system comprises: S301, based on the DOM tree parsing module, the DOM structure is hierarchically traversed, and an XPath / CSS selector is generated based on element tags, class names, ID attributes and semantic understanding to determine the element positioning strategy; S302, the OCR screenshot recognition module is used to obtain a web screenshot and combine with the OCR technology to recognize button text and picture embedded text, and cross verification is performed with the DOM structure to enhance the positioning accuracy; S303, the API data parsing module parses and processes HTTP requests and responses, matches data interfaces according to standardized requirements, and establishes semantic association between interface response content and requirement data fields; S304, the authentication parameter parsing module extracts and parses the website encrypted JS file, restores the encryption algorithm process through the large language model, and generates executable equivalent encryption / decryption code.

5. The method of claim 4, wherein the method further comprises: The quality verification and intelligent repair method for the generated crawler script includes: S401, the large language model calls a script template based on the semantic and multi-dimensional analysis results of the large model, selects an adaptive script template through template-driven generation, and automatically generates a crawler script containing data extraction, anti-crawling strategy and exception handling logic based on the adaptive script template; S402, the generated crawler script is subjected to syntax legality detection and small-scale gray test, and the script execution result and data integrity index are monitored; S403, if the crawler script fails the verification, a two-level repair mechanism is used to intelligently repair the crawler script, wherein, for common errors, the pre-defined rules are used for automatic repair, for complex problems, the large model is used for deep analysis to generate a repair scheme, and if the repair scheme does not meet the requirements, the analysis results are re-analyzed and the script strategy is adjusted.

6. The method of claim 1, wherein the method further comprises: The method for preprocessing multi-source data includes: S501, load multi-source data, and remove invalid data by combining regular expressions and rule engines; S502, use a large language model to perform semantic standardization processing on the multi-source data, wherein the semantic standardization processing includes unifying time formats and parsing ambiguous text; S503, convert heterogeneous data into a unified format through data mapping technology, and output a standardized data set that meets business requirements.

7. The method of claim 6, wherein the method further comprises: When the preprocessed multi-source data is stored in a target database, structured data is stored in a PostgreSQL, MySQL or MongoDB database, and data consistency and query performance are guaranteed through database transactions and indexes; unstructured data is stored in an S3, MinIO or OSS database, and metadata indexes such as file type, creation time and data tag are established.

8. A system for generating an intelligent crawler based on a large language model and an MCP protocol, for implementing the method for generating an intelligent crawler based on a large language model and an MCP protocol according to any one of claims 1-7, characterized in that: The intelligent crawler generation system based on the large language model and the MCP protocol includes: A requirement description module, a user inputs a collection requirement through natural language, and the requirement description module analyzes the collection requirement based on the large language model and generates a standardized requirement description document; An MCP protocol interaction system, which realizes the interaction between the large language model and the browser based on the MCP protocol, drives the browser to complete the target web content loading and dynamic data capture, and outputs the page basic information and complete DOM tree structure; The crawler script generation system is used to load basic information and complete DOM tree structure of a page, and the crawler script generation system is used for parsing and processing the complete DOM tree structure of the page; The script verification system is used to combine a large language model and multi-dimensional analysis results, generate adaptive crawler scripts through a template, verify the quality of the generated crawler scripts and intelligently repair them, form executable customized data collection scripts, and the script verification system includes a script generation module and a script quality verification module; The data storage module is based on data collection scripts to crawl multi-source data, pre-process the multi-source data, store the pre-processed multi-source data to a target database, and form a traceable data processing chain.

9. The intelligent crawler generation system based on a large language model and an MCP protocol according to claim 8, wherein: The MCP protocol interaction system includes: The MCP protocol initialization module calls the BrowserTools MCP service through the Anthropic model context protocol (MCP) to establish a communication channel with the Chrome browser. The browser page loading module sends a navigation instruction to the BrowserTools MCP service through the MCP protocol to drive the Chrome browser to open a target website page and monitor the page loading status. The context connection establishment module establishes a persistent context connection with the browser page through the MCP protocol after the page is successfully loaded, obtains basic information and complete DOM tree structure of the page, and stores them in the memory.

10. The intelligent crawler generation system based on a large language model and an MCP protocol according to claim 9, wherein: The crawler script generation system includes: The DOM tree analysis module is used to traverse the DOM structure layer by layer, generate XPath / CSS selectors based on element tags, class names, ID attributes and semantic understanding, and determine element positioning strategies. The OCR screenshot recognition module is used to obtain a web screenshot and recognize button text and picture embedded text through OCR technology, and cross-verify with the DOM structure to enhance positioning accuracy. The API data analysis module is used to analyze and process HTTP requests and responses, match data interfaces according to standardized requirements, and establish semantic association between interface response content and requirement data fields. The authentication parameter analysis module is used to extract and analyze encrypted JS files of a website, restore the encryption algorithm process through a large language model, and generate executable equivalent encryption / decryption code.

Images