Crawler code generation method and system based on Chrome MCP server

Abstract

The invention relates to the field of crawler development, and relates to a crawler code generation method and system based on a Chrome MCP server, and the method comprises the steps: determining target website information and target data description information; generating a browser control instruction based on the target website information, and sending the browser control instruction to a Chrome MCP server so as to drive the Chrome MCP server to control a browser to access the target website; receiving page interaction data returned by the Chrome MCP server, wherein the page interaction data comprises a network request log generated in the process of accessing the target website by the browser; positioning a target network request containing the target data from a plurality of network requests contained in the network request log; and extracting request parameter information of the target network request, and generating a crawler code for the target website. By adopting the method, the technical threshold and the operation complexity of crawler development can be effectively reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of crawler development. More particularly, the present application relates to a crawler code generation method and system based on a Chrome MCP server. BACKGROUND

[0002] With the rapid development of Internet big data, web crawler technology has become an important means to obtain public data resources. In the existing crawler development process (commonly known as "traditional crawler development"), developers usually need to go through tedious manual operation steps.

[0003] Specifically, the developer first needs to manually analyze the packet capture through the browser's Developer Tools, and then needs to check and filter each network request to locate the application program interface (API) containing the target data or analyze the HTML structure of the webpage. Subsequently, the developer needs to manually write code (such as using the requests library or BeautifulSoup library of Python) to simulate network requests or parse pages according to the results of manual analysis.

[0004] This traditional development method has significant technical defects: High technical threshold: requires developers to have a good command of HTTP protocol, anti-crawler strategy, DOM structure analysis, etc.

[0005] Low development efficiency and high repeatability: the manual packet capture and manual coding process is time-consuming and laborious, especially when facing complex dynamic web pages, the analysis process often takes several hours or even several days.

[0006] High maintenance cost: once the structure or interface of the target website changes, the developer must repeat the manual packet capture and code modification process, which is heavily dependent on personal experience.

[0007] Therefore, how to reduce the technical threshold of crawler development and realize the automation from requirement understanding to code generation is a technical problem to be solved. SUMMARY

[0008] To solve the above technical problems of how to reduce the technical threshold of crawler development and realize the automation from requirement understanding to code generation, the present application provides solutions in the following aspects.

[0009] In a first aspect, the present application provides a crawler code generation method based on a Chrome MCP server, comprising: obtaining a natural language requirement input by a user, performing semantic analysis on the natural language requirement, and determining target website information and target data description information; generate a browser control instruction based on the target website information, and send the browser control instruction to a Chrome MCP server to drive the Chrome MCP server to control the browser to access the target website; receive page interaction data returned by the Chrome MCP server, the page interaction data including network request logs generated by the browser in the process of accessing the target website; analyze the network request logs according to the target data description information, and locate a target network request containing the target data from a plurality of network requests contained in the network request logs; extract request parameter information of the target network request, and generate a crawler code for the target website according to the request parameter information.

[0010] Preferably, the generating a browser control instruction based on the target website information, and sending the browser control instruction to a Chrome MCP server includes: constructing a standard MCP protocol instruction containing a navigation action; establishing an MCP connection with the Chrome MCP server, and sending the standard MCP protocol instruction through the MCP connection to instruct the Chrome MCP server to load the target website in a real browser environment and perform a page rendering operation.

[0011] Preferably, the performing a page rendering operation includes: generating an interaction instruction simulating user behavior, the interaction instruction including a page scrolling instruction or a clicking instruction; sending the interaction instruction through the MCP connection to trigger the browser to load dynamic loading data in the target website; wherein the network request logs contain asynchronous network request records corresponding to the dynamic loading data.

[0012] Preferably, the locating a target network request containing the target data from a plurality of network requests contained in the network request logs includes: traversing the plurality of network requests in the network request logs to obtain response data corresponding to each network request; calculating semantic similarity between content of the response data and the target data description information; determining a network request corresponding to response data with the highest semantic similarity as the target network request.

[0013] Preferably, the extracting the request parameter information of the target network request comprises: parsing the target network request to obtain a uniform resource locator (URL), request headers (Headers), a request body (Payload), and cookies information used to maintain a session state; and combining the obtained URL, Headers, Payload, and cookies information into the request parameter information.

[0014] Preferably, the generating the crawler code for the target website based on the request parameter information comprises: obtaining a preset crawler code template, the crawler code template being constructed based on a network request library of a predetermined programming language; and filling the request parameter information into a corresponding variable position of the crawler code template to generate executable script code, wherein the script code is configured to reproduce the target network request in a browser-independent environment to obtain the target data.

[0015] Preferably, the method further comprises: determining whether response data of the target network request is encrypted data; if the response data is encrypted data, performing parsing based on a page document object model (DOM) structure returned by the Chrome MCP server; generating page element extraction code based on a DOM parsing rule, and taking the page element extraction code as the crawler code.

[0016] Preferably, after the crawler code for the target website is generated, the method further comprises: executing the crawler code in a local sandbox environment to obtain an execution result; verifying whether the execution result contains data content conforming to the target data description information; and if the verification fails, adjusting a positioning strategy or a code generation parameter of the target network request, and regenerating the crawler code.

[0017] Preferably, the method further comprises: in response to a website structure change signal of the target website, retriggering the step of generating the browser control instruction based on the target website information; and re-capturing an updated network request log through the Chrome MCP server, and generating updated crawler code.

[0018] In a second aspect, the present application provides a crawler code generation system based on a Chrome MCP server, comprising: a processor and a memory, the memory storing computer executable instructions; wherein the processor executes the computer executable instructions to implement the crawler code generation method based on the Chrome MCP server.

[0019] The beneficial effects of this invention are as follows: The method in this embodiment receives users' natural language requests and automatically converts them into crawler code, enabling users without a strong programming background or knowledge of network protocols to quickly obtain the required data, thus democratizing crawler development and effectively reducing the technical threshold and operational complexity of crawler development. Furthermore, by introducing a Chrome MCP server to control the browser, this invention can automatically capture network request logs and use intelligent algorithms to automatically locate target data requests, replacing the tedious manual packet capture and filtering process in traditional development, significantly improving development efficiency. Moreover, analysis based on interaction data from a real browser environment (Chrome MCP server) ensures that the obtained network request parameters (such as dynamically generated tokens or cookies) are authentic and valid, avoiding the problem that static analysis struggles to cope with complex anti-crawling strategies. Attached Figure Description

[0020] Figure 1 This is a schematic flowchart illustrating a web crawler code generation method based on a Chrome MCP server according to an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the structure of a web crawler code generation system based on a Chrome MCP server according to an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0023] Example of a method for generating web crawler code based on Chrome MCP server: like Figure 1 As shown, the web crawler code generation method based on the Chrome MCP server of the present invention includes: S101. Determine the target URL information and target data description information, specifically: obtain the natural language requirements input by the user, perform semantic parsing on the natural language requirements, and determine the target URL information and target data description information; The target data description information is a structured description of "what fields you want to crawl" extracted after semantic parsing of the natural language requirements input by the user.

[0024] S102. Driving the Chrome MCP server to control the browser to access the target URL, specifically: generating browser control instructions based on the target URL information, and sending the browser control instructions to the Chrome MCP server to drive the Chrome MCP server to control the browser to access the target URL; S103. Obtain network request logs, specifically: receive page interaction data returned by the Chrome MCP server, the page interaction data including network request logs generated by the browser during the access to the target URL; Typically, network request logs include XHR / Fetch requests, detailed parameters, and response content. Detailed parameters include the URL of each request, Request Headers, Request Body, and Cookies. Response content includes the actual data returned by the server (usually structured data in JSON format, such as a dictionary of product prices).

[0025] S104. Locate the target network request containing the target data from the multiple network requests contained in the network request log, specifically: analyze the network request log according to the target data description information, and locate the target network request containing the target data from the multiple network requests contained in the network request log. In this embodiment, locating the target network request containing the target data from the multiple network requests contained in the network request log includes: traversing the multiple network requests in the network request log and obtaining response data corresponding to each network request; calculating the semantic similarity between the content of the response data and the target data description information; and determining the network request corresponding to the response data with the highest semantic similarity as the target network request.

[0026] By calculating the "semantic similarity" between response data and user needs, the system can intelligently identify target requests that truly contain core business data (such as product prices and sales volume) from hundreds or thousands of network request candidates, eliminating reliance on human experience and improving the accuracy and robustness of positioning.

[0027] S105. Extract the request parameter information of the target network request, and generate crawler code for the target URL based on the request parameter information.

[0028] In this embodiment, extracting the request parameter information of the target network request includes: parsing the target network request to obtain the Uniform Resource Locator (URL), request headers, request body payload, and cookies used to maintain session state; and combining the obtained URL, headers, payload, and cookies into the request parameter information.

[0029] It not only extracts the URL but also automatically extracts key information such as headers, payload, and cookies, perfectly replicating the browser's state when initiating a request to construct a complete request context. Furthermore, this comprehensive parameter extraction mechanism effectively bypasses server verification of non-browser requests (such as User-Agent checks or session verification), resulting in code with higher usability and success rates.

[0030] In this embodiment, generating crawler code for the target URL based on the request parameter information includes: obtaining a preset crawler code template, the crawler code template being constructed based on a network request library of a predetermined programming language; filling the request parameter information into the corresponding variable positions of the crawler code template to generate executable script code; wherein, the script code is configured to reproduce the target network request in an environment without a browser to obtain the target data.

[0031] By filling parameters with preset templates, it can generate standardized, well-structured, and executable script code in seconds, avoiding syntax errors that may occur with manual coding. The generated code is configured to run in an environment independent of a browser (such as a pure Python script), making the final web crawler lightweight, consuming fewer resources and running faster during actual deployment, making it suitable for large-scale data collection.

[0032] In this embodiment, the entity executing each step is an intelligent agent.

[0033] The method in this embodiment is applicable to data API scraping from mainstream websites, especially in scenarios with clear structures and data returned through interfaces.

[0034] This embodiment's method receives users' natural language requests and automatically converts them into crawler code, enabling users without deep programming backgrounds or network protocol knowledge to quickly obtain the required data, thus democratizing crawler development and effectively reducing the technical threshold and operational complexity. Furthermore, by introducing a Chrome MCP server to control the browser, this invention can automatically capture network request logs and use intelligent algorithms to automatically locate target data requests, replacing the tedious manual packet capture and filtering process in traditional development, significantly improving development efficiency. Moreover, analysis based on interaction data from a real browser environment (Chrome MCP server) ensures that the obtained network request parameters (such as dynamically generated tokens or cookies) are authentic and valid, avoiding the problem of static analysis's inability to cope with complex anti-crawling strategies. Finally, code written by engineers requires manual re-analysis and modification once the website structure changes, resulting in poor reusability and high maintenance costs. Using this embodiment's method, when the website changes, only the crawler code generation method process in this embodiment needs to be re-executed to generate new code, resulting in lower maintenance costs.

[0035] In one embodiment, generating browser control instructions based on the target URL information and sending the browser control instructions to the Chrome MCP server includes: S201. Construct standard MCP protocol instructions that include navigation actions; The standard MCP protocol command acts as an intermediary bridge between "natural language intent" and "specific program code execution." It is typically a structured JSON object. It translates the vague natural language of "Help me open this website" into precise machine commands that the MCP server can understand.

[0036] The standard MCP protocol instructions that construct navigation actions include: (1) Intent parsing and parameter extraction: Extract core actions and parameters from the user's natural language (e.g., "Open TikTok sales chart").

[0037] (2) Tool matching: Find the corresponding MCP tool in the list of available tools.

[0038] In this embodiment, the MCP tool used is the MCP Browser tool.

[0039] (3) Encapsulation Instructions: Encapsulate the above information into a request format conforming to the MCP standard. A standard MCP instruction containing navigation actions typically includes the following three key parts: Method (method type): usually tools / call (calling tools).

[0040] Name (Tool Name): Specifies the specific tool capability to be invoked, such as browser_navigate or puppeteer_goto.

[0041] Arguments (parameter payload): Contains the specific URL and navigation configuration.

[0042] S202. Establish an MCP connection with the Chrome MCP server, and send the standard MCP protocol command through the MCP connection to instruct the Chrome MCP server to load the target URL in a real browser environment and perform page rendering operations.

[0043] By establishing connections using the standard MCP (Model Context Protocol), communication between the agent and the browser control service is decoupled, enhancing the system's scalability and compatibility. Furthermore, by instructing the ChromeMCP server to load the URL in a "real browser environment," the JavaScript code within the webpage can be fully executed, ensuring that the page structure and data are rendered correctly. This solves the problem that traditional static crawlers cannot handle client-side rendered pages (CSR).

[0044] In this embodiment, the page rendering operation includes: generating interactive instructions simulating user behavior, the interactive instructions including page scrolling instructions or click instructions; sending the interactive instructions through the MCP connection to trigger the browser to load dynamically loaded data from the target URL; wherein the network request log contains asynchronous network request records corresponding to the dynamically loaded data.

[0045] By generating scrolling or clicking commands, this invention can simulate real user actions, thereby triggering the "lazy loading" or asynchronous loading mechanism of web pages. This allows the system to capture asynchronous network requests (XHR / Fetch) that only occur after specific interactions, effectively solving the problem of crawling hidden or paginated data and significantly improving the data capture rate.

[0046] In one embodiment, the method further includes: determining whether the response data of the target network request is encrypted data; if the response data is encrypted data, parsing it based on the page document object model structure returned by the Chrome MCP server; generating page element extraction code based on DOM parsing rules, and using the page element extraction code as the crawler code.

[0047] When encountering complex scenarios where API data is encrypted (making it difficult to directly parse plaintext), the system automatically switches to a DOM-based page element extraction strategy. This provides a multi-dimensional parsing solution, ensuring that data can still be obtained by parsing the webpage's visual structure even when the API interface is unavailable, greatly improving the success rate of the crawler.

[0048] In one embodiment, the method further includes: after generating crawler code for the target URL, the method further includes: executing the crawler code in a local sandbox environment to obtain execution results; verifying whether the execution results contain data content that conforms to the target data description information; if the verification fails, adjusting the positioning strategy or code generation parameters of the target network request, and regenerating the crawler code.

[0049] In this embodiment, a local sandbox environment is introduced for trial operation and result verification, forming a closed-loop system of "generation-verification-optimization". This system can promptly detect code errors or data extraction failures and automatically adjust strategies (such as changing the target request or correcting parameters) to retry, ensuring that the code finally delivered to the user is verified and truly usable.

[0050] In this embodiment, adjusting the positioning strategy of the target network request includes: The agent enables a candidate list, which includes: During the initial analysis, the agent calculates the semantic similarity score between all network requests (XHR / Fetch) and the user's demand ("product price"). It typically selects the request with the highest score (Top 1) as the target. When Top 1 verification fails, the system marks the request as a "negative sample." The agent then falls back to the network request log and automatically selects the **second-highest similarity candidate (Top 2)** as the new target network request.

[0051] Adjusted code generation parameters, including: enhanced request headers and corrected parsing rules.

[0052] Enhanced request headers, including: comparing "Headers in the generated code" with "Headers in the raw logs captured by the MCP Server". Fields present in the raw logs but ignored in the code (such as x-csrf-token or cookie) are added to the code generation parameter configuration.

[0053] The revised parsing rules include: the Agent reads the sandbox error messages and the actual returned JSON structure. The dictionary value path (JSONPath) in the Python code has been modified, for example, response['price'] has been changed to response['data']['products'][0]['price'].

[0054] In one embodiment, the method further includes: in response to a website structure change signal for the target URL, re-triggering the step of generating browser control instructions based on the target URL information; and re-capturing the updated network request logs through the Chrome MCP server and generating updated crawler code.

[0055] When changes in the target website structure cause the original crawler to fail, the system can respond to the signal and re-trigger the automated process without the need for manual intervention and re-analysis, which greatly reduces the long-term maintenance cost and operation and maintenance pressure of the crawler project.

[0056] Example of a web crawler code generation system based on Chrome MCP server: This invention also provides a web crawler code generation system based on the Chrome MCP server. For example... Figure 2 As shown, the web crawler code generation system based on the Chrome MCP server includes a processor and a memory. The memory stores computer program instructions, which, when executed by the processor, implement a web crawler code generation method based on the Chrome MCP server according to the first aspect of the present invention.

[0057] The web crawler code generation system based on the Chrome MCP server also includes other components well known to those skilled in the art, such as communication buses and communication interfaces. Their settings and functions are known in the art and will not be described in detail here.

[0058] While this specification has shown and described numerous embodiments of the invention, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will occur to those skilled in the art without departing from the spirit and essence of the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed in the practice of this invention.

Claims

1. A method for generating web crawler code based on a Chrome MCP server, characterized in that, include: Obtain the user's input in natural language, perform semantic parsing on the natural language requirement, and determine the target URL information and target data description information; Based on the target URL information, a browser control command is generated and sent to the Chrome MCP server to drive the Chrome MCP server to control the browser to access the target URL; Receive page interaction data returned by the Chrome MCP server, the page interaction data including network request logs generated by the browser during the access to the target URL; The network request log is analyzed based on the target data description information, and the target network request containing the target data is located from the multiple network requests contained in the network request log. Extract the request parameter information of the target network request, and generate crawler code for the target URL based on the request parameter information.

2. The web crawler code generation method based on Chrome MCP server as described in claim 1, characterized in that, The step of generating browser control instructions based on the target URL information and sending the browser control instructions to the Chrome MCP server includes: Construct standard MCP protocol instructions that include navigation actions; Establish an MCP connection with the Chrome MCP server, and send the standard MCP protocol instructions through the MCP connection to instruct the Chrome MCP server to load the target URL in a real browser environment and perform page rendering operations.

3. The web crawler code generation method based on the Chrome MCP server as described in claim 2, characterized in that, The page rendering operation includes: generating interactive instructions simulating user behavior, including page scrolling instructions or click instructions; sending the interactive instructions through the MCP connection to trigger the browser to load dynamically loaded data from the target URL; wherein, the network request log contains asynchronous network request records corresponding to the dynamically loaded data.

4. The web crawler code generation method based on Chrome MCP server as described in claim 2, characterized in that, The step of locating the target network request containing the target data from the multiple network requests contained in the network request log includes: traversing the multiple network requests in the network request log and obtaining response data corresponding to each network request; calculating the semantic similarity between the content of the response data and the target data description information; and determining the network request corresponding to the response data with the highest semantic similarity as the target network request.

5. The web crawler code generation method based on Chrome MCP server as described in claim 1, characterized in that, The step of extracting the request parameter information of the target network request includes: parsing the target network request to obtain the Uniform Resource Locator (URL), request headers, request body payload, and cookies used to maintain session state; and combining the obtained URL, headers, payload, and cookies into the request parameter information.

6. The web crawler code generation method based on Chrome MCP server as described in claim 5, characterized in that, The step of generating crawler code for the target URL based on the request parameter information includes: obtaining a preset crawler code template, the crawler code template being constructed based on a network request library of a predetermined programming language; filling the request parameter information into the corresponding variable positions of the crawler code template to generate executable script code; wherein, the script code is configured to reproduce the target network request in an environment without a browser to obtain the target data.

7. The web crawler code generation method based on Chrome MCP server as described in claim 1, characterized in that, Also includes: Determine whether the response data of the target network request is encrypted; If the response data is encrypted, it is parsed based on the page document object model structure returned by the Chrome MCP server; page element extraction code based on DOM parsing rules is generated, and the page element extraction code is used as the crawler code.

8. The web crawler code generation method based on Chrome MCP server as described in claim 1, characterized in that, After generating crawler code for the target URL, the method further includes: executing the crawler code in a local sandbox environment to obtain execution results; verifying whether the execution results contain data content that matches the target data description information; if the verification fails, adjusting the positioning strategy or code generation parameters of the target network request, and regenerating the crawler code.

9. The web crawler code generation method based on the Chrome MCP server as described in any one of claims 1 to 8, characterized in that, Also includes: In response to a website structure change signal for the target URL, the step of generating browser control instructions based on the target URL information is retried; The updated network request logs are recaptured using the Chrome MCP server, and the updated crawler code is generated.

10. A web crawler code generation system based on the Chrome MCP server, comprising: A processor and a memory, the memory storing computer-executable instructions; characterized in that, when the processor executes the computer-executable instructions, it implements the web crawler code generation method based on the Chrome MCP server as described in any one of claims 1 to 9.

Images