Automatic network data acquisition method based on ScrapeGraphAI

Abstract

The invention discloses an automatic network data acquisition method based on ScrapeGraphAI. The method comprises the following steps: receiving an identifier of at least one target network resource and a data extraction demand; constructing or selecting a processing graph (Graph) comprising at least one acquisition node, an AI analysis node and a formatting node based on a data extraction requirement; executing the acquisition node to acquire the content of the target network resource; executing an AI analysis node, including: utilizing at least one large language model (LLM) to perform semantic analysis on the content of the acquired target network resource based on a data extraction demand to identify and extract target data, where the identification and extraction of the target data are based on semantics of the content rather than a predefined fixed structure selector; executing a formatting node, and converting the target data extracted by the AI analysis node into predetermined or inferred structured format data; and outputting the structured format data. The invention further discloses a system, electronic equipment and a computer readable storage medium.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network data processing, and particularly relates to a method for automatically collecting webpage data by using an artificial intelligence model. BACKGROUND

[0002] With the explosive growth of Internet information, the network has become an indispensable data source for many application fields such as information acquisition, knowledge discovery, business intelligence, market analysis, public opinion monitoring, and artificial intelligence model training. Accurately, efficiently and continuously extracting structured data from massive, heterogeneous and dynamic network information, i.e. network data collection (Web Scraping or Web Crawling and Extraction), has become a key basic technology. The value of webpage data is increasingly prominent. For example, e-commerce platforms need to collect information such as competitor product prices, sales, and user reviews to develop pricing strategies and optimize products; financial institutions need to monitor news, company announcements, and market data in real time to make risk assessments and investment decisions; scientific research institutions need to obtain research papers and experimental data from academic websites and literature databases to support scientific research; and artificial intelligence companies need a large amount of text, image, and audio / video data to train and optimize various AI models. Therefore, the demand for efficient, intelligent, and automated network data collection technology is increasingly urgent.

[0003] Web data collection techniques have evolved from manual copy-pasting to automated scripts, and then to specialized frameworks and platforms. In the early days, users could only manually browse web pages and copy the desired information, which was inefficient and not scalable. Later, developers began writing simple scripts (such as using regular expressions to match) to extract data from HTML source code, but this required programming skills and was very sensitive to changes in page structure. To improve development efficiency and code reusability, HTML / XML parsing libraries such as BeautifulSoup, lxml, and more comprehensive crawler frameworks such as Scrapy and Requests-HTML emerged. These tools simplify the process of sending HTTP requests, parsing HTML documents, defining data extraction rules (usually using CSS selectors or XPath expressions), and have become the most widely used technical means. However, with the evolution of web technologies, especially the popularity of dynamic loading technologies such as AJAX and SPA (Single Page Application), parsing static HTML source code alone cannot obtain complete page content. Therefore, browser automation tools such as Selenium, Puppeteer, and Playwright have emerged, which simulate real user operations to load pages, execute JavaScript, click buttons, scroll pages, and other operations to obtain the complete DOM structure after dynamic rendering, and then extract data. This has solved the problem of dynamic content collection to some extent, but also brought new challenges such as low running efficiency, high resource consumption (requiring the launch and management of browser instances), complex configuration, and being more easily detected by website anti-crawler mechanisms.

[0004] In recent years, in order to further reduce the use threshold, some visual, low-code / no-code web data collection tools have appeared on the market. These tools usually provide a graphical interface, allowing users to select the data elements they need to collect by clicking and dragging in browser plugins or desktop applications. The tool will automatically generate the corresponding selector rules in the background. This allows business personnel without programming backgrounds to perform simple data collection tasks. However, the underlying logic of these tools is still mostly based on CSS selectors or XPath, and they are essentially still dependent on specific DOM structures, and their robustness and ability to handle complex scenarios (such as nested data, data that requires complex interactions to display, and websites with strict anti-crawler strategies) are still limited.

[0005] The main problems faced by current web data collection techniques are concentrated in the following aspects:

[0006] 1. Vulnerability and high maintenance cost: Rules based on CSS selectors or XPath are extremely sensitive to minor changes in the website's frontend code (HTML structure, CSS class names, IDs, etc.). Website developers often adjust page layouts and code structures for optimization, redesign, or anti-crawler purposes, leading to the failure of preset collection rules, requiring a large amount of manpower for continuous monitoring, debugging, and updating maintenance, with high costs.

[0007] 2. Challenges in processing dynamic and complex web pages: Modern web pages commonly use JavaScript to dynamically generate or load content, making it difficult for traditional static parsing methods to obtain complete data. Although browser automation tools can be used, they are slow and resource-intensive, making them unsuitable for large-scale, high-concurrency collection tasks. At the same time, it is very difficult to define precise extraction rules for data with complex structures, deep nesting levels, or dependence on specific user interactions.

[0008] 3. Technical threshold and manpower dependence: Whether using a crawler framework or a browser automation tool, developers need to have a solid programming foundation, web frontend knowledge (HTML, CSS, JavaScript, DOM), and anti-crawler combat experience. This limits the popular application of data collection technology, making many teams or individuals with data needs but lack of technical ability hesitant. Visual tools, while lowering the threshold, have obvious ability boundaries and cannot meet complex or customized needs.

[0009] 4. Lack of semantic understanding and intelligent adaptation: Existing mainstream methods mainly rely on structural information rather than semantic information. They cannot understand the true meaning of elements such as "product main picture", "user rating", "news release date", etc., and can only locate them by their position in the DOM tree or specific attributes. This makes them difficult to adapt to pages with different layouts but similar semantics, and unable to handle fuzzy collection requirements based on natural language descriptions (e.g., "help me find the main points of this article").

[0010] 5. Continuous confrontation with anti-crawler mechanisms: As the value of data increases, website operators deploy increasingly complex anti-crawler measures such as CAPTCHA, IP restrictions, User-Agent detection, behavior analysis, JavaScript obfuscation encryption, browser fingerprinting, etc. This increases the difficulty and cost of data collection, requiring collectors to invest more effort in technical confrontation.

[0011] In summary, current web data collection techniques still have significant shortcomings in terms of automation level, intelligence level, ease of use, robustness, and ability to deal with complex dynamic web pages and anti-crawler strategies. There is an urgent need for a more intelligent, more adaptive, easier-to-use, and lower-maintenance web data automation collection solution to meet the growing demand for diverse data. The rapid development of artificial intelligence, especially large language models (LLM) and vision models (Vision Model), provides a new opportunity to solve the above problems. By leveraging the natural language understanding, visual recognition, and pattern learning capabilities of AI, it is possible to transition from rule-based "structured extraction" to content-understanding-based "intelligent extraction", thereby significantly improving the automation level and adaptability of web data collection.

[0012] To better understand the background and innovation of the present invention, several existing mainstream web data collection technology solutions related to the present invention are introduced as follows:

[0013] Solution One: Rule-based static / dynamic web page data collection (using Scrapy+Requests / Selenium as an example)

[0014] This is one of the most widely used technology solutions at present. The core idea is that the developer analyzes the HTML structure of the target web page in advance, writes precise CSS selectors or XPath expressions to locate the required data elements.

[0015] For static web pages (content directly in HTML source code), the requests library in Python is usually used to send HTTP requests to obtain the web HTML source code, and then the BeautifulSoup or lxml library is used to parse the HTML document and build a DOM tree. Then, the predefined CSS selector or XPath expression is used to find the matching nodes in the DOM tree, and the text content, attribute values, etc. of the nodes are extracted. For tasks that require processing multiple pages or following specific links for crawling, a crawler framework like Scrapy is usually used. Scrapy provides functions such as asynchronous processing, request scheduling, middleware, and data pipeline, which can efficiently manage the crawling process.

[0016] For dynamic web pages (content generated by JavaScript rendering on the client side), the above methods cannot obtain complete data. At this time, it is necessary to introduce browser automation tools such as Selenium or Playwright. This solution controls a real browser kernel (such as Chrome, Firefox) through programming, loads the target URL. The browser will execute the JavaScript code in the page and render the complete DOM. Then, developers use CSS selectors or XPath expressions to locate and extract data in the rendered DOM through the API provided by the browser automation tool. This way can handle most dynamic content, but the cost is that starting and running a browser instance consumes a lot of CPU and memory resources, and the execution speed is relatively slow. For example Figure 1 A simple flowchart is shown in the following figure. Figure 1 A simplified flowchart is shown in the following figure. The actual framework such as Scrapy will be more complex, including request scheduling, middleware, etc.

[0017] Solution two: visual low-code / no-code data collection tools

[0018] In order to reduce the technical threshold of data collection, many visual data collection tools have appeared on the market, such as Octoparse, ParseHub, WebScraper (browser extension), etc. These tools usually provide an embedded browser or browser plug-in, and users can directly operate on the target web page.

[0019] After the user opens the target web page, he can "select" the data elements (such as title, price, link, etc.) he needs to extract by clicking the buttons on the interface. The tool will try to automatically identify the CSS selector or XPath of the element clicked by the user and record it. Users can define the extraction of list data (such as product list, article list), set the page turning rules (click the "next page" button or scroll to load), and even simulate some simple interactions (such as clicking the drop-down menu). After completing the rule configuration, the user can start the collection task, and the tool will automatically execute the data scraping in the background (locally or in the cloud) according to the configured rules.

[0020] The advantage of such tools is intuitive and easy to use, without the need to write code, suitable for non-technical personnel to perform some routine data collection tasks. They usually also have some basic capabilities to handle dynamic content and anti-crawler.

[0021] Solution three: early attempts based on templates or machine learning

[0022] Before the popularity of LLM, there were also some researches trying to use machine learning or template-based methods to improve the automation of data collection. For example, some systems tried to automatically identify stable templates or regions containing target data by comparing the structural differences of multiple similar pages (such as multiple product detail pages of the same website). Some other researches used supervised learning, which required users to pre-annotate a large number of sample pages to train the model to recognize specific types of data fields (such as address, phone number). These methods achieved certain results in specific scenarios, but generally required a large amount of sample data, had limited generalization ability, and were not suitable for websites with drastic structural changes, and failed to achieve large-scale commercial application.

[0023] These existing technical solutions, although to varying degrees, solve the problem of network data collection, but as described in the background section, they generally have the defects of strong dependence on page structure, high maintenance cost, limited ability to handle complex dynamic content, high technical threshold, lack of semantic understanding ability, etc. These defects limit the efficiency, scope and intelligence level of network data collection, making it difficult to meet the growing demand for massive and diversified network data.

[0024] Existing technical solutions, whether based on rule-based programming methods, visual tools, or early machine learning attempts, have exposed many defects in practical applications, which are specifically described as follows:

[0025] 1. High vulnerability and maintenance nightmare: The core defect of existing technologies lies in their strong dependence on specific structures of web pages (HTML tags, CSS class names, IDs, XPath paths). Any minor changes in the website's front-end code, such as the developer changing a CSS class name, adjusting the DOM hierarchy structure, or even just adding an insignificant Tags, all of which can cause predefined CSS selectors or XPath expressions to fail, resulting in failed data collection or the extraction of incorrect data. Due to website redesigns, feature iterations, A / B testing, anti-crawler strategy adjustments, and other reasons, web page structure changes are the norm rather than the exception. This means that rule-based collection scripts or configurations need constant monitoring and maintenance. Once the rules are no longer valid, technical personnel need to reanalyze the page, locate the problem, modify the rules, and redeploy them. For scenarios that require the collection of a large number of different websites or frequent changes in website structure, the maintenance workload is enormous, the cost is high, and the timeliness and economy of data collection are severely affected.

[0026] 2. Efficiency and stability bottlenecks in handling complex dynamic content: Modern web pages use JavaScript extensively for asynchronous data loading (AJAX), client-side rendering (such as SPA built with React, Vue, Angular frameworks), and user interaction-driven content updates. Traditional static HTML parsing methods are ineffective. Although browser automation tools such as Selenium and Playwright can simulate browser behavior to execute JavaScript and obtain the final rendered DOM, this brings significant performance overhead. Starting and managing browser instances requires a large amount of CPU and memory resources, and the execution speed is much slower than direct HTTP requests, making it difficult to meet large-scale, high-concurrency data collection needs. In addition, the loading timing, rendering process of dynamic content may be affected by network delays, JavaScript execution errors, and specific interaction logic, resulting in unstable data extraction, sometimes getting data, sometimes not getting data or incomplete. Defining precise rules to capture these dynamically changing data is also extremely challenging.

[0027] 3. High technical threshold and human resource limitations: Using mainstream frameworks such as Scrapy and Selenium for data collection requires developers to be proficient in programming languages such as Python, and to have a deep understanding of HTTP protocols, HTML, CSS, JavaScript, DOM operations, browser working principles, and some anti-crawler and anti-anti-crawler experience. This requires a high level of technical ability from employees, a long training period, and a relatively small number of qualified personnel. This makes it difficult for many businesses or individuals with data needs but lack of professional technical teams to carry out effective data collection on their own, or they need to pay a high outsourcing cost.

[0028] 4. Lack of semantic understanding and intelligent adaptability: Existing technical solutions are essentially "pattern matching", not "content understanding". They locate data by structure and cannot understand the true meaning of data. For example, a rule may be set to extract all class="price" elements The label content is used as the price, but it cannot determine whether the price is the original price, a discounted price, or a member price, nor can it automatically adapt after changing the class name of the price label to class = "sale-price" on the website. This lack of semantic understanding leads to rigid collection systems that cannot handle natural language-based ambiguous requirements (such as "extract the main contact information on the page") and makes it difficult to transfer knowledge learned on one website to another website with similar content types but different structures.

[0029] 5. Limitations of visualization tools: While visualization tools lower the threshold for use, they often just hide the complexity of rule definition behind a graphical interface, and the underlying logic is still mostly CSS / XPath. Therefore, they also inherit the fragility of the rule method. And for pages that require complex login processes, multi-step interactive operations, special anti-crawler mechanisms (such as sliding verification codes, Canvas rendering content), or exceptionally complex nested data structures, visualization tools often struggle to perform, with much less flexibility and scalability than programming solutions.

[0030] 6. Continuous war of anti-crawler confrontation: Websites constantly upgrade their anti-crawler technology to protect their data and server resources. This forces data collectors to invest a lot of effort in researching and deploying anti-anti-crawler strategies, such as using proxy IP pools, maintaining User-Agent libraries, optimizing request headers, simulating real user behavior, cracking JavaScript obfuscation, recognizing and processing verification codes, etc. This not only increases technical difficulty and implementation costs, but also makes the data collection process full of uncertainty, like an endless "cat and mouse game".

[0031] In summary, the inherent defects of existing technical solutions in robustness, efficiency, ease of use, intelligence, and adaptability severely restrict the development and application potential of network data collection technology, and cannot meet the strong demand for large-scale, high-quality, real-time network data in modern society. Therefore, there is an urgent need for a new type of network data automatic collection method that can overcome the above defects. SUMMARY

[0032] The present application aims to overcome the defects of the prior art and proposes a network data automatic collection method and system based on ScrapeGraphAI. The core idea of the method is to use ScrapeGraphAI as a flexible graph execution engine, which is connected in series with a series of functional nodes. The key nodes use the powerful understanding ability of large language models (LLM) and / or vision models (VisionModel) to replace the traditional data extraction logic that relies on CSS / XPath rules, thereby realizing semantic understanding-based, robust and automated data collection. The core purpose of the present application is to provide a completely new network data collection method based on the ScrapeGraphAI framework, which deeply integrates the capabilities of large language models (LLM) and vision models (VisionModel) and is highly automated and intelligent. The method and system aim to: replace the traditional fixed structure rule-based data positioning method with the semantic understanding ability of AI models, achieve a deep understanding of the meaning of web page content, thereby greatly enhancing the robustness of data collection and enabling it to adapt to frequent changes in website structure; effectively address the data extraction challenges of complex dynamic web pages through intelligent content analysis and possibly combined with lightweight page rendering (if necessary), ensuring data integrity while seeking higher efficiency and stability than traditional browser automation solutions; support task definition methods based on natural language prompts (Prompts) or structured descriptions (Schemas), significantly reducing the user's usage threshold, allowing non-professional users to easily configure and execute data collection tasks; build a flexible and extensible graph-based processing flow that decomposes data collection tasks into a series of combinable nodes (such as acquisition, parsing, AI analysis, formatting, etc.), making it easy to customize and optimize, and intelligently dynamically constructing or adjusting the processing flow according to task requirements; integrate text and visual multi-modal information, use LLM to process text content and structure, and use VisionModel to understand page visual layout and image information, achieving a more comprehensive and accurate understanding of web page information and data extraction; ultimately realize a network data automatic collection solution with low maintenance cost, strong adaptability, user-friendly, and high intelligence, effectively overcoming the many defects of existing technologies and meeting the urgent needs of modern application scenarios for network data acquisition.

[0033] The first aspect of the present application is to provide a network data automatic collection method based on ScrapeGraphAI, which is implemented based on the ScrapeGraphAI framework or graph execution engine, wherein the data collection flow is represented as a graph composed of nodes (Nodes) and edges (Edges), facilitating the modularization, combination, execution and management of the flow, including:

[0034] S1, receiving an identifier (such as URL) of at least one target network resource and a data extraction requirement;

[0035] S2, constructing or selecting a processing graph containing at least one acquisition node, an AI analysis node and a formatting node based on the data extraction requirement; wherein the acquisition node has the ability of dynamic acquisition strategy selection, which can select or switch between standard HTTP request acquisition and post-rendering acquisition based on browser automation tools (such as Playwright, Selenium) according to the content characteristics of the target network resource, task configuration or historical execution situation;

[0036] S3, executing the acquisition node to acquire the content of the target network resource;

[0037] S4, executing the AI analysis node, including: using at least one large language model (LLM) to perform semantic analysis on the acquired content of the target network resource based on the data extraction requirement to identify and extract target data, wherein the identification and extraction of the target data are based on the semantics of the content rather than pre-defined fixed structure selectors (such as CSS or XPath);

[0038] S5, executing the formatting node to convert the target data extracted by the AI analysis node into a predetermined or inferred structured format data;

[0039] S6, outputting the structured format data.

[0040] Preferably, the data extraction requirement is provided in the form of a natural language prompt or a structured pattern.

[0041] Preferably, for the data extraction requirement provided in the form of the natural language prompt (Natural Language Prompt), the behavior and possible graph structure of the AI analysis node are automatically configured based on the data extraction requirement prompt, and the AI analysis node directly uses the natural language prompt to guide the LLM to perform semantic analysis and data extraction.

[0042] Preferably, the construction or selection of the processing graph is automatically performed according to the characteristics of the data extraction requirement, realizing the automatic mapping from user intention to execution flow; wherein the characteristics of the data extraction requirement include one or more of complexity and involved concepts.

[0043] Preferably, the processing graph further comprises a vision node (Vision Node) that is executed before or in parallel with the AI analysis node; the vision node is configured to obtain a visual representation (e.g., a screenshot) of the target network resource and analyze the visual representation using at least one vision model (Vision Model); the AI analysis node further combines the visual analysis result output by the vision node as a visual context when performing semantic analysis using the LLM, so as to improve the accuracy of data extraction or process visual-dependent information.

[0044] Preferably, the processing graph further comprises a parsing node (Parse Node) that is executed after the obtaining node and before the AI analysis node; the parsing node is configured to clean up, simplify (e.g., convert to Markdown), or block the content of the target network resource obtained, so as to generate a format more suitable for inputting into the LLM for analysis.

[0045] The second aspect of the present application provides a network data automatic collection system based on ScrapeGraphAI, which is configured to implement the method of the first aspect, and comprises:

[0046] An interface module configured to receive a user task comprising a URL and an extraction requirement;

[0047] A graph definition module configured to construct or select a processing graph according to the user task;

[0048] A node library comprising basic nodes including an obtaining node, an AI analysis node, and a formatting node;

[0049] A graph executor configured to schedule and execute the basic nodes in the processing graph;

[0050] An AI model service interface configured to call an external or local LLM and / or Vision Model; wherein the AI analysis node is configured to perform data extraction based on the semantic understanding capability of the LLM rather than a fixed structure selector;

[0051] A data processing module configured to store or output structured format data as a result.

[0052] Preferably, the system further comprises a vision node and an interface for calling a vision model, wherein the vision node is configured to provide visual analysis capability, and the interface for calling the vision model is configured to assist the LLM of the AI analysis node to perform more accurate data extraction.

[0053] The third aspect of the present application provides an electronic device comprising a processor and a memory, wherein the memory stores a plurality of instructions, and the processor is configured to read the instructions and perform the method according to the first aspect.

[0054] The fourth aspect of the present application provides a computer-readable storage medium storing a plurality of instructions, wherein the instructions can be read and executed by a processor to perform the method according to the first aspect.

[0055] Advantages of the method and system of the present application:

[0056] Compared with the closest prior art solutions, i.e., rule-based static / dynamic web page data collection methods (such as using Scrapy+Requests / Selenium, solution one) and visual low-code / no-code data collection tools (solution two), as well as early attempts based on templates or machine learning (solution three), the "network data automatic collection method and system based on ScrapeGraphAI" proposed by the present application has significant technical advantages and effects, which are directly derived from the technical solution of the present application and effectively overcome the defects of the prior art:

[0057] 1. High robustness and low maintenance cost (corresponding to defect 1: high fragility and maintenance nightmare):

[0058] Advantages: The most core advantage of the present application lies in its data extraction mechanism based on AI semantic understanding. Since it does not rely on specific, changeable CSS class names, IDs or XPath paths in the page HTML, but understands the meaning of the content (for example, "this is a price tag", "this is the publication date of the article"), it has natural immunity to regular modifications of the website front-end code (such as style adjustment, layout fine-tuning, label replacement, class name reconstruction). As long as the semantic context and visual presentation of the information do not undergo a revolutionary change, the AI model will usually still be able to correctly identify and extract data.

[0059] Effects: This directly leads to a significant improvement in the robustness of the collection task. Collection rules are no longer easily invalidated by frequent updates to the website, greatly reducing the time and effort required for manual monitoring, debugging and repairing rules, and significantly reducing long-term maintenance costs. This advantage is particularly prominent for scenarios that require monitoring of a large number of websites or websites with variable structures.

[0060] 2. Enhanced complex and dynamic web page processing capability (corresponding to defect 2: efficiency and stability bottleneck for processing complex dynamic content):

[0061] Advantages: The present application can seamlessly integrate browser rendering capabilities to handle JavaScript dynamic content when needed through intelligent acquisition node strategies, ensuring data integrity. More importantly, even in the face of complex DOM structures after rendering, the present application does not require manual writing of complex selectors for precise navigation, but relies on AI (LLM combined with optional VisionModel) to understand the overall content and layout to find target data. AI models are particularly good at handling unstructured text, understanding contextual relationships, and even visually identifying important areas, which is very effective for extracting deeply nested, irregularly structured, or visually dependent data (such as charts and special control data).

[0062] Effects: Improves the probability and stability of successfully extracting data from modern complex web pages (SPA, AJAX heavy applications, Canvas drawing, etc.). Although there is still performance overhead when browser rendering is needed, due to the intelligence of the extraction logic, it may require less interaction simulation and DOM exploration than traditional methods, and subsequent AI analysis can tolerate some uncertainty of DOM. The multi-modal analysis capability of text and vision further broadens the types of pages that can be processed.

[0063] 3. Significantly reduced technical threshold and improved user friendliness (corresponding to defect 3: high technical threshold and human resource limitations):

[0064] Advantages: The present application supports the use of natural language prompts as the main input method for data extraction requirements. Users only need to describe what data they want in everyday language ("give me all the product names and prices on this page"), without the need to learn and write any CSS, XPath or programming code. The system automatically converts this intent into AI model instructions and possible execution processes in the background.

[0065] Effects: This greatly reduces the operational threshold of web data collection, allowing business personnel, researchers without technical backgrounds to directly and quickly configure and run data collection tasks. This greatly expands the application population and scenarios of this technology, improves work efficiency, and reduces dependence on professional crawler engineers.

[0066] 4. Semantic understanding and intelligent adaptability (corresponding to defect 4: lack of semantic understanding and intelligent adaptability):

[0067] Advantages: The invention introduces the semantic understanding ability of AI. Instead of mechanically matching structural patterns, the system attempts to understand the meaning of web page content. This enables it to handle more ambiguous and closer to human thinking way of collection requests ("extract the core idea of the article" is challenging, but the direction is so). More importantly, this semantic understanding ability gives the system a certain adaptability. For example, training or prompting the model to identify "product main image" on an e-commerce website, the model may learn to identify a combination of features such as image size, location, context (such as close to the title and price), etc. When encountering another e-commerce website with a different layout but similar function, it may rely on this learned "conceptual" knowledge to find the main image faster on the new website without starting from scratch to define rules.

[0068] Effects: Improve the adaptability of the collection method to different website styles and layout changes. Make it possible to handle diversified and non-standardized web page data. Lay the foundation for cross-site knowledge transfer and more intelligent adaptive collection.

[0069] 5. Higher level of automation and flexibility (corresponding to defect 5: limitations of visual tools' capabilities & defect 6: part of the continuous attrition war of anti-scraping confrontation):

[0070] Advantages: Through the graph flow engine of ScrapeGraphAI, the entire collection process (acquisition, parsing, AI analysis, formatting, etc.) is clearly modularized and automatically arranged. Natural language driven task definition reduces the human intervention in the early configuration. Although the invention itself does not directly solve all anti-scraping problems, its flexibility allows easier integration of various anti-anti-scraping strategies as nodes in the graph (such as nodes that automatically handle Captcha, intelligent proxy switching nodes, and interactive nodes that simulate human behavior). And because it is not sensitive to changes in HTML structure, it is less likely to trigger anti-scraping detection based on DOM structure stability.

[0071] Effects: Provide more powerful and flexible customization capabilities than visual tools while maintaining a high level of automation. Make it more systematic to build and maintain complex collection processes that include anti-scraping strategies. May reduce the risk of being detected by simple anti-scraping mechanisms to some extent.

[0072] In summary, the present application realizes the paradigm shift from "structure" based extraction to "semantic" based extraction by deeply integrating the graph execution framework of ScrapeGraphAI with advanced AI (LLM and VisionModel) technologies. This transformation brings a comprehensive improvement in robustness, ease of use, adaptability, and automation, effectively overcoming the main defects of existing technical solutions, and providing a more intelligent, efficient, and future-oriented solution for the field of network data automated collection. It makes data collection tasks that previously required a lot of professional skills and continuous maintenance investment easier to implement and manage, thereby better meeting the huge demand for massive network data in various industries. BRIEF DESCRIPTION OF DRAWINGS

[0073] In order to more clearly illustrate the technical solutions in the specific embodiments or related art, the following will briefly introduce the drawings needed to be used in the specific embodiments or related art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0074] Figure 1 A static / dynamic web page data collection schematic diagram based on rules according to the prior art solution, taking Scrapy+Requests / Selenium as an example;

[0075] Figure 2 A network data automated collection system architecture diagram based on ScrapeGraphAI according to an embodiment of the present application;

[0076] Figure 3 A network data automated collection core process diagram based on ScrapeGraphAI according to an embodiment of the present application;

[0077] Figure 4 A core data extraction flow sequence diagram according to an embodiment of the present application;

[0078] Figure 5 An electronic device structure diagram according to an embodiment of the present application. DETAILED DESCRIPTION

[0079] The technical solutions of the present application will be described in detail below with reference to the drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0080] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the indicated device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second", "third" are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0081] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connection" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, or it can be connected inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0082] (I) In view of the above-mentioned defects existing in the prior art network data collection technical solutions, the present application aims to solve the following core technical problems:

[0083] 1. To solve the problem of vulnerability of collection rules: the existing technical solutions based on CSS selector or XPath are extremely vulnerable because they rely too much on the specific DOM structure of the webpage, and even a small code change in the website can cause the collection to fail, resulting in high maintenance costs. The technical problem to be solved by the present application is: how to reduce or even eliminate the hard dependence on the specific HTML structure, CSS class name or ID of the webpage in the data collection process, so that the collection method can better tolerate changes in the website front-end code, thereby significantly improving the robustness and stability of the collection task, and greatly reducing the frequency and cost of manual maintenance.

[0084] 2. To solve the efficiency and stability problem of processing complex dynamic web pages: the existing technology cannot obtain complete data when processing content dynamically loaded or rendered by JavaScript (static parsing), or relies on browser automation tools (such as Selenium) that consume a lot of resources, execute slowly and have poor stability. The technical problem to be solved by the present application is: how to effectively and efficiently extract the required data from dynamic web pages containing complex JavaScript logic, asynchronous loading, and client-side rendering, while minimizing the use of heavy browser simulation, and improving the stability and reliability of data acquisition.

[0085] 3. Addressing the High Technical Barrier and User-Friendliness Issues: Existing mainstream data collection methods require deep knowledge of programming and web technologies, limiting their application scope. While visualization tools lower the barrier to entry, their functionality is limited. The technical problem this invention aims to solve is: how to provide a more user-friendly data collection method, allowing users to specify collection requirements through natural language descriptions or other simple and intuitive methods, without writing complex code or configuring cumbersome rules, thus enabling more users without technical backgrounds to easily achieve automated data collection.

[0086] 4. Addressing the lack of semantic understanding and intelligent adaptability: Existing technologies, based on structure matching, cannot understand the meaning of the data content itself, struggle to adapt to structural changes across different websites or even after a website's redesign, and are unable to handle ambiguous semantic data collection requests. The technical problem this invention aims to solve is: how to endow the data collection system with a certain degree of semantic understanding, enabling it to locate and extract information based on the meaning of the content rather than merely its structure. This would allow it to better adapt to diverse webpage layouts, handle more flexible data collection tasks, and potentially apply knowledge learned in one scenario to other scenarios.

[0087] 5. Enhance automation and reduce human intervention: From task definition, rule generation (if necessary), execution monitoring to anomaly handling and adaptive rule adjustment, existing technologies still require significant human intervention throughout the entire data acquisition lifecycle. The technical problem this invention aims to solve is: how to further enhance the automation level of the entire network data acquisition process, particularly by achieving automated acquisition logic generation and execution based on advanced user intent (such as natural language descriptions), and by providing a certain degree of adaptive adjustment capability when encountering unexpected situations (such as changes in page structure), thereby minimizing human intervention.

[0088] (II) Invention Point

[0089] The main improvement and innovation of this invention compared to existing technical solutions is that it is not merely a simple combination or optimization of existing technologies, but rather proposes a completely new paradigm for automated network data collection driven by AI semantic understanding.

[0090] Specific inventive points include:

[0091] 1. Selectorless data extraction based on AI semantic understanding:

[0092] Technical details: The core innovation of the present application lies in that the data extraction process mainly relies on the natural language understanding ability of large language models (LLM) for web page content, rather than traditional structured positioning based on CSS selectors or XPath expressions. The AI analysis node (AIN) receives processed web page content (text, Markdown or simplified HTML) and the user's natural language Prompt or Schema, and through careful Prompt engineering, guides the LLM to understand the semantics of concepts such as "title", "price", "author", "release date", etc., and find the corresponding information in the text. Even if the location, tag, class name of these information in the HTML structure changes, as long as the semantic context remains relatively stable, the LLM still has a high probability of successful extraction. For scenarios that require visual information assistance, the visual node (VN) uses a multi-modal model to analyze page screenshots, further enhancing the understanding of layout, style, and non-text elements, providing richer context for the LLM and improving the extraction accuracy on complex pages.

[0093] Technical effects: This semantic understanding-based approach fundamentally breaks away from the dependence on specific and fragile DOM structures, making the collection method extremely robust to changes in website frontend code. It significantly reduces the risk of collection rule failure due to website updates, greatly reducing manual maintenance costs and workload. At the same time, it makes it possible to extract data from web pages with irregular structures, lack of clear CSS class names or IDs.

[0094] 2. Natural language-driven task definition and automated process generation:

[0095] Technical details: The present application supports users using natural language Prompts to describe data collection requirements. The system can parse this natural language input, understand the user's intent, and automatically build or select a suitable ScrapeGraphAI processing graph (Graph) accordingly (combined with possible heuristic rules or pre-defined templates). For example, if the Prompt contains words such as "review" and "rating", the system may automatically add logic to handle pagination or scroll loading in the graph; if the Prompt mentions "logo in the picture", the visual node will be automatically enabled. This means that users do not need to worry about the implementation details, such as which node to choose and how to connect them.

[0096] Technical effects: It greatly reduces the technical threshold of web data collection, making it easy for non-programming background business analysts, market researchers, etc. to define and execute complex collection tasks. It improves usability, shortens task configuration time, and achieves higher-level automation - from user intent to execution process.

[0097] 3、Graph-based Composable Intelligent Flow Engine:

[0098] Technical details: The invention adopts a ScrapeGraphAI or similar graph execution framework. The data collection process is modeled as a directed acyclic graph (DAG) composed of a series of pluggable, reusable function nodes (acquisition, parsing, AI analysis, visual analysis, formatting, login, proxy, etc.). The graph definer can flexibly combine these nodes according to task requirements to build a customized processing pipeline. The graph executor is responsible for scheduling node execution, managing data transmission and state between nodes. This architecture has high flexibility and scalability, making it easy to add new function nodes (such as specific anti-crawler strategy nodes, data verification nodes) or replace existing node implementations (such as switching different LLM models, different HTML parsers).

[0099] Technical effects: Provides a highly flexible, modular and scalable system architecture. Facilitate optimization and customization for different types of websites and task requirements. Make it easy to model and manage complex data collection logic (such as conditional branching, loops, parallel processing). At the same time, the graph structure also provides a foundation for future adaptive adjustments (for example, automatically trying to replace nodes or adjust the graph structure after detecting extraction failures).

[0100] 4、Text and visual multi-modal fusion analysis:

[0101] Technical details: The invention innovatively integrates text analysis (mainly completed by LLM in the AI analysis node) and visual analysis (completed by VisionModel in the visual node). The visual node not only can independently extract visual layout-based information (such as chart data, specific style elements), more importantly, it can provide strong visual context for LLM. For example, the visual model can tell LLM: "The user comment area is in the lower half of the page, and the background color is light gray". LLM combines these visual clues when processing text, can more accurately locate and understand relevant content, especially when dealing with pages with insufficient structural information but clear visual layout.

[0102] Technical effects: By combining text and visual modal information, a more comprehensive and in-depth understanding of web page content is achieved, overcoming the limitations of single modal analysis. Significantly improve the data extraction accuracy and robustness on complex layout, contain chart images, or rely on visual style to distinguish information web pages.

[0103] 5、Intelligent acquisition strategy and resource optimization:

[0104] Technical details: FetchNode is designed to intelligently select the fetching method. By default, it attempts a lightweight HTTP request, and only when it detects the need or when specified by the user, it enables the more resource-consuming browser rendering mode. This dynamic switching strategy aims to balance data integrity and execution efficiency. In the future, even historical data or page complexity assessment models can be combined to make more intelligent decisions.

[0105] Technical effects: Compared with always using browser automation methods, this invention can handle dynamic content while striving to use more efficient fetching methods when possible, thereby optimizing overall resource consumption and execution speed and improving the feasibility of large-scale collection.

[0106] Embodiment one

[0107] This embodiment provides a network data automatic collection method based on ScrapeGraphAI, which is implemented based on the ScrapeGraphAI framework or graph execution engine. The data collection process is represented as a graph composed of nodes and edges, facilitating modularization, combination, execution and management of the process, including:

[0108] S1, receiving at least one identifier (such as URL) of a target network resource and a data extraction requirement;

[0109] S2, based on the data extraction requirement, constructing or selecting a processing graph containing at least one fetch node, one AI analysis node and one formatting node; wherein the fetch node has the ability to dynamically select the fetching strategy, and can select or switch between standard HTTP request fetching and rendering-based browser automation tool (such as Playwright, Selenium) fetching according to the content characteristics of the target network resource, task configuration or historical execution situation;

[0110] S3, executing the fetch node to obtain the content of the target network resource;

[0111] S4, executing the AI analysis node, including: using at least one large language model (LLM) to perform semantic analysis on the obtained content of the target network resource based on the data extraction requirement to identify and extract target data, wherein the identification and extraction of the target data is based on the semantics of the content rather than a pre-defined fixed structure selector (such as CSS or XPath);

[0112] S5, executing the formatting node to convert the target data extracted by the AI analysis node into a predetermined or inferred structured format data;

[0113] S6, outputting the structured format data.

[0114] Preferably, the data extraction requirements are provided in the form of natural language prompts or structured patterns.

[0115] Preferably, for the data extraction requirements provided in the form of natural language prompts, the behavior and possible graph structure of the AI analysis node are automatically configured based on the data extraction requirement prompts, and the AI analysis node directly utilizes the natural language prompts to guide the LLM for semantic analysis and data extraction.

[0116] Preferably, the construction or selection of the processing graph is automatically performed according to the characteristics of the data extraction requirements, realizing automatic mapping from user intent to execution flow; wherein the characteristics of the data extraction requirements include one or more of complexity and involved concepts.

[0117] Preferably, the processing graph further comprises a vision node (Vision Node), which is executed before or in parallel with the AI analysis node; the vision node is used to obtain a visual representation (such as a screenshot) of the target network resource and analyze the visual representation using at least one vision model (Vision Model); when the AI analysis node utilizes the LLM for semantic analysis, the visual analysis results output by the vision node are further combined as visual context to improve the accuracy of data extraction or process visual-dependent information.

[0118] Preferably, the processing graph further comprises a parsing node (Parse Node), which is executed after the acquisition node and before the AI analysis node; the parsing node is used to clean, simplify (such as converting to Markdown), or block the content of the acquired target network resource, to generate a format more suitable for input to the LLM for analysis.

[0119] Embodiment Two

[0120] The embodiment provides a network data automatic collection system based on ScrapeGraphAI, which is used to implement the method of embodiment one, comprising:

[0121] An interface module for receiving a user task comprising a URL and extraction requirements;

[0122] A graph definition module for constructing or selecting a processing graph according to the user task;

[0123] A node library comprising basic nodes including an acquisition node, an AI analysis node, and a formatting node;

[0124] a graph executor for scheduling and executing the base nodes in the processing graph;

[0125] an AI model service interface for invoking external or local LLM and / or VisionModel; wherein the AI analysis node is configured to extract data with core LLM-based semantic understanding capability, rather than fixed structure selector;

[0126] a data processing module for storing or outputting structured format data as results.

[0127] Preferably, the system further comprises a vision node and an interface for invoking a vision model, the vision node being configured to provide visual analysis capability, and the interface for invoking the vision model being configured to assist the LLM of the AI analysis node to extract data more accurately.

[0128] Application Embodiments

[0129] The method proposed by the present application can be implemented by a system, and the overall architecture of the system is as shown in Figure 2 .

[0130] Figure 2 :Based on ScrapeGraphAI, the architecture diagram of the network data automatic collection system

[0131] As shown in Figure 1 , the system mainly includes the following core parts:

[0132] 1. User Interface Layer (User Interface Layer): provides an entry for user interaction, which can be a graphical user interface (GUI), a command line interface (CLI), or an application programming interface (API). Users submit data collection tasks through this layer, including target URLs (s), collection requirements (such as natural language description prompts or structured schema), and possible configuration parameters (such as the AI model used, whether login is required, proxy settings, etc.).

[0133] 2. Task Management and Scheduling Layer (Task Management & Scheduling Layer): responsible for receiving, parsing, storing, and scheduling user-submitted tasks. The task manager (TM) checks the validity of the tasks, prioritizes them, and puts the tasks to be executed into the task queue (TQ). At the same time, it will call the graph definer (GD) to construct or select a suitable execution graph (Graph) for the task.

[0134] 3. ScrapeGraphAI Core Engine: This is the core of the method, based on the ScrapeGraphAI library or similar concepts.

[0135] o Graph Definer (GD): According to the task requirements (such as Prompt complexity, whether visual information is needed, target website characteristics, etc.), a processing flowchart (Graph) is dynamically generated, or a predefined graph template is loaded from the configuration and model library (CS). The graph consists of a series of interconnected nodes (Node), defining the steps and order of data processing.

[0136] o Graph Executor (GE): Responsible for calling and executing each node in the graph in the logical order defined by the graph. It manages state transmission between nodes (such as passing the HTML obtained by the acquisition node to the parsing node) and execution flow control (such as conditional branching, loops, error handling).

[0137] o Base Node Library: Contains a series of reusable functional nodes, each of which performs a specific subtask. Key nodes include:

[0138] ■ Fetch Node (FN): Responsible for obtaining network content from a specified URL. It can implement various acquisition strategies, such as: simple HTTP GET request (using libraries such as requests); or when JavaScript rendering is detected, automatically or according to configuration, switch to using lightweight or heavyweight browser automation tools (such as Playwright) to obtain the rendered DOM or screenshot. This node needs to handle various issues related to network requests, such as timeouts, redirects, request header settings, cookie management, proxy usage, etc.

[0139] ■ Parse Node (PN): Receives the original HTML content output by the acquisition node, performs preprocessing and parsing. Work may include: removing unnecessary tags (such as <script>,<style>)、提取主要的文本内容、将HTML转换为更适合LLM处理的格式(如Markdown)、或者根据需要提取特定的结构化信息(如链接列表)。其目的是为后续的AI分析节点提供干净、有效的信息输入。

[0140] ■AI分析节点(AIAnalysisNode,AIN)：这是实现本发明核心优势的关键节点。它接收解析后的文本内容(或部分HTML片段)以及用户的采集Prompt / Schema。该节点会构造合适的请求，调用AI模型服务层(LLMS)提供的LLM接口。LLM被指示根据Prompt的要求，在提供的文本内容中理解、识别并提取出目标数据。例如，如果Prompt是"提取文章的标题和作者”，LLM会阅读文本并返回识别出的标题和作者信息。此节点需要精心设计Prompt工程，可能包含上下文管理(处理超出LLM单次输入限制的长文本)、结果校验等逻辑。

[0141] ■视觉节点(VisionNode,VN)：可选但非常重要的节点，用于处理需要视觉信息辅助的场景。当配置启用或AI分析节点判断需要时，获取节点会提供页面的截图。视觉节点将截图和相应的Prompt发送给AI模型服务层(VMS)的视觉模型接口(如GPT-4V、Gemini等)。视觉模型可以识别页面布局、控件样式、图表内容、图片中的文字等，这些信息可以单独用于提取数据，或者作为上下文信息辅助LLM进行更精准的文本内容提取。例如，识别visuallyprominent的价格标签，或者从图表中提取数据点。

[0142] ■格式化节点(FormatNode,FMT)：接收来自AI分析节点或视觉节点提取出的原始数据(可能是自然语言文本、列表、键值对等)，根据用户最初指定的Schema或默认规则，将其整理、清洗并格式化为结构化的输出，如JSON、CSV、XML或数据库记录。

[0143] ■其他节点(OtherNodes,ON)：根据具体需求，还可以包含登录节点、验证码处理节点、数据存储节点、条件判断节点、循环控制节点等，以支持更复杂的采集流程。

[0144] 4.AI模型服务层(AIModelServiceLayer)：提供对底层AI模型的访问接口。LLM服务接口(LLMS)和VisionModel服务接口(VMS)负责与实际的AI模型API(可以是OpenAI、Google、Anthropic等云服务，也可以是本地部署的模型)进行通信，处理API请求、认证、速率限制、错误处理等。这使得核心引擎与具体的AI模型实现解耦。

[0145] 5.数据与资源层(Data&ResourceLayer)：提供支撑系统运行所需的数据和资源。数据存储(DS)用于保存采集结果、任务状态、日志信息等。配置与模型库(CS)存储系统配置、预定义的图模板、Prompt模板、AI模型接入信息等。代理池(PROXY)提供IP地址资源以应对反爬虫的IP限制。

[0146] 核心工作流程

[0147] 本发明提出的方法的典型工作流程如图3所示。

[0148] 图3:基于ScrapeGraphAI的网络数据自动化采集核心流程图

[0149] 下面详细阐述流程中的关键步骤：

[0150] (1)接收任务(Step1)

[0151] ●执行主体:用户接口层(UI) / 任务管理器(TM)

[0152] ●触发条件:用户通过UI或API提交新的数据采集请求。

[0153] ●执行动作:系统接收并解析任务参数，主要包括：

[0154] ○目标标识符:一个或多个目标网页的URL。

[0155] ○采集需求:

[0156] ■自然语言提示(Prompt):用户用自然语言描述想要提取的数据，例如："请提取这个页面上所有新闻的标题、发布时间和摘要信息。”或者"找出这个商品页面的价格、评分和用户评论数量。”

[0157] ■结构化模式(Schema):用户提供一个期望输出数据的结构定义，例如一个JSONSchema，明确指出需要哪些字段及其类型。

[0158] ○可选配置:如使用的AI模型(如gpt-4o,gemini-pro-vision,local_llama3)、是否强制使用浏览器渲染、代理设置、登录凭据、结果存储方式等。

[0159] ●运行环境参数:无特定要求，取决于接口实现。

[0160] ●运行原理 / 机制:将用户友好的输入转换为内部可处理的任务对象。

[0161] ●作用 / 意义:定义了数据采集的目标和要求。

[0162] ●结束条件:任务参数被成功解析并验证。

[0163] ●运行结果:生成一个内部任务对象，包含所有必要信息。

[0164] ●技术效果:实现了用户友好的任务定义方式，特别是通过自然语言Prompt，极大地降低了使用门槛。

[0165] (2)初始化ScrapeGraphAI图(Step2)

[0166] ●执行主体:图定义器(GD) / 任务管理器(TM)

[0167] ●触发条件:接收到有效的任务对象。

[0168] ●执行动作:根据任务的特性(例如，Prompt的复杂性、是否提及视觉元素、是否已知目标网站需要JS渲染、是否指定了输出Schema等)，动态构建一个ScrapeGraphAI的处理图，或者从库中选择一个最匹配的预定义图模板。图定义了将要执行的节点序列及其连接关系。例如，一个简单的任务可能只需要Fetch->Parse->AIAnalysis->Format的线性图；一个需要处理动态内容并提取视觉信息的任务可能需要Fetch(Browser)->Vision-

[0169] >AIAnalysis(withvisualcontext)->Format的图。

[0170] ●运行环境参数:可能需要访问配置库中的图模板。

[0171] ●运行原理 / 机制:基于规则或简单的智能判断来选择或构建最合适的处理流程。

[0172] ScrapeGraphAI本身提供了定义这种图状结构的机制。

[0173] ●作用 / 意义:为任务定制化地编排了数据处理的流水线。

[0174] ●结束条件:一个代表处理流程的图对象被成功创建或加载。

[0175] ●运行结果:生成一个待执行的图实例。

[0176] ●技术效果:提供了灵活、可定制的数据处理流程，能够适应不同任务的需求。

[0177] (3)执行获取节点(FetchNode)(Step3&3a)

[0178] ●执行主体:图执行器(GE)调用获取节点(FN)。

[0179] ●触发条件:图执行器调度到获取节点。

[0180] ●执行动作:

[0181] ○默认尝试使用标准HTTP库(如requests)向目标URL发送请求，获取HTML源码。处理重定向、设置User-Agent、管理Cookies、使用代理(如果配置了)。

[0182] ○如果HTTP请求失败，或者配置了强制浏览器渲染，或者初步分析HTML源码发现可能是JS重度依赖的页面(例如，body为空或只有少量脚本标签)，则自动或根据配置切换到使用浏览器自动化工具(如Playwright)(Step3a)。启动一个浏览器实例(可能是无头模式)，加载URL，等待页面完全渲染(可能需要配置等待特定元素出现或等待一段时间)，然后获取渲染后的DOM结构和 / 或页面截图。

[0183] ●运行环境参数:网络连接、代理服务器(可选)、浏览器环境(如果使用Playwright)。

[0184] ●运行原理 / 机制:模拟网络请求或浏览器行为来获取网页内容。动态选择获取方式以平衡效率和效果。

[0185] ●作用 / 意义:获取数据提取的基础——网页内容。

[0186] ●结束条件:成功获取到网页的HTML源码、渲染后的DOM或截图；或达到重试次数上限后失败。

[0187] ●运行结果:返回获取到的内容(文本、DOM对象、截图文件路径 / 数据)给图执行器，或者抛出错误。

[0188] ●技术效果:能够处理静态和动态网页，并通过智能切换策略尝试优化效率。

[0189] (4)执行解析节点(ParseNode)(Step4)

[0190] ●执行主体:图执行器(GE)调用解析节点(PN)。

[0191] ●触发条件:获取节点成功返回网页内容，图执行器调度到解析节点。

[0192] ●执行动作:对获取到的HTML源码或DOM进行清洗和预处理。这可能包括：

[0193] ○移除不相关的HTML标签(<script>,<style>,<header>,<footer>等，根据任务需求可配置)。

[0194] ○提取主要的文本内容，去除广告、导航栏等干扰信息。

[0195] ○将HTML转换为更简洁、对LLM更友好的格式，如Markdown。这有助于减少输入Token数量并突出内容结构。

[0196] ○如果需要，也可以保留DOM结构供后续更精细的分析。

[0197] ○对于非常长的页面，可能需要进行分块(chunking)处理，以适应LLM的上下文窗口限制。

[0198] ●运行环境参数:无特殊要求，主要是CPU密集型操作。

[0199] ●运行原理 / 机制:应用HTML解析库(如BeautifulSoup,lxml)和文本处理技术。

[0200] ●作用 / 意义:准备干净、有效、格式合适的输入数据给AI分析节点。

[0201] ●结束条件:完成对输入内容的解析和处理。

[0202] ●运行结果:返回处理后的文本、简化HTML、Markdown或其他格式的数据。

[0203] ●技术效果:提高了后续AI分析的效率和准确性，降低了噪声干扰。

[0204] (5)判断是否需要视觉信息(Step5)

[0205] ●执行主体:图执行器(GE)或AI分析节点(AIN)的前置逻辑。

[0206] ●触发条件:解析节点完成之后，准备进入AI分析阶段。

[0207] ●执行动作:根据任务配置(是否明确要求使用视觉模型)、Prompt中是否包含视觉相关的描述(如"图片中的文字”、"按钮的颜色”)、或者基于对页面复杂度 / 类型的启发式判断，决定是否需要调用视觉节点。

[0208] ●运行环境参数:无。

[0209] ●运行原理 / 机制:基于预设规则或简单逻辑判断。

[0210] ●作用 / 意义:决定是否引入多模态信息来辅助数据提取。

[0211] ●结束条件:做出决策(是 / 否)。

[0212] ●运行结果:控制流程转向视觉节点或直接跳到AI分析节点。

[0213] ●技术效果:实现了根据需要智能启用视觉分析，优化资源使用。

[0214] (6)执行视觉节点(VisionNode)(Step6)(如果Step5判断需要)

[0215] ●执行主体:图执行器(GE)调用视觉节点(VN)。

[0216] ●触发条件:Step5决策需要视觉信息，且获取节点已提供了页面截图。

[0217] ●执行动作:

[0218] ○接收页面截图和相关的Prompt(可能是原始任务Prompt的一部分或专门为视觉分析设计的子Prompt，例如："请描述这张截图中主要商品区域的布局”或"识别图片中红色的购买按钮”)。

[0219] ○调用AI模型服务层(VMS)提供的视觉模型接口(如GPT-4VAPI)，将截图和Prompt发送给模型。

[0220] ○接收视觉模型的分析结果，这可能是对图像的描述、识别出的特定元素的位置或内容、OCR提取的文字等。

[0221] ●运行环境参数:需要访问视觉模型API，可能需要处理图像数据。

[0222] ●运行原理 / 机制:利用先进的多模态AI模型理解图像内容。

[0223] ●作用 / 意义:提取纯文本分析无法获取的视觉信息，或提供视觉上下文。

[0224] ●结束条件:收到视觉模型的响应或处理超时 / 失败。

[0225] ●运行结果:返回视觉分析的结果(文本描述、坐标、识别内容等)，可能作为后续AI分析节点的附加输入。

[0226] ●技术效果:增强了对富含视觉信息或布局复杂的网页的处理能力。

[0227] (7)执行AI分析节点(AIAnalysisNode)(Step7)

[0228] ●执行主体:图执行器(GE)调用AI分析节点(AIN)。

[0229] ●触发条件:解析节点(以及可能的视觉节点)完成处理，图执行器调度到AI分析节点。

[0230] ●执行动作:

[0231] ○准备输入:整合来自解析节点的处理后文本内容(如Markdown或简化HTML)和用户的原始Prompt / Schema。如果视觉节点被执行且返回了结果，也可能将视觉分析结果作为额外的上下文信息(例如，在Prompt中加入"根据之前的视觉分析，价格位于页面右上角的红色区域...”)。

[0232] ○构造LLM请求:精心设计发送给LLM的最终Prompt。这可能包括：

[0233] ■角色设定:指示LLM扮演一个"网页数据提取助手”。

[0234] ■任务描述:清晰地陈述需要提取的数据项(来自用户Prompt或Schema)。

[0235] ■输入内容:提供处理后的网页文本内容(可能需要分块处理长文本，并维护上下文)。

[0236] ■输出格式要求:指示LLM以特定格式返回结果(如JSON对象、列表等)，即使最终由格式化节点处理，预先指示有助于提高LLM输出的结构性。

[0237] ■附加指令:可能包括处理模糊情况的策略、数据清洗要求等。

[0238] ○调用LLM:通过AI模型服务层(LLMS)将构造好的请求发送给选定的LLM(如GPT-4,Claude3,Llama3等)。

[0239] ○接收并初步处理响应:获取LLM返回的包含提取数据的响应文本。可能需要进行初步的解析或校验。

[0240] ●运行环境参数:需要访问LLMAPI。处理可能涉及大量文本，对内存有一定要求。

[0241] ●运行原理 / 机制:核心步骤，利用LLM的自然语言理解和生成能力，根据用户意图从非结构化或半结构化的网页内容中智能提取信息。这是替代传统CSS / XPath规则的关键所在。

[0242] ●作用 / 意义:实现基于语义理解的智能数据提取，是本发明鲁棒性和易用性的主要来源。

[0243] ●结束条件:收到LLM的响应或处理超时 / 失败。

[0244] ●运行结果:返回LLM提取出的原始数据(通常是文本形式，可能包含某种程度的结构)。

[0245] ●技术效果:实现了不依赖固定页面结构的数据提取，提高了鲁棒性、适应性，并支持自然语言任务定义。

[0246] (8)执行格式化节点(FormatNode)(Step8)

[0247] ●执行主体:图执行器(GE)调用格式化节点(FMT)。

[0248] ●触发条件:AI分析节点成功返回提取的原始数据。

[0249] ●执行动作:

[0250] ○接收来自AI分析节点的原始输出(可能是一段自然语言描述、一个简单的列表、或者LLM尝试生成的JSON / Markdown等)。

[0251] ○根据用户最初在任务中定义的Schema(如果提供了)，或者根据Prompt中隐含的结构信息，或者使用预定义的规则，对原始数据进行解析、清洗、转换和格式化。

[0252] ○例如，如果LLM返回"标题是'X',作者是'Y'"，格式化节点会将其转换为{"title":"X","author":"Y"}的JSON对象。

[0253] ○可能需要处理数据类型转换、去除多余空格、标准化日期格式等。

[0254] ●运行环境参数:无特殊要求。

[0255] ●运行原理 / 机制:应用数据处理逻辑将AI的自由格式输出转化为规整的结构化数据。

[0256] ●作用 / 意义:确保输出的数据符合预期格式，便于后续使用。

[0257] ●结束条件:完成数据格式化。

[0258] ●运行结果:返回最终的结构化数据(如JSON对象、CSV行、字典列表等)。

[0259] ●技术效果:提供了规范、一致的数据输出。

[0260] (9)存储 / 返回结果(Step9)

[0261] ●执行主体:图执行器(GE) / 数据存储模块 / 任务管理器(TM)。

[0262] ●触发条件:格式化节点成功返回结构化数据，或流程中发生不可恢复的错误。

[0263] ●执行动作:

[0264] ○将格式化后的结构化数据存储到指定位置，如数据库、文件系统(CSV,JSONLines等)、消息队列等。

[0265] ○更新任务状态(成功、失败、部分成功)。

[0266] ○记录相关的日志信息，包括执行的图、使用的模型、消耗的资源、遇到的错误等。

[0267] ○如果通过API调用，将结果或任务状态返回给调用方。

[0268] ○处理错误情况(ErrorHandler)：记录详细错误信息，根据策略可能进行有限次数的重试(例如，针对网络波动或AI模型临时不可用)，或者将错误信息和部分成功的结果(如果有的话)一起保存。

[0269] ●运行环境参数:可能需要访问数据存储系统。

[0270] ●运行原理 / 机制:完成数据采集任务的收尾工作。

[0271] ●作用 / 意义:使采集到的数据持久化并可供后续分析使用，同时提供任务执行反馈。

[0272] ●结束条件:数据和状态信息被成功保存 / 返回。

[0273] ●运行结果:任务完成，数据可用。

[0274] ●技术效果:完成了整个数据采集闭环。

[0275] 错误处理与重试机制(ErrorHandler)

[0276] 贯穿整个流程，每个节点在执行过程中都可能遇到错误(网络超时、解析失败、AI模型API错误、速率限制、内容不符合预期等)。本方法包含健壮的错误处理机制：

[0277] ●节点级错误捕获:每个节点执行时都应能捕获预期的异常。

[0278] ●重试策略:对于可恢复的错误(如临时网络问题、AI模型繁忙)，可配置基于指数退避的重试策略。

[0279] ●状态记录:记录详细的错误信息、发生错误的节点、当时的输入数据等，便于调试。

[0280] ●流程控制:根据错误严重性和配置，决定是终止当前任务、跳过当前URL / 数据项、还是尝试替代路径(例如，如果LLM提取失败，是否尝试用更简单的规则或另一个模型)。

[0281] ●部分成功:即使任务整体失败，也应尽量保存已成功提取的部分数据。

[0282] 序列图示例

[0283] 为了更清晰地展示核心交互，图4提供一个简化的序列图，重点关注用户、ScrapeGraphAI引擎、AI模型之间的交互。

[0284] 图4:核心数据提取流程序列图(简化版)

[0285] 通过上述详细阐述的技术方案，本发明利用ScrapeGraphAI的图执行框架，有机地结合了传统的网页获取、解析技术与先进的AI(LLM和VisionModel)分析能力，构建了一个全新的、智能化的网络数据自动化采集流程。该流程以AI的语义理解为核心驱动力，旨在克服现有技术的诸多局限性，实现更鲁棒、更易用、更自适应的数据采集。

[0286] 本发明还提供了一种存储器，存储有多条指令，指令用于实现如实施例一的方法。

[0287] 如图5所示，本发明还提供了一种电子设备，包括处理器301和与处理器301连接的存储器302，存储器302存储有多条指令，指令可被处理器加载并执行，以使处理器能够执行如实施例一的方法。

[0288] 最后应说明的是：以上各实施例仅用以说明本发明的技术方案，而非对其限制；尽管参照前述各实施例对本发明进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分或者全部技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本发明各实施例技术方案的范围。< / script>

Claims

1. A ScrapeGraphAI-based network data automatic collection method, characterized in that, Based on the ScrapeGraphAI framework or graph execution engine, the data collection process is represented as a graph composed of nodes and edges, facilitating the modularization, combination, execution and management of the process, including: S1, receiving an identifier of at least one target network resource and a data extraction requirement; S2, based on the data extraction requirement, constructing or selecting a processing graph containing at least one acquisition node, an AI analysis node and a formatting node; wherein the acquisition node has the ability to dynamically select the acquisition strategy, which can select or switch between standard HTTP request acquisition and rendering based on browser automation tools according to the content characteristics of the target network resource, task configuration or historical execution; S3, executing the acquisition node to acquire the content of the target network resource; S4, executing the AI analysis node, including: using at least one large language model LLM to perform semantic analysis on the acquired content of the target network resource based on the data extraction requirement to identify and extract target data, wherein the identification and extraction of the target data is based on the semantics of the content rather than a pre-defined fixed structure selector; S5, executing the formatting node to convert the target data extracted by the AI analysis node into a predetermined or inferred structured format data; S6, outputting the structured format data.

2. The network data automatic collection method based on ScrapeGraphAI according to claim 1, characterized in that, The data extraction requirement is provided in the form of a natural language prompt or a structured pattern.

3. The network data automatic collection method based on ScrapeGraphAI according to claim 2, characterized in that, For the data extraction requirement provided in the form of a natural language prompt, the behavior and possible graph structure of the AI analysis node are automatically configured based on the data extraction requirement prompt, and the AI analysis node directly uses the natural language prompt to guide the large language model LLM to perform semantic analysis and data extraction.

4. The network data automatic collection method based on ScrapeGraphAI according to claim 3, characterized in that, The construction or selection of the processing graph is automatically performed according to the characteristics of the data extraction requirement, which realizes the automatic mapping from user intent to execution process; wherein the characteristics of the data extraction requirement include one or more of complexity and involved concepts.

5. The ScrapeGraph AI-based network data automatic collection method according to claim 4, characterized in that, The processing graph also contains a visual node, which is executed before or in parallel with the AI analysis node; the visual node is used to acquire the visual representation of the target network resource and analyze it using at least one visual model; the AI analysis node further combines the visual analysis results output by the visual node as visual context when performing semantic analysis using LLM, to improve the accuracy of data extraction or process visual-dependent information.

6. The network data automatic collection method based on ScrapeGraphAI according to claim 5, characterized in that, The processing graph also contains a parsing node, which is executed after the acquisition node and before the AI analysis node; the parsing node is used to clean up, simplify or block the acquired content of the target network resource, to generate a format more suitable for input to the large language model LLM for analysis.

7. A ScrapeGraph AI-based network data automatic collection system for implementing the method of any one of claims 1-6. Including: Interface module for receiving user tasks containing URL and extraction requirements; Graph definition module for constructing or selecting a processing graph according to the user tasks; A node library containing base nodes including an acquisition node, an AI analysis node, and a formatting node; A graph executor for scheduling and executing the base nodes in a processing graph; An AI model service interface for calling external or local large language models (LLMs) and / or visual models; wherein the AI analysis node is configured to core rely on the semantic understanding capability of the large language model (LLM) for data extraction, rather than a fixed structure selector; A data processing module for storing or outputting structured format data as a result.

8. The system of claim 7, wherein, The system further contains a visual node for providing visual analysis capability and an interface for calling a visual model to assist the large language model (LLM) of the AI analysis node for more accurate data extraction.

9. An electronic device, comprising: A processor and a memory, the memory storing a plurality of instructions, the processor being configured to read the instructions and perform the method of any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions that can be read and executed by a processor to perform the method of any one of claims 1-6.

Images