A website data analysis method and device based on large model technology

Abstract

The application discloses a website data analysis method and device based on large model technology, and belongs to the field of network crawler and artificial intelligence fusion. The method drives a large model to generate accurate XPath rules through a local knowledge base, and encapsulates the rules into decoupled plug-ins to be integrated into an existing crawler framework. A dynamic feedback optimization mechanism is innovatively designed, a composite verifier is used to monitor rule validity in real time, and XPath reconstruction and knowledge base updating are automatically triggered. The application can retain the high-concurrency performance of a traditional crawler, significantly improve rule generation efficiency and system robustness, reduce manual maintenance cost, and be suitable for website data collection scenes with complex structure or frequent updates.

Description

Technical Field

[0001] This invention relates to the field of web crawling and artificial intelligence integration technology, and mainly to a website data parsing method and apparatus based on large model technology. Background Technology

[0002] Currently, most data acquisition companies still rely on traditional frameworks such as Scrapy and BeautifulSoup for data acquisition. These systems offer mature high-concurrency capabilities. However, with the rapid increase in website complexity and frequency of changes, manual rule maintenance is costly. Directly using large models to replace traditional solutions for data acquisition results in significantly longer single-page processing times compared to traditional systems, high token costs, and missing rates of some key fields due to model output illusions, failing to meet the accuracy requirements of data acquisition. Therefore, an innovative solution is urgently needed: while retaining frameworks like Scrapy, intelligently upgrading only the XPath generation stage to achieve the dual advantages of "fast acquisition using traditional frameworks + dynamic parsing of large models."

[0003] Existing data collection systems based on traditional XPath solutions offer high concurrency and efficient parsing capabilities, but heavily rely on manual writing and maintenance of XPath rules. When faced with a large number of new websites or frequently redesigned web pages, the labor costs increase exponentially and the system has poor adaptability. On the other hand, LLM-based suggestion engineering requires the construction of different suggestion words for different types of websites, resulting in slow execution speed, high token costs, and the inherent uncertainty of large models can lead to insufficient system stability. Both methods struggle to balance automation and economic feasibility. Summary of the Invention

[0004] To address the above shortcomings, this invention upgrades the web crawler system through "intelligent XPath rule generation": On the one hand, based on a local knowledge base and Prompt engineering guidance technology, only the target URL needs to be input to automatically open the website and output high-precision XPath rules (such as precise positioning of the landing page, pagination, and data elements), covering diverse scenarios such as news and e-commerce, significantly reducing manual maintenance costs; on the other hand, it innovatively integrates a dynamic adaptation and feedback closed-loop mechanism, monitors the validity of rules (such as content null value rate) through a real-time verification module, automatically triggers the XPath reconstruction mechanism, and supports manual review and confirmation in key scenarios, forming a self-evolving system of "execution-feedback-optimization," seamlessly connecting to existing web crawler frameworks with decoupled plugins (compatible with JS rendering, proxy rotation, etc.), solving the adaptability problem of dynamic changes in website structure while retaining the advantages of high concurrency. According to the first aspect of this invention, a website data parsing method based on large model technology is proposed, the specific steps of which are as follows:

[0005] Receive the target webpage URL and perform a knowledge base query on the URL; if a corresponding XPath rule exists in the knowledge base, output the existing rule; if not, perform intelligent semantic analysis and dynamic Prompt construction to generate XPath structured rules.

[0006] The generated XPath rules are validated. If the validation is successful, they are stored in the XPath knowledge base of each domain. If the validation fails, re-analysis and rule updates are triggered.

[0007] The generated or matched XPath rules are applied to the data collection system, and data is extracted from the HTML content of the target webpage through the XPath plugin;

[0008] The extraction results are subjected to composite verification. If multiple web pages are detected to have extraction deviations or data anomalies, the automatic reconstruction unit is triggered to regenerate XPath rules based on changes in the web page structure.

[0009] Preferably, receiving the target webpage URL specifically involves: receiving the target website URL input by the user, calling a lightweight headless browser to obtain the initial HTML document; after receiving the HTML document, performing the following steps:

[0010] Extract Open Graph protocol and traditional metadata; if a definite type identifier exists, return the result directly.

[0011] If there is no definite metadata type identifier, the keyword features in the title and body are analyzed; if there is a definite type identifier, the results are returned directly.

[0012] If the semantic features still cannot determine the type, then analyze the DOM structural features and return the type result.

[0013] Preferably, if the semantic features still cannot determine the type, then the DOM structure features are analyzed to return the type result. Specifically, this further includes: performing DOM tree normalization processing on the HTML document, including removing comment nodes, whitespace text nodes, and visually hidden elements, and generating a set of visible nodes V = {v1, v2, ..., v...} m}, where m≤500; perform similarity matching between the standardized DOM feature vector and the pre-set vertical domain DOM structure feature library in the local knowledge base to obtain the final type.

[0014] Preferably, the dynamic Prompt construction specifically includes: customizing the Prompt based on the domain; extracting key anchors and path patterns from the XPath structural features of each template according to the identified website type and the Top-3 high similarity templates; the key anchors are a set of domain-specific keywords in the CSS class name or ID attribute, and the path pattern is the relative position expression of the target field in the DOM tree; using the feature samples as target references, constructing structured prompt words according to the preset Prompt template.

[0015] Preferably, the generation of XPath structured rules specifically includes: inputting HTML fragments and Prompt templates into a large model, outputting constraints as a pure XPath rule set, and saving the corresponding URL and the parsed XPath as keys to a knowledge base.

[0016] Preferably, the step of extracting data from the HTML content of the target webpage using an XPath plugin further includes: encapsulating the generated XPath rules into an independent plugin file and decoupling and integrating it into the crawler framework; the crawler framework calling the independent plugin file to perform webpage parsing and extract target field data based on the XPath rules.

[0017] Preferably, the composite verification of the extraction results further includes: real-time monitoring of rule validity through a composite validator; automatically triggering a large model reconstruction process when the null value rate of a field exceeds a threshold for a consecutive preset number of pages; extracting the HTML fragment of the current page and generating a new XPath rule by combining it with historical successful rules; and after verification, archiving the new rule to the local knowledge base and replacing the invalid rule, which also includes manual review intervention.

[0018] Preferably, the automatic reconstruction unit further includes: extracting the HTML content of the current page, generating a new Prompt by combining it with historical success rules, outputting a new XPath in the large model, performing sandbox testing with a composite validator, and entering the knowledge base if the verification is successful, which also includes manual review before entering the knowledge base.

[0019] According to a second aspect of the present invention, a website data parsing apparatus based on large model technology is proposed, specifically comprising:

[0020] The intelligent rule generation module is used to receive the target webpage URL, perform a knowledge base query on the URL; if the corresponding XPath rule exists in the knowledge base, the existing rule is output; if it is not included, intelligent semantic analysis and dynamic Prompt construction are performed to generate XPath structured rules.

[0021] The verification module is used to verify the generated XPath rules. If the verification is successful, the rules are stored in the XPath knowledge base of each domain. If the verification fails, the re-analysis and rule update are triggered.

[0022] The data extraction module is used to apply the generated or matched XPath rules to the data acquisition system and extract data from the HTML content of the target webpage through the XPath plugin;

[0023] The dynamic feedback optimization module is used to perform composite verification on the extraction results. If multiple web pages are detected to have extraction deviations or data anomalies, the automatic reconstruction unit is triggered to regenerate XPath rules based on changes in the web page structure.

[0024] According to a third aspect of the present invention, a computer program product is provided, on which one or more computer programs are stored, which, when executed by a computer processor, implement the method described above.

[0025] The above-described one or more technical solutions in the embodiments of this application have at least one of the following technical effects:

[0026] Highly efficient and accurate rule generation: Based on the domain features of the local knowledge base, the large model outputs high-precision XPath rules (such as / / h1[@id="title"] / text()), without the need for manual feature engineering or seed pages, covering diverse website scenarios, and with high rule generation accuracy.

[0027] Dynamic adaptive capability: Through the "execution-feedback-optimization" closed-loop mechanism, the repair function is automatically activated when the missing rate of key fields decreases, which significantly enhances the robustness of the system.

[0028] Seamless system integration: It adopts a decoupled plug-in design, which is fully compatible with the high-concurrency architecture, JS rendering and proxy rotation functions of existing crawler frameworks.

[0029] Economic feasibility optimization: Intelligent upgrades are made only to the XPath generation process, avoiding large-scale model calls across the entire chain, significantly reducing operation and maintenance costs and token consumption, making it suitable for large-scale data collection scenarios. Attached Figure Description

[0030] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain the principles of the invention. Other embodiments and many anticipated advantages of the embodiments will be readily recognized as they become better understood through reference to the following detailed description. Elements in the drawings are not necessarily to scale. The same reference numerals refer to corresponding similar parts.

[0031] Figure 1A flowchart illustrating a website data parsing method based on large model technology according to an embodiment of the present invention is shown.

[0032] Figure 2 A schematic diagram of the process for calculating an integrity score using a composite verifier according to an embodiment of the present invention is shown.

[0033] Figure 3 This is a schematic diagram of the structure of a computer system suitable for implementing the electronic devices of the present application embodiments. Detailed Implementation

[0034] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0035] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0036] The purpose of this invention is to provide a website data parsing method and apparatus based on large model technology. Through a local knowledge base-driven XPath intelligent generation and dynamic feedback optimization mechanism, it solves the problems of high rule maintenance costs, weak generalization ability, and insufficient system integration in existing technologies. This invention achieves automation, accuracy, and adaptability in web page data extraction while maintaining the high concurrency performance of traditional crawler frameworks. Figure 1 A flowchart illustrating a website data parsing method based on large model technology according to an embodiment of the present invention is shown, such as... Figure 1 As shown:

[0037] S1. Obtain the initial HTML from the received target website URL using a lightweight headless browser; based on the vertical domain DOM structure feature library (covering news, e-commerce, social media, etc.) in the local knowledge base, automatically identify the website type and dynamically construct a targeted Prompt template; the local knowledge base stores triple structure data <website type, DOM template ID, XPath structure feature>, where the XPath structure feature is represented by a parameterized regular expression template, and the key anchor points and path patterns in the XPath structure feature; input the HTML content and Prompt template into the large model to generate a structured XPath rule set, which includes target field positioning rules, pagination navigation rules, and page turning control rules, avoiding the problem of insufficient generalization caused by general prompt words. The specific steps are as follows:

[0038] S1.1 Initial HTML Acquisition and Website Type Identification;

[0039] Receive the target website URL (e.g., https: / / example-news.com) input by the user, and invoke a lightweight headless browser (such as Playwright) to obtain the initial HTML document. After receiving the HTML document, execute the following steps in sequence:

[0040] S1.1.1 Extract Open Graph protocol and traditional metadata; if there is a clear type identifier, return the result directly.

[0041] Open Graph protocol metadata: identification <meta property＝"og:type”> Extract the content attribute value from the tag;

[0042] Traditional metadata: identification <meta name＝"description”> and <meta name＝"keywords”> Extract the content attribute value from the tag.

[0043] S1.1.2 If there is no explicit metadata type identifier, analyze the keyword features in the title and body; if there is an explicit type identifier, return the results directly.

[0044] Title characteristics: Analysis <title> Extract title text features from tag content;< / title>

[0045] Text features: Analyze the visible text content in the analysis area and identify a set of preset keyword types.

[0046] S1.1.3 If the semantic features still cannot determine the type, analyze the DOM structure features and return the type result. Perform DOM tree normalization processing on the HTML document, including removing comment nodes, whitespace text nodes, and visually hidden elements, and generate a set of visible nodes V = {v1, v2, ..., v...} m}, where v i Let m represent the i-th node, where m ≤ 5000; perform similarity matching between the standardized DOM feature vector and the pre-set vertical domain DOM structure feature library in the local knowledge base to obtain the final type.

[0047] S1.2, Dynamic Prompt Construction;

[0048] Customized Prompts are constructed based on the domain. According to the identified website type and the top-3 highly similar templates, key anchor points and path patterns are extracted from the XPath structural features of each template. The key anchor points are a set of domain-specific keywords in CSS class names or ID attributes, and the path patterns are the relative position expressions of the target fields in the DOM tree. Using a small number of feature samples as examples, structured prompts are constructed according to a preset Prompt template, which includes the following mandatory constraints:

[0049] Please strictly follow the instructions below:

[0050] 1) Target field: {field_name} (Example: News title);

[0051] 2) Anchor point constraint: Must contain {anchor_points} (e.g., class or id contains "headline|title");

[0052] 3) Path pattern: Prefer {path_pattern} (e.g., / h1 or / div>h2);

[0053] 4) Fault tolerance requirement: When the primary path fails, the alternative path is {fallback_path};

[0054] 5) Prohibition rules: Exclude nodes containing advertising characteristics (classes containing "ad", "sponsored", etc.);

[0055] 6) Output format: Only output the XPath expression, without any interpretation.

[0056] HTML snippet:

[0057] {Truncation of HTML, limited to <2000 characters}.

[0058] The dynamic Prompt construction technique of this invention pre-embeds domain semantic features, structural anchors, and path constraints into a Prompt template, enabling the model to accurately identify key nodes in the webpage DOM structure and avoid noise interference when generating XPath rules, thereby significantly improving the accuracy and generality of rule generation. This mechanism combines knowledge guidance and structural constraints at the input stage, reducing the randomness and ambiguity of model output, minimizing the cost of manual correction and repeated training, and giving the generated XPath rules higher transferability, adaptability, and validation success rate.

[0059] S1.3, Structured rule output;

[0060] The HTML fragment and Prompt template are input into the large model, and the constraints are output as a pure XPath rule set. The corresponding URL and the parsed XPath are saved as key pairs in the knowledge base. For example, XPath rules generated for e-commerce websites:

[0061] {

[0062] "item_list":" / / ul[contains(@class,'product-list')] / li",

[0063] "product_name":". / / h2[@class='product-title'] / text()",

[0064] "next_page":" / / a[@rel='next'] / @href"

[0065] }

[0066] S2. Encapsulate the generated XPath rules into an independent plugin file and decouple and integrate it into the crawler framework; the crawler framework calls the plugin file to perform webpage parsing and extract target field data based on the XPath rules; this achieves decoupling and integration with the existing crawler system while retaining the performance advantages of traditional frameworks. The specific steps are as follows:

[0067] S2.1. Convert the generated XPath rule set into a standardized plugin file that conforms to the Scrapy middleware interface specification; the plugin file contains:

[0068] The `parse_item` method extracts target fields based on XPath rules.

[0069] The `follow_pagination` method handles pagination logic.

[0070] Metadata configuration: Specify the knowledge base version and verification threshold.

[0071] The validated XPath rule set is automatically converted into a standardized plugin file that conforms to the Scrapy middleware interface specification, specifically including:

[0072] S2.1.1 will perform semantic layering and field mapping using XPath rules generated and verified by dynamic prompts, and generate an executable Scrapy plugin file;

[0073] S2.1.2 Automatically generate the parse_item method based on the field extraction logic of XPath rules, which is used to dynamically extract target field data based on node paths during the crawling process;

[0074] S2.1.3 Automatically analyze the pagination element path based on the page structure pattern, generate the follow_pagination method, and realize automatic identification and recursive crawling of multi-page content;

[0075] S2.1.3. Write the knowledge base version identifier, rule generation timestamp, verification confidence threshold and adapted website structure summary into the plugin file to achieve cross-version rule tracking and consistency verification.

[0076] S2.1.4 Automatically monitor data extraction deviations during Scrapy runtime. When the field matching rate is detected to be lower than the confidence threshold, trigger the dynamic feedback optimization module to reconstruct rules and regenerate plugins.

[0077] This invention eliminates the need for manual intervention throughout the entire process from rule generation to Scrapy plugin formation, ensuring universality and cross-website portability. The plugin file itself contains verification thresholds and triggering conditions, enabling it to "self-detect" anomalies during operation and trigger model optimization in reverse. Each plugin can be traced back to a specific knowledge base version, facilitating rule tracing and difference management across multiple domains or multi-source websites.

[0078] S2.2. After loading the corresponding plugin file, the existing web crawling system (such as a Scrapy cluster) initiates an HTTP request and processes JS rendering according to the usual process. When the webpage response is obtained, the plugin is invoked to perform XPath parsing:

[0079] defparse(self, response):

[0080] item={}

[0081] for key,value in plugin_rules.items():

[0082] item[key]=response.xpath(plugin_rules[key]).get()#'title'

[0083] #...Extracting other fields

[0084] return item

[0085] The `parse()` method is responsible for implementing the intelligently generated XPath rules within the physical structure of the webpage. Through plugin files loaded by the Scrapy cluster, this method maps the abstracted XPath rules from the knowledge base onto the DOM tree of the webpage response, achieving the transformation from HTML structure to semantic data fields. This allows the system to automatically locate and extract target information in the actual structural space of the webpage, completing the physical transformation from "rule knowledge" to "data entities."

[0086] S3. During the data extraction process, the validity of rules is monitored in real time through a composite validator. When the null value rate of a field exceeds a threshold for a consecutive preset number of pages, the large model reconstruction process is automatically triggered. The current page's HTML fragment is extracted and combined with historically successful rules to generate new XPath rules. After verification, the new rules are archived to the local knowledge base and replace invalid rules. Manual review is supported in key scenarios to ensure the long-term stable operation of the system. The specific steps are as follows:

[0087] S3.1, Real-time monitoring layer;

[0088] Embed a composite validator in the data pipeline to calculate an integrity score for each page of extracted results:

[0089] Field integrity check: Number of non-null fields / Total number of target fields;

[0090] Content format validation: Detects whether the data format is correct using a lightweight NLP model;

[0091] Content rationality check: This includes factors such as sentence length in the main text, next page links, and title content.

[0092] When multiple consecutive pages have scores below a threshold, such as Figure 2 As shown, this triggers the XPath rule reconstruction process.

[0093] S3.2 Automatic reconfiguration unit;

[0094] The current page's HTML content is extracted, and a new Prompt is generated by combining it with historical success rules. After the new XPath is output from the large model, the composite validator performs sandbox testing (using cached page samples). If the verification passes, it is added to the knowledge base, and in some key scenarios, manual review is supported before it is added to the knowledge base.

[0095] According to one embodiment of the present invention, taking the parsing of a news portal website as an example:

[0096] Input the target webpage URL https: / / news.example.com, and the system will quickly identify the URL and page structure using a pre-trained webpage classification model. It will recognize the website as a "news" page type and automatically load the XPath knowledge template and rule priority parameters corresponding to the news domain.

[0097] The system retrieves structured feature samples related to "news title," "author information," and "publication time" from the local knowledge base. For example, the reference rule for the title field is / / h1[contains(@class,'main-headline')]. Based on the matched template, the dynamic Prompt construction module automatically generates a targeted Prompt, injecting domain semantic anchors (such as headline and title), path style constraints, and fault tolerance strategies into the Prompt template, providing structured guidance for subsequent model generation.

[0098] Under the constraints of the Prompt, the large model performs structural analysis and XPath path generation, outputting an XPath rule set containing multiple candidate rules. The generated results undergo syntax validation and adaptability screening, selecting high-confidence rules that pass validation and automatically encapsulating them into a standardized plugin file conforming to the Scrapy middleware interface specification. This plugin file contains the `parse_item()` and `follow_pagination()` methods, enabling the invocation of corresponding XPath expressions during crawler execution to achieve data extraction and pagination tracking.

[0099] Once the Scrapy cluster loads the plugin, it initiates HTTP requests and handles JavaScript rendering according to the standard crawling process. When the webpage response arrives, the crawler automatically calls the XPath rules in the plugin to parse the webpage's DOM tree nodes and extract the target field data. Simultaneously, the system's built-in composite verification module performs real-time monitoring and statistical analysis of the collected results. For example, if the system detects that the proportion of empty values ​​in the title field's extracted results exceeds 80% across five consecutive pages, it determines that the current rule is invalid, speculating that the webpage structure may have been redesigned, causing the original path (e.g., id="title") to be removed.

[0100] Once the rule expiration trigger condition is met, the system automatically activates the dynamic optimization module. This module extracts the latest webpage HTML structure, re-executes the intelligent prompt construction and XPath rule generation process, and outputs new valid rules after verification (e.g., updated to / / header / h1 / text()). After the generated new rules undergo stability and accuracy verification in a sandbox environment, they are automatically written into the XPath knowledge base to replace the old version rules. The entire reconstruction and update process requires no manual intervention, achieving self-learning, self-repair, and continuous evolution of the rules, thereby maintaining the long-term stability and high-precision data extraction capabilities of the crawling system.

[0101] Based on the above-described parsing method, this invention also discloses a website data parsing device based on large model technology, specifically comprising:

[0102] The intelligent rule generation module is used to receive the target webpage URL, perform a knowledge base query on the URL; if the corresponding XPath rule exists in the knowledge base, the existing rule is output; if it is not included, intelligent semantic analysis and dynamic Prompt construction are performed to generate XPath structured rules.

[0103] The verification module is used to verify the generated XPath rules. If the verification is successful, the rules are stored in the XPath knowledge base of each domain. If the verification fails, the re-analysis and rule update are triggered.

[0104] The data extraction module is used to apply the generated or matched XPath rules to the data acquisition system and extract data from the HTML content of the target webpage through the XPath plugin;

[0105] The dynamic feedback optimization module is used to perform composite verification on the extraction results. If multiple web pages are detected to have extraction deviations or data anomalies, the automatic reconstruction unit is triggered to regenerate XPath rules based on changes in the web page structure.

[0106] In summary, the innovation of this invention lies in:

[0107] By combining a knowledge base from a vertical domain, an intelligent XPath generation module was designed. This module requires only the target URL as input, determines the type, constructs a dynamic Prompt based on similar templates, and automatically generates high-precision XPath rules, significantly improving the automation and generalization capabilities of website structure parsing.

[0108] A composite validation-rule reconstruction mechanism was designed. This mechanism uses a composite validator to monitor multi-dimensional indicators such as field coverage, content rationality, and content format in real time, enabling automatic detection of invalid XPath rules. Simultaneously, invalid XPath rules are intelligently repaired, enhancing the system's robustness in scenarios with dynamic changes in website structure.

[0109] We propose a decoupled plugin integration architecture that encapsulates the XPath rules generated by large models into standardized middleware, seamlessly connecting with traditional crawling frameworks such as Scrapy and fully inheriting their high-concurrency scheduling capabilities.

[0110] While retaining the high efficiency of traditional web crawler frameworks, this invention achieves a breakthrough in website data parsing technology through the dual innovation of intelligent XPath rule generation and dynamic feedback optimization, providing a technical solution that combines accuracy, adaptability, and economic feasibility for data collection from large-scale, multi-type websites.

[0111] The following is for reference. Figure 3 It shows a schematic diagram of the structure of a computer system 300 suitable for implementing electronic devices according to embodiments of the present application. Figure 3The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0112] like Figure 3 As shown, the computer system 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 302 or programs loaded from storage section 308 into random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the system 300. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0113] The following components are connected to I / O interface 305: an input section 306 including a keyboard, mouse, etc.; an output section 307 including a liquid crystal display (LCD) and speakers, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN card and a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to I / O interface 305 as needed. A removable medium 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 310 as needed so that computer programs read from it can be installed into storage section 308 as needed.

[0114] Specifically, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 309, and / or installed from removable medium 311. When the computer program is executed by central processing unit (CPU) 301, it performs the functions defined in the methods of this application. It should be noted that the computer-readable storage medium of this application can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable storage medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0115] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0116] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0117] The modules described in the embodiments of this application can be implemented in software or in hardware.

[0118] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently without being assembled into the electronic device. The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by the electronic device, the electronic device causes the following: receiving a target webpage URL; querying the URL in a knowledge base; if a corresponding XPath rule exists in the knowledge base, outputting the stored rule; if not, performing intelligent semantic analysis and dynamic prompt construction to generate XPath structured rules; verifying the generated XPath rules, storing them in the XPath knowledge base of each domain if successful, and triggering re-analysis and rule updates if verification fails; applying the generated or matched XPath rules to the acquisition system, extracting data from the target webpage's HTML content through an XPath plugin; performing composite verification on the extraction results, and if multiple webpages are detected to have extraction deviations or data anomalies, triggering an automatic reconstruction unit to regenerate XPath rules based on changes in the webpage structure.

[0119] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A website data parsing method based on large model technology, characterized in that, include: Receive the target webpage URL and perform a knowledge base query on the URL; If a corresponding XPath rule exists in the knowledge base, output the existing rule. If not included, intelligent semantic analysis and dynamic Prompt construction are performed to generate XPath structured rules; The generated XPath structured rules are validated. If the validation is successful, they are stored in the XPath knowledge base of each domain. If the validation fails, re-analysis and rule updates are triggered. The generated or matched XPath structured rules are applied to the data collection system, and data is extracted from the HTML content of the target webpage through the XPath plugin; The extraction results are subjected to composite verification. If multiple web pages are detected to have extraction deviations or data anomalies, the automatic reconstruction unit is triggered to regenerate XPath structured rules based on changes in the web page structure. The specific steps for receiving the target webpage URL are: receiving the target webpage URL input by the user, calling a lightweight headless browser to obtain the initial HTML document; and after receiving the HTML document, performing the following steps: Extract Open Graph protocol and traditional metadata; if a definite type identifier exists, return the result directly. If there is no definite metadata type identifier, the keyword features in the title and body are analyzed; if there is a definite type identifier, the results are returned directly. If the semantic features still cannot determine the type, then analyze the DOM structural features and return the type result; The dynamic Prompt construction specifically includes: customizing the Prompt based on the domain; extracting key anchors and path patterns from the XPath structural features of each template according to the identified website type and the Top-3 high similarity templates; the key anchors are a set of domain-specific keywords in the CSS class name or ID attribute, and the path pattern is the relative position expression of the target field in the DOM tree; using the feature samples as target references, constructing structured prompt words according to the preset Prompt template; The process of extracting data from the HTML content of a target webpage using an XPath plugin further includes: encapsulating the generated XPath structured rules into an independent plugin file and decoupling and integrating it into the crawler framework; the crawler framework then calls the independent plugin file to perform webpage parsing and extract target field data based on the XPath structured rules.

2. The website data parsing method according to claim 1, characterized in that, If the semantic features still cannot determine the type, then the DOM structure features are analyzed to return the type result. Specifically, this also includes performing DOM tree normalization on the HTML document, including removing comment nodes, whitespace text nodes, and visually hidden elements, generating a set of visible nodes V={ , ,…, }, where m≤500; perform similarity matching between the standardized DOM feature vector and the pre-set vertical domain DOM structure feature library in the local knowledge base to obtain the final type.

3. The website data parsing method according to claim 1, characterized in that, The specific steps for generating XPath structured rules include: inputting HTML fragments and Prompt templates into a large model, outputting constraints as a pure XPath structured rule set, and saving the corresponding webpage URL and the parsed XPath as keys to a knowledge base.

4. The website data parsing method according to claim 1, characterized in that, The composite verification of the extraction results specifically includes: real-time monitoring of rule validity through a composite validator; automatically triggering a large model reconstruction process when the null value rate of a field exceeds a threshold for a consecutive preset number of pages; extracting the HTML fragment of the current page and generating a new XPath structured rule by combining it with historical successful rules; and archiving the new rule to the local knowledge base and replacing the invalid rule after verification, which also includes manual review intervention.

5. The website data parsing method according to claim 1, characterized in that, The automatic reconstruction unit further includes: extracting the HTML content of the current page, generating a new Prompt by combining it with historical success rules, outputting a new XPath in the large model, and then performing sandbox testing with a composite validator. If the verification is successful, the content is entered into the knowledge base, which also includes manual review before being added to the knowledge base.

6. A website data parsing device based on large model technology, characterized in that, Specifically, it includes: The intelligent rule generation module is used to receive the target webpage URL and perform a knowledge base query on the URL; If a corresponding XPath rule exists in the knowledge base, output the existing rule. If not included, intelligent semantic analysis and dynamic Prompt construction are performed to generate XPath structured rules; The verification module is used to verify the generated XPath structured rules. If the verification is successful, the rules are stored in the XPath knowledge base of each domain. If the verification fails, the re-analysis and rule update are triggered. The data extraction module is used to apply the generated or matched XPath structured rules to the acquisition system and extract data from the HTML content of the target webpage through the XPath plugin; The dynamic feedback optimization module is used to perform composite verification on the extraction results. If multiple web pages are detected to have extraction deviations or data anomalies, the automatic reconstruction unit is triggered to regenerate XPath structured rules based on changes in the web page structure. The specific steps for receiving the target webpage URL are: receiving the target webpage URL input by the user, calling a lightweight headless browser to obtain the initial HTML document; and after receiving the HTML document, performing the following steps: Extract Open Graph protocol and traditional metadata; if a definite type identifier exists, return the result directly. If there is no definite metadata type identifier, the keyword features in the title and body are analyzed; if there is a definite type identifier, the results are returned directly. If the semantic features still cannot determine the type, then analyze the DOM structural features and return the type result; The dynamic Prompt construction specifically includes: customizing the Prompt based on the domain; extracting key anchors and path patterns from the XPath structural features of each template according to the identified website type and the Top-3 high similarity templates; the key anchors are a set of domain-specific keywords in the CSS class name or ID attribute, and the path pattern is the relative position expression of the target field in the DOM tree; using the feature samples as target references, constructing structured prompt words according to the preset Prompt template; The process of extracting data from the HTML content of a target webpage using an XPath plugin further includes: encapsulating the generated XPath structured rules into an independent plugin file and decoupling and integrating it into the crawler framework; the crawler framework then calls the independent plugin file to perform webpage parsing and extract target field data based on the XPath structured rules.

7. A computer program product, characterized in that, It stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-5.

Images