Method and system for intelligent collection and semantic understanding of internet data based on large language model

Abstract

The application discloses an internet data intelligent collection and semantic understanding method and system based on a large language model, belongs to the technical field of intelligent collection and semantic understanding, and comprises data acquisition and preprocessing, structure perception and hierarchical scheduling, large language model semantic analysis and extraction, and three-level checking and secondary verification. The application initiates an HTTP request to acquire target webpage HTML source code and pre-processes by a collection task queue to obtain a purified page description document; structure feature digitization is performed on the purified page description document, hierarchical analysis scheduling determination is performed based on the result, if the structure is stable and cache hits, fields are directly extracted and large model calling is skipped, otherwise, a large language model analysis mode is entered; in the mode, prompt words are dynamically assembled, segments are divided and positions are marked, each segment target field key-value pair is extracted by inputting the model; three-level composite checking is performed on the output result, whether secondary verification is triggered is determined according to the checking result, and the final collection result is output after passing.

Description

Technical Field

[0001] This invention relates to the field of intelligent data acquisition and semantic understanding technology, and in particular to a method and system for intelligent data acquisition and semantic understanding of the Internet based on a large language model. Background Technology

[0002] As digital transformation deepens across industries, enterprises' demand for large-scale and continuous collection of publicly available information on the Internet continues to surge. Currently, the mainstream Internet data collection methods mainly include rule-driven web crawlers and basic large language model parsing collection, both of which have significant technical defects and are difficult to adapt to the complex and ever-changing web collection scenarios of today.

[0003] Traditional rule-driven web crawlers require manually writing fixed extraction rules such as XPath, CSS selectors, and regular expressions, and are custom-developed for different websites and page templates, only adapting to static web pages with stable structures. Currently, most websites rely on front-end frameworks such as Vue and React to dynamically generate the DOM (Document Object Model) structure, and frequently undergo page redesigns, layout testing, and random class name updates. Even minor changes in the web page structure can cause the collection rules to fail, leading to data loss, misalignment, and collection interruptions. Technical personnel need to continuously maintain and iterate scripts, significantly increasing manpower costs and extending project delivery cycles in multi-site collection scenarios. Furthermore, traditional crawlers only support simple text matching, lacking deep semantic understanding capabilities, and cannot effectively filter redundant noise such as navigation bars, advertisements, and footers. They struggle to complete complex tasks such as entity extraction, semantic classification, and cross-page information fusion, and have extremely poor adaptability to unstructured and semi-structured web page content. While existing collection solutions combining large language models have broken free from the dependence on fixed rules, they have not yet formed a complete technical system with structure awareness, adaptive control, and closed-loop verification, and still have many shortcomings. Existing solutions often directly input the complete webpage source code into the model for parsing without noise preprocessing. This not only leads to significant consumption of token resources and increased operating costs, but also reduces data recognition accuracy due to redundant code interference. Furthermore, these solutions lack the ability to proactively detect changes in webpage structure, failing to automatically identify page redesigns, field migrations, etc., still requiring manual anomaly detection and parsing strategy adjustments, and cannot achieve dynamic adaptive data collection. In addition, large language models suffer from inherent probability illusion problems, easily generating erroneous data when faced with fragmented and non-standardized webpages. Current technologies lack robust semantic verification and field comparison mechanisms, allowing erroneous data to directly flow into downstream systems, severely impacting the accuracy and reliability of big data analysis. Moreover, existing solutions employ a single full-data inference mode, failing to allocate computing power in a tiered manner based on webpage complexity and stability. Even for static and stable webpages, high-overhead model inference is still used, resulting in wasted computing resources and low overall collection efficiency. This fails to balance collection accuracy, cost, and response speed, making it difficult to meet the low-cost, highly automated, and highly stable continuous data collection needs of the Internet of Things and enterprise digitalization fields. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method and system for intelligent collection and semantic understanding of Internet data based on a large language model. This system enables the collection task queue to initiate HTTP requests to obtain the HTML source code of the target webpage and preprocess it to obtain a cleaned page description document. The cleaned page description document undergoes structural feature digitization, and hierarchical parsing scheduling is performed based on the results. If the structure is stable and the cache is hit, fields are directly extracted and the large model call is skipped; otherwise, the large language model parsing mode is entered. In this mode, prompt words are dynamically assembled, segments are segmented, and positions are marked. The model is then used to extract key-value pairs of target fields for each segment. A three-level composite verification is performed on the output results, and a secondary verification is triggered based on the verification results. Upon successful verification, the final collection result is output.

[0005] On the one hand, a method for intelligent collection and semantic understanding of Internet data based on a large language model is provided. This method includes: according to a preset collection task queue, initiating an HTTP request to the target website to obtain the original HTML source code of the target webpage and preprocessing it to obtain a cleaned page description document; performing structural feature digitization on the cleaned page description document, and determining whether to perform hierarchical parsing scheduling based on the result of the structural feature digitization. If yes, the cleaned page is extracted into fields, and the large language model call step is skipped. If no, the large language model parsing mode is entered. In the large language model parsing mode, according to the target field definition of the collection task, structured prompt words are dynamically assembled, the cleaned page description document is divided into multiple segments according to length, and a position mark is added to each segment. The assembled structured prompt words and segments are input into the large language model, and the model is requested to output the key-value pairs of the target fields in each segment. A three-level composite verification is performed on the initial parsing result output by the large language model. Based on the result of the three-level composite verification, it is determined whether a secondary verification is triggered. If yes, the local DOM segment where the field is located is re-extracted and the large language model is requested again for verification. If no, the data that passes the verification is taken as the final collection result.

[0006] On the other hand, a system for intelligent internet data acquisition and semantic understanding based on a large language model is provided. This system includes: a data acquisition and preprocessing module, a structure awareness and hierarchical scheduling module, a large language model semantic parsing and extraction module, and a three-level verification and secondary verification module. The data acquisition and preprocessing module initiates HTTP requests to the target website according to a pre-defined acquisition task queue to obtain the original HTML source code of the target webpage and performs preprocessing to obtain a cleaned page description document. The structure awareness and hierarchical scheduling module performs structural feature digitization on the cleaned page description document. Based on the result of the structural feature digitization, it determines whether to perform hierarchical parsing and scheduling. If so, it extracts fields from the cleaned page and skips the large language model call step; otherwise, it... Entering the large language model parsing mode; the large language model semantic parsing and extraction module is used to dynamically assemble structured prompt words according to the target field definition of the collection task during the large language model parsing mode, divide the cleaned page description document into multiple segments according to length, and add position marks to each segment. The assembled structured prompt words and segments are input into the large language model, and the model is requested to output the key-value pairs of the target field in each segment; the three-level verification and secondary verification module is used to perform three-level composite verification on the initial parsing results output by the large language model. Based on the result of the three-level composite verification, it is determined whether secondary verification is triggered. If so, the local DOM segment where the field is located is re-extracted and the large language model is requested again for verification. If not, the data that passes the verification is taken as the final collection result.

[0007] Beneficial effects The beneficial effects of the technical solutions provided by the embodiments of the present invention include at least the following: Compared to existing traditional rule-based web crawlers and conventional large-scale model collection solutions, this invention achieves comprehensive and systematic beneficial technical effects through a full-link collaborative technical solution. This solution includes pre-processing of webpages with layered noise reduction, digitization of DOM structural features and intelligent similarity discrimination, hierarchical scheduling of rule caching and large-scale model inference, segmented parsing of structured and accurate prompt words, and a triple-checking and secondary review of field format, logical relationships, and historical deviations. By intelligently removing noise blocks and compressing and purifying redundant nodes in the original HTML source code, this invention effectively filters out invalid and interfering content such as navigation, advertisements, pop-ups, and footers, reducing the redundant token consumption of the input large-scale model. This significantly reduces the computing power cost and operational expenses of routine batch data collection, while avoiding interference from massive amounts of invalid code on semantic parsing, thus improving the purity of effective information and the accuracy of field recognition from the input end. This invention innovatively introduces a webpage DOM structure fingerprinting and Jaccard similarity comparison mechanism, enabling fully automatic and non-intrusive perception of subtle changes and redesigns in webpage node hierarchy, tag attributes, and page layout. It eliminates the need for continuous manual monitoring of site status, analysis of page structure, and iterative updates to the collection script. This addresses the industry pain points of traditional fixed-rule crawlers, which are prone to rule failure, data misalignment, and task interruption due to dynamic webpage rendering, random class name updates, and page redesigns. It significantly reduces manual maintenance costs and project iteration time, comprehensively improving the collection system's adaptability and robustness to dynamic webpage scenarios. This invention constructs a hierarchical parsing and scheduling mechanism. For structurally stable static webpages, it reuses cached effective XPath template extraction and employs a lightweight rule engine for rapid collection. Large-model semantic reasoning is only used for complex webpages with structural changes and high unstructured nature. This overcomes the shortcomings of existing large-model collection schemes, which rely on single full-scale reasoning and waste computing power. While ensuring collection accuracy, it significantly shortens the single collection response latency, achieving refined scheduling and efficient utilization of computing resources. This invention effectively enhances the semantic understanding and fragmented information extraction capabilities of large models by using customized multi-constraint structured prompts, segmented parallel parsing with position tags, and cross-segment result fusion deduplication strategies. It suppresses the problem of information illusion in models at the instruction level and can accurately adapt to various Internet page data collection scenarios with semi-structured, unstructured, and scattered content. It breaks through the technical limitations of traditional crawlers that lack semantic parsing capabilities and cannot complete high-order information fusion.This invention establishes a three-level composite verification system based on field format, logical consistency, and historical deviation comparison, and is equipped with a secondary verification mechanism for local DOM fragments of abnormal fields. It can comprehensively screen various collection problems such as format errors, logical conflicts, data mutations, and null value anomalies, automatically correct distorted model data, and remove erroneous samples. It solves the defects of existing large model collection results, such as poor reliability, lack of closed-loop verification, and direct flow of erroneous data into downstream systems, significantly improving the accuracy, standardization, and credibility of the final collected data. At the same time, this invention achieves autonomous learning and continuous optimization of the collection strategy by iteratively updating the parsed template and page structure fingerprint associated with the verified data, forming an adaptive closed-loop collection system that can adapt to the continuous iterative updates of major portal websites and industry vertical platforms for a long time. Attached Figure Description

[0008] Figure 1 A flowchart of an intelligent Internet data acquisition and semantic understanding method based on a large language model provided in this application embodiment; Figure 2 A flowchart illustrating the hierarchical parsing and scheduling process of the Internet data intelligent collection and semantic understanding method based on a large language model, provided for embodiments of this application. Figure 3 This is a schematic diagram of the structure of the Internet data intelligent acquisition and semantic understanding system based on a large language model provided in the embodiments of this application. Detailed Implementation

[0009] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0010] like Figure 1 The diagram shown is a flowchart of an intelligent internet data acquisition and semantic understanding method based on a large language model provided in this application embodiment. The method includes the following steps: As the first step of this invention, according to a preset collection task queue, an HTTP request is sent to the target website to obtain the original HTML source code of the target webpage and preprocess it to obtain a cleaned page description document.

[0011] It should be understood that, through distributed crawler scheduling, HTTP / HTTPS requests are initiated to the target website according to a pre-set collection task queue to obtain the complete original HTML source code of the target webpage. Simultaneously, response header information, status codes, page load times, and the current request proxy IP are recorded to form the original page dataset. The following preprocessing sub-steps are performed on the original HTML source code: the HTML source code is converted into a traversable node tree structure based on a DOM tree parser; a pre-trained noise recognition classifier is used, combined with label type (such as...). <nav> 、 <footer> 、 <aside>The system automatically identifies and removes non-business-target noise blocks such as navigation bars, pop-up ads, footers, and recommendation lists by analyzing CSS class name keywords (e.g., "ad", "banner", "menu"), location features (e.g., top, bottom, sidebar), and text density thresholds, resulting in a purified HTML fragment of the main content subtree. Redundant nodes in the main content subtree are compressed, adjacent text nodes are merged, and empty nodes, comment nodes, and style script nodes are removed to generate a lightweight purified page description document.

[0012] As the second step of this invention, the structural features of the cleaned page description document are digitized. Based on the result of the structural feature digitization, it is determined whether to perform hierarchical parsing scheduling. If yes, the cleaned page fields are extracted and the large language model calling step is skipped. If no, the large language model parsing mode is entered.

[0013] It should be noted that the specific process for digitizing the structural features of the cleaned-up page description document is as follows: Parse and clean the page description document to build a standardized DOM tree; traverse the DOM tree from the root node to the node where each target field candidate area is located, and extract the tag name and preset key attribute combination of each level node. The key attributes include the stable part with business semantics in the id and class attributes (removing dynamically generated suffixes such as numbers and timestamps). For the randomly changing part of the class attribute value, normalization and replacement are performed using regular expression matching rules, and a general placeholder is used to obtain a stable node path sequence. The label name and normalized attribute value of each path sequence are concatenated into a string, and a fixed-length binary digest is generated using the SHA-256 hash algorithm, which serves as the fingerprint of the path; the fingerprint set of all candidate region paths within the page is arranged in a fixed order to form the structural feature vector of the page. The structural feature vector of the current page is compared with the historical feature vector of the previous version of the same site, which is stored in the historical structural fingerprint database. The similarity between the two was calculated using the Jaccard similarity coefficient and converted into a percentage score. Set a similarity threshold (preferably 95%). If the calculated similarity score is greater than or equal to the set similarity threshold, the structure is determined to be unchanged. If the similarity score is lower than the set similarity threshold, it is determined that the webpage structure has changed, and a structure change event signal is generated.

[0014] In this embodiment, the system reads the cleaned page description document that has been denoised, deredundant, and node compressed. It then uses a built-in standardized DOM tree parser to fully parse the clean HTML fragments, discarding interfering content such as dynamic scripts, random styles, and invalid comments from the original webpage. The system constructs a standardized DOM node tree with regular hierarchy, unique nodes, and forward traversal capability, ensuring that subsequent structural feature extraction relies solely on the actual business content nodes of the page, completely avoiding interference from redundant front-end dynamic rendering code on structural recognition. After constructing the DOM tree, the system starts from the root node and traverses downwards layer by layer to the candidate region nodes corresponding to each preset target field. It fully extracts the tag names of parent, child, and sibling nodes at each level, as well as key attribute combinations that identify the page structure. The key attributes are limited to the `id` and `class` attributes. Simultaneously, these two types of attributes undergo refined cleaning, precisely removing invalid fields in the `class` attribute such as random number strings, timestamps, dynamically generated random suffixes, and random session characters—fields without business semantics and changing with each page load. Only fixed semantic fields that stably represent the page structure layout, module functions, and business partitions are retained, ensuring that the extracted node attributes have long-term structural representation value. For the commonly present random variable fragments in the `class` attribute, this embodiment uses preset regular expression matching rules to perform normalization and unified replacement processing. It accurately matches and identifies dynamic variable fragments such as number combinations, millisecond timestamps, and randomly ordered characters, uniformly replacing them with universal fixed placeholders. This completely eliminates the structural feature disturbance problem caused by dynamic random attributes from the front end, ultimately obtaining a derandomized, highly stable, and vertically comparable standardized node path sequence. After obtaining the node path sequences of all target candidate regions, the system concatenates the node label name corresponding to each path with the normalized stable attribute value to form a unique structured path string. Then, it calls the SHA-256 hash algorithm to encrypt each path string and generate a fixed-length binary hash digest. This hash digest is used as the unique structural fingerprint of a single node path, effectively achieving lightweight and unique representation of long text path information. Subsequently, the system arranges and integrates the path fingerprints corresponding to all target field candidate regions in the page according to a preset fixed sorting rule, constructing a page structure feature vector that can fully represent the overall layout, node hierarchy, and field distribution characteristics of the current page, realizing the digital and vectorized conversion of the webpage visual structure and DOM hierarchical structure.After constructing the current page structure feature vector, the system retrieves the historical page structure feature vectors corresponding to the previous collection period of the same target site from the local historical structure fingerprint database and performs point-to-point feature comparison analysis. In order to accurately quantify the degree of page structure difference, this embodiment introduces the Jaccard similarity coefficient algorithm to complete the similarity calculation. By comparing the intersection and union ratio of the fingerprint sets of the current feature vector and the historical feature vector, the system accurately outputs the structural similarity value of the two and converts it into a percentage similarity score. This embodiment presets a structural stability similarity threshold of 95% as the sole criterion for determining whether a webpage structure has undergone substantial changes. When the calculated page structure similarity score is greater than or equal to 95%, it is determined that the current webpage's DOM hierarchy layout, field node positions, and core module structure have not undergone substantial changes, and the overall page structure is in a stable state. When the similarity score is less than 95%, it is accurately determined that the webpage has undergone substantial structural changes, covering various page redesign and dynamic rendering change scenarios such as DOM node hierarchy additions and subtractions, core tag name replacements, key attribute failures, node arrangement order adjustments, and field candidate area migrations. A standardized structural change event signal is generated simultaneously and pushed to the front-end hierarchical scheduling module, providing accurate and quantitative structural decision-making basis for subsequent adaptive selection of rule-based rapid collection mode or large model semantic parsing mode. This completely solves the technical defects of traditional collection schemes, which cannot quantitatively identify minor structural changes in pages and can only passively discover collection anomalies afterward, achieving fully automatic, proactive, and accurate perception of webpage structural changes.

[0015] It should be understood that, such as Figure 2 The flowchart shown is a hierarchical parsing and scheduling process for the Internet data intelligent collection and semantic understanding method based on a large language model provided in this application embodiment. The specific process is as follows: Obtain the cleaned page description document, first determine whether the webpage structure has changed. If the page structure has not changed, use the site identifier combined with the structural fingerprint hash to build a joint index to retrieve the parsing rule cache library. When the index is successfully hit and the extracted template status is valid, call the lightweight rule engine to complete field extraction based on XPath, directly output the collection results, and skip the large language model. If the page structure has changed, the cache index is not hit, or the template is invalid or in a pending review state, the large language model parsing mode is started, and prompt word assembly, document fragmentation, and field extraction are completed in sequence. Then, the parsing results are sent to the three-level composite verification. If the verification fails, a local DOM fragment is extracted for re-verification. After successful verification, the extracted template and page structure fingerprint are bound, the cache library is updated, and finally the collected data is output. The whole process realizes hierarchical scheduling of rapid rule collection and intelligent parsing of the large model, as well as the autonomous updating of the collection template.

[0016] It should be further explained that the specific process for determining whether to execute hierarchical parsing scheduling based on the results of digitizing the execution structure characteristics is as follows: If the structure is determined to be unchanged, the parsing rule cache is queried using the site identifier of the current page and the hash digest of the structural feature vector as a joint index. The parsing rule cache stores extraction templates that have been verified in historical collection tasks. Each extraction template contains at least one set of XPath expressions corresponding to the target fields. When a composite index is hit in the parsing rule cache and the corresponding validity flag is valid, the historical parsing rule is determined to be available. The lightweight rule engine is then called to apply the XPath expression extracted from the template to clean the page description document, perform node location and text content extraction on each target field, and obtain the field value. After the field extraction is completed, the extraction results are output directly, skipping the large language model parsing mode and no longer calling the large language model interface, thereby reducing token consumption and shortening the single collection response time.

[0017] Determining whether to execute hierarchical parsing scheduling based on the results of digitizing execution structure characteristics also includes: The historical parsing rules are deemed unavailable and the process is triggered to enter the large language model parsing mode when any of the following conditions occur: A similarity score below the structural stability threshold indicates that the webpage has undergone substantial structural changes, including the addition or removal of DOM levels, changes in tag names, replacement of key attribute names, or adjustment of node order. Although the similarity score reached the structural stability threshold, the composite index did not find any matching records in the parsing rule cache library; The composite index hit a matching record, but the corresponding validity flag is invalid or pending review, indicating that the template was marked as an extraction error or failed verification during the historical collection process; After triggering the entry into the large language model parsing mode, the historical extraction templates in the parsing rule cache library are abandoned, and the intelligent parsing process of the large language model is executed. This process dynamically assembles structured prompt words, segments and cleans page fragments, calls the large language model for semantic understanding and field extraction, and outputs the extraction results to the subsequent verification steps. At the same time, after the large language model parsing is completed and verified, the newly generated extraction template is associated with the structural fingerprint of the current page and stored in the parsing rule cache library to update the cache for quick reuse of subsequent pages with the same structure.

[0018] In this embodiment, after completing the digitization of webpage structural features, the quantitative calculation of structural similarity, and the determination of page structure status, the system automatically executes an intelligent hierarchical parsing scheduling process. Based on the stable state of the webpage structure and the matching status of the local rule cache, it adaptively switches between lightweight rule acquisition mode and large-model semantic parsing mode, achieving a dynamic balance between acquisition efficiency and accuracy. The specific implementation process is as follows: The system first reads the aforementioned page structure comparison results. Under the premise that the current target webpage structure has not undergone substantial changes and the page structure similarity score is greater than or equal to a preset 95% stable threshold, it immediately generates a unique joint index. This joint index is composed of the unique identifier of the target site and the SHA-256 hash digest of the current page structure feature vector, enabling... The precise and unique positioning of "site + page structure" effectively avoids cache matching errors caused by different sites and similar page structures. Subsequently, the system retrieves the local preset parsing rule cache library based on this composite index. The parsing rule cache library is a structured cache database that has been built up over a long period of system iteration. It is specifically used to store standardized field extraction templates that have been verified and confirmed to be accurate and effective in each collection task through three levels of composite validation. Each set of extraction templates corresponds to a specific web page structure fingerprint, and the template encapsulates a set of XPath expressions that accurately correspond to each business target field. It is also equipped with an independent validity identifier field to mark the current availability status of the template. The validity status includes three categories: valid, invalid, and pending review. The historical collection performance of the template is recorded throughout the process. When the composite index accurately matches a template in the parsing rule cache during the retrieval process, and the validity flag corresponding to that template is valid, the system can determine that the current webpage structure is completely consistent with the structure of historically successfully collected pages, and that the historical parsing rules are fully reusable. It then triggers a lightweight, fast collection branch, calling the built-in high-performance, lightweight rule engine. This bypasses the high-overhead large model call process and directly applies the mature XPath expression set pre-stored in the cache template to the already cleaned page description document. Through precise DOM node positioning, text extraction, and field cleaning operations, it batch-completes the rapid extraction and assignment of all preset target fields and directly outputs structured collection results. This mode eliminates the need for time-consuming operations such as large model token encoding, semantic reasoning, and result decoding, significantly reducing token resource consumption and interface response latency for a single collection and significantly improving the overall throughput efficiency of large-scale, routine collection tasks.Meanwhile, this embodiment sets up a comprehensive abnormal branch scheduling mechanism to automatically switch to high-precision large model parsing mode for three types of scenarios where historical rule templates cannot be reused. Specifically, these include: the first type is the page structure change scenario, where the page structure similarity score is lower than the structural stability threshold of 95%, indicating that the target webpage has undergone substantial redesign behaviors such as adding or removing DOM node levels, replacing core tag names, changing key Class and Id attributes, adjusting node arrangement order, and migrating business field areas, and the historical fixed XPath rules have completely become invalid; the second type is the cache miss scenario, where although the webpage structure has not changed and the page layout is stable, the current site structure fingerprint has never appeared in historical collection tasks, and there is no corresponding reusable template in the parsing rule cache library; the third type is the template failure scenario, where although the composite index successfully matches the historical template, the template validity is marked as invalid or pending review, indicating that the template has had problems such as field extraction misalignment, data missing, and verification anomalies in previous collection jobs, and does not meet the conditions for reuse. When any of the above abnormal scenarios are triggered, the system automatically abandons the caching rule call, terminates the lightweight rule collection process, and fully enters the large model intelligent parsing workflow. It sequentially performs structured prompt word dynamic assembly, cleansing page document fragmentation and segmentation, fragment position marking, parallel large model semantic reasoning, multi-fragment result fusion and deduplication, and missing field completion operations. Relying on the deep semantic understanding capabilities of the large model, it breaks through the limitations of fixed rules, completes accurate data extraction for complex dynamic pages and new page structures, and sends the preliminary parsing results to the backend three-level composite verification process for error correction and optimization. Notably, this embodiment possesses autonomous learning and cache iteration update capabilities. After the large model parsing results have passed three levels of verification and the data is confirmed to be accurate, the system automatically associates and binds the newly generated field extraction XPath template adapted to the new page structure with the structural feature fingerprint of the current page, and synchronously updates and writes it into the parsing rule cache library. It iteratively supplements and updates the original cached data, so that when encountering subsequent collection tasks with the same page structure, the new template can be directly reused without calling the large model again. It continuously accumulates effective collection rules, gradually improves the overall rapid collection coverage of the system, and achieves a closed-loop intelligent collection effect of self-adaptation of web page structure, self-optimization of collection strategy, and self-reduction of computing power cost. It effectively solves the technical defects of traditional collection solutions, such as poor adaptability of fixed rules, high manual maintenance costs, waste of computing power in full inference of traditional large models, and inability to differentiate scheduling between old and new page structures.

[0019] As the third step of this invention, during the large language model parsing process, structured prompt words are dynamically assembled according to the target field definition of the acquisition task. The cleaned page description document is divided into multiple segments according to length, and a position mark is added to each segment. The assembled structured prompt words and segments are input into the large language model, and the model is requested to output the key-value pairs of the target field in each segment.

[0020] It needs to be explained that the specific process of requesting the model to output the key-value pairs of the target field in each fragment is as follows: Based on the predefined target field pattern of the data collection task, structured prompt words are dynamically assembled; the prompt words should include at least: system role instructions, target field definition list, output format constraints, few sample examples, and negative constraint instructions; System role instructions are used to restrict large language models to perform tasks as data extraction experts; The target field definition list includes the field name, data type, field meaning description, and value examples; Output format constraints explicitly require the model to return the extracted results in JSON or XML format, and specify the key-value pair mapping relationship between fields and values; For a small number of sample examples, the parsing results of typical pages of the same type and domain as the current target page are selected as a reference to guide the model to understand the extraction rules; Negative constraint instructions explicitly state that models must not fabricate, speculate, or supplement information that is not explicitly present on the page, thus suppressing information illusions at the instruction level; The pre-processed cleaned page description document is divided into continuous segments according to a preset length threshold. The preset length threshold is determined based on the upper limit of the context window of the large language model used, and token space is reserved to accommodate prompt words and output results. Each segment is assigned a unique location marker, which includes the segment number, the segment's start and end line numbers in the original document, and the connection identifier between the segment and the preceding and following segments. This is used to restore the contextual association across segments in the subsequent merging stage.

[0021] The request model outputs key-value pairs of the target field in each fragment, including: Structured prompts are concatenated with each segment to form multiple independent model input units, which are then sent to the large language model in parallel based on the segment order. For each segment, the large language model performs semantic understanding and information extraction after receiving the input, and outputs a set of target field key-value pairs corresponding to the segment. The key in the key-value pair is consistent with the field name in the target field definition, and the value is the corresponding text content that the model recognizes and extracts from the current segment. By attaching a forced output instruction to the inference request, the large language model, while outputting each key-value pair, returns the line number range of the value in the cleaned page description document or the corresponding simplified XPath location path as supporting information for the credibility of the extraction result; the simplified XPath path at least includes the sequence of tag names from the document root node to the node where the target field is located and the necessary index information. The request model outputs key-value pairs of the target field in each fragment, including: For each segment, the set of key-value pairs returned by the large language model and the corresponding location evidence information are temporarily stored according to the segment number to form multiple segment-level extraction result units; each unit contains the segment identifier, all target field key-value pairs extracted from the segment, and the location evidence information corresponding to each value; All segment-level extraction result units are merged in order of segment number; for target fields that appear repeatedly across segments, deduplication and conflict resolution are performed based on the positional evidence information corresponding to each field and the preset field value priority rules (including taking the first occurrence value, the last occurrence value, or merging into an array), forming a unified extraction result set that covers the complete and cleaned page description document; The unified extraction result set is compared with the target field definition list to identify missing fields that were not successfully extracted from any segment. Missing fields are assigned a preset null value identifier (including null or empty string), and the list of missing fields and the reason code for the missing field are appended to the returned result.

[0022] In this embodiment, when the system determines that the webpage structure has changed or there are no valid historical parsing rules that can be reused, it automatically enters the intelligent parsing and extraction process of the large language model. Through the refined processing of the entire process, including customized structured prompt words, adaptive document segmentation, parallel reasoning with location tracing, and cross-segment result fusion and error correction, it achieves accurate field extraction of complex dynamic webpages and unstructured webpage content. It comprehensively suppresses the information illusion of the large model from the instruction layer, data layer, reasoning layer, and result layer, ensuring the standardization, completeness, and accuracy of the extraction results. The specific implementation process is as follows: The system first pre-configures fixed target field extraction rules and business field patterns according to the current collection task, and intelligently and dynamically assembles standardized structured prompt words adapted to the current collection scenario. These prompt words are a multi-dimensional constrained composite prompt system, covering five core components, comprehensively standardizing the reasoning behavior and output standards of the large model. Among them, the system role instructions are used to accurately limit the task identity and capability boundaries of the large language model, enabling the model to carry out semantic parsing work as a professional Internet data extraction expert, abandoning general chatter and invalid reasoning logic, and focusing on the core task of data extraction; the target field definition list is the current collection. The core business specifications clearly list the exclusive names, standard data types, precise field definitions, legal value ranges, and real compliant value examples for all fields to be extracted. This allows the model to clearly understand the extraction standards and business meanings of each field, avoiding misunderstandings. Output format constraints enforce standardization on the model's output format, strictly requiring the model to uniformly use standard JSON or XML structured formats to output extraction results. This clearly defines the one-to-one mapping relationship between constraint field keys and extracted values, completely eliminating subsequent parsing chaos, inconsistent formats, and machine recognition problems caused by free text output. The few-sample example module accurately selects historical high-quality parsing cases that are highly consistent with the current target webpage's industry field, page type, and content paradigm as reference samples. This allows the model to adapt to the current page characteristics by combining mature extraction logic from similar scenarios, significantly improving the field recognition and adaptation capabilities for unfamiliar and redesigned pages. Negative constraint instructions are the core constraint rules for preventing illusions. They explicitly prohibit the model from making subjective inferences, fabricating, or fictitious extensions of blank information not displayed or recorded on the webpage, avoiding the inherent probabilistic output defects of large language models from the source and minimizing the generation of false data.After assembling the prompt words, the system, based on the maximum capacity limit of the official context window of the invoked large language model, reserves sufficient token redundancy space to accommodate structured prompt words, location marker information, and the final output results. Based on this dynamic threshold, the purified lightweight page description document is continuously and non-overlappingly segmented into independent text fragments that conform to the model input specifications. This solves the technical problems of long text exceeding the limit and context truncation distortion. At the same time, each segmented text fragment is assigned a unique location marker, which includes the fragment's unique sequence number, the precise start and end line numbers of the fragment content in the original purified document, and the connection marker between adjacent fragments. This fully preserves the spatial location information and contextual relationships of the original document, providing core location basis for subsequent merging of multi-segment results, cross-segment field tracing, and conflict correction. Based on this, the system precisely concatenates the standardized structured prompts one by one with each text fragment to generate multiple independent and complete model inference input units. It then strictly follows the fragment arrangement order of the original document, initiating parallel inference requests to multiple input units to improve parsing efficiency while ensuring the correct context order. After receiving each independent input unit, the large language model leverages deep semantic understanding capabilities to comprehensively analyze the text content, DOM node information, and business semantics within the fragments. It accurately identifies the valid content corresponding to each preset target field, outputting a standardized set of field key-value pairs for the corresponding fragment. Simultaneously, a forced source tracing output instruction is executed during the inference process. While outputting each set of field key-value pairs, the model proactively returns the precise line number range of the field content in the cleaned page document and a simplified XPath location path. This path fully preserves the tag hierarchy sequence and core index information from the document root node to the terminal node where the target field is located, forming a dual source tracing credential of "extracted content + physical location," providing credible evidence for subsequent data verification, anomaly review, and error location. After all segments have completed inference, the system caches the extracted key-value pair set and location tracing evidence information corresponding to each segment in sequence according to the segment's unique sequence number, constructing independent segment-level extraction result units to achieve orderly storage and independent management of the segmented results. Subsequently, the system performs full-domain integration and splicing of all segment-level result units. For situations where business fields are displayed across multiple segments and appear repeatedly, the system flexibly adopts methods such as taking the first valid value, taking the last updated value, and merging multiple values ​​into an array to complete the deduplication of duplicate fields and the resolution of data conflicts based on preset field priority resolution rules. This completely solves the problems of field duplication and data fragmentation caused by long text segmentation parsing, and finally generates a unified full-domain extraction result set that covers all valid content of the complete webpage, is conflict-free, and ordered.Finally, the system will perform a precise field-by-field comparison between the integrated unified extraction result set and the preset target field definition list, comprehensively traversing and verifying the extraction status of all preset collection fields, accurately identifying target fields that were not successfully extracted or whose content is missing, uniformly assigning a standardized null or empty string as a null value identifier to missing fields, and generating a corresponding exclusive missing reason code, accurately recording the specific reasons for the field missing type such as no corresponding content on the page, missing segments due to truncation, or failure of semantic recognition, and finally outputting a complete initial collection result set with a list of missing fields and exception codes, providing a comprehensive, standardized, and traceable original data foundation for subsequent three-level composite verification and secondary manual review.

[0023] As the fourth step of this invention, a three-level composite verification is performed on the initial parsing result output by the large language model. Based on the result of the three-level composite verification, it is determined whether a secondary verification is triggered. If so, the local DOM fragment containing the field is re-extracted and the large language model is requested to perform a second verification. If not, the data that passes the verification is taken as the final collection result.

[0024] It should be further explained that the three-level composite verification includes field-level format verification, logical consistency verification, and historical comparison verification. Field-level format validation specifically involves performing format checks on each target field output by the large language model, according to the data type specifications predefined in the data collection task. If the target field's data type is defined as date, then the extracted value is checked to see if it conforms to the preset date format template (including YYYY-MM-DD, YYYY / MM / DD, MM-DD-YYYY); if it is defined as numeric, then the extracted value is checked to see if it is a valid number; if it is defined as email, then the extracted value is checked to see if it matches the RFC 2822 regular expression; if it is defined as URL, then the extracted value is checked to see if it matches the standard URL regular expression. Based on the minimum and maximum length constraints of the field, check whether the character length of the extracted value is within the legal range, and mark values ​​with lengths exceeding the range as having a format error; If the range of values ​​for the target field is a preset finite enumeration set (including status codes and category labels), then check whether the extracted value belongs to the enumeration set, and mark the values ​​that do not belong to the set as illegal values; For fields constrained by preset regular expressions, the extracted values ​​are matched against the fields constrained by preset regular expressions. If the match fails, the format is determined to be invalid. All fields that fail validation are marked as having format errors and the specific error reason code is recorded.

[0025] Logical consistency verification specifically involves performing logical rationality verification based on combinations of fields that have inherent logical relationships. For field pairs containing time sequence relationships (including "publication time" and "update time", "start time" and "end time"), compare the time sequence of the two. If the end time is earlier than the start time, or the update time is earlier than the publication time, it is determined to violate the time sequence logic and is marked as a logic exception. For field pairs with mathematical relationships (including "unit price" and "total price", "quantity" and "total amount"), verify whether the values ​​meet the preset mathematical equations or inequalities. If the deviation exceeds the preset tolerance threshold, it is judged as a logical anomaly. For fields with hierarchical relationships (including "parent category" and "child category", "province" and "city"), check the subordinate matching between them. If the default subordinate mapping rules are not met, it is judged as a logical anomaly. For fields with existence dependencies (including "discount price" depending on "original price" and "remarks" depending on "main field"), if the dependent field exists but the dependent field is empty or missing, it is considered a logical anomaly.

[0026] Historical comparison and verification specifically involves: using the target URL and field name of the current data collection task as a composite index, retrieving the baseline value of the corresponding field from the previous data collection period (including the previous day, previous week, or previous preset data collection batch) on the same page from the historical data collection database, and performing deviation detection. For numeric fields, calculate the absolute and relative rates of change between the current extracted value and the historical baseline value. If the rate of change exceeds the preset deviation threshold, the field is marked as having an abnormal historical deviation. The deviation threshold is a dynamic threshold that is automatically adjusted based on the historical fluctuation range of the field. For text-type fields, calculate the edit distance or cosine similarity between the current extracted value and the historical baseline value. If the similarity is lower than the preset similarity threshold, it is determined to be an abnormal historical deviation, and it is then determined whether it is a normal business change (including price adjustment and title update). For fields with a stable non-empty rate in historical records, if the current extracted value suddenly becomes an empty value or an empty string, and the field has not been empty in the past several collection periods, it is judged as an abnormal empty value mutation and triggers a key attention marker. If the current page structure fingerprint changes compared to the historical structure fingerprint, the weight of the historical comparison verification will be automatically reduced, and the current extracted value will be temporarily marked as pending manual review to avoid normal business data being misjudged as abnormal due to page redesign. The various anomaly markers generated by field-level format validation, logical consistency validation, and historical comparison validation are summarized by field dimension. An independent validation status label is generated for each target field. The validation status labels include validation passed, format anomaly, logical anomaly, historical deviation anomaly, and pending manual review. All validation results are associated with the extracted data and packaged to form a structured validation report with confidence scores.

[0027] In this embodiment, to address the acquisition defects caused by the probabilistic generation characteristics of large language models, such as information illusion, field mismatch, format chaos, data mutation, and logical contradiction, and to eliminate the data distortion problem caused by single semantic parsing, the system performs comprehensive, multi-level, three-stage composite verification processing on the initial fragmented fusion extraction results output by the large model. This involves sequentially performing field-level format verification, logical consistency verification, and historical comparison verification. Through multi-dimensional cross-validation, various abnormal data are accurately identified and status-marked, ultimately generating a standardized verification report with credibility levels. The specific implementation process is as follows: The system first performs the first-level field-level format verification, rigorously verifying each target field result output by the large language model. Based on the predefined data type specifications, format templates, length thresholds, enumeration rules, and custom regular expression constraints for this data collection task, the system conducts refined compliance checks. Specific validation rules are applied to different data types. For date fields, three mainstream standard date format templates are uniformly matched: YYYY-MM-DD, YYYY / MM / DD, and MM-DD-YYYY. Any date data with incorrect formatting, missing characters, or illegal characters is considered format abnormal. For numeric fields, the system strictly validates whether the extracted content is a valid integer or floating-point number, filtering out invalid values ​​containing Chinese characters, special symbols, or empty characters. For email fields, internationally recognized RFCs are used. The system employs standard regular expressions (2822) for precise matching and validation. For URL-type fields, it leverages standard network resources to locate regular expression rules and validate the legality of link formats. Simultaneously, the system performs character length constraint validation, determining whether the extracted content is excessively long or too short based on the preset maximum and minimum legal character ranges for each field. For enumerated fields with limited value ranges, such as status labels and business categories, the system verifies whether each extracted value belongs to a preset legal enumeration set, eliminating illegal values ​​that deviate from the standard category. For special business fields with custom regular expression constraints, the system performs precise regular expression matching validation. All fields that fail format validation are precisely marked as formatting anomalies by the system, and the corresponding standardized anomaly reason codes are recorded synchronously, enabling traceable classification of formatting issues.After completing the basic format compliance check, the system performs a second-level logical consistency check. This check verifies the cross-field logical rationality of field combinations with inherent constraints in the business scenario, preventing issues where a single field has a correct format but the overall business logic is contradictory. Specifically, it covers four core logical constraint scenarios: For time-series field pairs such as release time and update time, start time and end time, the system strictly verifies the logical relationship between time sequences. If there are time sequence inversion issues such as update time being earlier than release time or end time being earlier than start time, it is directly judged as a logical anomaly; For numerical operation-related fields such as unit price, quantity, total price, and total amount, the system performs preset mathematical operations. Deviation checks are performed on relationships and tolerance thresholds. When the calculation deviation between fields exceeds the allowable error range, it is marked as a logical anomaly. For hierarchical subordinate fields such as parent and child categories, provinces and cities, the matching rationality of superior and subordinate fields is checked based on preset regional and industry classification subordinate mapping rules to eliminate hierarchical confusion. For field combinations with existential dependencies such as discount prices depending on original prices and remarks information depending on main business fields, dependency logic checks are strictly followed. If the main dependent field has valid data but the subordinate field is missing, or if the subordinate field exists but the core dependent field is empty, it is judged as a logical anomaly, and business logic vulnerabilities in the collected results are comprehensively screened. After format and logic validation, the system performs a third-level historical comparison validation to identify dynamic data deviations and abnormal changes. This level of validation uses a unique composite index formed by the target URL of the current data collection task and the name of the field to be validated. It automatically retrieves the baseline data of the same page in the historical data collection database for one or more recent preset data collection periods to conduct longitudinal time-series deviation detection. Different validation strategies are adapted for different field types: For numerical fields such as price, quantity, and click-through rate, the system accurately calculates the absolute change and relative change rate between the current extracted value and the historical baseline value. The system also adopts a dynamic deviation threshold mechanism, which can adaptively adjust the judgment standard according to the long-term historical fluctuation range of the field to avoid misjudgments caused by fixed thresholds. Fields exceeding the dynamic threshold are marked as historical deviation anomalies. For text fields such as titles, descriptions, and tags, the system uses a dynamic deviation threshold mechanism to automatically adjust the judgment criteria based on the long-term historical fluctuation range of the field, avoiding misjudgments caused by fixed thresholds. Fields exceeding the dynamic threshold are marked as historical deviation anomalies. The system employs a dual algorithm of logical distance and cosine similarity to quantify the differences between new and old texts. When the similarity falls below a preset threshold, an abnormal change is identified. Furthermore, it differentiates between normal business iteration updates and model extraction errors based on the business scenario. For core fields that have been consistently stable and non-empty, if the current collection result suddenly contains null values ​​or empty strings, and consecutive historical collection data shows valid non-empty values, it is identified as an anomaly due to a null value mutation and marked as a key area of ​​concern. Simultaneously, the system possesses intelligent fault tolerance and adaptation capabilities. If a comparison reveals a significant change in the current page structure fingerprint compared to historical fingerprints, indicating that the page has undergone a redesign and reconstruction, the system automatically reduces the weight of historical comparison verification and temporarily marks the corresponding data as awaiting manual review. This effectively avoids the problem of business data changes caused by normal page redesigns being misjudged as collection anomalies, significantly improving the fault tolerance and rationality of dynamic site collection verification.Finally, the system summarizes all anomaly markers and verification results generated by the three-level verification. Taking a single target field as the smallest statistical unit, a unique verification status label is generated for each field, specifically including five categories: verification passed, format anomaly, logical anomaly, historical deviation anomaly, and pending manual review. At the same time, the system associates and packages the extracted raw data, anomaly type, anomaly code, similarity score, deviation rate parameter, and review status of all fields, and generates corresponding confidence scores by combining various verification indicators. Finally, it outputs a structured, traceable, and complete verification report with a credibility rating, providing accurate decision-making basis for subsequent secondary review and error correction, data filtering and storage, and cache template updates. This fundamentally solves the technical problems of unstable semantic extraction accuracy of large models, inability to self-check hallucination errors, and inability to accurately locate abnormal data, and comprehensively improves the standardization, authenticity, and reliability of intelligent data collection from the Internet.

[0028] like Figure 3 The diagram shows the structure of the Internet data intelligent acquisition and semantic understanding system based on a large language model provided in this application embodiment. It includes: a data acquisition and preprocessing module, a structure awareness and hierarchical scheduling module, a large language model semantic parsing and extraction module, and a three-level verification and secondary verification module. The data acquisition and preprocessing module initiates HTTP requests to the target website according to a preset acquisition task queue to obtain the original HTML source code of the target webpage and performs preprocessing to obtain a cleaned page description document. The structure awareness and hierarchical scheduling module performs structural feature digitization on the cleaned page description document, and determines whether to perform hierarchical parsing and scheduling based on the result of the structural feature digitization. If so, it extracts fields from the cleaned page and skips the large language model call step. If not, it enters the large language model parsing mode; the large language model semantic parsing and extraction module is used to dynamically assemble structured prompt words according to the target field definition of the collection task during the large language model parsing mode, divide the cleaned page description document into multiple segments according to length, add position marks to each segment, input the assembled structured prompt words and segments into the large language model, and request the model to output the key-value pairs of the target field in each segment; the three-level verification and secondary verification module is used to perform three-level composite verification on the initial parsing results output by the large language model, and determine whether secondary verification is triggered based on the result of the three-level composite verification. If yes, it re-extracts the local DOM segment where the field is located and requests the large language model to verify it again. If no, the data that passes the verification is taken as the final collection result.

[0029] In this embodiment, the Internet data intelligent acquisition and semantic understanding system based on a large language model is composed of four core functional modules: a data acquisition and preprocessing module, a structure perception and hierarchical scheduling module, a large language model semantic parsing and extraction module, and a three-level verification and secondary verification module. These modules work in a hierarchical, closed-loop manner, linking and cooperating from top to bottom to complete a fully automated intelligent acquisition process, from raw webpage crawling, noise reduction, intelligent structure perception, hierarchical differentiated parsing, large model semantic extraction, multi-level verification and error correction to final compliant data output. This effectively solves the industry pain points of traditional acquisition systems, such as weak adaptability, low intelligence, high computing power consumption, and poor data accuracy. The specific module implementation and linkage process are as follows: The data acquisition and preprocessing module, as the system's front-end data access unit, is responsible for receiving batch acquisition tasks issued by the system. It orderly schedules distributed crawler resources according to a preset standardized acquisition task queue, following the compliant network... The network request protocol initiates standardized HTTP / HTTPS network requests to target websites across various industries, automatically acquiring the complete original HTML source code data of the target webpages. Simultaneously, it records the response header information, network status code, page loading time, and the currently used proxy IP information, constructing a complete and traceable original page dataset. This provides fundamental network log support for subsequent page analysis and anomaly investigation. After completing the original webpage data crawling, this module further performs refined webpage purification preprocessing. Relying on a standard DOM tree parser, it converts the messy original HTML source code into a hierarchical, traversable standardized DOM node tree structure. Combined with a pre-trained intelligent noise recognition classifier, it comprehensively utilizes multiple techniques such as webpage tag type recognition, key CSS class name keyword matching, page spatial location feature determination, and text density threshold filtering to accurately identify and batch-remove noise from webpages. <nav>Navigation block <footer>Footer information <aside>All invalid and noisy blocks that are not related to business objectives, such as sidebar ads, pop-up promotional content, and redundant recommendation lists, are eliminated. Only the main content DOM subtree carrying core business data is retained. At the same time, deep redundancy compression is performed on the main content subtree, automatically merging adjacent duplicate text nodes, clearing invalid empty nodes, removing code comment nodes, and removing redundant styles and script nodes. Finally, a lightweight, high-purity, interference-free, and cleaned page description document that is adapted for subsequent structure recognition and semantic parsing is generated, completely avoiding the problems of invalid code interference and redundant token consumption from the data source. The structure perception and hierarchical scheduling module, as the core module of the system's intelligent decision-making, plays a crucial role in the digital perception of webpage structure and the scheduling of differentiated acquisition strategies. After acquiring the cleaned page description document, it immediately performs full-dimensional structural feature digital processing on the cleaned page content. This involves parsing the standardized DOM tree, extracting stable features of hierarchical nodes, normalizing dynamic random attributes, generating page structure fingerprints and feature vectors, and combining the Jaccard similarity coefficient to perform quantitative comparison of the similarity between the old and new page structures. This accurately determines the stability and changes in the current webpage structure and, based on the quantitative judgment results, implements intelligent hierarchical parsing and scheduling decisions. When the detection determines that the webpage structure has not undergone substantial changes and the page layout and field positions are generally stable, the system... The system automatically retrieves and parses the rule cache library, reuses valid XPath extraction templates that have passed historical verification, and directly calls a lightweight rule engine to quickly locate target fields and extract content. It skips the large language model calling steps throughout the process, without consuming the large model's inference computing power and interface response time, which greatly improves the collection efficiency of stable pages and reduces the routine collection and operation costs. When the detection determines that the webpage structure has been changed, the cached rules have not been hit, or the historical template has expired, the system automatically disables the traditional rule collection mode and intelligently switches to the large language model semantic parsing working mode. Through adaptive strategy switching, it perfectly balances the collection efficiency of stable pages and the collection adaptability of dynamically changed pages, realizing the fine-grained and intelligent scheduling of computing resources.The semantic parsing and extraction module of the large language model, as the core unit of the system's high-precision data extraction, is specifically responsible for intelligent semantic acquisition of complex dynamic web pages with structural changes and no effective caching rules. After entering the parsing mode, the module first dynamically generates a comprehensive set of structured prompt words based on the pre-defined information of various target fields in the current acquisition task, such as their names, data types, business interpretations, and value specifications. These prompt words include system role-limiting instructions, standardized field definition lists, fixed output format constraints, few sample reference cases in the same scenario, and anti-hallucination negative constraint instructions. This standardizes the model's reasoning logic, constrains the output format, and suppresses fabrication behavior at the instruction level. At the same time, combined with the maximum carrying threshold of the context window of the adapted large language model, sufficient redundancy space is reserved for prompt words and result output tokens. The module then performs uniform, continuous, and non-overlapping adaptive segmentation of the overall cleaned page description document. The system processes and assigns a unique location marker to each text fragment, including fragment number, original document start and end line numbers, and adjacent fragment connection indicators, thus fully preserving the contextual and spatial information of the long document. Then, structured prompts are concatenated with each text fragment to form independent model input units, which are then sent to the large language model in parallel. This enables the large model to identify and extract target fields from each fragment based on its deep semantic understanding capabilities, outputting standardized key-value pairs. Simultaneously, the system returns the page line number range and simplified XPath location path corresponding to each extracted field as source tracing evidence. Finally, the system performs ordered merging of all fragmented parsing results, resolves duplicate field conflicts, and uniformly marks missing fields, forming a complete, logically consistent, and traceable initial extraction result set covering the entire webpage content. This overcomes the limitations of traditional fixed rules in adapting to the collection of unstructured, fragmented, and dynamic pages.The Level 3 and Secondary Verification module, serving as the system's closed-loop error correction unit for data quality, is specifically designed to address issues such as illusion errors, formatting inconsistencies, logical contradictions, and data mutations arising from the probability output of large models. It is responsible for performing a three-tiered composite verification process on the initial global parsing result set output by the large language model: field format verification, logical consistency verification, and historical comparison verification. This process comprehensively screens for abnormal data from three dimensions: format specifications, business logic, and temporal changes. It automatically generates corresponding status labels for each target field: verification passed, format abnormal, logical abnormal, historical deviation abnormal, and pending manual review, forming a structured verification report with confidence scores. Furthermore, it possesses an intelligent secondary review mechanism; for problem fields marked as abnormal or questionable during the verification process, the system will not further review the entire field. Instead of repeatedly parsing page content, this method precisely extracts local DOM fragments corresponding to abnormal fields and calls the large language model again for local fine-grained verification. This significantly reduces the computational consumption of secondary inference while ensuring error correction accuracy. For field data that passes verification and has no anomalies, the final output is a standardized and highly accurate final collection result. The cache template is updated iteratively in sync, thus building a full-link intelligent collection system of "front-end purification and preprocessing - mid-end intelligent hierarchical scheduling - precise semantic extraction - back-end closed-loop verification and error correction". This effectively overcomes many shortcomings of existing technologies, such as poor adaptability, high operation and maintenance costs, waste of computing power, low data accuracy, and lack of autonomous error correction capabilities. It comprehensively improves the automation level, adaptability, and data quality of continuous large-scale Internet data collection.

[0030] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the scope and intent of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and variations.< / aside> < / footer> < / nav> < / aside> < / footer> < / nav>

Claims

1. A method for intelligent collection and semantic understanding of internet data based on a large language model, characterized in that: Includes the following steps: According to the preset collection task queue, an HTTP request is sent to the target website to obtain the original HTML source code of the target webpage and preprocess it to obtain the cleaned page description document. The structural features of the cleaned page description document are digitized. Based on the result of the structural feature digitization, it is determined whether to perform hierarchical parsing scheduling. If yes, the cleaned page fields are extracted and the large language model calling step is skipped. If no, the large language model parsing mode is entered. During the parsing process of the large language model, structured prompt words are dynamically assembled according to the target field definition of the collection task. The cleaned page description document is divided into multiple segments according to length, and a position mark is added to each segment. The assembled structured prompt words and segments are input into the large language model, and the model is requested to output the key-value pairs of the target field in each segment. A three-level composite validation is performed on the initial parsing result output by the large language model. Based on the result of the three-level composite validation, it is determined whether a secondary validation is triggered. If so, the local DOM fragment containing the field is re-extracted and the large language model is requested again for verification. If not, the data that passes the validation is taken as the final collection result.

2. The method for intelligent collection and semantic understanding of Internet data based on a large language model as described in claim 1, characterized in that: The specific process for digitizing the structural features of the cleaned page description document is as follows: Parse and clean the page description document, and construct a standardized DOM tree; traverse the DOM tree from the root node to the node where each target field candidate area is located, and extract the tag name and preset key attribute combination of each level node. The key attributes include the stable part of the id and class attributes that have business semantics. For the randomly changing part of the class attribute value, normalization and replacement are performed using regular expression matching rules, and a general placeholder is used to obtain a stable node path sequence. The label name and normalized attribute value of each path sequence are concatenated into a string, and a fixed-length binary digest is generated using the SHA-256 hash algorithm, which serves as the fingerprint of the path; the fingerprint set of all candidate region paths within the page is arranged in a fixed order to form the structural feature vector of the page. The structural feature vector of the current page is compared with the historical feature vector of the previous version of the same site, which is pre-stored in the historical structural fingerprint database. The similarity between the two was calculated using the Jaccard similarity coefficient and converted into a percentage score. A similarity threshold is set. If the calculated similarity score is greater than or equal to the set similarity threshold, the structure is determined to be unchanged. If the similarity score is lower than the set similarity threshold, it is determined that the webpage structure has changed, and a structure change event signal is generated.

3. The method for intelligent collection and semantic understanding of Internet data based on a large language model as described in claim 1, characterized in that: The specific process for determining whether to execute hierarchical parsing scheduling based on the results of digitizing execution structure features is as follows: If the structure is determined to be unchanged, the parsing rule cache is queried using the site identifier of the current page and the hash digest of the structural feature vector as a joint index. The parsing rule cache stores extraction templates that have been verified in historical collection tasks. Each extraction template contains at least one set of XPath expressions corresponding to the target fields. When the composite index is hit in the parsing rule cache library and the corresponding validity identifier is valid, the historical parsing rule is determined to be available. The lightweight rule engine is called to apply the XPath expression in the extraction template to the cleaned page description document, perform node positioning and text content extraction on each target field, and obtain the field value. After the field extraction is completed, the extraction results are output directly, skipping the large language model parsing mode and no longer calling the large language model interface, thereby reducing token consumption and shortening the single collection response time.

4. The method for intelligent collection and semantic understanding of Internet data based on a large language model as described in claim 3, characterized in that: The method for determining whether to execute hierarchical parsing scheduling based on the results of digitizing execution structure features also includes: The historical parsing rules are deemed unavailable and the process is triggered to enter the large language model parsing mode when any of the following conditions occur: If the similarity score is lower than the structural stability threshold, it indicates that the webpage has undergone substantial structural changes, including the addition or removal of DOM levels, changes in tag names, replacement of key attribute names, or adjustment of node order. Although the similarity score reached the structural stability threshold, the composite index did not find any matching records in the parsing rule cache library; The composite index hit a matching record, but the corresponding validity flag is invalid or pending review, indicating that the template was marked as an extraction error or failed verification during the historical collection process; After triggering the entry into the large language model parsing mode, the historical extraction templates in the parsing rule cache library are abandoned, and the large language model intelligent parsing process is executed. This process dynamically assembles structured prompt words, segments and purifies page fragments, calls the large language model for semantic understanding and field extraction, and outputs the extraction results to the subsequent verification steps. At the same time, after the large language model parsing is completed and verified, the newly generated extraction template is associated with the structural fingerprint of the current page and stored in the parsing rule cache library to update the cache for quick reuse of subsequent pages with the same structure.

5. The method for intelligent collection and semantic understanding of Internet data based on a large language model as described in claim 1, characterized in that: The specific process by which the request model outputs key-value pairs of the target field in each fragment is as follows: Based on the predefined target field pattern of the data collection task, structured prompt words are dynamically assembled; the prompt words include at least: system role instructions, target field definition list, output format constraints, few sample examples, and negative constraint instructions; The system role instructions are used to restrict the large language model to perform tasks as a data extraction expert; The target field definition list includes field name, data type, field meaning description and value examples; The output format constraints explicitly require the model to return the extracted results in JSON or XML format and specify the key-value pair mapping relationship between fields and values; The few sample examples are selected from the parsing results of typical pages of the same type and field as the current target page as references to guide the model to understand the extraction rules. The negative constraint instructions explicitly state that the model must not fabricate, speculate, or supplement information that is not explicitly present on the page, thus suppressing information illusion at the instruction level. The pre-processed cleaned page description document is divided into continuous segments according to a preset length threshold. The preset length threshold is determined based on the upper limit of the context window of the large language model used, and token space is reserved to accommodate the prompt words and output results. Each segment is assigned a unique location marker, which includes the segment number, the start and end line numbers of the segment in the original document, and the connection identifier between the segment and the previous and next segments, to restore the contextual association across segments in the subsequent merging stage.

6. The method for intelligent collection and semantic understanding of Internet data based on a large language model as described in claim 5, characterized in that: The request model outputs key-value pairs of the target field in each fragment, and also includes: Structured prompts are concatenated with each segment to form multiple independent model input units, which are then sent to the large language model in parallel based on the segment order. For each segment, the large language model performs semantic understanding and information extraction after receiving the input, and outputs a set of target field key-value pairs corresponding to the segment. The key in the key-value pair is consistent with the field name in the target field definition, and the value is the corresponding text content identified and extracted by the model from the current segment. By attaching a forced output instruction to the inference request, the large language model, while outputting each key-value pair, returns the line number range of the value in the cleaned page description document or the corresponding simplified XPath location path as supporting information for the credibility of the extraction result; the simplified XPath path at least includes the sequence of tag names from the document root node to the node where the target field is located and the necessary index information. The request model outputs key-value pairs of the target field in each fragment, and also includes: For each segment, the set of key-value pairs returned by the large language model and the corresponding location evidence information are temporarily stored according to the segment number to form multiple segment-level extraction result units; each unit contains the segment identifier, all target field key-value pairs extracted from the segment, and the location evidence information corresponding to each value; All segment-level extraction result units are merged in order of segment number; for target fields that appear repeatedly across segments, deduplication and conflict resolution are performed based on the supporting information of each field's location and the preset field value priority rules to form a unified extraction result set that covers the complete and cleaned page description document. The unified extraction result set is compared with the target field definition list to identify missing fields that were not successfully extracted from any segment. The missing fields are assigned a preset null value identifier, and the list of missing fields and the reason code for the missing fields are appended to the returned result.

7. The method for intelligent collection and semantic understanding of Internet data based on a large language model as described in claim 1, characterized in that: The three-level composite verification includes field-level format verification, logical consistency verification, and historical comparison verification. The field-level format validation specifically involves performing a format check on each target field output by the large language model, according to the data type specifications predefined in the data collection task. If the target field is defined as a date type, then check whether the extracted value conforms to the preset date format template; if it is defined as a numeric type, then check whether the extracted value is a valid number. If defined as an email type, the extracted value is checked to see if it matches the RFC 2822 regular expression; if defined as a URL type, the extracted value is checked to see if it matches the standard URL regular expression. Based on the minimum and maximum length constraints of the field, check whether the character length of the extracted value is within the legal range, and mark values ​​with lengths exceeding the range as having a format error; If the range of values ​​for the target field is a preset finite enumeration set, then check whether the extracted value belongs to the enumeration set, and mark the value that does not belong to the set as an illegal value; For fields constrained by preset regular expressions, the extracted values ​​are matched against the fields constrained by preset regular expressions. If the match fails, it is determined that the format is invalid. All fields that fail validation are marked as having a format error and the specific error reason code is recorded.

8. The method for intelligent collection and semantic understanding of Internet data based on a large language model as described in claim 7, characterized in that: The logical consistency check specifically involves performing a logical rationality check based on a combination of fields that have an inherent logical relationship. For field pairs containing time sequence relationships, compare the time order of the two. If the end time is earlier than the start time, or the update time is earlier than the publication time, it is determined to violate the time sequence logic and is marked as a logic exception. For field pairs with mathematical operation relationships, verify whether the values ​​meet the preset mathematical equation or inequality constraints. If the deviation exceeds the preset tolerance threshold, it is judged as a logical anomaly. For fields with hierarchical subordinate relationships, check the subordinate matching between the two. If the preset subordinate mapping rules are not met, it is judged as a logical anomaly. For fields with existence dependencies, if the dependent field exists but the field being depended on is empty or missing, it is considered a logical anomaly.

9. The method for intelligent collection and semantic understanding of Internet data based on a large language model as described in claim 7, characterized in that: The historical comparison verification specifically involves: using the target URL and field name of the current data collection task as a combined index, retrieving the baseline value of the corresponding field from the previous data collection period on the same page from the historical data collection database, and performing deviation detection. For numeric fields, calculate the absolute and relative rates of change between the current extracted value and the historical baseline value. If the rate of change exceeds a preset deviation threshold, the field is marked as having a historical deviation anomaly. The deviation threshold is a dynamic threshold that is automatically adjusted based on the historical fluctuation range of the field. For text-type fields, calculate the edit distance or cosine similarity between the current extracted value and the historical baseline value. If the similarity is lower than the preset similarity threshold, it is determined to be an abnormal historical deviation and whether it is a normal business change. For fields with a stable non-empty rate in historical records, if the current extracted value suddenly becomes an empty value or an empty string, and the field has not been empty in the past several collection periods, it is judged as an abnormal empty value mutation and triggers a key attention marker. If the current page structure fingerprint changes compared to the historical structure fingerprint, the weight of the historical comparison verification will be automatically reduced, and the current extracted value will be temporarily marked as pending manual review to avoid normal business data being misjudged as abnormal due to page redesign. The various anomaly markers generated by field-level format validation, logical consistency validation, and historical comparison validation are summarized by field dimension, and an independent validation status label is generated for each target field. The validation status label includes validation passed, format anomaly, logical anomaly, historical deviation anomaly, and pending manual review. All validation results are associated with the extracted data and packaged to form a structured validation report with confidence score.

10. A system applying the Internet data intelligent acquisition and semantic understanding method based on a large language model as described in any one of claims 1-9, characterized in that, include: The module includes a data acquisition and preprocessing module, a structure awareness and hierarchical scheduling module, a large language model semantic parsing and extraction module, and a three-level verification and secondary verification module. The data acquisition and preprocessing module is used to initiate an HTTP request to the target website according to a preset collection task queue, obtain the original HTML source code of the target webpage and perform preprocessing to obtain a cleaned page description document. The structure perception and hierarchical scheduling module is used to perform structural feature digitization on the cleaned page description document, and determine whether to perform hierarchical parsing scheduling based on the result of the structural feature digitization. If yes, the cleaned page is used to extract fields and skip the large language model calling step. If no, the large language model parsing mode is entered. The semantic parsing and extraction module of the large language model is used to dynamically assemble structured prompt words according to the target field definition of the collection task during the parsing process of the large language model, divide the clean page description document into multiple segments according to the length, add position marks to each segment, input the assembled structured prompt words and segments into the large language model, and request the model to output the key-value pairs of the target field in each segment. The three-level verification and secondary verification module is used to perform three-level composite verification on the initial parsing result output by the large language model. Based on the result of the three-level composite verification, it determines whether secondary verification is triggered. If so, the local DOM fragment where the field is located is re-extracted and the large language model is requested again for verification. If not, the data that passes the verification is taken as the final collection result.

Images