A Webpage Data Acquisition Method and System Based on Dual-Path Integrity Verification and Completion
By using a dual-path integrity verification and completion method, the problem of missing dynamically rendered content and original response in web page data collection is solved, achieving accurate completion of browser and directly linked content, and improving the integrity and reliability of data collection.
Patent Information
- Application Number
- CN202611338822.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-31
- Publication Date
- 2026-11-17
AI Technical Summary
Existing web page data collection technologies struggle to accurately pinpoint the gap between dynamically rendered content and the original response, leading to missing content, duplicate data collection, and insufficient completeness of web page data.
A dual-path integrity verification and completion method is adopted. By executing simulated browser data collection and direct domain connection data collection in parallel, dual-path content anchor chains are generated. The browser and direct connection sections to be completed are identified and completed. Data completion is performed using the complete response records of the direct domain connection data collection and the simulated browser data collection path.
It achieves fine-grained correspondence between browser content units and directly connected content units, reduces the risk of mismatch, accurately locates local gaps, reduces duplicate collection, and improves the integrity and reliability of web page collection data.
Smart Images

Figure CN123115502A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of web page data acquisition technology, and in particular to a web page data acquisition method and system based on dual-path integrity verification and completion. Background Technology
[0002] With the development of internet applications, dynamic web page technology, and big data processing technology, web page data has gradually become an important source for information retrieval, content aggregation, business monitoring, and data analysis. Web page data includes not only the original page source code, but also the document object model, visible text, structured data responses, resource response data, and page screenshots generated after page script execution. Existing web page data acquisition technologies typically combine web crawlers, browser automation, web packet capture, template matching, and machine learning to complete web page content identification and extraction.
[0003] For example, application CN112579852B provides an interactive method for precise collection of web page data, which combines web page packet capture technology and machine learning technology. The steps include: S10: establishing a data structure model for the target web page; S20: identifying the target elements of the web page and performing model analysis; S30: intelligently evaluating the effective data range and defining the display area; S40: determining the existence of events in each element within the collection range, interacting with the user to confirm, and achieving precise collection of web page data.
[0004] For example, application CN108153817B discloses an intelligent web page data collection method, which includes the following steps: initializing the crawler database, traversing the web page records in the web page database, adding web page records that have not been crawled and meet the crawling conditions to the crawling queue, crawling the web page records in the crawling queue and storing them in the crawler database, reading a web page record until successful reading, filtering web page records according to settings, matching web page records with web page templates, if a match is successful, sending the current web page content and marking the web page record as sent and clearing the web page template content to complete the sending; otherwise, extracting the content feature code from the web page record and updating the web page template.
[0005] However, existing web page data collection technologies mainly focus on target element identification, web page template matching, and collection scope determination, lacking fine-grained construction of the content correspondence between browser rendered data and domain direct connection response data. When script-rendered content, structured data responses, or resource responses are missing, it is difficult to accurately locate the missing range and use another collection path to implement targeted completion, which can easily lead to omissions, duplicate collection, and insufficient completeness of web page content.
[0006] Therefore, in order to address the above issues, there is an urgent need to provide a web page data acquisition method and system based on dual-path integrity verification and completion. Summary of the Invention
[0007] To address the challenges of simultaneously capturing dynamically rendered content and the original response in existing web page data acquisition methods, as well as insufficient cross-sectional gap location and directional completion, this invention provides a web page data acquisition method and system based on dual-path integrity verification and completion. The technical solution is as follows: On the one hand, a web page data collection method based on dual-path integrity verification and completion is provided, which includes: S1 executes simulated browser data collection and direct domain connection data collection in parallel to obtain runtime document object model, network request logs, original page source code and structured data response, and generates request matching keys and complete response records; S2, the data from the two acquisition paths are split into browser content units and directly connected content units. Based on the relationship that the content identifier value, browser position value and directly connected position value are all increasing, dual-path content anchor chains are generated, and the content cross-section between anchor points is divided according to adjacent anchor points. S3 generates browser content gap values, browser text gap values, browser resource gap values, and direct connection response gap values for the content cross-sections between each anchor point, and determines the complete cross-section, browser to be completed cross-section, and direct connection to be completed cross-section based on each gap value; S4 uses direct domain connection to collect data to complete the browser's missing sections, uses the complete response records of the simulated browser's collection path to complete the direct connection's missing sections, recalculates each gap value and verifies the completion results, and outputs web page collection data and a collection validity report.
[0008] Furthermore, the specific steps for parallel execution of simulated browser data collection and direct domain connection data collection to obtain the runtime document object model, network request logs, original page source code, and structured data response are as follows: The target webpage address list and data collection requirement record are read from the data collection task table through the webpage data collection task configuration interface. The data collection requirement record includes source code collection markers, text collection markers, target webpage first-screen screenshot collection markers, and a set of key text fields. The target webpage address and data collection requirement record are associated by data collection task number to generate a webpage data collection task record. Based on the webpage data collection task record, the simulated browser data collection path and the direct domain connection data collection path are started in parallel using the same user agent field, language field, access credential field, and data collection start timestamp. The simulated browser data collection path loads the target webpage address and executes the page script, collecting the runtime document object model, visible text, node page coordinates, resource loading records, network request logs, and the target webpage first-screen screenshot through the browser debugging interface. The direct domain connection data collection path sends a hypertext transfer request to the target webpage address, collecting the original page source code, structured data response, hypertext fragment response, direct domain connection request logs, response headers, server address, and response time value.
[0009] Further, the specific steps for generating the request matching key and complete response record are as follows: Read the request method, final access address, request body, and request sending timestamp from the simulated browser network request log and domain direct connection request log, respectively. Concatenate the request method, final access address, request body digest value, and same-address request sequence number to form the request matching key. Same-address request sequence numbers are generated in ascending order based on the request sending timestamps of requests with the same request method and final access address within the same collection path. When the response header contains a declared byte length value, write the response whose actual received byte length value is the same as the declared byte length value and whose response body has been successfully parsed according to media type after decompression into the complete response record table. When the response header does not contain a declared byte length value, write the response with a transmission end marker and whose response body has been successfully parsed according to media type after decompression into the complete response record table.
[0010] Furthermore, the specific steps for splitting the data from the two acquisition paths into browser content units and directly connected content units are as follows: The runtime document object model is split into browser content units according to a depth-first traversal order, recording the tag name, parent path, content identifier value, text content, text digest value, absolute resource address, browser position value, and node page coordinates; The original page source code, structured data response, and hypertext fragment response are split into directly connected content units according to the response reception order and the tag position or field traversal order within the response, recording the tag name, parent path, content identifier value, field name, text content, text digest value, absolute resource address, source request matching key, and directly connected position value.
[0011] Furthermore, the specific steps for generating a dual-path content anchor chain based on the increasing relationship between content identifier values, browser position values, and direct connection position values are as follows: When the content identifier values of both the browser content unit and the directly connected content unit are non-empty and identical, a candidate pairing record is written; when at least one content identifier value is empty, the tag name, parent node path, text summary value, and absolute resource address are compared. When the number of identical items reaches the content matching threshold, a candidate pairing record is written, and the number of matching items is recorded; with the browser position value and direct connection position value both increasing as constraints, the candidate pairing record sequence with the largest number of records is selected from the candidate pairing records; when multiple sequences have the same number of records, the sequence with the largest cumulative value of matching items is selected; the starting boundary record of the page root node is written at the front of the candidate pairing record sequence, and the ending boundary record of the last content unit of the two collection paths is written at the back of the candidate pairing record sequence, and the records are connected to generate a dual-path content anchor chain.
[0012] Further, the specific steps for dividing the content cross-section between anchor points according to adjacent anchor points are as follows: According to the arrangement order of the dual-path content anchor chains, the browser content units and directly connected content units between two adjacent records are assigned to the same content cross-section between anchor points; the source request matching keys of the directly connected content units are summarized, and the request matching keys with the same request address as the absolute resource address in the content cross-section between anchor points are extracted from the simulated browser network request log to generate a cross-section request matching key set; the smallest bounding rectangle of the visible node page coordinates in the cross-section is taken as the cross-section page coordinate range, and the start boundary, end boundary, cross-section request matching key set, and cross-section page coordinate range are written into the content cross-section record between anchor points.
[0013] Furthermore, for each anchor point content section, browser content gap values, browser text gap values, browser resource gap values, and direct connection response gap values are generated. The specific steps for determining complete sections, browser-to-be-completed sections, and direct connection-to-be-completed sections based on these gap values are as follows: Count the number of directly connected content units within the anchor point content section that do not form candidate pairing records with browser content units, generating browser content gap values; compare the visible text in the simulated browser acquisition path and the text content of directly connected content units in the domain direct connection acquisition path with the key text field set item by item, counting the number of key text fields with identical text content; use the key text fields of the domain direct connection acquisition path... Subtracting the number of key text fields in the simulated browser's data collection path from the number of this field, and setting negative values to zero, generates a browser text gap value; counting the number of resource addresses in the directly connected content unit and the number of resources with the same address and a complete response record in the simulated browser's network log, subtracting the number of successfully responded resources from the number of resource addresses, and setting negative values to zero, generates a browser resource gap value; comparing each request matching key in the cross-section request matching key set with the complete response record table of the domain direct connection data collection path item by item, counting the number of request matching keys for which no complete response record was found, generates a direct connection response gap value; requiring that when collecting screenshots of the first screen of the target webpage, the cross-section should be cropped according to the page coordinate range. When the cross-sectional image is successfully decoded and the cross-sectional page coordinates are completely within the boundaries of the target webpage's first-screen screenshot, a cross-sectional screenshot overlay marker is generated; when the browser content gap value, browser text gap value, and browser resource gap value all do not exceed the corresponding browser content gap threshold, browser text gap threshold, and browser resource gap threshold, and no first-screen screenshot of the target webpage is required or a cross-sectional screenshot overlay marker has been generated, and the direct connection response gap value does not exceed the response gap threshold, the content cross-section between anchor points is written into the complete cross-section table; when any of the three browser gap values exceeds its respective threshold, or when a first-screen screenshot of the target webpage is required but no cross-sectional screenshot overlay marker is generated, and... When the direct connection response gap value does not exceed the response gap threshold, the content cross-section between anchor points is written into the browser's cross-section to be completed table; when the browser content gap value, browser text gap value, and browser resource gap value all do not exceed the corresponding browser content gap threshold, browser text gap threshold, and browser resource gap threshold, and no screenshot of the first screen of the target webpage is required or a cross-section screenshot has been generated to cover the marker, and the direct connection response gap value exceeds the response gap threshold, and there is a complete response record corresponding to the missing request matching key in the complete response record table of the simulated browser collection path, the content cross-section between anchor points is written into the direct connection cross-section to be completed table; other content cross-sections between anchor points generate cross-section collection failure records.
[0014] Furthermore, the specific steps for completing the browser's missing section using direct domain connection data collection and for completing the direct connection section using the complete response records of the simulated browser collection path are as follows: Read the start and end boundaries of the browser's missing section; extract the tag sequence from the direct content units between the two boundaries; and extract the structured data response and resource response body associated with the section request matching key set from the complete response record table of the direct domain connection collection path; copy the runtime document object model obtained from the simulated browser collection path; delete the browser content units between the two boundaries and insert the tag sequence; and return structured data according to the request matching key through the request interception interface. The system executes script tags in the tag sequence for the response and resource response body. When the number of incomplete network requests does not exceed the threshold for the number of incomplete requests, it saves the completed runtime document object model, re-collects visible text and network request logs, and re-collects the first screen screenshot of the target webpage when required. It also generates the browser position value of the completed content unit according to the depth-first traversal order. The system reads the set of cross-section request matching keys directly connected to the cross-section to be completed, extracts the complete response record associated with each request matching key from the complete response record table of the simulated browser collection path, and writes the response header, response body, and response time value into the complete response record table of the domain direct connection collection path.
[0015] Further, the specific steps for recalculating each gap value, verifying the completion results, and outputting webpage collection data and collection validity reports are as follows: After browser completion, browser content gap values, browser text gap values, and browser resource gap values are regenerated; if all three gap values do not exceed their respective thresholds, and it is not required that the first screen screenshot of the target webpage or the re-captured first screen screenshot of the target webpage can be decoded and cover the cross-sectional page coordinate range, and the browser position values of the completed content units are all between the browser position values of the start boundary and the end boundary, a browser cross-sectional completion pass record is generated; if browser completion does not generate a browser cross-sectional completion pass record, the browser completion data is cancelled and a cross-sectional collection failure record is generated; after direct connection completion, the direct connection response gap value is regenerated. If the direct connection response gap value does not exceed the response gap threshold, a direct connection cross-sectional completion pass record is generated; otherwise, a direct connection cross-sectional completion failure record is generated. When a cross-section completion record passes, the direct connection completion data is revoked and a cross-section acquisition failure record is generated. For complete cross-sections, cross-sections with browser completion pass records, and cross-sections with direct connection completion pass records, the runtime document object model, visible text, and target webpage first-screen screenshot in the simulated browser acquisition path are retained, as are the original page source code, structured data response, and response metadata in the domain direct connection acquisition path. Visible text is arranged according to the completed parent path and browser position value to generate structured text. The total number of cross-sections, the number of cross-sections that pass completion, and the number of cross-section acquisition failure records are counted, and an acquisition validity report is generated. The original page source code, runtime document object model, structured text, target webpage first-screen screenshot, original response metadata, browser cross-section completion pass records, direct connection cross-section completion pass records, and acquisition validity report are output according to the acquisition task number.
[0016] On the other hand, a web page data acquisition system based on dual-path integrity verification and completion is provided. This system is applied to the web page data acquisition method based on dual-path integrity verification and completion, and includes: The target input module is used to read the list of target web page addresses and the collection requirements record, and generate a web page collection task record; The network environment simulation unit is used to set network latency, bandwidth limits, packet loss, and response timeout. The dual-path acquisition module includes a simulated browser acquisition unit and a domain direct connection acquisition unit, which are used to collect runtime document object models, network request logs, raw page source code and structured data responses in parallel, and generate request matching keys and complete response records; The integrity assessment module is used to split the data from the two acquisition paths into browser content units and directly connected content units, generate dual-path content anchor chains and content cross-sections between anchor points, and calculate browser content gap values, browser text gap values, browser resource gap values and directly connected response gap values. The completion and fusion module is used to determine the browser sections to be completed and the direct connection sections to be completed based on the browser content gap value, browser text gap value, browser resource gap value and direct connection response gap value. It completes the completion and re-verification by using the domain direct connection data and the complete response record of the simulated browser collection path, respectively. The results output module is used to output the original page source code, the runtime document object model, structured text, the first screen screenshot of the target webpage, the original response metadata, and the collection validity report.
[0017] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: (1) By constructing a dual-path content anchor chain through the content identifier value and the double positional increment relationship, and dividing the content cross-section between the anchor points according to the adjacent anchor points, the browser content unit and the directly connected content unit are matched in a fine-grained manner, thereby reducing the risk of cross-path content mismatch.
[0018] (2) Different types of data missing are represented by browser content gap value, browser text gap value, browser resource gap value and direct connection response gap value, so that the local gap can be located to the content section between specific anchor points, and the whole page integrity evaluation can avoid the local omissions.
[0019] (3) The scope of completion is limited by the start boundary, the end boundary and the cross section request matching key set. The cross section to be completed in the browser is completed by directly collecting data from the domain name, and the direct connection cross section to be completed is completed by using the complete response record of the simulated browser collection path, thereby reducing the repeated collection of the whole page and the replacement of irrelevant content.
[0020] (4) By recalculating each gap value and verifying the positional boundaries of the completed content and the coverage of the first screen screenshot of the target webpage, the complete cross-section and the completed cross-section are preserved, thereby improving the integrity, reliability and verifiability of the webpage data collected. Attached Figure Description
[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0022] Figure 1 This is a flowchart of a web page data collection method based on dual-path integrity verification and completion. Figure 2 This is a flowchart of dual-path webpage data acquisition and cross-section completion; Figure 3 It is a closed loop diagram of anchor chain completion for dual-path content cross-section; Figure 4This is a structural diagram of a web page data acquisition system based on dual-path integrity verification and completion. Detailed Implementation
[0023] The technical solution of the present invention will now be described with reference to the accompanying drawings.
[0024] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.
[0025] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.
[0026] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.
[0027] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.
[0028] like Figure 1 As shown, this embodiment of the invention provides a web page data collection method based on dual-path integrity verification and completion. The processing flow of this method may include the following steps: S1 executes simulated browser data collection and direct domain connection data collection in parallel to obtain runtime document object model, network request logs, original page source code and structured data response, and generates request matching keys and complete response records; S2, the data from the two acquisition paths are split into browser content units and directly connected content units. Based on the relationship that the content identifier value, browser position value and directly connected position value are all increasing, dual-path content anchor chains are generated, and the content cross-section between anchor points is divided according to adjacent anchor points. S3 generates browser content gap values, browser text gap values, browser resource gap values, and direct connection response gap values for the content cross-sections between each anchor point, and determines the complete cross-section, browser to be completed cross-section, and direct connection to be completed cross-section based on each gap value; S4 uses direct domain connection to collect data to complete the browser's missing sections, uses the complete response records of the simulated browser's collection path to complete the direct connection's missing sections, recalculates each gap value and verifies the completion results, and outputs web page collection data and a collection validity report.
[0029] Optionally, the parallel execution of simulated browser data collection and direct domain connection data collection to obtain the runtime document object model, network request logs, original page source code, and structured data response involves the following steps: The target webpage address list and data collection requirement record are read from the data collection task table through the webpage data collection task configuration interface. The webpage data collection task configuration interface uses a structured data interface based on the Hypertext Transfer Security Protocol (HTTP), and the call request carries the data collection task number. The data collection task table records at least the data collection task number, target webpage address, task creation timestamp, and task execution status code value. After reading the target webpage address, the protocol name, domain name, port number, path, and query parameters in the target webpage address are parsed to identify target webpages that do not contain the protocol name. The address is supplemented with the protocol name specified by the task, and duplicate separators in the path are removed to ensure that both collection paths use the same standardized target webpage address. The collection requirement record includes source code collection markers, text collection markers, target webpage first screen screenshot collection markers, and a set of key text fields. The source code collection markers, text collection markers, and target webpage first screen screenshot collection markers are all recorded using binary encoded values. When the encoded value is 1, the corresponding data collection is executed; when the encoded value is 0, the corresponding data collection is skipped. The set of key text fields is written to the collection requirement record by the task initiator through the webpage collection task configuration interface. If the task initiator has specified each key text field, the user configuration is read; if the task initiator has not specified each key text field, the target webpage class is read. The corresponding field templates are determined based on stable business fields that need to be continuously collected and participate in integrity verification in similar web pages, and do not include navigation text, advertising text, or random recommendation text. The key text field set uses a unified character encoding to store the field name and content of each key text field. The key text field set can be empty; when it is empty, the number of key text fields in both the simulated browser collection path and the direct domain connection collection path is recorded as 0, and the browser text gap value is recorded as 0. Taking a dynamic product details webpage as an example, the key text field set includes product name, sales price, and inventory status. The standardized visible text and the text content of the directly connected content unit are respectively associated with the product name, sales price, and inventory status. The fields corresponding to the storage status are compared item by item. If the text content is the same, the corresponding key text field is considered to match. The target webpage address and the collection requirement record are associated according to the collection task number to generate a webpage collection task record. The webpage collection task record is written with the standardized target webpage address, collection requirement record, user agent field, language field, access credential field, collection start timestamp and task creation timestamp. The uniqueness of the combination of collection task number and target webpage address is checked to prevent the same target webpage address from being repeatedly associated in the same collection task. Based on the webpage collection task record, the simulated browser collection path and the domain direct connection collection path are started in parallel using the same user agent field, language field, access credential field and collection start timestamp.The data collection start timestamp is a base timestamp shared by both data collection paths. It is generated by the same data collection scheduling process based on the same system clock and written to the web page data collection task record. It is not used as the actual request sending timestamp for the simulated browser data collection path or the direct domain connection data collection path. After the data collection scheduling process writes the data collection start timestamp, it releases the dual-path startup barrier. Each of the two data collection paths establishes a data collection thread and reads the same data collection start timestamp. The actual request sending timestamps for each data collection path are written to the simulated browser network request log and the direct domain connection request log, respectively. The time difference between the actual access initiated by the two data collection paths does not exceed the maximum allowed time deviation, which is between 5 and 20 milliseconds, with the specific value depending on the data collection server thread scheduling delay. The delay in network client startup is determined; the user agent field contains the browser type, browser version number, and operating platform name; the language field contains the page's preferred language encoding; and the access credential field contains the session identifier value, access token value, and identity verification header field. This ensures that both data collection paths access the target webpage under the same access identity and content negotiation conditions, reducing the probability of the web server returning different content due to differences in access context. The simulated browser data collection path loads the target webpage address and executes the page script. It collects the runtime document object model, visible text, node page coordinates, resource loading records, simulated browser network request logs, and a screenshot of the target webpage's first screen through the browser debugging interface. The simulated browser data collection path uses a headless browser. The browser debugging interface sequentially enables page control, network listening, runtime environment listening, and document object model listening. The page script execution completion condition is that the document load event has been triggered and no new critical network requests have been added within 300 to 1000 milliseconds. The duration is determined based on the 95th percentile of the historical resource loading time of the target webpage. The runtime document object model reads the node hierarchy after the page script execution, recording node tag names, parent paths, node attributes, text content, and node arrangement order. Visible text is obtained by reading node rendering styles and node bounding boxes, retaining only text content that is not hidden, has a transparency greater than zero, and whose node page coordinates are within the current page range. Node page coordinates are obtained through... The node bounding box interface obtains the x-coordinate, y-coordinate, width, and height of the top-left corner and converts them into pixel coordinates relative to the top-left corner of the page. The resource loading record records the resource address, resource type, request timestamp, response timestamp, response status code, and actual received byte length. The simulated browser network request log records the request method, initial access address, final access address, request body, request timestamp, response headers, response body, and transmission end marker. The target webpage's first-screen screenshot is generated by the page screenshot interface after the first screen rendering is complete. The screenshot area uses the page viewport boundaries, the screenshot file is saved in a lossless image format, and the screenshot width, height, and generation timestamp are recorded.The direct domain connection data collection path sends hypertext transfer requests to the target webpage address, collecting the original page source code, structured data response, hypertext fragment response, direct domain connection request log, response headers, server address, and response time value. The direct domain connection data collection path uses a network client that supports connection reuse and redirection following. It records the webpage collection task in the request header and sends the request, sequentially writing the initial request address, each redirect address, and the final access address into the direct domain connection request log. The original page source code is extracted from the main document response body and retains the original byte data. Text decoding is performed based on the character encoding field in the response header. If the response header does not record a character encoding field, the byte order mark and page character encoding declaration are read sequentially. The structured data response is identified based on the media type field in the response header and saves the field hierarchy path, field name, field content, and source request matching key. The hypertext fragment response is collected from media types that are in hypertext format and have not been modified. Extracting and recording tag positions, tag content, and source request matching keys from the response body returned by the main document; recording request methods, final access addresses, request bodies, request timestamps, response timestamps, and transmission end markers from the direct domain request log; saving response headers according to field names and content, using the actual successful establishment of the network connection's remote Internet Protocol address for the server address, and the difference between the timestamp of the first byte of the request and the timestamp of the last byte of the response, in milliseconds; writing all collected data into the same collection batch according to the collection task number, target webpage address, and collection path encoding value, enabling the running document object model to be associated with the original page source code, structured data response, and network request logs in subsequent steps based on request matching keys, content identifier values, browser order values, and direct connection order values, thereby simultaneously preserving the running state after page script execution and the original response state returned by the target server.
[0030] This implementation plan, by simultaneously retaining the running state of the page script execution and the original server response state under unified access identity, unified time benchmark, and unified collection batch, ensures that the data formed by the two collection paths have consistent access context, comparable time relationship, and clear source association. This provides a stable data foundation for subsequent request matching key construction, dual-path content anchor chain generation, and content cross-section verification between anchor points, reducing mismatches and missed collections caused by differences in access conditions, character encoding differences, and asynchronous loading, and improving data integrity, association accuracy, and batch collection reliability in web page big data processing.
[0031] Optionally, the specific steps for generating request matching keys and complete response records are as follows: Read the request method, final access address, request body, and request timestamp from the simulated browser network request log and the domain direct connection request log, respectively; establish a composite index for both the simulated browser network request log and the domain direct connection request log according to the collection task number, collection path encoding value, and request timestamp, so that requests within the same webpage collection task can be sorted according to the collection path. Before reading the request data, convert all letters in the request method to uppercase; convert the protocol name and domain name in the final access address to lowercase, remove the URL fragment identifier and the default protocol port, and retain the path, query parameter name, query parameter content, and query parameter order to generate a standardized final access address, avoiding discrepancies in character case and default port syntax that could prevent the same request from being matched. Before generating the request body digest value, the request body is normalized according to the media type field in the request header. For structured data request bodies, after parsing the field hierarchy, the data is reserialized according to the character order of the field names while preserving the array element order. For form request bodies, the field names and content are decoded using percent signs and then arranged according to the character order of the field names and content. For other text request bodies, byte order markers are removed, newline characters are standardized to newline characters, and the data is converted to a byte sequence using UTF-8 encoding. Binary request bodies retain the original byte sequence. When the request body is empty, a zero-length byte sequence is used. The normalized byte sequence is input into the SHA-256 digest function to obtain the request body digest value, which consists of lowercase hexadecimal characters. This ensures that the same request body generates the same request body digest value in both acquisition paths and provides a fixed and reproducible digest result for empty request bodies. The request method, standardized final access address, request body summary value, and same-address request sequence number are concatenated sequentially to form the request matching key, with field separators written between adjacent fields during concatenation. The request matching key does not include the collection task number; the collection task number, collection path encoding value, and request matching key together form the retrieval index for the request record, preventing data commingling between different webpage collection tasks. Same-address request sequence numbers are generated in ascending order based on the timestamps of requests with the same request method and standardized final access address within the same collection path. When multiple requests exist within the same millisecond, they are arranged in ascending order of their sequence numbers according to the records in the corresponding network request logs, and then continuously assigned same-address request sequence numbers starting from 1, ensuring a reproducible order for repeated polling requests, pagination requests, and script retry requests.After generating the request matching key, the request and response records are associated according to the request matching key, and the response header, the original byte data of the response body, the actual received byte length value, the transmission end marker, and the response reception timestamp are read. The actual received byte length value is the number of response body payload bytes accumulated after deleting the transmission chunk boundaries and protocol frame headers and before decompressing the response body, ensuring that the actual received byte length value and the declared byte length value in the response header use the same measurement caliber. When the response header contains a declared byte length value, responses with the same actual received byte length value as the declared byte length value and whose response bodies are successfully parsed according to the media type after decompression are written to the complete response record table. When the response header does not contain a declared byte length value, responses with a transmission end marker and whose response bodies are successfully parsed according to the media type after decompression are written to the complete response record table. The declared byte length value is read from the content length field of the response header. The transmission end marker is written by the network acquisition interface according to the protocol layer end event corresponding to the current response: For simulated browser acquisition paths, the transmission end marker is written when the browser debugging interface returns the network loading completion event of the current request; for domain direct connection acquisition paths using chunked transmission, the transmission end marker is written after receiving a zero-length termination block and triggering a message completion event; for multiplexed stream transmission, the transmission end marker is written after receiving the end frame of the current response stream; for connection closure to define the end of the response body, the transmission end marker is written after receiving the end event of the current response read stream. In connection multiplexing scenarios, physical connection closure is not used as the transmission end condition. When parsing the response body, decompression is first performed according to the content encoding field in the response header. If the response header does not record a content encoding field, the raw byte data of the response body is read directly. Then, the parsing method is selected according to the media type field. Structured data responses are parsed at the field level, hypertext responses are parsed by character decoding and tag closure checks, plain text responses are parsed by character decoding and character integrity checks, and binary resources are parsed according to the file structure of the corresponding media type. For text character encoding, the character encoding field in the response header is read first. If the character encoding field is missing, the byte order mark and page character encoding declaration are read in sequence. Text parsing is considered successful when no illegal byte substitution, character truncation, or termination of structure parsing occurs. For binary resources, binary resource parsing is considered successful when the file header, body data, and end structure can be read. The complete response record table saves the response header, response body, response time value, actual received byte length value, and transmission end mark according to the collection task number, collection path encoding value, and request matching key. It forms complete response records for simulated browser collection paths and complete response records for direct domain connection collection paths, respectively, providing clear and verifiable response data for subsequent cross-section request matching key set comparison and direct connection to be completed cross-section completion.
[0032] This implementation scheme establishes a stable correspondence between the network request logs of the simulated browser and the direct connection request logs of the domain under a unified identification rule and a unified integrity criterion. This reduces the risk of mismatch caused by duplicate requests, differences in address representation, and incomplete response transmission. It ensures that complete response records have a clear source and are verifiable, providing a reliable data foundation for subsequent comparison of cross-section request matching key sets, calculation of direct connection response gap values, and completion of direct connection cross-sections to be completed.
[0033] Optionally, the specific steps for splitting the data from the two acquisition paths into browser content units and directly connected content units are as follows: Read the running document object model, original page source code, structured data response, and hypertext fragment response according to the acquisition task number, and use the final access address of the corresponding target webpage as the relative address conversion benchmark, so that the browser content unit and the directly connected content unit adopt the same text processing rules, resource address conversion rules, and position numbering rules. The runtime document object model is divided into browser content units according to a depth-first traversal order, recording tag names, parent paths, content identifier values, text content, text digest values, absolute resource addresses, browser order values, and node page coordinates. Starting from the root node of the runtime document object model, the traversal proceeds according to the order of child nodes in each parent node, identifying each element node as a browser content unit. Upon first access to an element node, browser order values are generated consecutively starting from 1. Tag names are recorded using lowercase characters of the element node tag names. Parent paths use a hierarchical path format separated by slashes. Each path segment consists of the node name and the index of the same-named node under the same parent node, and path segments are connected according to the hierarchical order from the page root node to the parent node of the current element node. The content identifier value is read according to the order of id, name, and data-key attributes, starting with the first non-empty attribute value. If all three attributes are empty, the content identifier value is written as an empty string. Before generating the text digest value, the text content is converted to a unified character standard format, tabs, carriage returns, and newlines are converted to spaces, leading and trailing whitespace characters are removed, consecutive whitespace characters are compressed into a single space, and Latin letters are converted to lowercase. Then, it is converted to a byte sequence according to UTF-8 encoding and input into the SHA-256 digest function. If the text content is empty, the SHA-256 digest operation is performed on the zero-length byte sequence. The absolute resource address is read according to the order of src, href, and poster attributes, starting with the first non-empty address. The relative address is merged with the final access address, and path normalization is performed. The node page coordinates adopt the absolute page coordinate system, with the top left corner of the target webpage document area as the origin, the horizontal rightward direction of the page as the positive X-axis, and the vertical downward direction of the page as the positive Y-axis. The boundary rectangle of the element node relative to the current viewport is read through the browser debugging interface. The coordinates of the boundary rectangle are added to the page scroll distance to generate the absolute horizontal and vertical coordinate values of the page, and the width and height values are recorded. Hidden nodes and zero-area nodes are not written to the node page coordinates, but the corresponding browser content unit is retained.Based on the response reception order and the tag position or field traversal order within the response, the original page source code, structured data response, and hypertext fragment response are split into directly connected content units, recording tag names, parent paths, content identifier values, field names, text content, text digest values, absolute resource addresses, source request matching keys, and directly connected position values. First, the complete response records of the domain's directly connected data collection path are arranged in ascending order of response reception timestamp; if response reception timestamps are the same, they are arranged according to the request matching key. For the original page source code and hypertext fragment responses, a hypertext parser is used to identify tags, tag attributes, and tag text, determining each starting tag as a directly connected content unit, and arranging them according to the starting tag's position in the response body. The parent path uses the same node name and node number hierarchical path format as the browser's content unit. For the structured data response, a structured data parser is used to perform a depth-first traversal, determining each leaf field as a directly connected content unit, and object fields are arranged according to... The response body is traversed in the order of appearance. Array elements are traversed from smallest to largest according to their array indices. Object field path segments in the parent path consist of field names and field numbers with the same name, while array path segments consist of array names and array indices. Field names are written to the current leaf field name, tag names are written to the field type name, and text content is written in the character form of the leaf field content. The id, name, and key fields in the same parent path provide content identifier values in order. Directly connected content units use the same text normalization rules, text digest value generation rules, and absolute resource address conversion rules as browser content units. The source request matching key is read from the complete response record carrying the corresponding response. Directly connected positional values are generated continuously starting from 1 according to the response reception order of the complete response record and the tag position or field traversal order within the response body. This provides browser content units and directly connected content units with consistent field structures and clear arrangement order for candidate pairing record generation and dual-path content anchor chain construction.
[0034] In this implementation plan, the runtime document object model, original page source code, structured data response, and hypertext fragment response are uniformly converted into browser content units and directly connected content units with consistent field structures and clear positional relationships. This enables the page content in the two acquisition paths to have comparable data representation and traceable source relationships, reducing pairing biases caused by differences in tag levels, text formats, and resource addresses. This provides a stable foundation for candidate pairing record screening, dual-path content anchor chain construction, and content cross-section division between anchor points, thereby improving the accuracy of cross-path content association and the reliability of subsequent gap identification.
[0035] Optionally, the specific steps for generating dual-path content anchor chains based on the increasing relationship between content identifier values, browser position values, and direct connection position values are as follows: Browser content units and directly connected content units are read according to the collection task number. Leading and trailing whitespace characters are removed from the content identifier values, character widths are standardized, and letter case is unified. Browser content units are arranged in ascending order of browser position value, and directly connected content units are arranged in ascending order of direct connection position value, ensuring that candidate pairing records are generated under a unified character comparison and positional comparison standard. When the content identifier values of both browser content units and directly connected content units are non-empty and identical, they are written into the candidate pairing record. The candidate pairing record includes the browser content unit number, directly connected content unit number, browser position value, direct connection position value, and content identifier value matching flag. Simultaneously, the tag name, parent path, text summary value, and absolute resource address are compared, and the number of identical items is recorded as the number of matching items, ensuring that candidate pairing records with identical content identifier values have matching strength data for subsequent sequence filtering. When at least one content identifier value is empty, the tag name, parent path, text digest value, and absolute resource address are compared. When the number of identical items reaches the content matching threshold, a candidate pairing record is written, and the number of matching items is recorded. Tag names are considered identical if all characters are identical; parent paths are considered identical if the number of path levels, the names of nodes at each level, and the sequence number of nodes with the same name are all identical; text digest values are considered identical if the digest character sequence is all identical; and absolute resource addresses are considered identical if the protocol name, domain name, path, and query parameters are all identical. The content matching threshold is an integer between 2 and 4, determined based on manually confirmed pairing samples of browser content units and directly connected content units in historical collection tasks of the same webpage type. The integer corresponding to the minimum sum of mismatches and omissions is selected as the content matching threshold. When a browser content unit corresponds to multiple candidate pairing records, all candidate pairing records are retained, and candidate pairing records are not deleted during the local comparison stage. This allows subsequent sequence filtering to combine the global positional relationship between the two collection paths to eliminate local mismatches caused by duplicate tags, duplicate text, and duplicate absolute resource addresses.Constrained by both browser positional value and direct connection positional value increasing, the candidate pairing record sequence with the largest number of records is selected from the candidate pairing records. First, the candidate pairing records are arranged in ascending order of browser positional value and ascending order of direct connection positional value. For each candidate pairing record, a sequence length value, cumulative matching item value, and predecessor record number are established. Using the current candidate pairing record as the last record of the sequence, candidate pairing records with browser positional value less than the current browser positional value and direct connection positional value less than the current direct connection positional value are retrieved. The retrieval results are used as optional predecessor records. The sequence length value of the optional predecessor record is increased by 1, and the cumulative matching item value is increased by the current number of matching items. The optional predecessor record with the largest sequence length value is selected and written into the predecessor record number. The sequence length value and cumulative matching item value of the current candidate pairing record are updated to ensure that the candidate pairing records maintain the same arrangement direction in the two acquisition paths and to eliminate content disorder caused by the intersection of pairing relationships. When there are multiple candidate pairing record sequences with the same number of records, the candidate pairing record sequence with the largest cumulative value of the number of matching items is selected; when the cumulative value of the number of matching items is still the same, the candidate pairing record sequences with smaller browser position values and smaller direct connection position values are selected in turn; starting from the selected last record, backtracking is performed according to the predecessor record number, and the backtracked records are arranged in reverse order to generate a candidate pairing record sequence in which both browser position values and direct connection position values are increasing. Write the page root node start boundary record at the front of the candidate pairing record sequence, and write the end boundary record of the last content unit of the two collection paths at the back of the candidate pairing record sequence. Connect the records to generate a dual-path content anchor chain. The page root node start boundary record is the start boundary record used by both collection paths. The browser side is associated with the document root node of the runtime document object model, and the direct connection side uses the virtual response root boundary located before the first directly connected content unit. The original page source code uses the document root tag as the internal parsing starting point, the structured data response uses the response root object as the internal parsing starting point, and the hypertext fragment response uses the fragment virtual root tag as the internal parsing starting point. Each internal parsing starting point is only used to determine the parent path and direct connection position value of the directly connected content unit and is not written as a candidate pairing record into the dual-path content anchor chain.The starting boundary record of the page root node is written with the boundary type encoding value, the browser starting position value, and the direct connection starting position value, both of which are set to 0. The ending boundary record of the last content unit is written with the ending boundary type encoding value, the browser ending position value, and the direct connection ending position value. The browser ending position value is the maximum browser position value of the browser content unit plus 1, and the direct connection ending position value is the maximum direct connection position value of the direct connection content unit plus 1. Anchor chain position values are continuously written in the order of the starting boundary record of the page root node, the candidate pairing record sequence, and the ending boundary record of the last content unit, and anchor chain connection relationships are written between adjacent records to generate a dual-path content anchor chain that simultaneously carries browser position relationship and direct connection position relationship. This provides a consistent and reproducible data foundation for subsequent division of content sections between anchor points according to adjacent anchor points.
[0036] In this implementation scheme, a globally consistent dual-path content anchor chain is formed by utilizing the increasing relationship between content identifier values, the number of matching items, and both browser position values and direct connection position values. This establishes a unique and verifiable correspondence between browser content units and directly connected content units while maintaining their original arrangement relationships. This reduces the risk of local mismatches and order overlaps caused by duplicate tags, duplicate text, and duplicate resource addresses. It also provides a continuous boundary basis for the stable division of content sections between anchor points, the location of local gaps, and cross-path directional completion, thereby improving the accuracy of dual-path webpage data association and the reliability of subsequent integrity verification.
[0037] Optionally, the specific steps for dividing the content cross-section between anchor points according to adjacent anchor points are as follows: Read the dual-path content anchor chain, browser content unit and directly connected content unit according to the collection task number; read the page root node start boundary record, candidate pairing record and last content unit termination boundary record in order of anchor chain position value from small to large; and read the browser position value and directly connected position value in each record respectively, so that the same group of adjacent records simultaneously defines the content interval in the two collection paths. Based on the arrangement order of the dual-path content anchor chains, the browser content units and directly connected content units between two adjacent anchor chain records are collectively divided into a content section between anchor points. The anchor chain record on the left is used as the starting boundary, and the anchor chain record on the right is used as the ending boundary. Browser content units with browser position values greater than the starting boundary browser position value and less than the ending boundary browser position value are extracted. Directly connected content units with direct position values greater than the starting boundary direct position value and less than the ending boundary direct position value are also extracted. The two types of content units are grouped according to the collection task number and the anchor chain position interval. When the starting boundary is the starting boundary record of the page root node, the browser starting position value 0 and the direct starting position value 0 are used as the lower boundary of the interval. When the ending boundary is the ending boundary record of the last content unit, the browser ending position value and the direct ending position value are used as the upper boundary of the interval, so that the first segment, the middle segment, and the last segment of the page all have a defined division range. When the number of browser content units is zero, the directly connected content units in the content section between anchor points are retained. When the number of directly connected content units is zero, the browser content units in the content section between anchor points are retained, so that missing content on one side can generate corresponding gap values in the future.The source request matching keys of directly connected content units are aggregated, and request matching keys that match the absolute resource addresses within the content segments between anchor points are extracted from the simulated browser network request logs, generating a segment request matching key set. When generating the segment request matching key set, the source request matching keys of each directly connected content unit within the content segments between anchor points are first read, then the absolute resource addresses in the browser content units and directly connected content units are aggregated. The absolute resource addresses, as well as the initial access address, redirect address, and final access address in the simulated browser network request logs, are uniformly normalized. Protocol names and domain names are converted to lowercase, default protocol ports and URL fragment identifiers are removed, current directory markers, parent directory markers, and repeating separators in the path are parsed, and percent sign encoding is standardized. The system retains the query parameter names, content, and order. When the normalized absolute resource address matches any of the initial, redirected, or final access addresses, the corresponding request matching key is read. When the final access address changes due to redirection or content distribution network address translation, the corresponding request matching key is traced back based on the initial access address and redirected address chain in the same network request log. When the resource is returned by the browser cache, if the simulated browser network request log has recorded the cache hit flag and request matching key, the corresponding request matching key is written into the cross-section request matching key set. When a match cannot be completed by address and the source request matching key has been recorded in the directly connected content unit, the source request matching key is used as the basis for associating the cross-section content with the response record. Duplicate values are removed from all request matching keys, and the cross-section request matching key set is written according to the first occurrence order of the source request matching key in the directly connected positional value and the first occurrence order of the simulated browser request matching key in the request sending timestamp, so that the same request matching key is retained only once. The smallest bounding rectangle of the valid visible node page coordinates within the cross-section is taken as the cross-section page coordinate range. The start boundary, end boundary, cross-section request matching key set, and cross-section page coordinate range are written into the content cross-section record between anchor points. Before calculating the cross-section page coordinate range, node page coordinates whose coordinate values cannot be converted to finite values, whose width values are not greater than zero, or whose height values are not greater than zero are deleted. When the node page coordinates exceed the page document boundary, the excess part is truncated to the page document boundary. If the width and height values are both greater than zero after truncation, the node page coordinates are retained. For visible nodes that span multiple scrolling areas, since the node page coordinates use the page absolute coordinate system, the minimum bounding rectangle is calculated directly based on the absolute page coordinates of each node, without segmenting according to the boundaries of a single viewport. The horizontal, vertical, width, and height values of each valid visible node's page coordinates are read, and the minimum value among all horizontal coordinate values is used as the left boundary, the minimum value among all vertical coordinate values is used as the top boundary, the maximum value of the sum of the horizontal coordinate value and the width value is used as the right boundary, and the maximum value of the sum of the vertical coordinate value and the height value is used as the bottom boundary. The left, top, right, and bottom boundaries are then written into the cross-sectional page coordinate range.If there are no browser content units within the content cross-section between anchor points, no valid node page coordinates for any browser content units, or all node page coordinates are invalid after verification, the page coordinate range of the cross-section will be set to null. Subsequent verification of the target webpage's first-screen screenshot overlay will not generate a cross-section screenshot overlay marker. The start and end boundaries are written to the browser position value, direct connection position value, and anchor chain position value in the corresponding anchor chain record, respectively. The cross-section request matching key set retains the collection path source for each request matching key. The content cross-section record between anchor points establishes a retrieval index according to the collection task number and the anchor chain position range, enabling browser content units, direct connection content units, request response sources, and page space ranges to be associated under the same cross-section boundary. This provides a consistent data foundation for subsequent calculations of browser content gap values, browser text gap values, browser resource gap values, and direct connection response gap values.
[0038] This implementation scheme establishes a complete association between browser content units, directly connected content units, cross-section request matching key sets, and cross-section page coordinate ranges under unified start and end boundaries. This reduces misjudgments of gaps caused by cross-section content concatenation, misplaced response sources, and page space range offsets. It also ensures that content cross-sections between anchor points simultaneously possess three verifiable criteria: content order, response source, and page position. This provides a stable boundary foundation for the accurate generation of browser content gap values, browser text gap values, browser resource gap values, and directly connected response gap values, and improves the reliability of subsequent cross-section classification and targeted completion.
[0039] Optionally, the specific steps for generating browser content gap values, browser text gap values, browser resource gap values, and direct connection response gap values for each anchor point content cross-section, and determining the complete cross-section, browser to be completed cross-section, and direct connection to be completed cross-section based on each gap value are as follows: Read the browser content unit, direct connection content unit, candidate pairing record, cross-section request matching key set, cross-section page coordinate range, simulated browser network request log, complete response record table of domain direct connection collection path, target webpage first screen screenshot collection marker, and key text field set according to the collection task number and anchor point content cross-section record. Then, read the browser content gap threshold, browser text gap threshold, browser resource gap threshold, and response gap threshold from the integrity parameter table. Each threshold is taken from 0 to 3, with the specific value determined based on gap samples manually reviewed in historical collection tasks of the same webpage type. Select the gap number with the fewest cross-section misclassifications as the current threshold. The number of directly connected content units within the content cross-section between anchor points that do not form a candidate pairing record with the browser content unit is counted, generating a browser content gap value. Before counting, candidate pairing records are retrieved based on the directly connected content unit number. If there is at least one candidate pairing record for the same directly connected content unit, it is recorded as paired; if there is no candidate pairing record, it is recorded as unpaired. The same unpaired directly connected content unit is counted only once, so that the browser content gap value represents the number of content units that the domain direct connection acquisition path has obtained but the simulated browser acquisition path has not obtained. The visible text in the simulated browser acquisition path and the text content of directly connected content units in the domain direct acquisition path are compared item by item with the key text field set to count the number of key text fields with the same text content. Before comparison, the visible text, the text content of directly connected content units, and the key text field content are processed by hypertext entity restoration, unified character encoding conversion, Unicode normalization, unification of full-width and half-width characters, conversion of tabs and newlines, deletion of leading and trailing whitespace characters, compression of consecutive whitespace characters, unification of Latin letter capitalization, and normalization of punctuation characters. The processed key text field content is compared with the processed text content of each visible text and directly connected content unit. If the character sequence is completely consistent, the text content is considered to be the same. The same key text field is counted only once in the same acquisition path. Multilingual text retains the original language characters and character arrangement order. Only unified character encoding conversion and Unicode normalization are performed. No language translation or semantic replacement is performed to make the two acquisition paths adopt a reproducible text comparison standard. The browser text gap value is generated by subtracting the number of key text fields in the simulated browser collection path from the number of key text fields in the direct domain collection path and setting the negative value to zero. When the key text field set is empty, the number of key text fields in both collection paths and the browser text gap value are both recorded as 0.The system counts the number of absolute resource addresses in directly connected content units and the number of resources with the same address that form complete response records in the simulated browser network request logs. The browser resource gap value is generated by subtracting the number of resources forming complete response records from the number of absolute resource addresses and setting negative values to zero. Before counting, duplicate absolute resource addresses in directly connected content units are removed, and the absolute resource addresses and the final access addresses in the simulated browser network request logs are processed by lowercase protocol names, lowercase domain names, deleting default ports, deleting URL fragment identifiers, unifying percent sign encoding, and normalizing paths. When the normalized addresses are consistent and the corresponding responses meet the conditions for writing complete response records, the corresponding resources are included in the number of resources forming complete response records. The same absolute resource address is counted only once, so that the browser resource gap value represents the number of resources for which the domain direct connection acquisition path has been identified but the simulated browser acquisition path has not formed complete response records. Each request matching key in the cross-section request matching key set is compared item by item with the complete response record table of the direct domain connection collection path. The number of request matching keys for which no complete response record was found is counted, and a direct connection response gap value is generated. During the comparison, complete response records are retrieved according to the collection task number, collection path code value and request matching key. If the same request matching key has a response header, response body and response time value, it is recorded as a complete response record that has been found. If no complete response record is found, it is recorded as a missing request matching key. The same missing request matching key is counted only once. When capturing a screenshot of the first screen of a target webpage, the image should be cropped according to the cross-sectional page coordinate range. Before cropping, the width and height of the first screen screenshot of the target webpage should be read. The left, top, right, and bottom boundaries of the cross-sectional page coordinate range should be converted into pixel coordinates of the first screen screenshot of the target webpage. Cropping should be performed when the left boundary is not less than zero, the top boundary is not less than zero, the right boundary is not greater than the width of the first screen screenshot of the target webpage, and the bottom boundary is not greater than the height of the first screen screenshot of the target webpage. A cross-sectional screenshot overlay marker should be generated when the cross-sectional image is successfully decoded and the cross-sectional page coordinate range is completely within the boundaries of the first screen screenshot of the target webpage. No cross-sectional screenshot overlay marker should be generated when the cross-sectional page coordinate range is empty, the cross-sectional image width is zero, the cross-sectional image height is zero, or the image decoding fails. If the browser content gap value, browser text gap value, and browser resource gap value all do not exceed their respective thresholds, and a screenshot of the first screen of the target webpage is not required, and the direct connection response gap value does not exceed the response gap threshold, then the content cross-section between anchor points will be written into the complete cross-section table. If the browser content gap value, browser text gap value, and browser resource gap value all do not exceed their respective thresholds, and a screenshot of the first screen of the target webpage is required and a cross-section screenshot overlay mark has been generated, and the direct connection response gap value does not exceed the response gap threshold, then the content cross-section between anchor points will be written into the complete cross-section table.If at least one of the browser content gap value, browser text gap value, and browser resource gap value exceeds the corresponding threshold, and the direct connection response gap value does not exceed the response gap threshold, the content cross-section between anchor points will be written into the browser's cross-section to be completed table; if a screenshot of the first screen of the target webpage is required but no cross-section screenshot overlay mark is generated, and the direct connection response gap value does not exceed the response gap threshold, the content cross-section between anchor points will be written into the browser's cross-section to be completed table. If the browser content gap value, browser text gap value, and browser resource gap value all do not exceed their respective thresholds, and a screenshot of the first screen of the target webpage is not required, but the direct connection response gap value exceeds the response gap threshold, and there are complete response records corresponding to each missing request matching key in the complete response record table of the simulated browser collection path, then the content cross-section between anchor points will be written into the direct connection to be completed cross-section table. If the browser content gap value, browser text gap value, and browser resource gap value all do not exceed their respective thresholds, and a screenshot of the first screen of the target webpage is required and a cross-section screenshot overlay mark has been generated, but the direct connection response gap value exceeds the response gap threshold, and there are complete response records corresponding to each missing request matching key in the complete response record table of the simulated browser collection path, then the content cross-section between anchor points will be written into the direct connection to be completed cross-section table. The remaining anchor points generate cross-section collection failure records; the cross-section collection failure records are written with the collection task number, the content cross-section number between anchor points, the start boundary, the end boundary, the browser content gap value, the browser text gap value, the browser resource gap value, the direct connection response gap value, the cross-section screenshot overlay marker generation status, and the missing request matching key, so that the complete cross-section, the browser cross-section to be completed, the direct connection cross-section to be completed, and the cross-section collection failure records form mutually exclusive categories, providing a clear processing entry point for subsequent cross-path targeted completion.
[0040] This implementation plan uses a multi-dimensional integrity assessment of content sections between anchor points based on browser content gap values, browser text gap values, browser resource gap values, direct connection response gap values, and section screenshot overlay markers. This allows missing content units, missing key text, missing resources, and missing responses to be uniformly mapped to complete sections, browser sections to be completed, direct connection sections to be completed, and sections collection failure records. This reduces misjudgments and omissions caused by single evaluation indicators, provides a clear processing entry point for subsequent cross-path targeted completion, and improves the accuracy of section classification, the reliability of completion decisions, and the completeness of collection results in web page big data processing.
[0041] Optionally, the specific steps for using direct domain connection data collection to complete the browser's missing sections and using the complete response records of the simulated browser collection path to complete the directly connected missing sections are as follows: Read the browser's missing section table, the directly connected missing section table, the dual-path content anchor chain, the browser content unit, the directly connected content unit, the section request matching key set, the complete response record table of the direct domain connection collection path, and the complete response record table of the simulated browser collection path according to the collection task number, and read the threshold for the number of incomplete requests from the integrity parameter table; the threshold for the number of incomplete requests is set to 0 to 3, determined based on the distribution of the number of incomplete network requests after the execution of the page script in historical collection tasks of the same webpage type, selecting the number that ensures the completed page content remains complete and has the shortest processing time as the current value. The system reads the start and end boundaries of the browser's cross-section to be completed, extracts tag sequences from the directly connected content units between the two boundaries, and extracts the structured data response and resource response body associated with the cross-section request matching key set from the complete response record table of the domain direct connection collection path. When extracting the tag sequence, the directly connected content units between the two boundaries are arranged in ascending order of their direct connection position values. Tag names, text content, and parent paths are read from the directly connected content units, and tag attributes are read from the original page source code and hypertext fragment response corresponding to the directly connected content units, recombining them into a tag sequence. The directly connected content units corresponding to the structured data response are retained as request return data and are not directly written into the tag sequence. When multiple directly connected content units exist under the same parent path, the order of tags at the same level is restored according to the direct connection position value. The system searches the complete response record table of the domain direct connection collection path item by item according to the cross-section request matching key set, extracting only complete response records that simultaneously record the response header, response body, and response time value, and whose response body has been successfully parsed according to media type after decompression. The structured data response and resource response body are distinguished based on the media type field in the response header. The runtime document object model obtained by simulating the browser's crawling path is copied. Browser content units between the two boundaries are deleted, and a tag sequence is inserted. The structured data response and resource response body are returned via the request interception interface according to the request matching key, and the script tags in the tag sequence are executed. When copying the runtime document object model, the hierarchical relationship between the nodes corresponding to the starting boundary, the ending boundary, the page root node, and the nodes outside the two boundaries is preserved. The insertion position after the node corresponding to the starting boundary is taken as the starting point of the tag sequence, and the insertion position before the node corresponding to the ending boundary is taken as the ending point of the tag sequence. A copy of the cross-section node before completion is saved. When deleting browser content units between the two boundaries, the node references and event listener references corresponding to the browser content units are deleted simultaneously. After inserting the tag sequence, nodes are built layer by layer according to the parent path. If the parent path does not exist, the missing hierarchy is filled down from the most recent parent node, and the insertion order of sibling nodes is determined according to the direct connection position value.Script tags execute within the independent page context corresponding to the copied runtime Document Object Model. This independent page context uses the user agent, language, and access credential fields from the web page scraping task record and disables direct network access to the target server. Only script tags in the tag sequence between the start and end boundaries are executed; script tags outside these boundaries are not executed. Script tags are executed sequentially according to their order of appearance in the tag sequence. Inline scripts execute directly within the independent page context, while resource requests triggered by external scripts are handled through the request interception interface. The request interception interface reads the request method, final access address, request body digest value, and request sequence number at the same address from the script-triggered request, generates a request matching key, and compares the request matching key with the cross-sectional request matching key set item by item. If a match is successful, the response header and response body from the corresponding complete response record are returned; if no match is found, the current network request is blocked and counted as an incomplete network request. The nodes, text content, and resource requests generated by the script execution are written to a copied runtime document object model, retaining only the node changes between the start and end boundaries. If a script execution exception occurs, the browser debugging interface returns an execution timeout status, or the script modifies nodes outside the two boundaries, subsequent script execution is stopped, the current completion result is discarded, and a copy of the cross-section nodes before completion is restored. If the number of incomplete network requests does not exceed the incomplete request threshold, the completed runtime document object model is saved, visible text and simulated browser network request logs are re-collected, and when a screenshot of the target webpage's first screen is required, it is re-collected, generating browser position values for the completed content units according to depth-first traversal order. The number of incomplete network requests is obtained by counting requests that have been sent but have not yet received a transmission end marker. During the counting process, persistent connection requests and monitoring reports are deleted. If the number of incomplete network requests does not exceed the incomplete request threshold for 300 to 1000 milliseconds, the current completion cross-section is considered to have entered a stable state. The duration is determined based on the 95th percentile of resource loading completion time in historical collection tasks of the same webpage type. After saving the completed runtime document object model, the nodes between the start and end boundaries are re-split according to the browser content unit generation rules. Tag names, parent paths, content identifier values, text content, text digest values, absolute resource addresses, browser position values, and node page coordinates are regenerated, and candidate pairing records are updated based on the regenerated browser content units. When re-capturing the first screen screenshot of the target webpage, the page viewport width, page viewport height, and page scaling ratio before completion are used to ensure that the cross-sectional page coordinate range before and after completion uses the same pixel reference.The system reads the set of request matching keys for the directly connected segments to be completed. It extracts the complete response records associated with each request matching key from the complete response record table of the simulated browser's collection path, and writes the response header, response body, and response time value into the complete response record table of the domain's directly connected collection path. When extracting complete response records, it searches by collection task number, target webpage address, and request matching key, and confirms that the source complete response record meets the conditions for writing complete response records. The complete response record table of the simulated browser's collection path and the complete response record table of the domain's directly connected collection path use the same field format. Before writing, the response header field names are converted to a unified character format, the original byte content of the response body remains unchanged, the response time value is uniformly converted to milliseconds, the actual received byte length value is generated based on the number of original bytes in the response body, and the transmission end marker in the source complete response record is inherited. During writing, a unique index is established using the collection task number, target webpage address, domain's directly connected collection path encoding value, and request matching key. Writing is only performed on request matching keys for which no complete response record is found in the complete response record table of the domain's directly connected collection path; if a corresponding complete response record already exists, the original record is retained and not written again. The newly added complete response record includes the source acquisition path encoding value and the completion timestamp. The source acquisition path encoding value is a simulated browser acquisition path encoding value, while the domain direct connection acquisition path encoding value continues to serve as the attribution acquisition path encoding value for the current record. This ensures that the completed response record has both a clear attribution and source. The browser content writing range and the direct connection response writing range are limited by the start boundary, end boundary, and cross-section request matching key set, respectively. This ensures that bidirectional completion is performed around the content cross-section between the same anchor point, and provides a verifiable data foundation for subsequent recalculation of gap values, verification of completed content boundary positions, and revocation of completed data that fails verification.
[0042] This implementation scheme enables the browser to complete the missing sections and directly connected missing sections to complete bidirectionally under the boundary constraints of the content sections between anchor points. This ensures that the original content of the complete section is not overwritten, and that the completed data is consistent with the request version, content order, and response source in the current collection task. This reduces the risk of content duplication, version mismatch, and structural disorder caused by whole-page replacement. It also provides a data foundation with a clear source for recalculating each missing value, verifying the completion result, and canceling the completed data, thereby improving the integrity, accuracy, and reliability of web page collection data.
[0043] Optionally, the specific steps for recalculating each gap value and verifying the completion results, and outputting the webpage collection data and collection validity report are as follows: Read the completed running document object model, the completed browser content unit, the updated candidate pairing record, the simulated browser network request log, the complete response record table of the domain direct connection collection path, the cross-section request matching key set, the cross-section page coordinate range, the target webpage first screen screenshot collection mark, the key text field set, the start boundary and the end boundary according to the collection task number, and read the browser content gap threshold, browser text gap threshold, browser resource gap threshold and response gap threshold from the integrity parameter table. After browser completion, browser content gap values, browser text gap values, and browser resource gap values are regenerated. When regenerating browser content gap values, the number of directly connected content units within the content cross-section between anchor points that do not form candidate pairing records with browser content units is counted. When regenerating browser text gap values, the text content of the completed visible text and directly connected content units is compared item by item with the key text field set according to the character standardization processing method used before completion. The number of key text fields in the direct domain collection path is subtracted from the number of key text fields in the simulated browser collection path, and negative values are set to zero. When regenerating browser resource gap values, duplicate values are removed from the absolute resource addresses of directly connected content units within the content cross-section between anchor points. The absolute resource addresses are compared with the final access addresses in the network request logs of the completed simulated browser. The number of absolute resource addresses and the number of resources with the same address that form a complete response record are counted. The number of resources with the same address that form a complete response record is subtracted from the number of absolute resource addresses, and negative values are set to zero. This ensures that the three gap values before and after completion use the same calculation objects, text comparison rules, and address normalization rules. If the browser content gap value, browser text gap value, and browser resource gap value do not exceed the browser content gap threshold, a browser cross-section completion record is generated when the browser content gap value, browser text gap value, and browser resource gap value all fall between the browser position values of the completed content units, no screenshot of the target webpage is required, and the browser position values of all completed content units are between the browser position values of the start and end boundaries. If the browser content gap value, browser text gap value, and browser resource gap value all fall between the browser content gap threshold, a screenshot of the target webpage is required, the newly captured screenshot of the target webpage can be decoded and covers the cross-section page coordinate range, and the browser position values of all completed content units are between the browser position values of the start and end boundaries, a browser cross-section completion record is generated.When re-capturing the first screen of the target webpage, the page viewport width, page viewport height, and page scaling ratio before completion are used. The left, top, right, and bottom boundaries of the cross-sectional page coordinate range are read. If the left boundary is not less than zero, the top boundary is not less than zero, the right boundary is not greater than the width of the first screen of the target webpage screenshot, and the bottom boundary is not greater than the height of the first screen of the target webpage screenshot, the first screen of the target webpage is considered to cover the cross-sectional page coordinate range. Browser cross-sectional completion records the acquisition task number, the content cross-sectional number between anchor points, the browser content gap value after completion, the browser text gap value, the browser resource gap value, the browser position value of the starting boundary, the browser position value of the ending boundary, the cross-sectional screenshot coverage marker generation status, and the completion timestamp. If browser completion fails to generate a browser cross-section completion success record, the browser completion data is undone and a cross-section capture failure record is generated. When browser completion data is undone, the runtime document object model copy, browser content unit snapshot, visible text snapshot, simulated browser network request log snapshot, and target webpage first screen screenshot snapshot saved before completion are read. Only the nodes, text content, and browser content units that were deleted or inserted between the start and end boundaries are restored. Node references, event listener references, candidate pairing records, and resource response caches generated between the two boundaries during the completion process are deleted. The simulated browser network request log and target webpage first screen screenshot before completion are restored. The cross-section capture failure record is retained independently and is not deleted when the browser completion data is rolled back. After direct connection completion, a new direct connection response gap value is generated. Each request matching key in the cross-section request matching key set is compared item by item with the complete response record table of the completed domain direct connection collection path, and the number of request matching keys for which no complete response record was found is counted. If the direct connection response gap value does not exceed the response gap threshold, a direct connection cross-section completion pass record is generated. The direct connection cross-section completion pass record is written with the collection task number, anchor content cross-section number, completed direct connection response gap value, completion source collection path encoding value, completion request matching key, and completion timestamp. If no direct connection cross-section completion pass record is generated, the direct connection completion data is revoked and a cross-section collection failure record is generated. When the direct connection completion data is revoked, the newly added complete response record with the completion source collection path encoding value of the simulated browser collection path encoding value is deleted from the complete response record table of the domain direct connection collection path according to the collection task number, anchor content cross-section number, and completion request matching key. The original complete response record of the domain direct connection collection path and the domain direct connection request log are retained, and the cross-section collection failure record is retained independently. The cross-section acquisition failure record is written with the acquisition task number, the content cross-section number between anchor points, the completion type encoding value, the browser content gap value after completion, the browser text gap value, the browser resource gap value, the direct connection response gap value after completion, the start boundary, the end boundary, and the failure timestamp.For complete sections, sections with browser completion records, and sections with direct connection completion records, the sections are aggregated according to the section order in the dual-path content anchor chain. The runtime document object model, visible text, and target webpage screenshot from the simulated browser acquisition path are retained, along with the original page source code, structured data response, and original response metadata from the domain direct connection acquisition path. The original response metadata includes response headers, response status codes, server addresses, response timestamps, and source request matching keys, extracted from the domain direct connection request log and the complete response record table of the domain direct connection acquisition path, respectively. After browser completion is successful, the completed runtime document object model is retraced in depth-first traversal order. The parent path is regenerated based on the tag name, node number, and node level of each element node, and the browser position value of the completed content unit is updated synchronously. When only direct connection completion is performed, the runtime document object model remains unchanged, and the parent path and browser position value from the original browser content unit are used. The visible text is arranged according to the completed parent path and browser position value to generate structured text. When generating structured text, the parent path is used as the basis for text hierarchy, and the browser position value is used as the basis for text arrangement. Duplicate and empty text content is deleted, and hierarchy separators are written at locations where the parent path changes, ensuring that the structured text maintains the node hierarchy and page order in the runtime document object model. A collection validity report is generated by counting the total number of sections, the number of sections that passed completion, and the number of sections that failed to collect data. The total number of sections is obtained from the number of content section records between anchor points. The number of sections that passed completion is counted by deduplicating the content section numbers between anchor points in browser section completion records and directly connected section completion records. The number of sections that failed to collect data is obtained from the number of section failure records retained under the same collection task number. The section completeness rate is generated by dividing the sum of the number of complete sections and the number of sections that passed completion by the total number of sections. The output includes the original page source code, runtime document object model, structured text, target webpage first screen screenshot, original response metadata, browser cross-section completion pass records, direct connection cross-section completion pass records, and a collection validity report, all based on the collection task number. The output file is named using the collection task number and data type encoding value. The content between anchor points of the cross-section collection failure records is not written into the structured text and the completed runtime document object model. The original page source code and target webpage first screen screenshot are output separately as original collection data, ensuring that the processed webpage collection data only contains complete cross-sections and cross-sections that have passed completion verification.
[0044] This implementation plan ensures that the content cross-sections between completed anchor points form a verifiable closed verification chain under unified gap calculation criteria, boundary constraints, and screenshot coverage conditions. This prevents failed verification data from entering the final webpage data collection, while maintaining the source association between complete cross-sections, browser cross-section completion success records, and direct-connection cross-section completion success records. This reduces result distortion caused by gap residue, out-of-order positions, and failed data contamination, providing a consistent, quality-controllable, and easily statistically traceable data output foundation for webpage big data processing. It also enhances the credibility of collection validity reports and the stability of batch webpage data applications.
[0045] like Figure 2 As shown, the system executes simulated browser data collection and direct domain connection data collection in parallel, generating request matching keys and complete response records. The data from the two collection paths is split into browser content units and direct connection content units. A dual-path content anchor chain is constructed based on the increasing relationship between content identifier values and both browser and direct connection positional values. Content sections between anchor points are divided according to adjacent anchor points. For each content section between anchor points, browser content gap values, browser text gap values, browser resource gap values, and direct connection response gap values are calculated. The completeness of the content sections between anchor points is determined by combining the section screenshot overlay markers. Complete sections are written to a complete section table. Browser sections to be completed are completed using direct domain connection data collection, and direct connection sections to be completed are completed using the complete response records of the simulated browser collection path. For other content sections between anchor points, section collection failure records are generated. After completion, each gap value is recalculated and the completion results are verified. Data collection data generated from complete sections and completed sections is collected, and statistics on section collection failure records are compiled to generate a collection validity report.
[0046] In this embodiment, a dynamic product detail webpage is used as the data collection target, and simulated browser data collection and direct domain connection data collection are performed in parallel. The simulated browser data collection path obtains the running document object model, visible text, node page coordinates, simulated browser network request logs, and a screenshot of the target webpage's first screen. The direct domain connection data collection path obtains the original page source code, structured data response, hypertext fragment response, and original response metadata. Nine anchor points A0 to A8 are generated based on the candidate pairing records between browser content units and directly connected content units, and the data between adjacent anchor points is divided into eight content sections between anchor points D1 to D8. The browser content gap threshold, browser text gap threshold, browser resource gap threshold, and response gap threshold are all set to 0; B(a, b, c) represent the browser content gap value, browser text gap value, and browser resource gap value, respectively, and R(d) represents the direct connection response gap value. According to each gap value, D1, D2, and D8 are written into the complete section table, D3 to D5 are written into the browser section table to be completed, D6 is written into the direct connection section table to be completed, and D7 generates a section collection failure record. The browser profile completion for D3 to D5 is completed using direct domain connection data collection. The direct connection profile completion for D6 is completed using the complete response record of the simulated browser collection path. After recalculating each gap value and verifying the browser position value of the browser completion content and the coverage of the first screen screenshot of the target webpage, D3 to D6 pass the completion verification. D7 is not written into the final webpage collection data.
[0047] like Figure 3 As shown, the content cross-sections between the eight anchor points D1 to D8 are arranged horizontally according to the adjacent boundary intervals from A0-A1 to A7-A8. Vertically, they represent four states: before browser path completion, before direct domain connection path completion, after browser path completion, and after direct domain connection path completion. Solid rectangles indicate complete cross-sections, forward diagonal rectangles indicate gaps in the browser acquisition path, reverse diagonal rectangles indicate gaps in the direct domain connection acquisition path, intersecting grid rectangles indicate failed cross-section completion, vertical dashed lines represent cross-section boundaries in dual-path content anchor chains, and arrows indicate the direction of cross-path local completion. Figure 3 This indicates that the dual-path content anchor chain can limit data gaps to the content cross-section between specific anchor points and perform targeted completion based on the acquisition path where the gap is located. At the same time, it generates a cross-section acquisition failure record for the content cross-section between anchor points where both paths lack available completion data.
[0048] like Figure 4 As shown, another aspect of the present invention provides a web page data acquisition system based on dual-path integrity verification and completion, the system comprising: The target input module is used to read the list of target web page addresses and the collection requirements record, and generate a web page collection task record; The network environment simulation unit is used to set network latency, bandwidth limits, packet loss, and response timeout. The dual-path acquisition module includes a simulated browser acquisition unit and a domain direct connection acquisition unit, which are used to collect runtime document object models, network request logs, raw page source code and structured data responses in parallel, and generate request matching keys and complete response records; The integrity assessment module is used to split the data from the two acquisition paths into browser content units and directly connected content units, generate dual-path content anchor chains and content cross-sections between anchor points, and calculate browser content gap values, browser text gap values, browser resource gap values and directly connected response gap values. The completion and fusion module is used to determine the browser sections to be completed and the direct connection sections to be completed based on the browser content gap value, browser text gap value, browser resource gap value and direct connection response gap value. It completes the completion and re-verification by using the domain direct connection data and the complete response record of the simulated browser collection path, respectively. The results output module is used to output the original page source code, the runtime document object model, structured text, the first screen screenshot of the target webpage, the original response metadata, and the collection validity report.
[0049] This implementation plan forms a closed processing chain from web page task reception, dual-path acquisition, cross-section-level integrity identification, cross-path completion to unified result output. This enables the runtime document object model to complete association verification and quality control with the original page source code, structured data response, and original response metadata under the same acquisition task. This reduces the impact of missing dynamic content, incomplete response data, and distortion of completion results on the final web page acquisition data, providing a data foundation with clear sources, unified structure, and verifiable integrity for web page big data processing, and improving the execution reliability and result verification efficiency of batch acquisition tasks.
[0050] The following points need to be explained: (1) The accompanying drawings of the embodiments of the present invention only involve the structures involved in the embodiments of the present invention. Other structures can refer to the general design.
[0051] (2) For clarity, the thickness of layers or regions is enlarged or reduced in the drawings used to describe embodiments of the invention, i.e., these drawings are not drawn to scale. It is understood that when an element such as a layer, film, region or substrate is referred to as being “above” or “below” another element, the element may be “directly” located “above” or “below” the other element or there may be intermediate elements.
[0052] (3) Where there is no conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other to obtain new embodiments.
[0053] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. The scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A web data collection method based on double-path integrity check complement, characterized in that, The method includes: S1 executes simulated browser data collection and direct domain connection data collection in parallel to obtain runtime document object model, network request logs, original page source code and structured data response, and generates request matching keys and complete response records; S2, the data from the two acquisition paths are split into browser content units and directly connected content units. Based on the relationship that the content identifier value, browser position value and directly connected position value are all increasing, dual-path content anchor chains are generated, and the content cross-section between anchor points is divided according to adjacent anchor points. S3 generates browser content gap values, browser text gap values, browser resource gap values, and direct connection response gap values for the content cross-sections between each anchor point, and determines the complete cross-section, browser to be completed cross-section, and direct connection to be completed cross-section based on each gap value; S4 uses direct domain connection to collect data to complete the browser's missing sections, uses the complete response records of the simulated browser's collection path to complete the direct connection's missing sections, recalculates each gap value and verifies the completion results, and outputs web page collection data and a collection validity report.
2. The method of claim 1, wherein, The specific steps for parallel execution of simulated browser data collection and direct domain connection data collection to obtain the runtime document object model, network request logs, original page source code, and structured data response are as follows: The web page collection task configuration interface reads the target web page address list and collection requirement record from the collection task table. The collection requirement record includes source code collection markers, text collection markers, target web page first screen screenshot collection markers, and a set of key text fields. Generate web page collection task records by associating the target web page address with the collection requirement record according to the collection task number; Based on the web page collection task records, use the same user agent field, language field, access credential field and collection start timestamp to start the simulated browser collection path and the domain direct connection collection path in parallel. Simulate a browser to collect path loading of target web page address and execute page script, and collect runtime document object model, visible text, node page coordinates, resource loading records, network request logs and first screen screenshot of target web page through browser debugging interface; The direct domain connection data collection path sends hypertext transfer requests to the target webpage address to collect the original page source code, structured data response, hypertext fragment response, direct domain connection request logs, response headers, server address, and response time value.
3. The method of claim 2, wherein, The specific steps for generating the request matching key and complete response record are as follows: The request method, final access address, request body, and request timestamp are read from the simulated browser network request log and the domain direct connection request log, respectively. The request method, final access address, request body digest value, and request sequence number with the same address are concatenated to form the request matching key. The request sequence number with the same address is generated in ascending order according to the timestamp of the request sent for requests with the same request method and final access address in the same collection path. When the response header contains a declared byte length value, the response whose actual received byte length value is the same as the declared byte length value and whose response body is successfully parsed according to the media type after decompression is written into the complete response record table. When the response header does not contain a declared byte length value, the response with a transmission end marker and whose response body is successfully parsed according to the media type after decompression is written into the complete response record table.
4. The method of claim 3, wherein, The specific steps for splitting the data from the two acquisition paths into browser content units and directly connected content units are as follows: The runtime document object model is split into browser content units according to the depth-first traversal order, recording tag names, parent paths, content identifier values, text content, text digest values, absolute resource addresses, browser position values, and node page coordinates. The original page source code, structured data response, and hypertext fragment response are split into directly connected content units according to the response reception order and the tag position or field traversal order within the response, recording tag names, parent paths, content identifier values, field names, text content, text digest values, absolute resource addresses, source request matching keys, and directly connected position values.
5. The method of claim 4, wherein, The specific steps for generating dual-path content anchor chains based on the increasing relationship between content identifier values, browser position values, and direct connection position values are as follows: When both the browser content unit and the directly connected content unit have non-empty and identical content identifier values, a candidate pairing record is written; when at least one content identifier value is empty, the tag name, parent node path, text digest value, and absolute resource address are compared. When the number of identical items reaches the content matching threshold, a candidate pairing record is written, and the number of matching items is recorded. With the constraint that both the browser bit sequence value and the direct connection bit sequence value are increasing, the candidate pair record sequence with the largest number of records is selected from the candidate pair records. When multiple sequences with the same number of records exist, the sequence with the largest cumulative number of matching items is selected; the starting boundary record of the page root node is written at the front of the candidate pairing record sequence, and the ending boundary record of the last content unit of the two collection paths is written at the back of the candidate pairing record sequence, and the records are connected to generate a dual-path content anchor chain.
6. The method of claim 5, wherein, The specific steps for dividing the content section between anchor points according to adjacent anchor points are as follows: Following the arrangement order of dual-path content anchor chains, browser content units and directly connected content units between two adjacent records are grouped into the same anchor-point content cross-section. The source request matching keys of directly connected content units are summarized, and request matching keys whose request addresses are the same as the absolute resource addresses within the anchor-point content cross-section are extracted from the simulated browser network request logs to generate a cross-section request matching key set. The smallest bounding rectangle of the visible node page coordinates within the cross-section is taken as the cross-section page coordinate range, and the start boundary, end boundary, cross-section request matching key set, and cross-section page coordinate range are written into the anchor-point content cross-section record.
7. The method of claim 6, wherein, The specific steps for generating browser content gap values, browser text gap values, browser resource gap values, and direct connection response gap values for the content cross-sections between each anchor point, and determining the complete cross-section, the browser cross-section to be completed, and the direct connection cross-section to be completed based on each gap value are as follows: The browser content gap value is generated by counting the number of directly connected content units within the content cross-section between anchor points that do not form candidate pairing records with browser content units. The visible text in the simulated browser acquisition path and the text content of directly connected content units in the domain direct acquisition path are compared item by item with the key text field set to count the number of key text fields with identical text content. The browser text gap value is generated by subtracting the number of key text fields in the simulated browser acquisition path from the number of key text fields in the domain direct acquisition path and setting the negative value to zero. Finally, the browser resource gap value is generated by counting the number of resource addresses in directly connected content units and the number of resources with the same address and forming complete response records in the simulated browser network logs, and subtracting the number of resources with successful responses from the number of resource addresses and setting the negative value to zero. Each request matching key in the cross-section request matching key set is compared item by item with the complete response record table of the domain direct connection collection path. The number of request matching keys for which no complete response record was found is counted, and a direct connection response gap value is generated. When collecting the first screen screenshot of the target webpage, the cross-section image is to be captured according to the cross-section page coordinate range. When the cross-section image is successfully decoded and the cross-section page coordinate range is completely within the boundary of the first screen screenshot of the target webpage, a cross-section screenshot overlay mark is generated. If the browser content gap value, browser text gap value, and browser resource gap value all do not exceed their respective browser content gap thresholds, browser text gap thresholds, and browser resource gap thresholds, and no screenshot of the target webpage's first screen is required or a cross-sectional screenshot overlay marker has been generated, and the direct connection response gap value does not exceed the response gap threshold, then the content cross-section between anchor points will be written into the complete cross-section table. If any of the three browser gap values exceeds its respective threshold, or if a screenshot of the target webpage's first screen is required but a cross-sectional screenshot overlay marker has not been generated, and the direct connection response gap value does not exceed the response gap threshold, then the content cross-section between anchor points will be written into the complete cross-section table. The cross-section is written to the browser's cross-section completion table; if the browser content gap value, browser text gap value, and browser resource gap value do not exceed the corresponding browser content gap threshold, browser text gap threshold, and browser resource gap threshold, and no screenshot of the first screen of the target webpage is required or a cross-section screenshot has been generated to cover the marker, and the direct connection response gap value exceeds the response gap threshold, and there is a complete response record corresponding to the missing request matching key in the complete response record table of the simulated browser collection path, the content cross-section between anchor points is written to the direct connection cross-section completion table; other content cross-sections between anchor points generate cross-section collection failure records.
8. The method of claim 7, wherein, The specific steps for completing the browser's missing sections by collecting data through direct domain connection and for completing the direct connection's missing sections by collecting complete response records of the simulated browser path are as follows: Read the start and end boundaries of the browser section to be completed, extract the tag sequence from the directly connected content units between the two boundaries, and extract the structured data response and resource response body associated with the section request matching key set from the complete response record table of the domain direct connection collection path; copy the running document object model obtained by simulating the browser collection path, delete the browser content units between the two boundaries and insert the tag sequence, return the structured data response and resource response body according to the request matching key through the request interception interface, and execute the script tags in the tag sequence; When the number of incomplete network requests does not exceed the threshold for the number of incomplete requests, save the completed runtime document object model, re-collect visible text and network request logs, re-collect the first screen screenshot of the target webpage when required, and generate the browser position value of the completed content unit according to the depth-first traversal order. Read the set of cross-section request matching keys for the directly connected cross-section to be completed, extract the complete response record associated with each request matching key from the complete response record table of the simulated browser collection path, and write the response header, response body and response time value into the complete response record table of the domain direct connection collection path.
9. The method of claim 8, wherein, The specific steps for recalculating each gap value, verifying the completion results, and outputting the webpage data collection data and collection validity report are as follows: After browser completion, browser content gap values, browser text gap values, and browser resource gap values are regenerated. If all three gap values do not exceed their respective thresholds, and it is not required that the captured or re-captured first-screen screenshot of the target webpage be decoded and cover the cross-sectional page coordinate range, and the browser position values of the completed content units are all between the browser position values at the start and end boundaries, a browser cross-sectional completion pass record is generated. If browser completion does not generate a browser cross-sectional completion pass record, the browser completion data is cancelled and a cross-sectional acquisition failure record is generated. After direct connection completion, direct connection response gap values are regenerated. If the direct connection response gap value does not exceed the response gap threshold, a direct connection cross-sectional completion pass record is generated. If no direct connection cross-sectional completion pass record is generated, the direct connection completion data is cancelled and a cross-sectional acquisition failure record is generated. For complete sections, sections with browser-completed completion records, and sections with direct-connection-completed completion records, retain the runtime document object model, visible text, and target webpage screenshot in the simulated browser acquisition path, and retain the original page source code, structured data response, and response metadata in the domain direct-connection acquisition path; arrange the visible text according to the completed parent path and browser position value to generate structured text; The system counts the total number of cross-sections, the number of cross-sections that passed completion, and the number of cross-section collection failure records. It generates a collection validity report and outputs the original page source code, runtime document object model, structured text, target webpage first screen screenshot, original response metadata, browser cross-section completion success records, direct connection cross-section completion success records, and collection validity report according to the collection task number.
10. A web data collection system based on double path integrity check completion, applying the web data collection method based on double path integrity check completion according to any one of claims 1-9, characterized in that, include: The target input module is used to read the list of target web page addresses and the collection requirements record, and generate a web page collection task record; The network environment simulation unit is used to set network latency, bandwidth limits, packet loss, and response timeout. The dual-path acquisition module includes a simulated browser acquisition unit and a domain direct connection acquisition unit, which are used to collect runtime document object models, network request logs, raw page source code and structured data responses in parallel, and generate request matching keys and complete response records; The integrity assessment module is used to split the data from the two acquisition paths into browser content units and directly connected content units, generate dual-path content anchor chains and content cross-sections between anchor points, and calculate browser content gap values, browser text gap values, browser resource gap values and directly connected response gap values. The completion and fusion module is used to determine the browser sections to be completed and the direct connection sections to be completed based on the browser content gap value, browser text gap value, browser resource gap value and direct connection response gap value. It completes the completion and re-verification by using the domain direct connection data and the complete response record of the simulated browser collection path, respectively. The results output module is used to output the original page source code, the runtime document object model, structured text, the first screen screenshot of the target webpage, the original response metadata, and the collection validity report.
Citation Information
Patent Citations
A method for intelligent web page data collection
CN108153817B
A method for precise data collection from interactive web pages
CN112579852B
