Internet data real-time acquisition and capture method

Abstract

The invention relates to the technical field of Internet data acquisition, and discloses an Internet data acquisition method with high real-time performance and low resource consumption. Traditional polling is replaced by a website update event monitoring mechanism, second-level data acquisition is realized in combination with a streaming processing framework, and acquisition stability is guaranteed by adopting a dynamic anti-crawling strategy. The problems of high delay and serious resource waste in the prior art are solved, and the method is particularly suitable for scenes such as financial real-time quotation.

Description

Technical Field

[0001] The present invention relates to the field of Internet data collection technology, and in particular to a real-time data collection and capture method based on streaming processing and dynamic scheduling, which is suitable for scenarios with high timeliness requirements such as financial monitoring, public opinion analysis, and e-commerce price comparison. Background Art

[0002] Current Internet data collection has the following pain points: Poor timeliness: Traditional polling-based data collection suffers from minute-level delays, failing to meet real-time requirements. Resource waste: Static crawlers repeatedly crawl unupdated pages, wasting bandwidth and computing resources. Weak adaptability: Unable to dynamically respond to changes in website anti-crawling policies (such as verification codes and IP blocking). Low data integrity: Inadequate crawling capabilities for dynamic pages rendered with JavaScript. While incremental crawling solutions have been proposed, they still rely on fixed-cycle scanning and cannot achieve sub-second response times. Summary of the Invention

[0003] In order to overcome the above-mentioned defects of the prior art, the present invention provides a real-time data collection and capture method based on streaming processing and dynamic scheduling, which is suitable for scenarios with high timeliness requirements such as financial monitoring, public opinion analysis, and e-commerce price comparison.

[0004] The present invention provides the following technical solution: a method for real-time collection and crawling of Internet data, comprising the following steps: S1 dual-channel update monitoring: establish an independent HTTP long polling channel for each target site, maintain the connection for 60-120 seconds and the WebSocket push channel; Real-time monitoring of updates through the DOM tree hash comparer: triggering a collection task when the difference between the old and new hash values ​​exceeds a preset threshold; S2 streaming pipeline construction: Uses a time window slicing mechanism to process data streams; defines a rolling window with a window size of 500ms and a sliding step size of 100ms; integrates an FPGA acceleration module for XPath expression compilation; S3 dynamic scheduling engine; S4 dynamic page rendering: based on the headless browser rendering state machine; S5 data cleaning and output: Use the regular expression + CSS selector dual engine to extract data and output it to the time series database.

[0005] Furthermore, the DOM tree hash comparer in S1 performs three layers of filtering: removing tags, normalizing whitespace characters, and ignoring numeric attributes.

[0006] Furthermore, the proxy IP pool management in S3 adopts a health scoring mechanism. When the health score is <60, the IP is automatically discarded. The User-Agent rotation follows the Markov chain model: state transition probability: PC → mobile probability = 0.2, mobile → PC probability = 0.1.

[0007] Furthermore, in S4: the headless browser includes three rendering modes: extreme mode, standard mode, and complete mode.

[0008] Furthermore, the abnormal circuit breaker mechanism: when the number of consecutive failures is greater than 5, it automatically switches to the degraded collection mode, sends the site abnormality code to the dispatch center, and starts exponential backoff retry.

[0009] This invention solves the latency problem of traditional polling through dual-channel monitoring and DOM hash comparison (measured latency ≤ 800ms vs. ≥ 30s for traditional solutions). It also reduces the invalid collection rate by 37% by filtering non-content updates (such as ad carousels) through hash filtering. The scoring model in dynamic scheduling introduces a business weight coefficient to ensure high-value data is collected first, and the agent health score formula has been tested to increase the effective IP utilization rate to 91.5%. The headless browser state machine dynamically switches between three modes, increasing the dynamic page collection success rate from 68% to 94%. Resource loading control reduces invalid bandwidth consumption by 42%. DETAILED DESCRIPTION

[0010] The technical solutions in the present invention will be described clearly and completely below. In addition, the forms of the various structures described in the following embodiments are merely examples. The Internet-based survey information service management system involved in the present invention is not limited to the various structures described in the following embodiments. All other implementations obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Example

[0011] A method for real-time collection and crawling of Internet data comprises the following steps: S1 dual-channel update monitoring: establish an independent HTTP long polling channel for each target site, maintain the connection for 60-120 seconds and the WebSocket push channel; Real-time monitoring of updates through the DOM tree hash comparer: triggering a collection task when the difference between the old and new hash values ​​exceeds a preset threshold; The DOM tree hash comparer performs three layers of filtering: removing tags, normalizing whitespace, and ignoring numeric attributes.

[0012] S2 streaming pipeline construction: Uses a time window slicing mechanism to process data streams; defines a rolling window with a window size of 500ms and a sliding step size of 100ms; integrates an FPGA acceleration module for XPath expression compilation; S3 dynamic scheduling engine; proxy IP pool management adopts a health scoring mechanism. When the health score is <60, the IP is automatically discarded. User-Agent rotation follows the Markov chain model: state transition probability: PC → mobile probability = 0.2, mobile → PC probability = 0.1.

[0013] S4 dynamic page rendering: based on the headless browser rendering state machine; the headless browser includes three rendering modes: extreme mode, standard mode, and full mode.

[0014] S5 data cleaning and output: Use the regular expression + CSS selector dual engine to extract data and output it to the time series database.

[0015] Abnormal circuit breaker mechanism: When the number of consecutive failures is greater than 5, it automatically switches to degraded collection mode, sends the site abnormality code to the dispatch center, and starts exponential backoff retry.

[0016] Dual-channel monitoring and DOM hash comparison solve the latency issue of traditional polling (measured latency ≤ 800ms vs. ≥ 30s for traditional solutions). Hash filtering of non-content updates (such as ad carousels) reduces the invalid collection rate by 37%. The scoring model in dynamic scheduling introduces a business weight coefficient to ensure high-value data is collected first. The proxy health score formula has been tested and increased the effective IP utilization rate to 91.5%. The headless browser state machine dynamically switches between three modes, increasing the dynamic page collection success rate from 68% to 94%. Resource loading control reduces invalid bandwidth consumption by 42%.

[0017] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for real-time collection and capture of Internet data, characterized in that The following steps are involved: S1 dual-channel update monitoring: establish an independent HTTP long polling channel for each target site, maintain the connection for 60-120 seconds and the WebSocket push channel; Real-time monitoring of updates through the DOM tree hash comparer: triggering a collection task when the difference between the old and new hash values ​​exceeds a preset threshold; S2 stream processing pipeline construction: Use time window slicing mechanism to process data streams; Define a rolling window with a window size of 500ms and a sliding step size of 100ms; integrate an FPGA acceleration module for XPath expression compilation; S3 dynamic scheduling engine; S4 dynamic page rendering: based on the headless browser rendering state machine; S5 data cleaning and output: Use the regular expression + CSS selector dual engine to extract data and output it to the time series database.

2. The method for real-time collection and capture of Internet data according to claim 1, characterized in that: The DOM tree hash comparer described in S1 performs three layers of filtering: removing tags, normalizing whitespace, and ignoring numeric attributes.

3. The method for real-time collection and capture of Internet data according to claim 1, characterized in that: The proxy IP pool management in S3 adopts a health scoring mechanism. When the health score is less than 60, the IP is automatically discarded. User-Agent rotation follows the Markov chain model: state transition probability: PC → mobile probability = 0.2, mobile → PC probability = 0.

1.

4. The method for real-time collection and capture of Internet data according to claim 1, characterized in that: In S4: The headless browser includes three rendering modes: extreme mode, standard mode, and complete mode.

5. The method for real-time collection and capture of Internet data according to claim 1, characterized in that: Abnormal circuit breaker mechanism: When the number of consecutive failures is greater than 5, it automatically switches to degraded collection mode, sends the site abnormality code to the dispatch center, and starts exponential backoff retry.