Multi-task cooperative incremental web page data collection method and system

Abstract

The application discloses a multi-task cooperative incremental webpage data collection method and system, and particularly relates to the technical field of webpage data collection, and comprises the following steps: performing initial evaluation on a target webpage to calculate a structure uncertainty score, judging whether to enter deep evaluation to generate a page structure consistency index and a content update index, outputting a collection scheduling coefficient through a collection scheduling prediction model and selecting a structure reconstruction marker tracking or a content difference snapshot comparison path; classifying task strategy types in combination with the combination relationship of the two indexes, and realizing multi-task cooperative collection according to the classification results and preset cooperative strategies; the application can dynamically perceive webpage structure and content changes, generate a structure consistency index and a content update index, and improve the judgment accuracy before collection; the collection scheduling coefficient is used to select an adaptive path, and the collection stability in a complex webpage environment is enhanced; in combination with the classification results and the cooperative strategies, resource allocation and time scheduling are optimized, and efficient cooperation among multi-tasks is realized.

Description

Technical Field

[0001] This invention relates to the field of web page data acquisition technology, and more specifically, to a multi-task collaborative incremental web page data acquisition method and system. Background Technology

[0002] With the ever-increasing demand for information, web page data collection, as a fundamental step in data mining and knowledge construction, is widely used in various intelligent systems. Especially in scenarios such as public opinion monitoring, product price comparison, policy tracking, and content aggregation, higher demands are placed on the rapid, accurate, and stable collection of web page content.

[0003] Traditional web scraping methods are mostly based on static rules or timed full-scale crawling, which struggles to handle frequent changes in webpage structure and irregular content updates. Especially in environments processing multiple webpages simultaneously, the frequency of structural changes and content refresh cycles vary significantly across different webpages. Using a uniform scheduling strategy can easily lead to resource waste, duplicate crawling, or scraping failures. In recent years, incremental web scraping methods have gained increasing attention. Their core idea is to only scrape webpage content that has changed since the last scraping. However, existing incremental scraping methods mostly target single webpages, lacking collaborative judgment and dynamic scheduling capabilities between tasks, and still suffer from the following shortcomings in practical deployment:

[0004] Existing solutions often rely on time intervals or content summaries to determine whether data collection is necessary, lacking a systematic evaluation process based on webpage structure hierarchy and visible changes in behavior. This results in insufficient response to pages with high-frequency structural disturbances. For changing webpages, data collection is often triggered simply by whether there has been a change, lacking comprehensive modeling of structural evolution trends and content evolution patterns, making it difficult to formulate precise collection paths. Therefore, this paper proposes a multi-task collaborative incremental webpage data collection method and system to address the aforementioned problems. Summary of the Invention

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A multi-task collaborative incremental web page data acquisition method includes the following steps:

[0007] Before scheduling the web page collection task, an initial evaluation operation is performed on the target web page. The evaluation operation includes obtaining the range of structural hierarchy changes, reconstruction frequency of the visible area, and rendering position fluctuation of the web page within a preset historical period, calculating the structural uncertainty score, and determining whether to enter the deep evaluation process based on the score.

[0008] When the structural uncertainty score exceeds the preset disorder threshold, a deep evaluation operation is performed. The deep evaluation includes analyzing the historical change trend of web page structural nodes, element rearrangement distribution and regional stability patterns, and generating a page structure consistency index. At the same time, multiple historical content snapshots are compared to calculate field update density, semantic variation amplitude and time distribution pattern, and generate a content update index.

[0009] The page structure consistency index and content update index are fed into the pre-trained data collection scheduling prediction model as input parameters, and the data collection scheduling coefficient is output. This coefficient is used to evaluate the execution priority and strategy tendency of the current task in the current scheduling cycle.

[0010] Based on the numerical range of the collection scheduling coefficient, the collection path of the task is selected. The collection path includes a marker tracking and localization process based on structural reconstruction tolerance enhancement, or an incremental comparison and snapshot update process based on content difference identification.

[0011] After determining the collection path, the collection tasks are classified into execution strategies based on the combination of the page structure consistency index and the content update index. The classification results are structure-controlled, content-controlled, or scheduling-responsive. According to the classification results and the preset collaborative strategy, the resource allocation order and scheduling time window of the tasks are adjusted to achieve collaborative scheduling and strategy adaptation among multiple collection tasks.

[0012] In a preferred embodiment, the calculation of the structural uncertainty score includes the following steps:

[0013] The structural hierarchy information of the target webpage obtained in the three most recent consecutive collection cycles is compared, and the set of structural nodes in each cycle is extracted. Each node in the set of structural nodes contains three indicators: tag name, hierarchy depth and number of sibling nodes.

[0014] Map the set of structural nodes for any two periods one by one, compare the number of label changes, the number of hierarchical migrations, and the number of new or missing nodes, and sum them up according to a preset ratio to obtain the structural variation of any two periods. Use the average value of the structural variation of all periods as the structural variation benchmark value, divide the benchmark value by the total number of structural nodes of the current webpage, and multiply the quotient by the preset floating weight coefficient corresponding to the current webpage type to obtain the structural uncertainty score.

[0015] In a preferred embodiment, the process of generating the page structure consistency index includes the following steps:

[0016] Extract the structural hierarchy information of the target webpage at five consecutive historical collection time points, and transform the webpage structure at each time point into a set of node paths. Each path is represented by a combination of a tag sequence and its hierarchy depth.

[0017] Using the path set corresponding to the fifth time point as a reference, alignment operations are performed on each path in the first four time points in turn. Path alignment is based on the longest common substring of the label sequence. If the length of the common substring exceeds 60% of the total path length, it is considered a successful alignment.

[0018] After successful alignment, calculate the hierarchical distance difference between path pairs, i.e. the hierarchical difference of the end node of the label, and average the hierarchical differences of all path pairs to form the hierarchical difference value of that point in time relative to the baseline structure.

[0019] Extract the concentration of structural paths at each time point. The concentration is defined as the percentage of the number of paths at the same level divided by the total number of paths. Multiply the level difference value at each time point by its concentration, and then average the product values ​​of the five time points to finally obtain the page structure consistency index.

[0020] In a preferred embodiment, the process of generating the content update index includes the following steps:

[0021] Extract content snapshots of the target webpage at five consecutive historical collection time points consistent with the structural hierarchy sampling. Divide the content of each snapshot into multiple content blocks according to paragraph boundaries, and perform difference analysis on the content blocks at the same position of adjacent snapshots at each time point.

[0022] For each pair of adjacent content blocks, calculate the text editing span, which is defined as the ratio of the minimum editing distance between the two versions of the content block to the total number of characters in the block in the current snapshot;

[0023] The magnitude of keyword group change is calculated as the ratio of the size of the symmetric difference between the sets of keywords extracted from the two versions of the content to the size of the union of the two sets;

[0024] Calculate the publication time arrangement offset, which is defined as the difference between the standard deviation of the publication time of all paragraphs in the content block and the standard deviation of the average publication time of all paragraphs at that point in time.

[0025] For each group of paragraphs, the three values ​​mentioned above—text editing span, keyword group change range, and publication time arrangement offset—are normalized to the range of zero to one. The normalized result is then multiplied by the display density of the paragraph on the webpage, which is the percentage of the paragraph's total content length on the webpage. The product values ​​of all paragraphs in each pair of snapshots are summed, and finally, the average of the sums across the five snapshots is taken to generate the content update index.

[0026] In a preferred embodiment, the acquisition scheduling prediction model used to generate acquisition scheduling coefficients is pre-trained through supervised learning. The training samples include page structure consistency index, content update index, actual acquisition success rate, scheduling delay time, and resource utilization rate in historical acquisition tasks. During training, the model objective is to minimize the comprehensive loss function of scheduling error rate, scheduling delay time, and resource waste rate. The final model is constructed using an ensemble regression algorithm, and the output acquisition scheduling coefficients are continuous values ​​between zero and one, used to represent the comprehensive urgency and resource adaptation difficulty of the current acquisition task within the scheduling cycle.

[0027] In a preferred embodiment, the tag tracking and localization process includes the following steps:

[0028] Retrieve the stored set of structural tag paths from the last successfully collected version. Each set of paths contains a sequence of tag names and their corresponding level index. Perform a path scan on the current webpage structure tree, calculate the matching similarity between each stored path and the current structural path, and mark paths with similarity below a preset ratio threshold as offset paths. Reconstruct the level pointer for each offset path and update it by replacing the old path with the new path.

[0029] In a preferred embodiment, the incremental comparison and snapshot update process includes the following steps:

[0030] Get the content area of ​​the current webpage and divide it into paragraphs based on the content area corresponding to the structural position in the previous version of the webpage. Each paragraph is a content block, and the content block is indexed by the dual coordinates of the paragraph's start position and length.

[0031] Perform bidirectional mapping on the content block indexes in the two versions. The conditions for a successful match are that the starting position offset of the current version paragraph does not exceed fifty characters and the length change does not exceed twenty percent. If these conditions are not met, the block is considered a failed match.

[0032] Keyword groups are extracted from all content blocks that fail to match. Each keyword group is selected from the title field and the first fifty words of the paragraph using a preset text analysis algorithm. The keyword group difference is defined as the ratio of the number of symmetric differences between the keyword sets of two versions to the number of their union. If the difference value is greater than 0.6, the content block is determined to be changed content. All paragraphs determined to be changed content are used as update paragraphs to replace the original paragraph content at the corresponding position in the snapshot, and the timestamp and keyword group of this change are recorded.

[0033] In a preferred embodiment, after determining the acquisition path, the acquisition task is classified into execution strategies by combining the combination relationship between the page structure consistency index and the content update index using a fuzzy logic classifier. The fuzzy logic classifier uses the page structure consistency index and the content update index as input variables and generates classification labels based on preset membership functions and decision rules. The classification results are structure-controlled, content-controlled, or scheduling-response type.

[0034] In a preferred embodiment, the multi-task collaborative incremental web page data acquisition system specifically includes:

[0035] The initial structural evaluation module is used to perform an initial evaluation of the target webpage before the webpage collection task is scheduled. The evaluation includes obtaining the range of structural hierarchy changes, reconstruction frequency of the visible area, and rendering position fluctuation of the webpage within a preset historical period, calculating the structural uncertainty score, and determining whether to enter the deep evaluation process based on the score.

[0036] The deep perception module is used to perform a deep evaluation operation when the structural uncertainty score exceeds the preset disorder threshold. The deep evaluation includes analyzing the historical change trend of web page structural nodes, element rearrangement distribution and regional stability patterns, and generating a page structure consistency index; at the same time, it compares multiple historical content snapshots, calculates field update density, semantic variation magnitude and time distribution pattern, and generates a content update index.

[0037] The scheduling prediction module is used to input the page structure consistency index and content update index as input parameters into the pre-trained collection scheduling prediction model and output the collection scheduling coefficient. This coefficient is used to evaluate the execution priority and strategy tendency of the current task in the current scheduling cycle.

[0038] The path execution module is used to select the acquisition path of the task based on the numerical range of the acquisition scheduling coefficient. The acquisition path includes a marker tracking and localization process based on structural reconstruction tolerance enhancement, or an incremental comparison and snapshot update process based on content difference identification.

[0039] The strategy assignment module is used to classify the execution strategy of the collection task after the collection path is determined, based on the combination of the page structure consistency index and the content update index. The classification result is structure-controlled, content-controlled, or scheduling-responsive. According to the classification result and the preset coordination strategy, the resource allocation order and scheduling time window of the task are adjusted.

[0040] The technical effects and advantages of this invention are as follows:

[0041] This invention performs an initial evaluation before webpage data collection tasks are scheduled, and calculates a structural uncertainty score by combining the range of structural hierarchy changes, the frequency of visible area reconstruction, and rendering position fluctuations. This allows the system to determine the stability of the webpage structure before collection. If the webpage exhibits obvious structural fluctuations, the system automatically enters a deep evaluation process to further extract historical structural change trends and element rearrangement distributions, generating a page structure consistency index. Simultaneously, it analyzes the field density, semantic variations, and temporal patterns of content based on historical snapshots to form a content update index. This method achieves proactive perception of dynamic changes in both structure and content, enabling the collection system to complete adaptation preparation before significant webpage changes occur.

[0042] This invention, after obtaining the page structure consistency index and content update index, inputs them into a pre-trained data collection scheduling prediction model to output data collection scheduling coefficients. Based on the numerical range of these coefficients, a suitable data collection path is dynamically selected. The data collection path includes two processing flows: one is a marker tracking and localization flow with enhanced tolerance for structural reconstruction, and the other is an incremental comparison and snapshot update flow based on content difference identification. This path selection mechanism can match the optimal execution method according to the actual change characteristics of the webpage, effectively reducing data collection failures or information loss caused by path mismatch, thereby enhancing the system's adaptability to heterogeneous webpage structures and frequent content changes.

[0043] After determining the acquisition path, this invention further classifies each acquisition task into execution strategies based on the combination of page structure consistency index and content update index, resulting in structure-driven, content-driven, or scheduling-responsive strategies. Based on the classification results and preset coordination strategies, the system adjusts the resource allocation order and scheduling time window of the tasks, thereby achieving differentiated scheduling when multiple acquisition tasks run in parallel. This mechanism not only avoids task resource contention and scheduling congestion but also provides an orderly and reasonable execution rhythm based on task characteristics, improving the efficiency and stability of the overall scheduling system in handling multiple tasks during complex web page acquisition. Attached Figure Description

[0044] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings;

[0045] Figure 1 This is a schematic diagram of the incremental web page data acquisition method with multi-task collaboration in this invention.

[0046] Figure 2 This is a schematic diagram of the multi-task collaborative incremental web page data acquisition system of the present invention. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] Reference Figure 1 - Figure 2 The following examples were obtained:

[0049] Example 1: A multi-task collaborative incremental web page data acquisition method, comprising the following steps:

[0050] Before scheduling webpage data collection tasks, an initial evaluation is performed on the target webpage. This evaluation includes acquiring the range of structural hierarchy changes, the reconstruction frequency of the visible area, and the fluctuation of rendering position within a preset historical period. A structural uncertainty score is calculated, and the score is used to determine whether to proceed to the in-depth evaluation process. The significance of this step is that, before system resources are allocated to data collection tasks, preliminary collection of structural and visual information allows the system to determine whether the webpage exhibits structural fluctuation characteristics or unstable layout behavior. Through statistical analysis of structural hierarchy, frequency of visible area changes, and rendering offset analysis, the system can obtain an overall stability profile of the webpage. The calculated structural uncertainty score serves as a quantitative indicator to determine whether further in-depth analysis and processing is necessary, thereby achieving risk assessment and screening before data collection tasks.

[0051] When the structural uncertainty score exceeds a preset disorder threshold, a deep evaluation is performed. This deep evaluation includes analyzing the historical change trends of webpage structural nodes, element rearrangement distribution, and regional stability patterns to generate a page structure consistency index. Simultaneously, it compares multiple historical content snapshots to calculate field update density, semantic variation magnitude, and temporal distribution patterns, generating a content update index. The significance of this step is that, for webpages whose initial evaluation results show structural anomalies or frequent adjustments, the system triggers a more precise analysis process. Historical tracking and rearrangement analysis of structural nodes help capture deep-seated change patterns and construct a structural consistency evaluation benchmark; while the comparison between content snapshots reveals the content evolution trajectory. These two indices reflect the dynamic behavior of both structure and content dimensions, becoming core input variables for subsequent decision-making and reasoning.

[0052] The page structure consistency index and content update index are fed into a pre-trained data acquisition scheduling prediction model as input parameters, outputting a data acquisition scheduling coefficient. This coefficient is used to evaluate the execution priority and strategy tendency of the current task within the current scheduling cycle. The significance of this step lies in introducing a learning-based scheduling mechanism to map the structure and content change indicators into a one-dimensional scheduling coefficient, thereby uniformly assessing the urgency and complexity of the current data acquisition task. The scheduling prediction model is trained based on historical data acquisition performance and can predict whether a task is worth executing immediately, whether it needs to occupy high-priority resources, and which type of data acquisition path should be prioritized based on different index combinations, demonstrating the system's intelligent scheduling decision-making capability.

[0053] Based on the numerical range of the data collection scheduling coefficient, the data collection path for the task is selected. This path includes either a marker-tracking and localization process based on enhanced tolerance for structural reconstruction, or an incremental comparison and snapshot update process based on content difference identification. The significance of this step lies in the fact that the system achieves adaptive matching between structure and content by selecting specific execution path strategies in segments according to the scheduling coefficient. If the webpage structure changes significantly, marker-tracking is prioritized to enhance position recovery capabilities; if the content changes frequently, a content comparison process is used to quickly identify incremental data. Path decision-making directly affects the success rate and accuracy of the data collection operation and is a crucial transition from scheduling to execution.

[0054] After determining the acquisition path, the system categorizes acquisition tasks into execution strategies based on the combined relationship between the page structure consistency index and the content update index. The categorization results are classified as structure-driven, content-driven, or scheduling-responsive. According to the categorization results and preset collaborative strategies, the resource allocation order and scheduling time window of the tasks are adjusted to achieve collaborative scheduling and strategy adaptation among multiple tasks. The significance of this step is that after the acquisition path is confirmed, the system intelligently classifies the nature of tasks based on the relative relationship between the two indices, enabling hierarchical management in multi-task concurrent scheduling. The three strategies are mapped to different resource allocation priorities and time window settings, allowing the system to adaptively handle high-frequency content update tasks, structurally variable webpage tasks, and ordinary tasks, ensuring the overall parallel efficiency and strategy consistency of the acquisition system.

[0055] To achieve collaborative scheduling and strategy adaptation among multiple tasks, the system executes the following two-level scheduling control process within each scheduling cycle based on the execution strategy classification results of all acquisition tasks and the corresponding page structure consistency index and content update index: Pre-set collaborative strategies refer to a set of scheduling rules pre-configured by the system to adjust resource allocation priorities and timing peak-shifting strategies for different types of acquisition tasks, taking into account their characteristic differences. Collaborative strategies are divided into three categories according to the task classification results:

[0056] Structure-controlled task strategy: The scheduling priority is high by default; due to the high cost of structure changes and marking reconstruction, the system prioritizes the allocation of execution threads with strong structure tolerance; if there are too many structure-controlled tasks in the same batch, the system restricts their concentrated execution through staggered peaks to avoid the structure marking reconstruction operation causing system concurrency bottlenecks.

[0057] Content-driven task strategy: Priority is automatically adjusted based on the content update index; if the index is higher than the preset high change threshold, the task will get an earlier scheduling time window; similar paragraph snapshots between content-driven tasks can be merged for processing, and the system will aggregate similar tasks into parallel batch processing groups to improve data processing efficiency.

[0058] Scheduled responsive task strategy: This type of task does not trigger emergency collection due to significant structural or content features; the system treats it as a filler task during the scheduling idle period; its execution order is based on the dynamic priority ranking of the collection scheduling coefficient in the low range, and it dynamically selects the opportunity to enter the execution window.

[0059] Resource allocation order adjustment mechanism: Before each acquisition task enters the scheduling engine, it is inserted into the resource allocation table for sorting based on its category label (structure-driven, content-driven, or scheduling-responsive) and acquisition scheduling coefficient. The specific logic is as follows: First, it is grouped according to the category label and queued in the order of structure-driven > content-driven > scheduling-responsive. Within each group, it is sorted from high to low according to the acquisition scheduling coefficient. The higher the acquisition scheduling coefficient, the higher the priority of the task. In case of resource contention and conflict, the system sets a thread limit threshold. For example, the structure-driven type can handle no more than 40% of the total number of threads at the same time, and the rest are occupied by content-driven and scheduling-responsive types.

[0060] Scheduling Time Window Adjustment Mechanism: The system divides each scheduling cycle into multiple time windows (e.g., minute-level, second-level granularity). The scheduling time window for a task is jointly determined by the following parameters: Classification Label: Structurally controlled tasks are given priority for earlier window allocation; Index Combination: Tasks with low structural consistency index and high content update index are allocated earlier; Coefficient Density Control: The system evaluates the distribution density of scheduling coefficients within the same window to avoid scheduling congestion; Clustering and Merging: For tasks with high similarity in structural path or content snapshot, scheduling windows are merged and processed in batches in parallel.

[0061] The multi-task collaboration implementation process is as follows: After initial and in-depth evaluation, all tasks obtain a structural consistency index and a content update index; these two indices are input into the scheduling prediction model to obtain the acquisition scheduling coefficient; a fuzzy logic classifier classifies the tasks according to a strategy, with the label being one of three categories; the task classification result + index value + coefficient → input to the preset collaboration strategy engine; the collaboration strategy engine outputs the resource allocation priority and scheduling time window of the task; the system executes task initiation according to the above scheduling table, dynamically monitors and adjusts the strategy adaptation. For example: Assume there are 10 acquisition tasks in the current scheduling cycle. The system evaluation finds that: Task A is a structure-driven task with an acquisition scheduling coefficient of 0.89; Tasks B and C are content-driven tasks with coefficients of 0.75 and 0.68 respectively; Task DJ is a scheduling response task with a coefficient ranging from 0.32 to 0.56; the system schedules A to be executed in the first window, allocating a high-performance structure marking thread; B and C are aggregated into the second window for snapshot difference comparison; DJ is sorted and inserted into subsequent idle windows in batches; if resources are scarce, J is postponed to the next cycle.

[0062] In this invention, the initial assessment serves as a "pre-judgment mechanism" for data acquisition decisions. The specific logic is as follows: The purpose of the initial assessment is to determine whether the current webpage exhibits sufficiently significant structural fluctuations. If the structural uncertainty score is lower than the "preset disorder threshold," it indicates that the webpage's recent structural changes are not significant, and the system determines that a deep assessment is unnecessary. Because the incremental acquisition strategy of this invention relies entirely on the "page structure consistency index" and "content update index" generated by the deep assessment as input, if a deep assessment is not performed, neither index will be generated; the prediction model cannot assess the acquisition scheduling coefficient; and the system will not execute the incremental acquisition task for that webpage within the current scheduling cycle to save resources and avoid redundant acquisition. This invention is explicitly applicable to scenarios involving parallel acquisition task scheduling and strategy collaboration for multiple webpages. Multiple webpage acquisition tasks are simultaneously scheduled within the current scheduling cycle; each task undergoes individual assessment → model prediction → classification decision; ultimately, the system constructs a task index combination matrix and executes operations such as resource allocation, parallel window peak shifting, and strategy clustering within the preset collaboration strategy.

[0063] The calculation of structural uncertainty score includes the following steps: comparing the structural hierarchy information of the target webpage obtained in the three most recent consecutive collection periods, and extracting the set of structural nodes in each period. Each node in the set of structural nodes contains three indicators: tag name, hierarchy depth, and number of sibling nodes. In this invention, the three behavioral characteristics in the initial evaluation operation are "range of structural hierarchy change", "reconstruction frequency of visible area", and "fluctuation of rendering position". Their judgment and calculation are based on the analysis of the set of structural nodes of the webpage, which is constructed from the structural hierarchy information extracted from the target webpage in the three most recent consecutive collection periods. Each structural node contains three core indicators: tag name, hierarchy depth, and number of sibling nodes. The "range of structural hierarchy change" is mainly obtained by comparing the hierarchy depth of the nodes vertically. The hierarchy depth reflects the nesting hierarchy of the nodes in the entire DOM structure tree. When the hierarchy depth of the node at the same position changes significantly in consecutive periods, or when the deep structure is reconstructed as a whole, it means that the structural hierarchy has fluctuated, thus reflecting the instability of the structural hierarchy range.

[0064] The "reconstruction frequency of the visible area" can be judged by the increase or decrease trend of the number of sibling nodes between adjacent periods. When the number of sibling nodes under the same parent node frequently increases or decreases, it indicates that there are frequent local structural updates in that area, especially structures appearing in the visible area, which may lead to page content reflow or re-rendering. This kind of structural fluctuation indirectly reflects the frequent changes in the composition within the webpage's visible area.

[0065] "Rendering position fluctuations" are related to changes in tag names. Changes in tag names often trigger changes in the corresponding element's appearance on the webpage, CSS style bindings, or behavioral properties.<di v> Become<secti on> , Become Such changes may lead to adjustments in the display position, stacking order, and visual logic of elements. The quantity and positional distribution of tag name changes over a continuous period can help infer the stability of the rendering position. While the three indicators in the structural node set are not directly equivalent to the three behavioral characteristics mentioned above, they provide underlying data support for quantifying and analyzing webpage structural change behavior. By comparing and analyzing these indicators during different collection periods, and combining them with preset weighted calculation rules, the structural hierarchy fluctuation trend, reconstruction frequency changes, and the stability of the rendering structure can be effectively derived, thereby generating a structural uncertainty score.

[0066] A structural node set refers to the collection of all identifiable markup elements that make up the web page document object model. Each structural node corresponds to an HTML tag, for example...<di v> ,<sect i on> , The tag name refers to the specific type of a node in the HTML structure, used to identify its structural function; the hierarchy depth is the nesting level of the node from the root node, with a larger value indicating deeper nesting; the number of sibling nodes refers to the number of nodes at the same level and under the same parent node, reflecting the horizontal density of the structure. In this step, the system captures the complete structural information of the target webpage in three adjacent collection periods, constructing three sets of structural node sets, with each set maintaining the integrity of node attributes. For example, for a news list page, the first period contains 20... Two new nodes were added during the second cycle due to content updates. Node, and remove an ad. If this happens, the node set will change in terms of label type, level, and number, reflecting structural change characteristics.

[0067] The system maps the structural node sets of any two cycles one by one, compares the number of label changes, the number of hierarchical migrations, and the number of newly added or missing nodes, and then sums them according to a preset weighted ratio to obtain the structural change between any two cycles. One-by-one mapping of structural nodes means that for nodes with the same label name or similar structural positions in two consecutive cycles, a correspondence is established, and their attribute changes are compared. The number of label changes refers to the number of times the label name of a node at the same position is changed; the number of hierarchical migrations represents the number of times the depth of the same label node changes in the hierarchical structure; and the number of newly added or missing nodes represents the total number of new nodes appearing or old nodes disappearing in the later cycle. The system sets weighting coefficients for these three types of changes, for example, label changes are weighted at 0.5, hierarchical migrations at 0.3, and newly added or missing nodes at 0.2, and sums these three weighted values ​​to obtain the structural change between the two cycles. Pairwise combinations of the three cycles will generate three different cycle-level change values. Continuing the example above, if there are 5 tag changes, 3 level migrations, and 2 new nodes from the first to the second cycle, then the structural variation during the cycle = 5 × 0.5 + 3 × 0.3 + 2 × 0.2 = 2.5 + 0.9 + 0.4 = 3.8. The average of the structural variations across all cycles is used as the baseline value for structural variation. This baseline value is then divided by the total number of nodes in the current webpage structure. The quotient, multiplied by the preset floating weight coefficient corresponding to the current webpage type, is the structural uncertainty score.

[0068] The structural variation baseline value is the arithmetic mean of the structural variations over three periods, reflecting the overall trend of structural change. The total number of structural nodes on the current webpage refers to the total number of structural nodes extracted in the latest collection period, used to standardize the variation value to a structural scale. Preset floating weight coefficients are weighting factors pre-defined for different webpage types (e.g., news, e-commerce, portal), used to improve the adaptability of the score in different application scenarios. For example, the floating weight is 1.0 for news pages, 0.8 for e-commerce pages, and 0.5 for portal information pages with relatively stable content. Finally, the ratio of the structural variation baseline value to the total number of structures is used as the variation ratio, multiplied by the corresponding floating weight, to obtain the structural uncertainty score. This score is a floating-point number between 0 and 1; a higher value indicates greater uncertainty in the webpage structure in the short term. If the three structural changes are 3.8, 2.6, and 4.1 respectively, with an average of 3.5, the current total number of webpage nodes is 120, the webpage type is news, and the floating weight is 1.0, then the structural uncertainty score = (3.5 / 120) × 1.0 = 0.029.

[0069] The process of generating the page structure consistency index includes the following steps: extracting the structural hierarchy information of the target webpage at five consecutive historical data collection points; converting the webpage structure at each time point into a set of node paths, where each path is represented by a combination of a tag sequence and its hierarchy depth; structural hierarchy information refers to the nesting of nodes in the DOM (Document Object Model) structure, with each path recorded from the root node down to the final structural unit. The set of node paths represents the sum of all paths in the webpage, where each path is a chain composed of a set of tag names in sequence, such as / / / / The tag sequence is the order in which the tags in the path are arranged, and the level depth is the nesting level of the terminal node of the path in the DOM tree (the root node is the first level). Each time point corresponds to a complete webpage collection. The system extracts its structure and converts it into the above path set form, forming a unified and comparable data unit.

[0070] Using the path set corresponding to the fifth time point as a reference, alignment operations are performed on each path in the first four time points sequentially. Path alignment is based on the longest common substring of the tag sequences. If the length of the common substring exceeds 60% of the total path length, the alignment is considered successful. The reference reference refers to the latest webpage structure selected from the five historical data collection time points as the structural alignment target. Path alignment compares paths from any two different time points to determine whether they represent the same or similar structural units. The longest common substring of the tag sequences refers to the longest consecutive identical tag segment in two tag sequences. The similarity is obtained by dividing the length of the longest common substring by the total path length. If this ratio exceeds 0.6 (i.e., 60%), the two paths are considered to be structurally stable and are considered successfully aligned. For example, a path A is / / / <section> / / Path B is / / / / The longest common substring of the two is / / / / The length of path A is 5, and the total length of path A is 6. The similarity is 5 / 6≈0.83, which meets the matching criteria.

[0071] After successful alignment, the hierarchical distance difference between path pairs is calculated, which is the hierarchical difference of the end nodes of the tags. The average of the hierarchical differences of all path pairs is taken to form the hierarchical difference value relative to the baseline structure at that point in time. A path pair refers to a combination of path pairs that are successfully matched during the alignment operation. The hierarchical distance difference is the absolute difference between the hierarchical levels of the terminal nodes (i.e., the end nodes of the paths) in two aligned paths. For example, if one path's end node is at level 6 and the other is at level 5, then the hierarchical distance difference is |6-5|=1. The system calculates this difference for all successfully aligned path pairs at each time point and the fifth time point, and takes the average to generate the hierarchical difference value for the current time point. The smaller this value, the higher the stability of the path hierarchy over time. The concentration of the structural paths at each time point is extracted. The concentration is defined as the percentage of the number of paths at the same level divided by the total number of paths. The hierarchical difference value at each time point is multiplied by its concentration, and then the product values ​​of the five time points are averaged to finally obtain the page structure consistency index.

[0072] The concentration of structural paths refers to the proportion of paths falling at the same level (e.g., the third, fourth, etc.) to the total number of paths on a webpage at a given point in time. Concentration reflects whether the webpage structure exhibits hierarchical aggregation characteristics; for example, content may be densely concentrated in the third to fifth levels, while other levels are sparser. The system uses the concentration at each point in time as a weighting factor for the structural hierarchy stability at that time, multiplying it by the corresponding level difference value to obtain the weighted level deviation. The average of the weighted level deviations at five time points is then taken as the page structure consistency index. For example, assuming the level difference values ​​for the five time points are 1.0, 1.2, 0.9, 1.3, and 0.8, and the corresponding concentrations are 0.7, 0.6, 0.8, 0.5, and 0.9, the weighted values ​​are 0.7, 0.72, 0.72, 0.65, and 0.72, with an average of 0.702. The smaller this value, the more stable the structural paths are over time, and the more consistent the page structure.

[0073] The content update index generation process includes the following steps: extracting content snapshots of the target webpage at five consecutive historical collection time points consistent with the structural hierarchy sampling; dividing the content of each snapshot into multiple content blocks according to paragraph boundaries; and performing difference analysis on the content blocks at the same position in adjacent snapshots at each time point; a content snapshot is a static copy of the webpage's main text area extracted at a specific collection time point, maintaining the original text order, paragraph structure, and location markers. Each snapshot is divided into content blocks according to paragraph boundaries, where paragraph boundaries are logical paragraph separators (e.g., ...). 、 、 <section>) or newline node (e.g. The text blocks are defined by boundaries () and (). Each content block is a paragraph unit within the segmentation result, possessing text continuity and page position stability. The system matches blocks within the same content area from snapshots at two adjacent time points, using a positional offset of no more than 20 characters and identical paragraph numbers as the criteria. Upon successful matching, a difference analysis is performed on the pair of blocks.

[0074] For each pair of adjacent content blocks, the text editing span is calculated, defined as the ratio of the minimum edit distance between two versions of the content block to the total number of characters in the current snapshot. The text editing span measures the degree of textual changes to content at the same location in two snapshots. The minimum edit distance refers to the minimum number of character operations required to transform content from one version to another, including insertion, deletion, and replacement operations. The total number of characters in the current snapshot is the total number of characters in the corresponding paragraph in the snapshot at the current time point. The ratio of the two constitutes the editing span index, representing the average degree of change per character. After normalization, it facilitates a unified comparison between paragraphs of different lengths. For example, a paragraph that reads "This system supports web page scraping" in the previous version and "This system can perform incremental web page scraping" in the current version has a minimum edit distance of 5 and a total number of characters of 15, resulting in an editing span of 5 / 15 = 0.33.

[0075] The magnitude of keyword group change is calculated as the ratio of the symmetric difference between the keyword sets extracted from two versions of the content to the union of the two sets. A keyword set refers to important nouns, verb phrases, or entity phrases extracted from paragraph content using algorithms such as word frequency statistics, part-of-speech tagging, and TF-IDF; these are typically used to describe the semantic center of the paragraph. After extracting keyword sets from each version of the content, their symmetric difference is calculated, which is the union of the unique words in each set. The union size represents the total number of unique words in both sets. The ratio reflects the proportion of word substitution or disappearance in the semantic content and is an indicator of semantic change in the paragraph. For example, if set A = {collection, webpage, increment} and set B = {collection, page, update}, then the symmetric difference = {webpage, increment, page, update}, with a size of 4; the union = {collection, webpage, increment, page, update}, with a size of 5. Therefore, the magnitude of keyword group change = 4 / 5 = 0.8. The publication time offset is calculated as the difference between the standard deviation of the publication times of all paragraphs within a content block and the standard deviation of the average publication time of all paragraphs at that point in time. Publication time refers to the explicit timestamp carried by a webpage paragraph, such as news time, comment publication time, update marker time, etc. A set of associated time data can be extracted for each paragraph. The standard deviation of this set measures whether the time distribution within the paragraph is concentrated. Compared with the average standard deviation of the publication times of all paragraphs at that point in time, a large difference indicates that the paragraph exhibits a significant anomaly in time distribution, possibly indicating a hotspot update area or a region undergoing batch replacement in the background. This quantification of the degree of anomaly in the time characteristics of the content block helps identify implicit change trends. For example, if the average standard deviation at a certain point in time is 2 hours, and the standard deviation within a certain paragraph is 5 hours, then the offset is 5-2=3 hours, which, after normalization, is included in the update index calculation.

[0076] The three values ​​for each paragraph—text editing span, keyword group variation, and publication time offset—are normalized to a range of 0 to 1. The normalized result is then multiplied by the paragraph's display density on the webpage, which is the percentage of the paragraph's total content length. The product of all paragraphs in each snapshot pair is summed, and the average of the sums across the five snapshots is used to generate the content update index. Normalization uses the minimum-maximum normalization method to map the original values ​​to the range [0,1]. Display density represents the relative proportion of the paragraph on the webpage, calculated by dividing the paragraph's word count by the total word count of the webpage. For example, a paragraph with 300 words and a total page count of 3000 words would have a display density of 0.1. The normalized result of the three indicators is multiplied by this density to form a weighted index, emphasizing that changes in the core area of ​​the webpage have a greater impact than those in the peripheral areas. In each snapshot pair, the weighted change values ​​of all paragraphs are summed to obtain the change score for that pair of snapshots. The average of the four change scores across the five snapshots is the content update index. The index ranges from [0,1], with values ​​closer to 1 indicating a higher frequency of webpage content updates and greater fluctuations.

[0077] The acquisition scheduling prediction model used to generate acquisition scheduling coefficients is pre-trained using supervised learning. Supervised learning refers to a machine learning method that optimizes model parameters based on known input-output sample pairs, requiring explicit labels as training targets. In this technical solution, training samples are derived from historical acquisition tasks, with each acquisition task serving as the basic sample unit. Each sample contains a set of input features and an output result. The training samples include the following five input feature variables: Page structure consistency index: used to quantify the stability of the webpage structure in the time series, derived from the analysis of structural hierarchy path change trends; Content update index: used to measure the update frequency and degree of variation of the webpage's main text area in historical snapshots; Actual acquisition success rate: refers to the ratio of the number of times the target data was successfully extracted from this type of page within the execution cycle to the total number of executions in historical tasks, reflecting task reachability; Scheduling delay time: refers to the average time interval between the acquisition task being scheduled and its actual completion, in seconds, used to characterize the degree of delay caused by the system; Resource utilization rate: refers to the ratio between the computing resources consumed by the task (such as the number of CPU cores, memory usage, etc.) and the system's allowed upper limit, reflecting resource pressure.

[0078] During training, the model aims to minimize the combined loss function of scheduling error rate, scheduling delay time, and resource waste rate. The following are the definitions and calculation methods for the three loss dimensions: Scheduling error rate: This refers to the ratio of scheduling failures caused by the deviation between the model's output scheduling coefficient and the actual execution result in historical tasks. It is calculated as: Scheduling error rate = Number of tasks exceeding the scheduling coefficient threshold but failing to complete successfully / Total number of tasks. Explanation: The system sets several thresholds based on the output scheduling coefficient (e.g., above 0.8 is recommended for immediate scheduling). If a task's scheduling coefficient is higher than the set threshold but fails to complete successfully (e.g., due to rapid webpage changes or resource conflicts), it is counted as a scheduling failure. The scheduling error rate is obtained by statistically analyzing these "scheduling prediction deviation" samples. Scheduling delay time: This is calculated as the average of the difference between the task scheduling completion time and the scheduling trigger time across all training samples. The larger this value, the more likely it is that the system has a bottleneck or insufficient prediction accuracy leads to low resource allocation efficiency. Resource waste rate: This represents the waste rate caused by the task failing to fully utilize the allocated resources after the model predicts that the task will use them. The calculation method is: Resource waste rate = (Total allocated resources - Actual consumed resources) / Total allocated resources. The comprehensive loss function is a weighted combination of the above three indicators to construct an objective function for model optimization. Let the three loss terms be L1 (error rate), L2 (delay time), and L3 (waste rate), then the comprehensive loss function can be expressed as: Loss = α*L1 + β*L2 + γ*L3, where α, β, and γ are non-zero weight coefficients set by the user or system based on scheduling sensitivity. For example, for a real-time acquisition system, β > α > γ might be set.

[0079] The final model is constructed using an ensemble regression algorithm, and the output acquisition scheduling coefficient is a continuous value between zero and one. An ensemble regression algorithm refers to obtaining a prediction model with higher accuracy and stability by integrating multiple basic prediction models (such as decision tree regressors, linear regressors, gradient boosting regressors, etc.). Commonly used methods include random forest regression and XGBoost. The output of this model is the acquisition scheduling coefficient, which is a continuous value in the normalized range [0,1]. This coefficient is used to represent the overall urgency and resource adaptation difficulty of the current acquisition task within the scheduling cycle. The closer the value is to 1, the more urgent the task, the more drastic the changes in structure or content, and the more priority the system needs to allocate resources for processing. For example, the characteristics of a historical task sample are as follows: page structure consistency index: 0.85, content update index: 0.76, actual collection success rate: 0.92, scheduling latency: 120 seconds, resource utilization rate: 0.55. This five-dimensional vector is input into the model during training as input features. The model uses its actual performance to update the comprehensive loss function, enabling the final model to predict a continuous output value (e.g., 0.89) for new task inputs. This collection scheduling coefficient itself is an urgency score referenced by the collection scheduling system. The higher the value, the more volatile the webpage structure and content are, the higher the historical success rate, and the more reasonable the current latency and resource utilization are, thus justifying priority scheduling.

[0080] After the acquisition scheduling prediction model outputs the acquisition scheduling coefficient, the system selects the acquisition path of the task according to the numerical range of the coefficient. Based on the preset path switching threshold, the acquisition path is divided into the following two types: (1) Mark tracking and positioning process (when the scheduling coefficient is ≥0.6). When the acquisition scheduling coefficient is greater than or equal to the structural path activation threshold (e.g., 0.6), the system selects the mark tracking and positioning process based on structural reconstruction tolerance enhancement as the acquisition path of the task. This process calculates the similarity of the tag sequence, path offset and hierarchical tolerance in the structural path, reconstructs the tag index and completes data positioning. It is suitable for web pages with high structural change frequency and fast structural evolution speed. (2) Incremental comparison and snapshot update process (when the scheduling coefficient is <0.6). When the acquisition scheduling coefficient is less than the structural path activation threshold, the system selects the incremental comparison and snapshot update process based on content difference identification as the acquisition path. This process focuses on the rapid analysis of the text editing span between content blocks, keyword group differences and publication time arrangement offset, accurately extracts the changed parts and replaces the snapshot locally. It is suitable for web pages with relatively stable structure and frequent content updates.

[0081] The tag tracking and localization process is used to re-identify and adjust the structural paths of target data on a webpage when the webpage structure changes frequently, in order to achieve accurate tracking of data tags. This process includes the following steps: First, the stored set of structural tag paths is retrieved from the last successfully acquired version. The set of structural tag paths refers to the set of paths pointing to the target data location in the webpage's structure tree. Each path consists of a sequence of tag names and their corresponding hierarchical index in the structure tree. The tag names are element names in the HTML structure, such as div, span, etc., and the hierarchical index is its depth position in the DOM tree, such as the level number from the root node. Then, a path scanning operation is performed on the current webpage's structure tree. The structure tree refers to the hierarchical representation of the current webpage's Document Object Model (DOM). Starting from the root node, the system traverses each level of nodes, extracts all identifiable paths in the current page, and matches them one by one with the stored paths. For each stored path, the system calculates its matching similarity with the current structural path. This similarity is calculated based on factors such as the consistency of the tag sequence, the degree of hierarchical alignment, and changes in path length. If the similarity value of a path is lower than a preset threshold (e.g., 0.7), the path is identified as an offset path. An offset path indicates that the page structure has changed, causing the element position pointed to by the original path to become invalid or drift. For each offset path, the system reconstructs the hierarchical pointer based on the structural nodes of the current webpage. This pointer represents a new path from the root node to the current matching position, replacing the old structural path. After the path reconstruction is completed, the system updates by replacing the old path with the new path, thereby ensuring the accuracy and stability of data labeling in subsequent data collection tasks.

[0082] The incremental comparison and snapshot update process is used to accurately update and partially replace snapshots when webpage content changes frequently but the structure remains relatively stable. This is achieved by detecting content differences. The process includes the following steps: First, the system extracts content areas from the current webpage. Content areas refer to the main text associated with the target data, typically the central body, excluding structural elements such as navigation bars and footers. Simultaneously, the system extracts the content areas from the previous version of the webpage at their corresponding structural positions. Each content area is divided into multiple content blocks according to paragraph boundaries. Paragraph boundaries can be determined using line break tags or text density in the HTML structure. Each content block is considered the smallest comparison unit and indexed by its paragraph start position and length on the page, facilitating subsequent mapping and comparison. Subsequently, a bidirectional mapping operation is performed on the content block indices of the two versions. The criteria for a successful match include: the starting position of the current version's paragraph is offset from the starting position of the corresponding paragraph in the previous version by no more than fifty characters, and the length change ratio is no more than twenty percent. Content blocks meeting these conditions are considered "successfully matched blocks"; those not meeting these conditions are considered "failed-match blocks." For each content block that fails to match, the system extracts the corresponding keyword group.

[0083] Keyword groups refer to a representative set of core words extracted from webpage content blocks, used to measure the degree of semantic shift during content changes. In the "Incremental Comparison and Snapshot Update Process," for content blocks that fail to match, the system extracts keyword groups using a pre-defined text analysis algorithm. This algorithm is a keyword extraction algorithm based on Term Frequency-Inverse Document Frequency (TF-IDF), optimized with a part-of-speech tagging mechanism. The specific steps are as follows: Keyword extraction for each content block originates from two semantic locations: the paragraph heading field: referring to the heading tag content to which the content block belongs in the webpage structure. Heading tags include HTML tags... <h1>、< / h1> <h2>、< / h2> <h3>Etc., the system uses the nearest title field above the content block as its semantic tag input; the first fifty words of the paragraph: refer to the first fifty consecutive words at the beginning of the body part of the current content block, which is considered the most representative and is a semantic core area with a high information density. The text contents from the above two sources will be combined as the original candidate corpus for keyword extraction. Calculate the TF-IDF score for each term in the candidate corpus, by calculating its TF-IDF value for each term and sorting all terms in descending order of the score. After the sorting is completed, the system filters the final keyword group from the top 20% of the terms. The filtering process combines a词性判断机制 (lexical category judgment mechanism), and only retains the terms in the following lexical categories: Noun categories: such as topic words, thing names, organization names, etc.; Verb categories: such as key actions, operation methods; Proper noun categories: such as brands, products, technical terms; The system eliminates stop words, punctuation marks, numbers, and non-semantic-bearing words (such as "的", "是", "了", etc.), and combines duplicate or variant words. The finally formed keyword set is used as the "keyword group" of this content block for semantic variation comparison between subsequent versions. The keyword group difference is defined as the ratio of the number of the symmetric difference set of the keyword sets of the two versions to the number of the union of the two sets. The symmetric difference set is the number of non-overlapping parts in the two sets, reflecting the degree of semantic change. If this difference value is greater than 0.6, the system marks this content block as changed content. All paragraphs determined to be changed content will be used as updated paragraphs to replace the original paragraph content at the corresponding position in the snapshot. At the same time, the system records the timestamp of this change (i.e., the collection time of the current version) and the corresponding keyword group for subsequent snapshot version management and change trend analysis. For example, a web page content block is a product introduction paragraph, and its所属标题 (associated title) is "H1: Product Function Description". The first fifty words in the body include words such as "Intelligent Recognition", "Image Enhancement", "Edge Detection", "Automatic Adjustment", etc. After TF-IDF calculation and lexical filtering, the system selects the terms with the top TF-IDF values and valid lexical categories as the keyword group: → The keyword group is {"Intelligent Recognition", "Image Enhancement", "Edge Detection", "Automatic Adjustment"}. When the keyword groups between two versions change significantly, that is, the proportion of the symmetric difference set is greater than 0.6, the system determines this block as a content block with semantic changes.

[0084] It should be noted that the term "词性判断机制" in the original text is not clearly defined. I have left it in Chinese first and marked it for you to fill in the correct English expression according to the actual situation. If you have any other questions, please feel free to let me know.After determining the data collection path, the system uses a fuzzy logic classifier to classify the execution strategy of the current data collection task. In this scheme, the fuzzy logic classifier uses the page structure consistency index and content update index as input variables, and generates classification labels through preset membership functions and decision rules to classify the execution strategy of the data collection task. Page structure consistency index: a value generated after deep evaluation, representing the stability of the target webpage structure within a historical period; a higher value indicates a more stable structural evolution. Content update index: a value generated based on historical content snapshots, reflecting the update frequency and change magnitude of the target webpage content in the time dimension; a higher value indicates faster content change. Both are continuous real-valued inputs between [0,1].

[0085] The system defines multiple membership functions for each of the two input variables, representing the fuzzy sets to which they semantically belong. For example, for the "page structure consistency index," three membership functions are defined: Low consistency, Medium consistency, and High consistency; for the "content update index," three membership functions are defined: Low update frequency, Moderate update frequency, and High update frequency. Each input value can belong to multiple membership functions simultaneously based on its position in the numerical range. The system uses triangular or trapezoidal membership functions for modeling, satisfying the principles of continuity and overlap. The system establishes a fuzzy rule base based on task characteristics. For example, if the structure consistency index is "high" and the content update index is "low," it is classified as "structure-controlled type"; if the structure consistency index is "low" and the content update index is "high," it is classified as "content-controlled type"; if both the structure consistency index and the content update index are "medium," or one is high and the other is low, it is classified as "scheduling response type." Each rule in the rule base is in the form of "IF-THEN", which describes the correspondence between the input fuzzy set and the output classification label.

[0086] The fuzzy logic classifier performs fuzzy inference calculations based on the membership values ​​of all activation rules, using the maximum membership principle as the output mechanism. That is, it selects the classification label with the highest membership as the final strategy category for the task. Classification results include: Structure-driven: Indicates that the current task's data collection is significantly affected by structural changes, requiring a tag-tracking strategy with strong tolerance for structural reconstruction; Content-driven: Indicates that the task's data collection mainly relies on content variation detection, suitable for incremental comparison and snapshot update processes; Scheduling-responsive: Indicates that the task is not significantly affected by structural or content-driven factors, and the execution priority and time window should be adjusted according to the data collection scheduling coefficient. Assume that a webpage, after deep evaluation, has a page structure consistency index of 0.82 and a content update index of 0.18. Based on the membership function, this structure consistency value has a "high consistency" membership of 0.9 and a "medium consistency" membership of 0.1; the content update value has a "low update frequency" membership of 0.85 and a "medium update frequency" membership of 0.15. The system matches the fuzzy rule "If structural consistency is high and content update is low, then the task is structurally dominant." This rule has the highest activation level, and the final output is classified as structurally dominant. The system then maps this task to a resource scheduling and acquisition path execution plan that matches the structurally dominant strategy.

[0087] Example 2: A multi-task collaborative incremental web page data acquisition system, specifically including:

[0088] The initial structural evaluation module is used to perform an initial evaluation of the target webpage before the webpage collection task is scheduled. The evaluation includes obtaining the range of structural hierarchy changes, reconstruction frequency of the visible area, and rendering position fluctuation of the webpage within a preset historical period, calculating the structural uncertainty score, and determining whether to enter the deep evaluation process based on the score.

[0089] The deep perception module is used to perform a deep evaluation operation when the structural uncertainty score exceeds the preset disorder threshold. The deep evaluation includes analyzing the historical change trend of web page structural nodes, element rearrangement distribution and regional stability patterns, and generating a page structure consistency index; at the same time, it compares multiple historical content snapshots, calculates field update density, semantic variation magnitude and time distribution pattern, and generates a content update index.

[0090] The scheduling prediction module is used to input the page structure consistency index and content update index as input parameters into the pre-trained collection scheduling prediction model and output the collection scheduling coefficient. This coefficient is used to evaluate the execution priority and strategy tendency of the current task in the current scheduling cycle.

[0091] The path execution module is used to select the acquisition path of the task based on the numerical range of the acquisition scheduling coefficient. The acquisition path includes a marker tracking and localization process based on structural reconstruction tolerance enhancement, or an incremental comparison and snapshot update process based on content difference identification.

[0092] The strategy assignment module is used to classify the execution strategy of the collection task after the collection path is determined, based on the combination of the page structure consistency index and the content update index. The classification result is structure-controlled, content-controlled, or scheduling-responsive. According to the classification result and the preset coordination strategy, the resource allocation order and scheduling time window of the task are adjusted.

[0093] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0094] It should be understood that, in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0095] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0096] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0097] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.< / h3> < / section> < / section>

Claims

1. A multi-task collaborative incremental web page data acquisition method, characterized in that, Includes the following steps: Before scheduling the webpage scraping task, an initial evaluation is performed on the target webpage. This initial evaluation includes obtaining the range of structural hierarchy changes, the reconstruction frequency of the visible area, and the rendering position fluctuations of the webpage within a preset historical period. A structural uncertainty score is calculated, and the decision to proceed to the deep evaluation process is based on this score. The calculation of the structural uncertainty score includes the following steps: The structural hierarchy information of the target webpage obtained in the three most recent consecutive collection cycles is compared, and the set of structural nodes in each cycle is extracted. Each node in the set of structural nodes contains three indicators: tag name, hierarchy depth and number of sibling nodes. Map the set of structural nodes for any two periods one by one, compare the number of label changes, the number of hierarchical migrations, and the number of new or missing nodes, and sum them up according to a preset ratio to obtain the structural variation of any two periods; take the average of the structural variation of all periods as the structural variation benchmark value, divide the benchmark value by the total number of structural nodes of the current webpage, and multiply the quotient by the preset floating weight coefficient corresponding to the current webpage type to obtain the structural uncertainty score. When the structural uncertainty score exceeds the preset disorder threshold, a deep evaluation operation is performed. The deep evaluation includes analyzing the historical change trends of webpage structural nodes, element rearrangement distribution, and regional stability patterns to generate a page structure consistency index; at the same time, it compares multiple historical content snapshots to calculate field update density, semantic variation magnitude, and time distribution patterns to generate a content update index; the generation process of the page structure consistency index includes the following steps: Extract the structural hierarchy information of the target webpage at five consecutive historical collection time points, and transform the webpage structure at each time point into a set of node paths. Each path is represented by a combination of a tag sequence and its hierarchy depth. Using the path set corresponding to the fifth time point as a reference, alignment operations are performed on each path in the first four time points in turn. Path alignment is based on the longest common substring of the label sequence. If the length of the common substring exceeds 60% of the total path length, it is considered a successful alignment. After successful alignment, calculate the hierarchical distance difference between path pairs, i.e. the hierarchical difference of the end node of the label, and average the hierarchical differences of all path pairs to form the hierarchical difference value of that point in time relative to the baseline structure. Extract the concentration of the structural path at each time point. The concentration is defined as the percentage of the number of paths at the same level divided by the total number of paths. Multiply the level difference value at each time point by its concentration value to obtain the product value. Then average the product values ​​at the five time points to finally obtain the page structure consistency index. The process of generating the content update index includes the following steps: Extract content snapshots of the target webpage at five consecutive historical collection time points consistent with the structural hierarchy sampling. Divide the content of each snapshot into multiple content blocks according to paragraph boundaries, and perform difference analysis on the content blocks at the same position of adjacent snapshots at each time point. For each pair of adjacent content blocks, calculate the text editing span, which is defined as the ratio of the minimum editing distance between the two versions of the content block to the total number of characters in the content block in the current snapshot; The magnitude of keyword group change is calculated as the ratio of the size of the symmetric difference between the sets of keywords extracted from the two versions of the content to the size of the union of the two sets; Calculate the publication time arrangement offset, which is defined as the difference between the standard deviation of the publication time of all paragraphs in the content block and the standard deviation of the average publication time of all paragraphs at that point in time. The three values ​​of each group of paragraphs—text editing span, keyword group change range, and publication time offset—are normalized to the range of zero to one. The normalized result is then multiplied by the display density of the paragraph on the webpage, which is the percentage of the paragraph's total content length on the webpage. The product of all paragraphs in each pair of snapshots is summed, and the average of the sums across the five snapshots is taken to generate the content update index. The page structure consistency index and content update index are fed into the pre-trained data collection scheduling prediction model as input parameters, and the data collection scheduling coefficient is output. This coefficient is used to evaluate the execution priority and strategy tendency of the current task in the current scheduling cycle. Based on the numerical range of the collection scheduling coefficient, the collection path of the task is selected. The collection path includes a marker tracking and localization process based on structural reconstruction tolerance enhancement, or an incremental comparison and snapshot update process based on content difference identification. After determining the collection path, the collection tasks are classified into execution strategies based on the combination of the page structure consistency index and the content update index. The classification results are structure-controlled, content-controlled, or scheduling-responsive. According to the classification results and the preset collaborative strategy, the resource allocation order and scheduling time window of the tasks are adjusted to complete the collaborative scheduling and strategy adaptation between multiple tasks.

2. The incremental web page data acquisition method with multi-task collaboration according to claim 1, characterized in that, The acquisition scheduling prediction model used to generate acquisition scheduling coefficients is pre-trained through supervised learning. The training samples include page structure consistency index, content update index, actual acquisition success rate, scheduling delay time, and resource utilization rate in historical acquisition tasks. During training, the model objective is to minimize the comprehensive loss function of scheduling error rate, scheduling delay time, and resource waste rate. The final model is constructed using an ensemble regression algorithm, and the output acquisition scheduling coefficients are continuous values ​​between zero and one, used to represent the comprehensive urgency and resource adaptation difficulty of the current acquisition task within the scheduling cycle.

3. The incremental web page data acquisition method with multi-task collaboration according to claim 2, characterized in that, The tag tracking and localization process includes the following steps: Retrieve the stored set of structural tag paths from the last successfully collected version. Each set of paths contains a sequence of tag names and their corresponding level index. Perform a path scan on the current webpage structure tree, calculate the matching similarity between each stored path and the current structural path, and mark paths with similarity below a preset ratio threshold as offset paths. Reconstruct the level pointer for each offset path and update it by replacing the old path with the new path.

4. The incremental web page data acquisition method with multi-task collaboration according to claim 3, characterized in that, The incremental comparison and snapshot update process includes the following steps: Get the content area of ​​the current webpage and divide it into paragraphs based on the content area corresponding to the structural position in the previous version of the webpage. Each paragraph is a content block, and the content block is indexed by the dual coordinates of the paragraph's start position and length. Perform bidirectional mapping on the content block indexes in the two versions. The conditions for a successful match are that the starting position offset of the current version paragraph does not exceed fifty characters and the length change does not exceed twenty percent. If these conditions are not met, the block is considered a failed match. Keyword groups are extracted from all content blocks that fail to match. Each keyword group is selected from the title field and the first fifty words of the paragraph using a preset text analysis algorithm. The keyword group difference is defined as the ratio of the number of symmetric differences between the keyword sets of two versions to the number of their union. If the difference is greater than 0.6, the content block is determined to be changed content. All paragraphs determined to be changed content are used as update paragraphs to replace the original paragraph content at the corresponding position in the snapshot, and the timestamp and keyword group of this change are recorded.

5. The incremental web page data acquisition method with multi-task collaboration according to claim 4, characterized in that, After determining the collection path, the collection task is classified into execution strategies by combining the page structure consistency index and the content update index. The fuzzy logic classifier uses the page structure consistency index and the content update index as input variables and generates classification labels according to the preset membership function and decision rules. The classification results are structure-controlled, content-controlled, or scheduling-response type.

6. A multi-task collaborative incremental web page data acquisition system, based on the multi-task collaborative incremental web page data acquisition method according to any one of claims 1-5, characterized in that, Specifically, it includes: The initial structural evaluation module is used to perform an initial evaluation of the target webpage before the webpage collection task is scheduled. The evaluation includes obtaining the range of structural hierarchy changes, reconstruction frequency of the visible area, and rendering position fluctuation of the webpage within a preset historical period, calculating the structural uncertainty score, and determining whether to enter the deep evaluation process based on the score. The deep perception module is used to perform a deep evaluation operation when the structural uncertainty score exceeds the preset disorder threshold. The deep evaluation includes analyzing the historical change trend of web page structural nodes, element rearrangement distribution and regional stability patterns, and generating a page structure consistency index; at the same time, it compares multiple historical content snapshots, calculates field update density, semantic variation magnitude and time distribution pattern, and generates a content update index. The scheduling prediction module is used to input the page structure consistency index and content update index as input parameters into the pre-trained collection scheduling prediction model and output the collection scheduling coefficient. This coefficient is used to evaluate the execution priority and strategy tendency of the current task in the current scheduling cycle. The path execution module is used to select the acquisition path of the task based on the numerical range of the acquisition scheduling coefficient. The acquisition path includes a marker tracking and localization process based on structural reconstruction tolerance enhancement, or an incremental comparison and snapshot update process based on content difference identification. The strategy assignment module is used to classify the execution strategy of the collection task after the collection path is determined, based on the combination of the page structure consistency index and the content update index. The classification result is structure-controlled, content-controlled, or scheduling-responsive. According to the classification result and the preset coordination strategy, the resource allocation order and scheduling time window of the task are adjusted.

Images