Science and technology information collection and dispatching system based on load awareness of computing nodes

Abstract

The present application belongs to the technical field of computer data processing and distributed task scheduling, and specifically relates to a scientific and technological information collection scheduling system based on load sensing of computing nodes. The system records the execution track of scientific and technological information collection tasks in stages, establishes the resource occupation increment, resource release lag and queue propagation relationship, and forms information timeliness debt according to the information source update interval, historical change density, continuous failure state and downstream dependency relationship. The to-be-scheduled task is mapped to the candidate computing node, the resource evolution track is generated in combination with the load causal fingerprint, the sustained congestion mapping is excluded, the task node mapping is determined according to the load propagation cost and timeliness debt reduction amount, and the load causal fingerprint is updated according to the attributable deviation between the actual and predicted tracks. The present application can reduce the queue sustained growth and resource release lag caused by static load judgment, and coordinate the recoverability of computing nodes and the timeliness of scientific and technological information collection.

Description

Technical Field

[0001] This invention belongs to the field of computer data processing and distributed task scheduling technology, specifically a scientific and technological information collection and scheduling system based on computing node load awareness. Background Technology

[0002] Scientific and technological information acquisition systems typically employ a combination of task queues and computing node resource monitoring for distributed scheduling. The system generates acquisition tasks based on information source type, acquisition cycle, data page count, and parsing process. It periodically reads the processor utilization, memory usage, input / output wait time, and queue length of each computing node, converting these states into node load values. The scheduler allocates unexecuted acquisition tasks to computing nodes with lower load values ​​based on rules such as round-robin, minimum load, weighted minimum queue, or preset priority. After a task enters a node, it sequentially performs data access, content acquisition, field parsing, duplicate information comparison, related information extraction, and result writing. Some systems also set resource requirement tags for tasks based on their historical execution duration or data size, performing simple matching during node selection. This approach primarily relies on the node state at the time of scheduling, avoiding significant resource concentration before tasks begin execution. However, its load assessment typically remains at the static snapshot level, failing to continuously describe resource usage changes, resource release delays, and queue propagation processes that occur at different execution stages after a task enters a node.

[0003] To mitigate allocation biases caused by static load assessments, existing technologies employ historical statistical or predictive models to estimate task execution time and resource consumption, combining the prediction results with the node's current remaining resources to form a scheduling score. A common implementation involves summarizing historical processor usage, memory residency time, data read volume, and average completion time by task type to create task feature templates. These templates are then selected based on the data size of the tasks to be scheduled, calculating the expected load on different nodes. For high-priority tasks in scientific and technological information collection, the system typically increases scheduling weights based on the collection cycle, information source level, or manually set order to shorten waiting time. Some solutions also update historical averages after task execution, gradually bringing subsequent predictions closer to actual execution conditions. However, these methods still treat the node's current load, expected task consumption, and task priority as independent evaluation metrics, determining nodes through linear weighting or sequential comparison. This makes it difficult to distinguish resource changes caused by the task itself, coupling changes caused by concurrent task competition, and propagation changes caused by delays in preceding stages. Furthermore, it fails to represent the continuous accumulation of time-sensitive information during the waiting process.

[0004] The inability to simultaneously characterize the load propagation trajectory of acquisition tasks after they enter candidate computing nodes and the time loss caused by the continued waiting of scientific and technological information before task allocation leads to a disconnect between node selection criteria and the actual task execution process. Scientific and technological information acquisition tasks involve multiple stages, including data access, content parsing, duplicate comparison, and result writing. Each stage has different usage patterns for processors, memory, input / output channels, and task queues, and the same task may result in different resource release orders under different concurrency combinations. A node being in a low-load state during scheduling does not guarantee that a recoverable queue state will be maintained after tasks enter. If the scheduler allocates tasks solely based on current idle capacity, multiple tasks with continuous resource consumption characteristics may be concentrated on the same node, leading to continuous queue growth and delayed resource release. Furthermore, fixed priorities or sorting solely based on task weight cannot reflect the cumulative time loss caused by information source update intervals, historical change density, continuous acquisition failures, and downstream dependency blockages, easily causing tasks with high resource consumption but strong time requirements to be continuously postponed. Therefore, existing solutions struggle to establish a unified and correctable scheduling basis that balances avoiding continuous node congestion with controlling delays in scientific and technological information acquisition. Summary of the Invention

[0005] The purpose of this invention is to provide a scientific and technological information collection and scheduling system based on computing node load awareness, which can solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: The technology information collection and scheduling system based on computing node load awareness includes a task status construction module, a load causal fingerprint generation module, an information timeliness debt generation module, a candidate node inference module, and a scheduling closed-loop correction module. The task status construction module records the execution trajectory of the scientific and technological information collection task in stages; The load causal fingerprint generation module establishes the resource usage increment, resource release lag, and queue propagation relationship after the task enters the computing node based on the execution trajectory. The information time-sensitive debt generation module generates time-sensitive debts based on the information source update status and task waiting status. The candidate node inference module maps the tasks to be scheduled to candidate computing nodes, generates resource evolution trajectories by combining the current task combination of the node with the load causal fingerprint, deletes persistent congestion mappings, and determines the task node mapping based on the load propagation cost and time-sensitivity debt reduction of the remaining mappings. The scheduling closed-loop correction module updates the load causal fingerprint based on the attributable deviation between the actual execution trajectory and the resource evolution trajectory.

[0007] Preferably, the load causal fingerprint generation module divides the execution trajectory of each scientific and technological information collection task into multiple stage segments, and establishes resource state transition records for adjacent stage segments; The resource state transition record includes at least processor usage increment, memory resident increment, input / output wait increment, task queue length change, cache hit status change, and resource release time difference; The load causal fingerprint generation module extracts the direct load path triggered by the task itself, the coupled load path triggered by shared resource contention, and the propagation load path triggered by delays in preceding stages based on the resource state transition records of the same type of task under different concurrent task combinations. It then establishes a set of load causal fingerprints with stage connection relationships according to task type, data size range, and node execution environment for the candidate node inference module to call.

[0008] Preferably, the information timeliness debt generation module establishes a timeliness status sequence corresponding to the scientific and technological information source for each task to be scheduled. The timeliness status sequence includes the information source's historical update interval, the most recent successful collection time, the density of historical content changes, the number of consecutive collection failures, the version differences of the collected information, and the downstream task dependencies. The information timeliness debt generation module determines the basic debt segment based on the relative position of the current waiting time in the historical update interval, adjusts the accumulation rate of the basic debt segment based on the density of historical content changes and the number of consecutive collection failures, and decomposes the task debt into ontology debt and dependency blocking debt according to the downstream task dependency relationship, forming a timeliness debt sequence that is updated with the waiting state, which is called by the candidate node inference module when comparing candidate mappings.

[0009] Preferably, the candidate node inference module copies the current task combination stage state for each candidate computing node, embeds the initial stage of the task to be scheduled into the stage state, and generates a prediction sequence of processor usage, memory residency, input / output wait and task queue length stage by stage according to the stage connection relationship in the load causal fingerprint. The candidate node inference module constructs a node recovery trajectory based on the continuous growth segment, resource release segment, and queue fallback segment of each resource prediction sequence, and marks candidate maps that do not have a queue fallback segment, whose resource release time is later than the predetermined task window, or whose multiple resource prediction sequences continue to grow simultaneously as continuous congestion maps. After the candidate node deduction module removes the persistent congestion mapping from the candidate mapping set, it forms a remaining mapping set for task node mapping determination.

[0010] Preferably, the scheduling closed-loop correction module performs time alignment between the actual execution trajectory and the resource evolution trajectory according to the stage segment, and generates processor occupancy residual, memory resident residual, input / output waiting residual and queue change residual; The scheduling closed-loop correction module uses the start and end records of concurrent tasks on the same computing node to identify the competing residuals that change synchronously with the state of concurrent tasks, uses the difference between the current data size and the corresponding historical data size to identify the scale mutation residuals, and marks the remaining residuals after deducting the competing residuals and the scale mutation residuals as task-attributable residuals. The scheduling closed-loop correction module corrects the direct load path of the corresponding task type only based on the task attributable residual, and corrects the coupled load path based on the contention residual, while keeping the propagation load path and the stage connection relationship updated respectively.

[0011] Preferably, the information timeliness debt generation module represents the downstream task dependency relationship as a task dependency graph composed of collection task nodes and dependency edges, and records the version identifier of the preceding collection result being parsed, associated, extracted, or indexed into the subsequent task call in the dependency edge; When multiple scheduled tasks block the same subsequent task, the information timeliness debt generation module divides the dependent blocking debt into mutually exclusive debt segments and overlapping debt segments according to the data range that each preceding task has not yet provided. Only one debt record is retained for the overlapping debt segment, and a debt ownership relationship is established according to the data coverage ratio of each preceding task. After any preceding task completes mapping, the candidate node deduction module deducts the corresponding debt fragment from the task dependency graph based on the data range released by the completed task, and synchronously updates the time-debt sequence of the remaining tasks to be scheduled.

[0012] Preferably, the candidate node inference module extracts the load propagation cost, node recovery time, time-limited debt reduction, and dependency blocking debt release for the remaining mapping set, constructs a resource constraint dimension according to the load propagation cost and the node recovery time, and constructs a time-limited benefit dimension according to the time-limited debt reduction and dependency blocking debt release. The candidate node deduction module first deletes candidate mappings that are simultaneously dominated by another mapping scheme in two dimensions, then determines the insertable execution interval based on the resource prediction sequence corresponding to each candidate mapping and the existing task release time of the node, and determines the task start order within the insertable execution interval according to the growth order of the time-debt of each task. When multiple candidate mappings have the same startup order, the task node mapping with the lowest overlap between the load propagation path and the existing task load path of the node is selected.

[0013] Preferably, the candidate node inference module establishes an inter-task load interference matrix for multiple tasks to be scheduled within the same scheduling period. The elements of the inter-task load interference matrix are generated by the shared resource occupancy increment formed by the resource evolution trajectories of two tasks during the stage overlap. The candidate node deduction module establishes a debt growth order based on the time-sensitive debt of each task to be scheduled, and adds the task node mapping one by one according to the debt growth order. Each time a task node mapping is added, the resource evolution trajectory of the remaining candidate mappings is corrected using the inter-task load interference matrix. When the corrected resource evolution trajectory is marked as the persistent congestion mapping, the mapping of the most recently added task node with a small amount of dependent blocking debt release is revoked, and the corresponding task is moved into the next insertable execution interval until a batch scheduling sequence that does not contain the persistent congestion mapping is formed.

[0014] Preferably, the scheduling closed-loop correction module maintains an updated reliable state for each direct load path, coupled load path, and propagation load path, consisting of the number of samples, the consistency of residual direction, and the coverage of the node environment. When the task attributable residuals continuously point to the same resource change direction and the corresponding node environment coverage meets the update conditions, the scheduling closed-loop correction module writes the task attributable residuals into the candidate fingerprint version and uses the candidate fingerprint version to re-deduce the resource evolution trajectory of the most recently completed tasks. The candidate fingerprint version replaces the original fingerprint version only when the stage deviation between the replayed trajectory and the actual execution trajectory is less than the original fingerprint version and no new continuous congestion misjudgment is added. Otherwise, the original fingerprint version is retained and the corresponding residual is transferred to the set of residuals to be reviewed for joint attribution of subsequent execution trajectories of the same type of task.

[0015] Preferably, the scheduling closed-loop correction module receives the actual resource status of each computing node according to the stage segment during the execution of the batch scheduling sequence, and continuously aligns the actual resource status with the corresponding resource evolution trajectory. When the queue fallback time of any computing node is shifted backward and the time-debt of unstarted tasks continues to increase, the scheduling closed-loop correction module freezes the mapping of task nodes that have not yet entered the data access stage and have low time-debt, keeps the tasks that have entered the content parsing stage from continuing to execute, and re-adds the frozen tasks to the candidate node inference module according to the updated time-debt. The candidate node deduction module regenerates the resource evolution trajectory using the remaining stages of the unfinished tasks as the initial stage, and forms the subsequent batch scheduling sequence based on the updated inter-task load interference matrix and the insertable execution interval.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. By dividing the historical execution trajectory of scientific and technological information collection tasks into stages, a load causal fingerprint is established, including resource consumption increments, resource release lags, queue propagation relationships, and concurrent coupling relationships. Before task allocation, the tasks to be scheduled are mapped to candidate computing nodes, generating corresponding resource evolution trajectories. The processor usage, memory residency, input / output waiting time, and queue changes of candidate nodes are continuously deduced from the task stage connection relationships. Mappings in the node recovery trajectory that do not have queue fallback segments, resource release times exceeding the task window, or multiple resource sequences continuously increasing are excluded. At the same time, the system forms information timeliness debt based on the historical update interval of the information source, the most recent successful collection time, content change density, consecutive failure states, and downstream dependencies. The task node mapping is determined from the remaining candidate mappings by combining the load propagation cost and the timeliness debt reduction. Thus, allocation results where nodes are currently idle but may cause continuous congestion after task entry will not enter the execution sequence, and collection tasks with rapidly increasing waiting times and blocking downstream tasks will not be delayed for a long time due to high resource consumption. This ensures that node recoverability and the timeliness of scientific and technological information are coordinated in the same scheduling process.

[0017] 2. After task execution, the system aligns the actual resource state with the pre-generated resource evolution trajectory according to the execution stage. It identifies residuals caused by the task itself, those caused by concurrent task contention, and those caused by sudden changes in data scale. Only residuals with corresponding attribution relationships are used to correct direct load paths, coupled load paths, and propagated load paths. Historical task trajectories are replayed using candidate fingerprint versions, and version replacement is completed when stage deviations decrease and no new persistent congestion misjudgments occur, reducing the interference of occasional resource fluctuations on subsequent scheduling models. For multiple tasks to be scheduled within the same scheduling cycle, the system continuously corrects subsequent candidate mappings using the inter-task load interference matrix and determines the insertable execution interval based on the existing task release time of nodes. During execution, when the queue fallback time shifts backward, tasks that have not yet entered the data access stage and have low time debt are re-participated in candidate node deduction, while tasks that have entered the content parsing stage continue to execute continuously. This processing allows the resource contention relationship between batch tasks to update with the actual state and restricts scheduling adjustments to tasks that have not yet reached an intermediate processing state, reducing duplicate execution, invalid migrations, and secondary congestion. Attached Figure Description

[0018] Figure 1 A flowchart of the load-aware closed-loop scheduling process for scientific and technological information collection tasks; Figure 2 A graph showing the growth curve of debt related to information timeliness; Figure 3 A diagram showing the allocation of overlapping debt segments; Figure 4A flowchart illustrating the process of constructing causal fingerprints for load and extrapolating the evolution of candidate node resources; Figure 5 Distribution of candidate mapping persistent congestion determinations; Figure 6 Stacked diagram of contribution to residual decomposition. Detailed Implementation

[0019] In one embodiment, a technology information collection and scheduling system based on computing node load awareness includes a task state construction module, a load causal fingerprint generation module, an information timeliness debt generation module, a candidate node inference module, and a scheduling closed-loop correction module. The task state construction module records the execution trajectory of technology information collection tasks in stages. The load causal fingerprint generation module establishes the resource consumption increment, resource release lag, and queue propagation relationship after a task enters a computing node based on the execution trajectory. The information timeliness debt generation module forms timeliness debt based on the information source update status and task waiting status. The candidate node inference module maps the tasks to be scheduled to candidate computing nodes. The current task combination and load causal fingerprint of the merged node generate the resource evolution trajectory, delete the persistent congestion mapping, and determine the task node mapping based on the load propagation cost and time debt reduction of the remaining mapping. The scheduling closed-loop correction module updates the load causal fingerprint based on the attributable deviation between the actual execution trajectory and the resource evolution trajectory. This forms a closed-loop processing chain that connects task generation, state encoding, candidate mapping deduction, task node matching, execution trajectory recycling, and fingerprint version correction. The entire processing chain is implemented through data interaction between software processes, task queues, shared state storage, and node execution agents, without relying on the modification of the physical structure of the computing nodes.

[0020] In this embodiment, please refer to Figure 1 The task status construction module generates a task status record when a data acquisition task enters the scheduling queue. The task status record includes at least the task identifier, information source identifier, task type, data range, acquisition cursor, parsing rule version, preceding task identifier, succeeding task identifier, generation time, most recently successful acquisition time, and current waiting time. After the task enters the computing node, the node execution agent writes the status changes at the following points: data access start, first batch content return, content parsing start, duplicate comparison start, association extraction start, result writing start, and task end. Stage segments are formed between adjacent stage boundaries. Within the stage segments, processor time slice usage, memory resident changes, input / output waiting time, queue length changes, cache access results, and resource reclamation delay are continuously summarized. The task status construction module corrects the above records with a unified time base, associates node-level resource records with task-level stage records by time interval to avoid incorrect stage attribution due to clock offsets of different nodes, and marks shared resource changes that cannot be uniquely attributed to a task as pending attribution records instead of directly writing them into the task's own trajectory.

[0021] Specifically, the load causal fingerprint generation module does not describe tasks with a single average load value. Instead, it connects the phase segments of the same type of task under different data scale ranges and different concurrency combinations into a directed load path. Each phase segment corresponds to a resource state vector, and each phase connection corresponds to the change relationship from the previous resource state vector to the next resource state vector. The direct load path records the changes in processor usage, memory residency, and input / output waiting caused by the execution of the task itself. The coupled load path records the changes in resource competition caused when the task shares parsing cache, network connection pool, index write channel, or storage queue with other tasks on the same node. The propagation load path records the time-series propagation relationship where the delay of the previous phase causes the start of the next phase to be delayed, and further causes queue accumulation. The load causal fingerprint saves multiple path branches with task type, data scale range, node execution environment, and concurrent task combination as retrieval keys. When deducing candidate nodes, the retrieval key is matched first, and then the starting phase is selected according to the current task state, so that the same task can call different load change paths in different node environments.

[0022] The resource state vector is constructed according to the following formula: ; in, This represents the resource state vector for task type a within stage segment b. This indicates the amount of processor time slices allocated to this stage segment. This represents the increase in memory resident time at the end of the phase relative to the beginning of the phase. This indicates the cumulative amount of input / output waiting time. This indicates the change in the length of the task queue. Indicates the cache hit status code. The vector represents the time difference required for resources to fall back to the level at the beginning of the phase after the phase ends. All quantities in the vector are represented using standardized values ​​within the same sampling window, and the symbols maintain the same meaning throughout the text.

[0023] In a specific processing step of this embodiment, a document metadata acquisition task includes four stages: data access, content parsing, duplicate comparison, and result writing. The data access stage has a high cumulative input / output waiting time but a low memory resident increment. The content parsing stage has a synchronous increase in processor time slice usage and memory resident increment. The duplicate comparison stage requires access to the summary index shared by the same node. The result writing stage may compete with other tasks for the storage queue. The task state construction module records the resource state vectors of the four stages. The load causal fingerprint generation module establishes stage connections based on stage boundaries and records the queue growth that occurs in the duplicate comparison stage and overlaps with other summary comparison tasks as the coupled load path. The record of the result writing stage that is shifted due to the previous parsing delay is written into the propagation load path. When subsequent similar tasks enter the candidate node for deduction, the node load is no longer estimated solely based on the historical average processing time. Instead, the corresponding path branch is selected according to whether there is a summary comparison task on the current candidate node, whether the storage queue is in the release stage, and the data size range of the task itself.

[0024] In one embodiment, the information timeliness debt generation module establishes a timeliness state sequence corresponding to the scientific and technological information source for each scheduled task. The timeliness state sequence includes the information source's historical update interval, the most recent successful acquisition time, historical content change density, the number of consecutive acquisition failures, the version differences of the acquired information, and the downstream task dependencies. The information timeliness debt generation module determines the basic debt segment based on the relative position of the current waiting time in the historical update interval, corrects the accumulation rate of the basic debt segment based on the historical content change density and the number of consecutive acquisition failures, and decomposes the task debt into ontology debt and dependency blocking debt based on the downstream task dependencies, forming a timeliness debt sequence that is updated with the waiting state. The candidate node inference module calls this timeliness debt sequence when comparing candidate mappings to distinguish acquisition tasks with different content change probabilities and different dependency blocking ranges under the same waiting time.

[0025] Information timeliness debt is calculated according to the following formula: ; in, This represents the information time-delay debt corresponding to the task u to be scheduled at time t. Indicates the most recent successful data acquisition time. Indicates the historical update interval of the corresponding information source. Indicates the basic debt coefficient. This represents a non-linear cumulative index determined by the density of historical content changes. This represents the normalized value indicating the number of consecutive failed data acquisitions. Indicates the failure correction factor. This indicates the amount of difference between the versions of the collected information. Indicates the version correction factor. This represents the dependency blocking amount obtained by aggregating downstream dependencies. Represents the blocking correction coefficient, when Less than At that time, the debt was in a slow accumulation phase, when Reaching or exceeding Time through Control the curvature of the cumulative curve.

[0026] Please refer to Figure 2 , Figure 2 The horizontal axis represents task waiting time in hours, with examples ranging from 0 to 12; the vertical axis represents information timeliness debt, with values ​​increasing from approximately 0.42 to 1.42. The curve illustrates the debt accumulation process for the same technology information collection task under different waiting times. Figure 2 The vertical reference lines are used to identify the historical acceptable data collection intervals for the task, while the horizontal reference lines are used to help observe whether the debt has entered the interval requiring priority scheduling. The curves are represented by continuous broken lines to facilitate observation of the impact of waiting time changes on task priority.

[0027] from Figure 2 As can be seen, the curve already has a basic debt of approximately 0.42 in the initial waiting period, indicating that even before a long waiting period occurs, this task is affected by the importance of the information source, the frequency of collection, information volatility, and business bias. As the waiting time increases, the debt curve shows a continuous upward trend, reaching approximately 1.42 when it approaches the historically acceptable collection interval, indicating that the task has gradually transformed from a routine collection requirement into a time-sensitive task that requires priority processing by the scheduler.

[0028] For example, the historical update interval of a certain technology project's public information source is 24 time units, the most recent successful collection time is 30 time units from the current time, the nonlinear cumulative exponent corresponding to the historical content change density is greater than 1, and the task fails twice in a row, blocking subsequent related extraction tasks. The information timeliness debt generation module maps the part of the waiting time exceeding the historical update interval to the accelerated accumulation interval, and adds the failure record and dependency blocking amount to the basic debt. Another task with a waiting time of 30 time units but a historical update interval of 72 time units and no downstream blocking is still in the low-speed accumulation interval. The candidate node inference module can thus distinguish the scheduling urgency of the two tasks without using fixed manual priority. Moreover, when the blocked subsequent task completes the replacement data, the dependency blocking amount will be deducted synchronously, so that the debt status is updated with the task dependency graph.

[0029] In a preferred embodiment, the information timeliness debt generation module represents the downstream task dependency relationship as a task dependency graph composed of collection task nodes and dependency edges. The dependency edge records the version identifier of the preceding collection results being parsed, associated, extracted, or indexed into the task call by the subsequent task. When multiple tasks to be scheduled block the same subsequent task, the information timeliness debt generation module divides the dependency blocking debt into mutually exclusive debt segments and overlapping debt segments according to the data range not yet provided by each preceding task. Only one debt record is retained for the overlapping debt segment, and a debt attribution relationship is established according to the data coverage ratio of each preceding task. After any preceding task completes the mapping, the candidate node inference module deducts the corresponding debt segment from the task dependency graph according to the data range released by the completed task, and synchronously updates the timeliness debt sequence of the remaining tasks to be scheduled, so as to avoid the same subsequent processing range being repeatedly included in the debt by multiple preceding tasks.

[0030] The attribution of overlapping debt segments is determined according to the following formula: ; in, This represents the attribution amount that the task u to be scheduled assumes for the overlapping debt segment j. This indicates the amount of data coverage that task u has not yet provided in debt fragment j. This indicates the amount of data coverage that the preceding task x has not yet provided in debt fragment j. This represents the set of preceding tasks that jointly influence debt segment j. Let represent the number of unique records of debt segment j in the task dependency graph. The denominator represents the sum of the unprovided coverage of the same debt segment by all related preceding tasks. When a task completes data collection and provides the corresponding data, the task's ______ becomes ______. The remaining amount becomes 0, and the remaining amount is redistributed based on the unfinished tasks.

[0031] Please refer to Figure 3 , Figure 3 The horizontal axis represents the overlapping segment number, ranging from 1 to 10, while the vertical axis represents the amount of debt or the remaining amount of debt to be allocated. The bars represent the debt assigned to a single acquisition task for the corresponding overlapping segment, and the line represents the remaining block debt or total debt reference amount for that segment after participating in allocation. Both types of graphs are displayed in the same coordinate system to simultaneously observe the segment-level attribution results and the allocation pressure of the overall debt block. The legend distinguishes the meanings of the bars and the line.

[0032] from Figure 3As can be seen, fragment number 6 has a remaining block debt of approximately 2.189 and an attribution debt of approximately 1.313, making it the fragment with the highest overlapping debt in the current example. Fragment number 5 has a remaining block debt of approximately 2.066 and an attribution debt of approximately 1.240, which is also at a relatively high level. Fragment number 2 has an attribution debt of approximately 1.196, indicating that although this fragment may originate from a duplicate entry or similar information block, it still has high timeliness processing value. The attribution debts of fragment numbers 1, 3, and 4 are relatively low, indicating that their urgency in the current task set is weaker than that of the aforementioned fragments.

[0033] Figure 3 This demonstrates the engineering significance of overlapping debt allocation. In scientific and technological information collection, it's common for multiple tasks to simultaneously access the same site, the same section, the same announcement list, or the same text fragment. If the system only calculates debt for each task individually, it easily leads to duplicate fetching and parsing. By allocating overlapping fragments by their affiliation, the scheduler can identify which fragments should be handled by higher-priority or more available tasks, and which fragments can reduce duplicate execution through caching, result sharing, or delayed merging.

[0034] Furthermore, the data range in the task dependency graph is not represented by an abstract task name, but is described using a range key composed of information source partitions, time windows, topic ranges, pagination intervals, and version identifiers. After the preceding task is completed, the system performs an intersection operation between the actual written information range and the range key in the dependency edge, releasing only the data fragments that have formed usable results. If the task only completes part of the pagination or only some fields pass the parsing verification, only the debt fragments of the corresponding range are deducted, and the incomplete range remains in the dependency graph. Before the subsequent task starts, it will also check whether the required version identifier is consistent with the version of the preceding result. If the versions are inconsistent, the preceding task is not marked as completely unblocked, but the debt fragments corresponding to the compatible fields are deducted and the debt fragments corresponding to the incompatible fields are retained, thereby keeping the information timeliness debt consistent with the actual consumable data range.

[0035] In one embodiment, please refer to Figure 4The candidate node inference module copies the current task combination stage state of each candidate computing node and embeds the initial stage of the task to be scheduled into this stage state. According to the stage connection relationship in the load causal fingerprint, it generates a prediction sequence of processor usage, memory residency, input / output wait and task queue length stage by stage. The candidate node inference module constructs the node recovery trajectory based on the continuous growth segment, resource release segment and queue fallback segment of each resource prediction sequence. Candidate mappings that do not have a queue fallback segment, whose resource release time is later than the predetermined task window or whose multiple resource prediction sequences continue to grow are marked as continuous congestion mappings. After the candidate node inference module deletes the continuous congestion mappings from the candidate mapping set, it forms the remaining mapping set for task node mapping determination. The whole process is carried out on the candidate state copy in memory and does not change the actual task queue of the computing node.

[0036] The persistent congestion determination of candidate mappings is calculated according to the following formula: ; in, Let W represent the congestion determination after the scheduled task u is mapped to the candidate computing node n, and let W represent the time window covered by this simulation. Indicates time Compared to the queue increment before mapping, Indicates time Compared to the memory residency increment before mapping, This indicates the recovery time required for the predicted resource state to fall back to the pre-mapping available range. This represents the number of resource sequences that maintain a monotonically increasing trend within the simulation window. to These represent the scale conversion coefficients corresponding to the four types of decision quantities. The identification of the continuous congestion mapping also requires that the queue does not have a fallback segment or the recovery time exceeds at least one of the structural conditions in the task window.

[0037] Please refer to Figure 5 , Figure 5 The horizontal axis represents the candidate mapping number, ranging from 1 to 24, and the vertical axis represents the persistent congestion determination. The height of the bars represents the congestion risk score of the corresponding candidate mapping within the observation window, and the horizontal dashed line represents the persistent congestion determination threshold. The legend explains the meaning of the bars and the threshold line. Figure 5 The use of a bar chart to display the data is to highlight the risk differences between different candidate mappings and to facilitate the scheduler in quickly eliminating high-risk mappings before formal assignment.

[0038] from Figure 5As can be seen, the highest candidate mapping risk scores in the current example appear near serial numbers 9 and 24, both around 0.302; scores near serial numbers 4 and 19 are around 0.300, also relatively high. Although the risk levels of these mappings are higher than other candidates, they are still significantly lower than the sustained congestion threshold of 0.72, indicating that the current batch has not yet formed a sustained congestion state requiring mandatory avoidance. The risk score near serial number 8 is around 0.242, at the second highest level, and can be considered as a key observation target for subsequent load changes.

[0039] The overall distribution of the bars shows multiple local high points, indicating that the risk of candidate mappings is not determined by a single node, but is jointly influenced by queue growth, decreasing resource availability, recovery delay, historical fluctuations, and load trends within the candidate window. Even if some mappings currently have sufficient resource availability, if their corresponding nodes experience continuous queue growth or input / output waiting within a short period, a high number of persistent congestion judgments can still be formed. Figure 5 By exposing this trend risk in advance, the system can complete scheduling adjustments before actual congestion occurs.

[0040] In this embodiment, the candidate node inference module reads the current stage, the estimated remaining amount of the stage, and the resource release order of the tasks being executed on the candidate nodes, and inserts the starting stage of the task to be scheduled into the current timeline. If the candidate node already has a result written task occupying the storage queue, and the task to be scheduled is expected to enter the result writing stage at a similar time, the coupled load path will cause the queue increment and recovery time to expand synchronously. If another candidate node currently has a high processor usage but its existing task will end before the task to be scheduled enters the content parsing stage, then a clear resource release segment and queue fallback segment will appear in the resource evolution trajectory. The former candidate node may be marked as a continuous congestion mapping, and the latter candidate node will enter the remaining mapping set, thereby avoiding allocation based solely on the node load at the inference starting point.

[0041] In a preferred embodiment, the candidate node deduction module extracts the load propagation cost, node recovery time, time-limited debt reduction, and dependency blocking debt release for the remaining mapping set. It constructs a resource constraint dimension based on the load propagation cost and node recovery time, and a time-limited benefit dimension based on the time-limited debt reduction and dependency blocking debt release. The candidate node deduction module first deletes candidate mappings that are simultaneously dominated by another mapping scheme in both dimensions. Then, it determines the insertable execution interval based on the resource prediction sequence corresponding to each candidate mapping and the release time of the node's existing tasks. Within the insertable execution interval, it determines the task start order according to the growth order of the time-limited debt of each task. When multiple candidate mappings have the same start order, the task node mapping with a lower degree of overlap between the load propagation path and the node's existing task load path is selected. The dominance judgment adopts a dimension-by-dimensional comparison instead of directly merging the resource constraint dimension and the time-limited benefit dimension into a single score.

[0042] Specifically, if the load propagation cost of candidate mapping A is not higher than that of candidate mapping B, the node recovery time is not longer than that of candidate mapping B, the amount of time-delayed debt reduction is not lower than that of candidate mapping B, and the amount of dependency blocking debt release is not lower than that of candidate mapping B, and at least one of the comparison quantities is better than that of candidate mapping B, then candidate mapping B is deleted. If the remaining candidate mappings still cannot determine a unique result through a one-dimensional relationship, the candidate node inference module compares the subsequent growth slope of the time-delayed debt of the task within the insertable execution interval formed by the release of node resources, and checks whether the high-occupancy phase of the task to be scheduled overlaps with the high-occupancy phase of the existing task of the node. If the time-delayed debt growth order is the same, the mapping with less phase overlap is selected first. If the phase overlap is still the same, the execution order is determined according to the task generation time to ensure that a definite scheduling result is obtained under the same input state.

[0043] In one embodiment, the candidate node inference module establishes an inter-task load interference matrix for multiple tasks to be scheduled within the same scheduling period. The elements of the inter-task load interference matrix are generated by the shared resource occupancy increment formed by the resource evolution trajectories of two tasks during the stage overlap. The candidate node inference module establishes a debt growth order based on the time-sensitive debt of each task to be scheduled, and adds the task node mapping one by one according to the debt growth order. Each time a task node mapping is added, the inter-task load interference matrix is ​​used to correct the resource evolution trajectory of the remaining candidate mappings. When the corrected resource evolution trajectory is marked as a persistent congestion mapping, the most recently added task node mapping with a small amount of dependent blocking debt release is withdrawn, and the corresponding task is moved into the next insertable execution interval until a batch scheduling sequence without persistent congestion mapping is formed.

[0044] The elements of the inter-task load interference matrix are generated according to the following formula: ; in, This represents the load interference when both scheduled task x and scheduled task y are mapped to the same computing node. This represents the set of phase combinations where two tasks have overlapping phases. This indicates the processor shared occupancy increment within stage combination b. This indicates the increase in memory co-residence within phase combination b. This indicates that the input / output increments within stage combination b are pending. Indicates stage combination The incremental growth of the queue within the queue. , , and These represent the scaling factors for the four types of resource quantities. If there is no phase overlap between two tasks, then the corresponding... It is an empty set and Take 0.

[0045] In this embodiment, the batch scheduling sequence does not directly superimpose the optimal mappings of each task. Instead, after each task is accepted, the overlapping range of stages of other tasks on the same node is recalculated. For example, when task A and task B are individually mapped to node C, there is a queue fallback segment. However, when the data access stage and result writing stage of the two tasks overlap, the input / output waiting increment and the storage queue growth increment will be written into the corrected resource evolution trajectory through the load interference matrix. If the corrected node C has a continuous congestion structure, the system compares the dependency blocking debt release amount of the most recently added task, moves the task with the smaller release amount into the next insertable execution interval, and retains the mapping of the other task. When re-deducing subsequent tasks, the latest stage time axis formed by the accepted tasks is used until the predicted trajectory of all nodes in this batch contains identifiable resource release segments and queue fallback segments.

[0046] In one embodiment, the scheduling closed-loop correction module performs time alignment between the actual execution trajectory and the resource evolution trajectory according to the stage segments, generating processor occupancy residuals, memory resident residuals, input / output waiting residuals, and queue change residuals. The scheduling closed-loop correction module uses the stage start and end records of concurrent tasks on the same computing node to identify competition residuals that change synchronously with the state of concurrent tasks, uses the difference between the current data size and the corresponding historical data size to identify scale mutation residuals, and marks the remaining residuals after deducting competition residuals and scale mutation residuals as task-attributable residuals. The scheduling closed-loop correction module only corrects the direct load path of the corresponding task type based on the task-attributable residuals, and corrects the coupled load path based on the competition residuals, keeping the propagation load path and stage connection relationship updated separately, avoiding mixing deviations of different causes into the same fingerprint parameter.

[0047] Residual decomposition is performed according to the following formula: ; in, This represents the total residual vector for task type a within stage segment b. Represents the actual resource state vector. This represents the resource state vector obtained through deduction. This represents the attributable residuals caused by changes in the task's own load path. This represents the competitive residual resulting from competition among concurrent tasks. This represents the scale mutation residual caused by the deviation of the current data collection scale from the historical scale range. The competing residual is identified by the change segments synchronized with the start and end of the concurrent task phase. The scale mutation residual is estimated by the difference in data volume and the unit data resource consumption record. The remaining part is written into the attributable residual.

[0048] Please refer to Figure 6 , Figure 6 The horizontal axis represents resource dimensions, including processors, memory, input / output, and queues; the vertical axis represents residual contribution. Each bar consists of three stacked parts, representing its own resource fluctuation contribution, task contention contribution, and scale disturbance contribution, respectively. The legend distinguishes the three types of contribution sources. The total height of the stacked bars represents the overall deviation level of that resource dimension in the current batch load forecast, used to determine the resource directions that the scheduling model needs to focus on correcting.

[0049] from Figure 6 As can be seen, the memory dimension has the highest total residual contribution, approximately 0.0427, of which its own resource fluctuations contribute approximately 0.0214, task contention contributes approximately 0.0145, and scale disturbances contribute approximately 0.0068. This indicates that the memory deviation in the current batch not only comes from the node's own load fluctuations but is also significantly affected by concurrent task contention. The total residual contribution of the queue dimension is approximately 0.0215, the processor dimension is approximately 0.0176, and the input / output dimension is approximately 0.0134, indicating that the contributions of different resource dimensions to prediction deviation are not balanced.

[0050] The stacked structure across various dimensions shows that fluctuations in internal resources are usually the main source of residuals, but the contribution of task competition in memory and queue dimensions is not negligible. For scientific and technological information collection tasks, text extraction, structured parsing, vectorized caching, deduplication indexing, and result writing will create interleaved loads on different nodes. When multiple tasks share node resources within similar time slices, the prediction model may underestimate the instantaneous pressure brought by competition. Figure 6 Decomposing this bias into explainable sources of contribution facilitates subsequent model correction.

[0051] For example, if the actual memory resident amount of a task during the content parsing phase is higher than the projected value, and another task enters the large-volume text parsing phase at the same time, and the two tasks share the parsing cache, then the scheduling closed-loop correction module will include the memory changes synchronized with the start and end of the other task's phase into the contention residual. If the amount of text collected in this instance exceeds the corresponding data size range, then the scale mutation residual will be estimated based on the historical memory increment per unit of text. Only the remaining memory residual after deducting the two parts will be used to correct the direct load path. If the same type of task in another execution batch still has a residual in the same direction when there are no concurrent parsing tasks and the data size falls within the original range, then the residual will continue to accumulate in the candidate correction record of the task's own path, without modifying the coupled load path related to the storage queue.

[0052] In a preferred embodiment, the scheduling closed-loop correction module maintains an updated credible state for each direct load path, coupled load path, and propagation load path, consisting of the number of samples, the consistency of residual direction, and the coverage of the node environment. When the task attributable residuals continuously point to the same resource change direction and the corresponding node environment coverage meets the update conditions, the scheduling closed-loop correction module writes the task attributable residuals into the candidate fingerprint version and uses the candidate fingerprint version to re-examine the resource evolution trajectory of the most recently completed tasks. The candidate fingerprint version replaces the original fingerprint version only when the stage deviation between the re-examined trajectory and the actual execution trajectory is less than the original fingerprint version and no new continuous congestion misjudgment is added. Otherwise, the original fingerprint version is retained and the corresponding residuals are transferred to the set of residuals to be reviewed for joint attribution of the execution trajectories of subsequent tasks of the same type.

[0053] Furthermore, the number of samples in the updated trusted state is accumulated according to the number of trajectories that have been attributed within the same task type, data size range, and node execution environment. The consistency of the residual direction is recorded according to the proportion of residuals with the same sign in each resource dimension. The node environment coverage is encoded according to whether the candidate correction record comes from different load levels, different concurrency combinations, and different cache states. Only when the candidate correction record covers both low-load and medium-load environments or covers at least two types of concurrency combinations is it allowed to generate a candidate fingerprint version. The replay process reads the node state snapshot of the completed task at its actual scheduling time, replaces the original fingerprint with the candidate fingerprint version, and re-executes the resource evolution deduction. The queue fallback position, resource release position, and deviation of each resource state vector are compared stage by stage. If the candidate version causes some tasks to generate a continuous congestion judgment that did not exist before, the candidate version will not enter the online retrieval index.

[0054] In one embodiment, the scheduling closed-loop correction module receives the actual resource status of each computing node according to the stage segment during the execution of the batch scheduling sequence, and continuously aligns the actual resource status with the corresponding resource evolution trajectory. When the queue fallback time of any computing node is delayed and the time debt of unstarted tasks continues to increase, the scheduling closed-loop correction module freezes the mapping of task nodes that have not yet entered the data access stage and have low time debt, keeps the tasks that have entered the content parsing stage running, and adds the frozen tasks back to the candidate node inference module according to the updated time debt. The candidate node inference module regenerates the resource evolution trajectory with the remaining stage of the unfinished tasks as the initial stage, and forms the subsequent batch scheduling sequence based on the updated inter-task load interference matrix and the insertable execution interval.

[0055] Specifically, before a task is about to enter the data access phase, the node execution agent requests a start permission. The scheduling closed-loop correction module determines whether the original mapping is still in the insertable execution interval based on the latest resource status. If the original mapping has lost the insertable condition due to the release and relocation of the preceding task, the task remains in an unstarted state and is marked as a remapped task. Tasks that have received data and entered the content parsing phase do not undergo forced migration. The system only writes the remaining phases into the resource evolution trajectory of the current node and re-deduces the time-delay of tasks that have not yet started. If the time-delay debt of an unstarted task enters a high growth range during the waiting period, its start position will be advanced accordingly during reordering. If the node resource status recovers and a new queue fallback segment is formed, the frozen task can be mapped back to the original node. The entire online reordering process does not change the result records of completed tasks and the intermediate states of tasks that have entered the parsing phase.

[0056] In a preferred embodiment, for new task types lacking historical execution trajectories, the task state construction module first divides the new task into stage segments that can match existing task stages based on the data access method, parsing rule category, repeated comparison method, and result writing method included in the task. Then, it selects path segments with the same stage processing method from the existing load causal fingerprint set to construct temporary load causal fingerprints. The candidate node inference module adds uncertainty intervals to the temporary load causal fingerprints and expands the prediction range of resource recovery time. Only new tasks are allowed to enter candidate nodes with continuous idle intervals and no highly interfering tasks. After the new task is completed, the scheduling closed-loop correction module establishes a formal fingerprint branch using the actual execution trajectory as an independent sample. The temporary fingerprint is replaced after subsequent tasks of the same type meet the requirements for updating the trusted state, thus avoiding the direct use of a single average value for scheduling due to insufficient historical samples.

[0057] In a preferred embodiment, when the scientific and technological information source returns empty content, access times out, page structure changes, or parsing rules cannot be matched, the task status construction module still records the data access stage and abnormal termination position, and writes the exception type into the execution trajectory. The information timeliness debt generation module does not update the most recent successful collection time for tasks that have not formed usable results, but only adjusts the debt accumulation status based on the number of consecutive collection failures. The load causal fingerprint generation module records the resource status changes that have occurred before the abnormal termination to the path branch corresponding to the exception type, and does not fill in the missing subsequent stages as zero load. When the candidate node inference module reschedules the same task, it calls the corresponding abnormal branch to estimate the resource consumption of the retry process, and decides whether to continue using the abnormal branch or return to the normal path branch based on whether the parsing rule version has changed.

[0058] In a preferred embodiment, task status records, stage segment records, load causal fingerprints, time-delay debt sequences, task dependency graphs, candidate mapping records, and actual execution trajectories are stored using data objects with version identifiers. At the start of the scheduling cycle, a consistent snapshot is generated and the snapshot version is recorded. The candidate node inference module only reads the task status and node status in the snapshot within the same scheduling cycle. After the batch scheduling sequence is generated, it is written to the task node mapping through atomic commit. If the node status version changes before the commit, the unexecuted mapping is abandoned and the status snapshot is reread. The mappings that have entered the execution stage remain unchanged. When the scheduling closed-loop correction module receives the actual trajectory, it associates it according to the task identifier, node identifier, stage boundary, and fingerprint version, so that each residual attribution can be traced back to the fingerprint version that generated the corresponding resource evolution trajectory.

[0059] Furthermore, the load causal fingerprint adopts a separate storage method for stage graphs and path parameter tables. The stage graph stores the allowed connections between task stages and the abnormal branches that can be entered in each stage. The path parameter table stores the resource state vectors and resource release time differences under different task types, data size ranges, node execution environments, and concurrency combinations. The information timeliness debt sequence is stored using time state objects indexed by task identifiers. The task dependency graph uses range key indexes for dependency edges. When candidate nodes are inferred, only records that match the range key, task type, and candidate node environment of the task to be scheduled are read. After scheduling, the deleted persistent congestion mappings and their judgment criteria are saved. When the actual execution trajectory shows that the node can recover within the corresponding window, the record is entered into the set of residuals to be reviewed without directly changing the online fingerprint, thereby maintaining the boundary between online scheduling data and model correction data.

[0060] In this embodiment, the entire scheduling process is based on the phased execution trajectory formed by the task state construction module. The load causal fingerprint describes the resource change path after the task enters the node. The information timeliness debt describes the version lag and dependency blocking state when the scientific and technological information continues to wait. The candidate node inference module constructs the multi-node resource evolution trajectory in the state replica and excludes the continuous congestion mapping. The scheduling closed-loop correction module completes residual attribution, fingerprint version replay and online batch reordering based on the actual execution results. The task node mapping only enters the execution queue when there is a resource release segment and a queue fallback segment in the node recovery trajectory, the task timeliness debt can be reduced within the insertable execution interval and the load interference between batch tasks does not form a new continuous congestion structure. This ensures that the node selection, execution order and model correction of the scientific and technological information collection task are all connected by the same set of traceable state records.

Claims

1. A scientific and technological information collection and scheduling system based on computing node load awareness, characterized in that, It includes a task status construction module, a load causal fingerprint generation module, an information timeliness debt generation module, a candidate node inference module, and a scheduling closed-loop correction module; The task status construction module records the execution trajectory of the scientific and technological information collection task in stages; The load causal fingerprint generation module establishes the resource usage increment, resource release lag, and queue propagation relationship after the task enters the computing node based on the execution trajectory. The information time-sensitive debt generation module generates time-sensitive debts based on the information source update status and task waiting status. The candidate node inference module maps the tasks to be scheduled to candidate computing nodes, generates resource evolution trajectories by combining the current task combination of the node with the load causal fingerprint, deletes persistent congestion mappings, and determines the task node mapping based on the load propagation cost and time-sensitivity debt reduction of the remaining mappings. The scheduling closed-loop correction module updates the load causal fingerprint based on the attributable deviation between the actual execution trajectory and the resource evolution trajectory.

2. The scientific and technological information collection and scheduling system based on computing node load awareness according to claim 1, characterized in that, The load causal fingerprint generation module divides the execution trajectory of each scientific and technological information collection task into multiple stage segments and establishes resource state transition records for adjacent stage segments. The resource state transition record includes at least the processor usage increment, memory resident increment, input / output wait increment, task queue length change, cache hit status change, and resource release time difference; The load causal fingerprint generation module extracts the direct load path triggered by the task itself, the coupled load path triggered by shared resource contention, and the propagation load path triggered by delays in preceding stages based on the resource state transition records of the same type of task under different concurrent task combinations. It then establishes a set of load causal fingerprints with stage connection relationships according to task type, data size range, and node execution environment for the candidate node inference module to call.

3. The scientific and technological information collection and scheduling system based on computing node load awareness according to claim 1, characterized in that, The information timeliness debt generation module establishes a timeliness status sequence corresponding to the scientific and technological information source for each task to be scheduled. The timeliness status sequence includes the information source’s historical update interval, the most recent successful collection time, the density of historical content changes, the number of consecutive collection failures, the version differences of the collected information, and the downstream task dependencies. The information timeliness debt generation module determines the basic debt segment based on the relative position of the current waiting time in the historical update interval, adjusts the accumulation rate of the basic debt segment based on the density of historical content changes and the number of consecutive collection failures, and decomposes the task debt into ontology debt and dependency blocking debt according to the downstream task dependency relationship, forming a timeliness debt sequence that is updated with the waiting state, which is called by the candidate node inference module when comparing candidate mappings.

4. The scientific and technological information collection and scheduling system based on computing node load awareness according to claim 1, characterized in that, The candidate node inference module replicates the current task combination stage state for each candidate computing node, embeds the initial stage of the task to be scheduled into the stage state, and generates a prediction sequence of processor usage, memory residency, input / output wait and task queue length stage by stage according to the stage connection relationship in the load causal fingerprint. The candidate node inference module constructs a node recovery trajectory based on the continuous growth segment, resource release segment, and queue fallback segment of each resource prediction sequence, and marks candidate maps that do not have a queue fallback segment, whose resource release time is later than the predetermined task window, or whose multiple resource prediction sequences continue to grow simultaneously as continuous congestion maps. After the candidate node deduction module removes the persistent congestion mapping from the candidate mapping set, it forms a remaining mapping set for task node mapping determination.

5. The scientific and technological information collection and scheduling system based on computing node load awareness according to claim 2, characterized in that, The scheduling closed-loop correction module performs time alignment between the actual execution trajectory and the resource evolution trajectory according to the stage segments, and generates processor occupancy residual, memory resident residual, input / output waiting residual and queue change residual; The scheduling closed-loop correction module uses the start and end records of concurrent tasks on the same computing node to identify the competing residuals that change synchronously with the state of concurrent tasks, uses the difference between the current data size and the corresponding historical data size to identify the scale mutation residuals, and marks the remaining residuals after deducting the competing residuals and the scale mutation residuals as task-attributable residuals. The scheduling closed-loop correction module corrects the direct load path of the corresponding task type only based on the task attributable residual, and corrects the coupled load path based on the contention residual, while keeping the propagation load path and the stage connection relationship updated respectively.

6. The scientific and technological information collection and scheduling system based on computing node load awareness according to claim 3, characterized in that, The information timeliness debt generation module represents the downstream task dependency relationship as a task dependency graph composed of collection task nodes and dependency edges, and records the version identifier of the preceding collection results being parsed, associated, extracted, or indexed into the subsequent task call in the dependency edge; When multiple tasks to be scheduled block the same subsequent task, the information timeliness debt generation module divides the dependent blocking debt into mutually exclusive debt segments and overlapping debt segments according to the data range that each preceding task has not yet provided. Only one debt record is retained for the overlapping debt segment, and a debt ownership relationship is established according to the data coverage ratio of each preceding task. After any preceding task completes mapping, the candidate node deduction module deducts the corresponding debt fragment from the task dependency graph based on the data range released by the completed task, and synchronously updates the time-debt sequence of the remaining tasks to be scheduled.

7. The scientific and technological information collection and scheduling system based on computing node load awareness according to claim 4, characterized in that, The candidate node deduction module extracts the load propagation cost, node recovery time, time-limited debt reduction, and dependency blocking debt release for the remaining mapping set. It constructs a resource constraint dimension based on the load propagation cost and the node recovery time, and constructs a time-limited benefit dimension based on the time-limited debt reduction and dependency blocking debt release. The candidate node deduction module first deletes candidate mappings that are simultaneously dominated by another mapping scheme in two dimensions, then determines the insertable execution interval based on the resource prediction sequence corresponding to each candidate mapping and the existing task release time of the node, and determines the task start order within the insertable execution interval according to the growth order of the time-debt of each task. When multiple candidate mappings have the same startup order, the task node mapping with the lowest overlap between the load propagation path and the existing task load path of the node is selected.

8. The scientific and technological information collection and scheduling system based on computing node load awareness according to claim 7, characterized in that, The candidate node inference module establishes an inter-task load interference matrix for multiple tasks to be scheduled within the same scheduling period. The elements of the inter-task load interference matrix are generated by the shared resource occupancy increment formed by the resource evolution trajectories of two tasks during the stage overlap. The candidate node deduction module establishes a debt growth order based on the time-sensitive debt of each task to be scheduled, and adds the task node mapping one by one according to the debt growth order. Each time a task node mapping is added, the resource evolution trajectory of the remaining candidate mappings is corrected using the inter-task load interference matrix. When the corrected resource evolution trajectory is marked as the persistent congestion mapping, the mapping of the most recently added task node with a small amount of dependent blocking debt release is revoked, and the corresponding task is moved into the next insertable execution interval until a batch scheduling sequence that does not contain the persistent congestion mapping is formed.

9. The scientific and technological information collection and scheduling system based on computing node load awareness according to claim 5, characterized in that, The scheduling closed-loop correction module maintains an updated reliable state for each direct load path, coupled load path, and propagation load path, consisting of the number of samples, the consistency of residual direction, and the coverage of the node environment. When the task attributable residuals continuously point to the same resource change direction and the corresponding node environment coverage meets the update conditions, the scheduling closed-loop correction module writes the task attributable residuals into the candidate fingerprint version and uses the candidate fingerprint version to re-deduce the resource evolution trajectory of the most recently completed tasks. The candidate fingerprint version replaces the original fingerprint version only when the stage deviation between the reenacted trajectory and the actual execution trajectory is less than the original fingerprint version and no new continuous congestion misjudgment is added. Otherwise, the original fingerprint version is retained and the corresponding residual is transferred to the set of residuals to be reviewed for joint attribution of subsequent execution trajectories of the same type of task.

10. The scientific and technological information collection and scheduling system based on computing node load awareness according to claim 8, characterized in that, The scheduling closed-loop correction module receives the actual resource status of each computing node according to the stage segment during the execution of the batch scheduling sequence, and continuously aligns the actual resource status with the corresponding resource evolution trajectory. When the queue fallback time of any computing node is shifted backward and the time-debt of unstarted tasks continues to increase, the scheduling closed-loop correction module freezes the mapping of task nodes that have not yet entered the data access stage and have low time-debt, keeps the tasks that have entered the content parsing stage from continuing to execute, and re-adds the frozen tasks to the candidate node inference module according to the updated time-debt. The candidate node deduction module regenerates the resource evolution trajectory using the remaining stages of the unfinished tasks as the initial stage, and forms the subsequent batch scheduling sequence based on the updated inter-task load interference matrix and the insertable execution interval.

Images