Stream computing resource scheduling method and system based on dynamic time window

Abstract

The invention provides a stream computing resource scheduling method and system based on a dynamic time window, and relates to the field of stream computing resource scheduling, and the method comprises the following steps: constructing a master-slave event time monitoring network to obtain a task feature vector, and establishing an adaptive time window scheduling mechanism to dynamically adjust the window size; and constructing a multi-level task priority queue, carrying out state transition by adopting a remote direct memory access mechanism of incremental transmission, and finally coordinating computing nodes through a distributed task arrangement protocol to complete resource switching. According to the method, computing resource competition can be effectively reduced, the flow computing task processing efficiency is improved, dynamic optimal configuration of resources is realized, and the system throughput is enhanced.

Description

Technical Field

[0001] The present invention relates to stream computing resource scheduling technology, and in particular to a stream computing resource scheduling method and system based on dynamic time windows. Background Art

[0002] As stream computing applications continue to expand, data stream processing places higher demands on real-time performance and resource efficiency. Existing stream computing frameworks use static resource allocation strategies that isolate resources between computing nodes, resulting in low overall resource utilization and an inability to adapt to dynamic load changes.

[0003] Existing technologies primarily rely on global backpressure detection mechanisms for stream computing task resource scheduling. These mechanisms require waiting for backpressure signals to propagate throughout the entire processing chain before triggering adjustments, resulting in significant response delays. Furthermore, the lack of fine-grained event time monitoring and state migration mechanisms can easily lead to data processing delays and state consistency issues during task reallocation.

[0004] Currently, the fault recovery strategy for stream computing frameworks relies primarily on a global restart. This approach results in the downtime and restart of all task instances, resulting in long recovery times and impacting service availability. Therefore, a stream computing resource scheduling method based on dynamic time windows is urgently needed to achieve fine-grained dynamic adjustment of computing resources and rapid fault recovery, thereby improving overall system performance and reliability. Summary of the Invention

[0005] The embodiments of the present invention provide a method and system for scheduling stream computing resources based on a dynamic time window, which can solve the problems in the prior art.

[0006] A first aspect of an embodiment of the present invention provides a method for scheduling stream computing resources based on a dynamic time window, comprising: Build a master-slave event time monitoring network. The master node aggregates the event time information collected by each slave node to obtain the global event time deviation rate. The slave node detects the processing delay status of the local computing task in real time and uses the global event time deviation rate and processing delay status as the task feature vector. An adaptive time window scheduling mechanism is established based on the task feature vector. The resource contention level of each computing node is obtained through a distributed state detector. The time window size is dynamically adjusted according to the resource contention level, and the window adjustment coefficient and task resource contention mark are output. A multi-level task priority queue is constructed based on the window adjustment coefficient and task resource competition mark, and the tasks are prioritized and task descriptors are generated. Based on the state location information in the task descriptor, a remote direct memory access mechanism with incremental transmission is used to perform state migration. The changed state area is marked with a state change bitmap, and a migration completion mark is generated. According to the target location and resource capacity information in the migration completion mark, the distributed task scheduling protocol is used to coordinate the computing nodes to complete resource switching and realize the dynamic optimization configuration of computing resources.

[0007] In an optional embodiment, Construct a master-slave event time monitoring network. The master node aggregates the event time information collected by each slave node to obtain the global event time deviation rate. The slave node detects the processing delay status of the local computing task in real time. The global event time deviation rate and processing delay status are used as task feature vectors, including: A master-slave event time monitoring network is constructed in a distributed computing environment, including a master node and multiple slave nodes, wherein the master node sends event time monitoring configuration information including a sampling period and a sampling frequency to each slave node; Each slave node sets a local sampling time window according to the event time monitoring configuration information, records the event arrival timestamp and processing completion timestamp within the sampling time window, and calculates the event time sequence based on the recorded timestamp information; Each slave node performs segmented statistics on the event time series to obtain the event time distribution series. The difference between the event arrival time distribution series and the processing completion time distribution series is used as the local processing delay status. The change rate of the local processing delay status is monitored in real time. When the change rate exceeds the dynamic change threshold calculated based on historical data, a status report containing the local processing delay status is sent to the master node. After the master node receives the status report of each slave node, it calculates the node weight coefficient based on the local processing delay status in the status report, and uses the product of the local processing delay status of each node and the corresponding weight coefficient to obtain the global event time deviation rate. The global event time deviation rate is combined with the local processing delay status of each slave node to construct a task feature vector.

[0008] In an optional embodiment, An adaptive time window scheduling mechanism is established based on the task feature vector. The resource contention level of each computing node is obtained through a distributed state detector. The time window size is dynamically adjusted according to the resource contention level. The output window adjustment coefficient and task resource contention mark include: Deploy a distributed status detector on each computing node to collect processor interrupt frequency, memory page fault exception number and network packet retransmission rate to form a resource contention sequence; Calculate the level deviation value of the resource contention sequence, determine the sampling time when the level deviation value exceeds the statistical benchmark value as the resource contention point, and calculate the resource contention intensity based on the time distribution of the resource contention point; Perform interval correlation calculation on each component in the task feature vector and the resource competition intensity. When the absolute value of the correlation coefficient is greater than the correlation threshold, the corresponding feature component is determined as the resource competition feature, and the product of the resource competition feature and the resource competition intensity is used as the resource competition degree. Calculate a resource contention change value according to a rate of change of the resource contention degree, update an adjustment step size based on the resource contention change value, and output a window adjustment coefficient; The time series of the window adjustment coefficient is decomposed into frequency components, the amplitude within the preset frequency interval is extracted as the steady-state resource competition component, the variance of the remaining frequency interval is extracted as the disturbance resource competition component, and the task resource competition mark is generated based on the ratio of the steady-state resource competition component to the disturbance resource competition component.

[0009] In an optional embodiment, The resource contention change value is calculated according to the change rate of the resource contention degree, and the adjustment step size is updated based on the resource contention change value. The output window adjustment coefficient includes: Acquire time series data of resource contention levels, construct a multi-level sliding window group, calculate the rate of change of resource contention levels based on the multi-level sliding window group, perform adaptive weight assignment on the rate of change of resource contention levels, and construct a resource contention change feature matrix by multiplying the assigned weights by the corresponding rate of change; Calculating the projection overlap of the resource competition change characteristic matrix at adjacent moments, and when the projection overlap exceeds a stability determination threshold, taking the product of the norm of the resource competition change characteristic matrix and the projection overlap as the cumulative amount of resource competition change; Based on the directionality and amplitude of the cumulative amount of resource competition change, a nonlinear compression function is used to generate a resource competition change value, and a dynamic attenuation function is constructed according to the projection overlap. The product of the dynamic attenuation function and the reference smoothing coefficient is used as the adjustment step coefficient; Multiply the current adjustment step size by the exponential function of the adjustment step size coefficient to obtain an updated adjustment step size, use the product of the updated adjustment step size and the resource competition change value as the window adjustment amount, perform a nonlinear combination of the benchmark coefficient and the window adjustment amount to obtain the window adjustment coefficient, and perform adaptive boundary restrictions on the window adjustment coefficient to output the final window adjustment coefficient.

[0010] In an optional embodiment, A multi-level task priority queue is constructed based on the window adjustment coefficient and task resource competition mark, and the tasks are prioritized. The generated task descriptors include: Generate a priority mapping interval according to the window adjustment coefficient, where the upper bound of the priority mapping interval is determined by the inverse of the window adjustment coefficient; calculate the mapping ratio corresponding to each task resource competition mark according to the distribution of the task resource competition mark within the priority mapping interval, and use the product of the mapping ratio and the task resource competition mark as the priority score of the task; Priority scores are graded according to preset priority grading intervals, tasks with the same grading results are grouped into priority sub-queues, and a multi-level task priority queue containing multiple priority sub-queues is constructed; A priority buffer threshold is set in the multi-level task priority queue, the final queue position of the task is determined according to the difference between the task priority score and the average priority score of the adjacent priority sub-queues, and a task descriptor is generated according to the final queue position of the task in the multi-level task priority queue.

[0011] In an optional embodiment, Based on the state location information in the task descriptor, the state migration is performed using the remote direct memory access mechanism of incremental transmission. The changed state area is marked by the state change bitmap, and the migration completion mark is generated, including: Divide the state space into multiple page storage units based on the state location information in the task descriptor, establish a page bitmap and a data block bitmap to form a state change bitmap, and mark the changed state area through the state change bitmap, wherein the page bitmap marks the changed page storage unit and the data block bitmap marks the specific changed area; Establishing a remote direct memory access connection, setting the source end state space as a remote access area, synchronizing the source end state change bitmap to the target end, selecting a page to be transferred based on the mark of the page bitmap in the state change bitmap, incrementally transferring the selected page based on the mark of the data block bitmap, and marking the transferred area in the target end state change bitmap; Monitor the source end status change, transfer the new change area to the target end and update the target end status change bitmap, compare the source end and target end status change bitmap, and when the status change bitmap is Figure 1 When the migration is complete and the incremental transfer is complete, a migration completion marker is generated.

[0012] In an optional embodiment, Based on the target location and resource capacity information in the migration completion marker, the distributed task orchestration protocol coordinates the computing nodes to complete resource switching, achieving dynamic optimization of computing resources. This includes: Determine the target computing node for resource switching and the available computing resources based on the target location and resource capacity information in the migration completion mark; Each computing node maintains local resource status and exchanges resource status through heartbeat messages. The resource status includes the allocatable computing resources specified by resource capacity information, and the status consistency is verified using version numbers and timestamps. Selecting computing nodes to participate in resource switching based on target location information, constructing a task dependency graph between computing nodes, and generating a task scheduling plan based on the task dependency graph and resource capacity information; Initialize a resource allocation token containing a task scheduling plan. The resource allocation token is passed between computing nodes. The computing node that receives the resource allocation token marks the allocable resources according to the local resource status and updates the resource allocation token. After completing one round of transmission, the resource allocation marks of each computing node are aggregated to generate resource allocation confirmation information. Coordinate each computing node to execute resource switching according to the resource allocation confirmation information, record the operation log of the switching process, and roll back according to the operation log when receiving the switching failure information to realize the dynamic optimization configuration of computing resources.

[0013] A second aspect of an embodiment of the present invention provides a stream computing resource scheduling system based on a dynamic time window, comprising: The first unit is used to build a master-slave event time monitoring network. The master node aggregates the event time information collected by each slave node to obtain the global event time deviation rate. The slave node detects the processing delay status of the local computing task in real time and uses the global event time deviation rate and processing delay status as the task feature vector. The second unit is used to establish an adaptive time window scheduling mechanism based on the task feature vector. It obtains the resource contention level of each computing node through a distributed state detector, dynamically adjusts the time window size according to the resource contention level, and outputs the window adjustment coefficient and the task resource contention mark. The third unit is used to build a multi-level task priority queue based on the window adjustment coefficient and the task resource competition flag, prioritize tasks, generate task descriptors, perform state migration using a remote direct memory access mechanism with incremental transmission based on the state location information in the task descriptor, mark the changed state area with a state change bitmap, and generate a migration completion flag; The fourth unit is used to coordinate the computing nodes to complete resource switching through the distributed task scheduling protocol based on the target location and resource capacity information in the migration completion mark, thereby realizing dynamic optimization configuration of computing resources.

[0014] According to a third aspect of an embodiment of the present invention, an electronic device is provided, including: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0015] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0016] In this embodiment, by constructing a master-slave event time monitoring network, the global event time deviation rate and processing delay status are obtained in real time. Combined with the resource competition level obtained by the distributed state detector, the time window size can be accurately and dynamically adjusted, thereby improving the adaptability of the stream computing system to data flow fluctuations, reducing processing delays, and enhancing system stability. A multi-level task priority queue is used to hierarchically manage computing tasks, and a remote direct memory access mechanism for incremental transmission is implemented based on the state location information of the task descriptor, which greatly reduces the amount of data transmission during state migration, reduces network bandwidth usage, speeds up state migration, and improves resource scheduling efficiency. Through the distributed task scheduling protocol, each computing node is coordinated to complete resource switching, realizing dynamic optimization configuration of computing resources, effectively solving the problems of uneven resource allocation, large task processing delay, and low system throughput in traditional stream computing systems, improving overall computing performance and resource utilization, and enabling the system to better cope with peak loads and sudden traffic. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flow chart of a method for scheduling stream computing resources based on dynamic time windows according to an embodiment of the present invention; Figure 2 This is a heat map comparing state transition times in an embodiment of the present invention; Figure 3 This is a flow chart of the distributed task orchestration protocol according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0019] The following specific embodiments are used to describe the technical solution of the present invention in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0020] Figure 1 FIG. 1 is a flow chart of a method for scheduling stream computing resources based on a dynamic time window according to an embodiment of the present invention. Figure 1 As shown, the method includes: Build a master-slave event time monitoring network. The master node aggregates the event time information collected by each slave node to obtain the global event time deviation rate. The slave node detects the processing delay status of the local computing task in real time and uses the global event time deviation rate and processing delay status as the task feature vector. An adaptive time window scheduling mechanism is established based on the task feature vector. The resource contention level of each computing node is obtained through a distributed state detector. The time window size is dynamically adjusted according to the resource contention level, and the window adjustment coefficient and task resource contention mark are output. A multi-level task priority queue is constructed based on the window adjustment coefficient and task resource competition mark, and the tasks are prioritized and task descriptors are generated. Based on the state location information in the task descriptor, a remote direct memory access mechanism with incremental transmission is used to perform state migration. The changed state area is marked with a state change bitmap, and a migration completion mark is generated. According to the target location and resource capacity information in the migration completion mark, the distributed task scheduling protocol is used to coordinate the computing nodes to complete resource switching and realize the dynamic optimization configuration of computing resources.

[0021] In an optional implementation, a master-slave event time monitoring network is constructed. The master node aggregates the event time information collected by each slave node to obtain a global event time deviation rate. The slave node detects the processing delay status of the local computing task in real time. The global event time deviation rate and the processing delay status are used as task feature vectors, including: A master-slave event time monitoring network is constructed in a distributed computing environment, including a master node and multiple slave nodes, wherein the master node sends event time monitoring configuration information including a sampling period and a sampling frequency to each slave node; Each slave node sets a local sampling time window according to the event time monitoring configuration information, records the event arrival timestamp and processing completion timestamp within the sampling time window, and calculates the event time sequence based on the recorded timestamp information; Each slave node performs segmented statistics on the event time series to obtain the event time distribution series. The difference between the event arrival time distribution series and the processing completion time distribution series is used as the local processing delay status. The change rate of the local processing delay status is monitored in real time. When the change rate exceeds the dynamic change threshold calculated based on historical data, a status report containing the local processing delay status is sent to the master node. After the master node receives the status report of each slave node, it calculates the node weight coefficient based on the local processing delay status in the status report, and uses the product of the local processing delay status of each node and the corresponding weight coefficient to obtain the global event time deviation rate. The global event time deviation rate is combined with the local processing delay status of each slave node to construct a task feature vector.

[0022] In a distributed computing environment, building a master-slave event time monitoring network involves configuring a master node and multiple slave nodes. The master node acts as a central coordinator, distributing event time monitoring configuration information and aggregating data reported by slave nodes. Slave nodes are responsible for locally collecting event time information and calculating processing latency.

[0023] The event time monitoring configuration information sent by the master node to each slave node mainly includes two key parameters: the sampling period and the sampling frequency. The sampling period defines a complete monitoring time window, for example, 300 seconds; the sampling frequency defines how often timestamps are recorded within that period, for example, every 10 seconds. This configuration information is sent to each slave node via a network transmission protocol such as TCP / IP.

[0024] After receiving the configuration information, the slave node immediately sets the sampling time window locally. For example, if a slave node receives a configuration with a sampling period of 300 seconds and a sampling frequency of 10 seconds, it creates a 300-second time window and records the arrival and processing completion timestamps of events every 10 seconds within this window. The recorded timestamps are recorded with millisecond precision. For example, if the arrival timestamp of event A is 1598452800000 and the processing completion timestamp is 1598452803500, the processing delay is 3.5 seconds.

[0025] Within the sampling window, the slave node continuously records the timestamp information of all events, forming an event time series. This series contains records at multiple time points, each of which includes an event identifier, arrival timestamp, and processing completion timestamp. For example, within a 300-second sampling window, 30 sets of timestamp data may be recorded, forming a complete event time series.

[0026] The slave node performs segmented statistical processing on the event time series, dividing the sampling time window into multiple time periods. For example, a 300-second window can be divided into ten 30-second time periods. Within each time period, the distribution of event arrival times and processing completion times is statistically analyzed to calculate the event arrival frequency and processing completion frequency. For example, if 15 events arrive and 12 are processed within a 30-second time period, the event arrival frequency is 0.5 events per second and the processing completion frequency is 0.4 events per second.

[0027] By comparing the event arrival time distribution series with the processing completion time distribution series, the local processing delay status is calculated from the node. The specific calculation method is to take the difference between the two distribution series to obtain a time series representing the processing delay. For example, if the event arrival frequency in a certain time period is 0.5 / second and the processing completion frequency is 0.4 / second, the processing delay status value for that time period is 0.1 / second, indicating a backlog of tasks.

[0028] The slave node monitors the rate of change of the local processing delay status in real time, calculates the difference between the current processing delay status and the processing delay status in the previous time period, and obtains the rate of change. At the same time, a dynamic change threshold is established based on historical data. This threshold is determined by analyzing the range of change in the processing delay status over a certain period of time. For example, the average value of the change rate over the past 10 sampling periods plus twice the standard deviation is taken as the dynamic change threshold. Assume that the calculated threshold is 0.05 per second. When the monitored rate of change of the processing delay status exceeds the dynamic change threshold, the slave node immediately sends a status report to the master node. For example, if the processing delay status in the current time period is 0.15 per second and the previous time period was 0.08 per second, the rate of change is 0.07 per second, exceeding the threshold of 0.05 per second, which triggers the sending of a status report. The status report contains information such as the node identifier, the current processing delay status, and a timestamp.

[0029] After receiving status reports from each slave node, the master node calculates a node weight based on the processing delay status reported. This weighting takes into account the node's processing power and current load, assigning higher weights to nodes with higher processing power and lower load. For example, if slave node A has a low processing delay status value, indicating high processing power or a light load, it might be assigned a weight of 0.4. Nodes B and C, with higher processing delay status values, might receive weights of 0.3 and 0.3, respectively.

[0030] The master node calculates the global event time deviation rate by multiplying each node's local processing delay by the corresponding weight coefficient. Assuming the processing delays of nodes A, B, and C are 0.05, 0.15, and 0.2 events per second, respectively, and the weight coefficients are 0.4, 0.3, and 0.3, respectively, the global event time deviation rate is 0.05 × 0.4 + 0.15 × 0.3 + 0.2 × 0.3 = 0.125 events per second, representing the event processing delay state of the entire system. Finally, the master node combines the calculated global event time deviation rate with the local processing delay state of each slave node to construct a task feature vector. This vector contains the global event time deviation rate and the local processing delay state of each slave node and can be expressed as [0.125, 0.05, 0.15, 0.2], where the first element is the global deviation rate, and the next three elements are the local processing delay states of nodes A, B, and C. This task feature vector can be used for subsequent tasks such as task scheduling, load balancing, and anomaly detection.

[0031] In this embodiment, the master-slave event time monitoring network can grasp the event processing status in the distributed system in real time and accurately capture the global time deviation rate. The slave node records the event timestamp information through the sampling time window and calculates the processing delay status. It reports to the master node only when the change rate exceeds the dynamic threshold, which greatly reduces the network communication overhead. The master node dynamically calculates the node weight based on the report of each slave node and synthesizes the global event time deviation rate, so that the system can adapt to the time deviation under different load conditions. The constructed task feature vector integrates the global time deviation and local processing delay information, providing an accurate decision-making basis for subsequent task scheduling and load balancing. This architecture performs well in high-concurrency environments, can effectively identify system bottlenecks and predict performance anomalies, improve the overall stability and reliability of the distributed computing environment, and reduce the resource consumption of time synchronization and monitoring.

[0032] In an optional embodiment, an adaptive time window scheduling mechanism is established based on the task feature vector. The resource contention level of each computing node is obtained through a distributed state detector. The time window size is dynamically adjusted according to the resource contention level. The output window adjustment coefficient and task resource contention flag include: Deploy a distributed status detector on each computing node to collect processor interrupt frequency, memory page fault exception number and network packet retransmission rate to form a resource contention sequence; Calculate the level deviation value of the resource contention sequence, determine the sampling time when the level deviation value exceeds the statistical benchmark value as the resource contention point, and calculate the resource contention intensity based on the time distribution of the resource contention point; Perform interval correlation calculation on each component in the task feature vector and the resource competition intensity. When the absolute value of the correlation coefficient is greater than the correlation threshold, the corresponding feature component is determined as the resource competition feature, and the product of the resource competition feature and the resource competition intensity is used as the resource competition degree. Calculate a resource contention change value according to a rate of change of the resource contention degree, update an adjustment step size based on the resource contention change value, and output a window adjustment coefficient; The time series of the window adjustment coefficient is decomposed into frequency components, the amplitude within the preset frequency interval is extracted as the steady-state resource competition component, the variance of the remaining frequency interval is extracted as the disturbance resource competition component, and the task resource competition mark is generated based on the ratio of the steady-state resource competition component to the disturbance resource competition component.

[0033] In actual implementation, distributed status probes are deployed on each compute node to collect underlying system metrics such as processor interrupt frequency, memory page fault count, and network packet retransmission rate. The status probes collect system status every 100 milliseconds and record the collected data in a time series format, forming a resource contention sequence. For example, for a given node, the processor interrupt frequency collected over 10 consecutive seconds was [2300, 2350, 2800, 3100, 3400, 3450, 3200, 2900, 2700, 2500] times / second, the memory page fault count was [45, 48, 52, 78, 95, 105, 98, 76, 60, 52] times / second, and the network packet retransmission rate was [0.02, 0.025, 0.03, 0.045, 0.055, 0.057, 0.048, 0.035, 0.028, 0.023].

[0034] The level deviation value of the resource contention sequence is calculated using a sliding window method, with a window size of three sampling points. For each sampling point, the average difference between it and the previous and next sampling points is calculated and divided by the standard deviation within the window to obtain the level deviation value. Taking the processor interrupt frequency as an example, the level deviation value of the fifth sampling point (3400) is calculated as the average difference between it and its adjacent points (3100 and 3450) (3400 - (3100 + 3450) / 2) divided by the standard deviation of the three points. When the level deviation value exceeds the statistical threshold (typically set to 1.5), the sampling moment is marked as a resource contention point. In the example data, the 4th, 5th, and 6th sampling points are marked as resource contention points. Resource contention intensity is calculated based on the temporal distribution of resource contention points, quantified by the density and duration of contention points. Resource contention intensity is calculated by dividing the number of contention points by the total number of sampling points, and then multiplying by the average deviation value of the contention points. In this example, the resource contention intensity is (3 / 10)×1.8=0.54, indicating a moderate degree of resource contention.

[0035] The interval correlation between each component in the task feature vector and the resource contention intensity is calculated using the sliding window Pearson correlation coefficient method. The window size is set to 20 sampling points, and the sliding window is 5 sampling points at a time. When the absolute value of the correlation coefficient is greater than the correlation threshold (set to 0.65), the corresponding feature component is identified as a resource contention feature. For example, the correlation coefficient between the global event time deviation rate and resource contention intensity is 0.78, and the correlation coefficient between the local processing delay status is 0.71. Both are identified as resource contention features. The resource contention degree is calculated by multiplying the resource contention feature by the resource contention intensity. For example, if the global event time deviation rate is 0.032, the corresponding resource contention degree is 0.032 × 0.54 = 0.017.

[0036] The rate of change of resource contention is calculated by dividing the difference between the current value and the previous value by the time interval. If the current resource contention level is 0.017, the previous value was 0.012, and the time interval is 100 milliseconds, the rate of change is (0.017 - 0.012) / 0.1 = 0.05. The resource contention change value is the ratio of the change rate to the historical average change rate. For example, if the historical average change rate is 0.03, the resource contention change value is 0.05 / 0.03 = 1.67.

[0037] The window adjustment coefficient is calculated based on the resource contention change using a piecewise linear mapping function. When the resource contention change is less than 0.8, the window adjustment coefficient decreases linearly within the range [0.6, 0.9]. When the resource contention change is between [0.8, 1.2], the window adjustment coefficient remains unchanged at 1.0. When the resource contention change is greater than 1.2, the window adjustment coefficient increases linearly within the range [1.1, 1.5]. For the resource contention change value of 1.67 in this example, the calculated window adjustment coefficient is 1.3.

[0038] The time series of window adjustment coefficients is decomposed into frequency components using a sliding window Fourier transform method with a window size of 64 sampling points. The preset frequency interval is [0, 0.2] Hz, and the amplitude average within this interval is extracted as the steady-state resource contention component, for example, 0.85. The variance of the remaining frequency interval [0.2, 5] Hz is calculated as the perturbation resource contention component, for example, 0.17. The ratio of the steady-state resource contention component to the perturbation resource contention component is 0.85 / 0.17 = 5. When this ratio is greater than the preset threshold of 3, the task resource contention is marked as "stable contention"; when the ratio is between [1, 3], it is marked as "fluctuating contention"; and when the ratio is less than 1, it is marked as "burst contention." In this example, the task resource contention is marked as "stable contention."

[0039] Based on the calculated window adjustment coefficient and the task resource contention flag, the stream computing system dynamically adjusts the task's execution window size. For a window adjustment coefficient of 1.3 and the "stable contention" flag, the system increases the current time window size from 500 milliseconds to 650 milliseconds (500 x 1.3) to accommodate the steadily increasing resource contention. This dynamic adjustment enables the system to better balance computing loads under resource constraints, improving overall task processing efficiency.

[0040] In this embodiment, it is possible to perceive the resource contention status in a distributed environment in real time, and to achieve a quantitative assessment of the degree of resource contention by accurately capturing key indicators such as processor interrupts, memory page faults, and network retransmissions. This solution correlates task feature vectors with resource contention intensity, dynamically identifies key factors affecting system performance, and provides an accurate basis for time window adjustment. Through the frequency component decomposition of the window adjustment coefficient, the system can distinguish between different competition modes such as stable competition, fluctuating competition, and sudden competition, and adjust the time window size in a targeted manner to effectively respond to different load scenarios. This adaptive scheduling mechanism significantly improves the system's throughput in high-load environments, reduces task processing delays, and optimizes resource utilization.

[0041] In an optional embodiment, a resource contention change value is calculated according to a rate of change of the resource contention degree, and an adjustment step size is updated based on the resource contention change value. The output window adjustment coefficient includes: Acquire time series data of resource contention levels, construct a multi-level sliding window group, calculate the rate of change of resource contention levels based on the multi-level sliding window group, perform adaptive weight assignment on the rate of change of resource contention levels, and construct a resource contention change feature matrix by multiplying the assigned weights by the corresponding rate of change; Calculating the projection overlap of the resource competition change characteristic matrix at adjacent moments, and when the projection overlap exceeds a stability determination threshold, taking the product of the norm of the resource competition change characteristic matrix and the projection overlap as the cumulative amount of resource competition change; Based on the directionality and amplitude of the cumulative amount of resource competition change, a nonlinear compression function is used to generate a resource competition change value, and a dynamic attenuation function is constructed according to the projection overlap. The product of the dynamic attenuation function and the reference smoothing coefficient is used as the adjustment step coefficient; Multiply the current adjustment step size by the exponential function of the adjustment step size coefficient to obtain an updated adjustment step size, use the product of the updated adjustment step size and the resource competition change value as the window adjustment amount, perform a nonlinear combination of the benchmark coefficient and the window adjustment amount to obtain the window adjustment coefficient, and perform adaptive boundary restrictions on the window adjustment coefficient to output the final window adjustment coefficient.

[0042] For example, the present invention first provides a method for calculating a resource contention change value based on the rate of change of the resource contention degree, updating an adjustment step size based on the resource contention change value, and outputting a window adjustment coefficient. Specific implementation is as follows: In a computer system, obtain time-series data on resource contention levels. This data can be data on changes over time in resource metrics such as CPU usage, memory utilization, and network bandwidth utilization. For example, the system CPU utilization can be sampled every 100 milliseconds, resulting in a data series of [45%, 48%, 52%, 53%, 51%, 49%, ...]. Construct a multi-level sliding window group, including at least three sliding windows with different time spans, such as a short window (5 sampling points), a medium window (15 sampling points), and a long window (30 sampling points). Calculate the rate of change of resource contention levels for the data within each window. The rate of change is calculated by dividing the difference between the latest value in the window and the window's starting value by the window's time span. For example, the rate of change for a short window is (51% - 45%) / 5 = 1.2% per point, the rate for a medium window is 0.8% per point, and the rate for a long window is 0.5% per point.

[0043] Adaptive weighting is applied to the calculated multi-level change rates, based on the stability and timeliness of each window. Stability is measured by the volatility of the data within the window; smaller fluctuations result in larger weights. Timeliness is measured by the base weight of shorter windows. Assume that the weights assigned are 0.5 for short windows, 0.3 for medium windows, and 0.2 for long windows. The product of the assigned weights and the corresponding change rates is used to construct a resource contention change feature matrix with the dimensions of the number of windows × the number of time points. For example, the feature vector at a given time t is [0.5 × 1.2%, 0.3 × 0.8%, 0.2 × 0.5%] = [0.6%, 0.24%, 0.1%].

[0044] Calculate the projected overlap of the resource contention change feature matrix at adjacent time points. Projected overlap indicates the similarity of the eigenvector directions. Assume that the eigenvectors at time t and t+1 are [0.6%, 0.24%, 0.1%] and [0.65%, 0.25%, 0.11%], respectively. Calculate the overlap by dividing the vector inner product by the vector modulus product, yielding a value of 0.98. When the projected overlap exceeds a preset stability threshold (e.g., 0.9), it indicates that the resource contention trend is relatively stable. The product of the norm of the resource contention change feature matrix and the projected overlap is used as the cumulative resource contention change. The norm can be the Euclidean norm of the vector. For example, the norm of [0.65%, 0.25%, 0.11%] is approximately 0.7%, resulting in a cumulative change of 0.7% × 0.98 = 0.686%.

[0045] Based on the directionality and magnitude of the cumulative amount of resource contention changes, a nonlinear compression function is used to generate the resource contention change value. A positive cumulative value indicates increased resource contention, while a negative value indicates decreased contention. The nonlinear compression function can be a transformation based on the hyperbolic tangent to ensure that the output value is within a reasonable range. Assume that the compressed resource contention change value is 0.42. Simultaneously, a dynamic attenuation function is constructed based on the projection overlap. The higher the projection overlap, the more stable the system state and the smaller the attenuation. For example, the attenuation function can be 1-0.2×(1-projection overlap)=1-0.2×0.02=0.996. The product of the dynamic attenuation function and the baseline smoothing coefficient (e.g., 0.8) is used as the adjustment step coefficient, that is, 0.996×0.8=0.7968.

[0046] The updated adjustment step size is obtained by multiplying the current adjustment step size (assuming it is 0.05) by the exponential function of the adjustment step size coefficient. The exponential function can be a power of 2, i.e. 0.05×2 0.7968 ≈0.087. The window adjustment amount is calculated by multiplying the updated adjustment step size by the change in resource contention, i.e., 0.087 × 0.42 = 0.03654. A nonlinear combination of the reference coefficient (e.g., 1) and the window adjustment amount is used to obtain the window adjustment coefficient. This nonlinear combination can be performed as a weighted sum, e.g., 1 + 0.03654 = 1.03654, indicating that the resource window should be expanded by approximately 3.65%.

[0047] Adaptive limits are applied to the window adjustment coefficient to ensure its value remains within a reasonable range, such as [0.7, 1.5], to prevent over-adjustment and system instability. Since 1.03654 is within the permitted range, the final output window adjustment coefficient is 1.03654. The system dynamically adjusts the resource allocation window based on this coefficient, expanding the window when the coefficient is greater than 1 and shrinking it when it is less than 1, thereby achieving efficient resource utilization.

[0048] The dynamic time window scheduling method described above, based on the calculation of the rate of change in resource contention, achieves multi-level resource competition awareness. Multi-level sliding windows capture competition trends at different time scales, effectively resolving the problem of a single time window being unable to simultaneously address both sensitivity and stability. An adaptive weight allocation mechanism dynamically adjusts the weights of each window based on the stability of the rate of change, improving the system's accuracy in identifying actual resource contention. Projection overlap calculation ensures the continuity of window adjustments, avoiding system turbulence caused by frequent fluctuations. The nonlinear compression function and dynamic attenuation mechanism enable the system to rapidly expand windows under intense competition and smoothly contract them when competition eases, significantly improving the completion rate and processing efficiency of stream computing tasks.

[0049] In an optional embodiment, a multi-level task priority queue is constructed based on the window adjustment coefficient and the task resource competition flag, and the tasks are prioritized. The generation of the task descriptor includes: Generate a priority mapping interval according to the window adjustment coefficient, where the upper bound of the priority mapping interval is determined by the inverse of the window adjustment coefficient; calculate the mapping ratio corresponding to each task resource competition mark according to the distribution of the task resource competition mark within the priority mapping interval, and use the product of the mapping ratio and the task resource competition mark as the priority score of the task; Priority scores are graded according to preset priority grading intervals, tasks with the same grading results are grouped into priority sub-queues, and a multi-level task priority queue containing multiple priority sub-queues is constructed; A priority buffer threshold is set in the multi-level task priority queue, the final queue position of the task is determined according to the difference between the task priority score and the average priority score of the adjacent priority sub-queues, and a task descriptor is generated according to the final queue position of the task in the multi-level task priority queue.

[0050] In one specific embodiment, a multi-level task priority queue is first constructed based on the window adjustment coefficient and task resource contention flags. The specific implementation method for prioritizing tasks and generating task descriptors is as follows. The window adjustment coefficient is a key indicator for measuring system resource pressure and typically fluctuates between 0.7 and 1.3. A larger window adjustment coefficient indicates that system resources are tight and the time window needs to be expanded to balance the load; a smaller window adjustment coefficient indicates that system resources are sufficient and the time window can be narrowed to improve processing efficiency. Task resource contention flags include "stable contention," "fluctuating contention," and "burst contention," reflecting the characteristics of tasks in resource competition.

[0051] The priority mapping interval is generated based on the inverse of the window adjustment coefficient. The lower bound of the interval is fixed at 0, and the upper bound is determined by the inverse of the window adjustment coefficient. For example, when the window adjustment coefficient is 1.25, the upper bound of the mapping interval is 1 / 1.25 = 0.8; when the window adjustment coefficient is 0.8, the upper bound of the mapping interval is 1 / 0.8 = 1.25. This design ensures that the priority interval is compressed when resources are scarce and expanded when resources are abundant, making task priority assignment more sensitive. The mapping ratio is calculated based on the distribution of task resource contention tags within the priority mapping interval. The relative positions of "stable contention," "fluctuating contention," and "burst contention" within the interval are typically set to 0.7, 0.5, and 0.3, respectively. When the priority mapping interval is [0, 0.8], the mapping ratio for "stable contention" is 0.7 × 0.8 = 0.56, the mapping ratio for "fluctuating contention" is 0.5 × 0.8 = 0.4, and the mapping ratio for "burst contention" is 0.3 × 0.8 = 0.24.

[0052] The mapping ratio is multiplied by the quantized value of the task's resource contention flag to obtain the task's priority score. The quantized value of the task's resource contention flag is determined by the intensity of the contention, for example, 5 for "stable contention," 3 for "fluctuating contention," and 1 for "burstful contention." For a task marked with "stable contention," its priority score is 0.56 × 5 = 2.8; for a task marked with "fluctuating contention," its priority score is 0.4 × 3 = 1.2; and for a task marked with "burstful contention," its priority score is 0.24 × 1 = 0.24.

[0053] Priority scores are typically divided into four levels using a preset priority grading interval: high priority (2.0, 3.0), medium-high priority (1.0, 2.0), medium-low priority (0.5, 1.0), and low priority (0, 0.5). Based on this grading standard, a "stable contention" task with a priority score of 2.8 is placed in the high priority queue, a "fluctuating contention" task with a score of 1.2 is placed in the medium-high priority queue, and a "sudden contention" task with a score of 0.24 is placed in the low priority queue. Tasks with the same grading results are grouped into priority sub-queues, creating a multi-level task priority queue.

[0054] In multi-level task priority queues, a priority buffer threshold is set to smooth task allocation at priority boundaries. The buffer threshold is typically set at 10% of the boundary between two adjacent priority intervals. For example, the boundary between medium-high priority and high priority is 2.0, and the buffer threshold is 0.2. When the difference between a task's priority score and the average priority score of an adjacent priority subqueue is less than the buffer threshold, the task is considered for movement to the adjacent queue. For example, if the average priority score of the high-priority queue is 2.5, a task with a priority score of 1.9, while classifying it as medium-high priority, remains in the medium-high priority queue because its difference with the high-priority queue's average score is |1.9 - 2.5| = 0.6, which is greater than the buffer threshold of 0.2. On the other hand, a task with a priority score of 1.95 remains in the medium-high priority queue because its difference with the high-priority queue's average score is |1.95 - 2.5| = 0.55, still greater than the buffer threshold. This mechanism prevents tasks at priority boundaries from frequently moving across queues, improving system stability.

[0055] A task descriptor is generated based on the task's final queue position in the multi-level task priority queue. The task descriptor contains information such as the task ID, priority level, priority score, resource contention flag, and execution window size. For example, a task with the "stable contention" flag and a priority score of 2.8 would have a descriptor like {Task ID: Task-123456, Priority Level: High, Priority Score: 2.8, Resource Contention Flag: Stable Contention, Execution Time Window: 625ms}. This information is used for subsequent task scheduling and resource allocation.

[0056] The stream computing system makes resource allocation and scheduling decisions based on task descriptors. Tasks in the high-priority queue receive more processor time and memory resources, tasks in the medium-priority queue receive a moderate resource quota, and tasks in the low-priority queue execute when resources are sufficient. For example, on an 8-core processor node, high-priority tasks can be allocated 4 cores, medium-high priority tasks 2 cores, and medium-low and low priority tasks 1 core each. As system load changes, resource allocation ratios adjust accordingly.

[0057] In this embodiment, a multi-level task priority queue constructed based on the window adjustment coefficient and task resource competition mark realizes the intelligent matching of resource pressure and task characteristics. The task classification ratio is dynamically adjusted through the priority mapping interval, so that the system automatically shrinks the priority interval to highlight key tasks when resources are tight, and expands the interval to improve overall throughput when resources are sufficient. The priority buffer threshold mechanism effectively smooths the jumping behavior of tasks at the priority boundary, reduces the frequent migration of tasks between queues, and improves scheduling stability. The multi-dimensional information contained in the task descriptor provides a basis for refined resource allocation, enabling the system to make optimal scheduling decisions based on task characteristics and current resource conditions.

[0058] In an optional embodiment, based on the state location information in the task descriptor, a remote direct memory access mechanism of incremental transmission is used to perform state migration, a state change area is marked by a state change bitmap, and a migration completion mark is generated, which includes: Divide the state space into multiple page storage units based on the state location information in the task descriptor, establish a page bitmap and a data block bitmap to form a state change bitmap, and mark the changed state area through the state change bitmap, wherein the page bitmap marks the changed page storage unit and the data block bitmap marks the specific changed area; Establishing a remote direct memory access connection, setting the source end state space as a remote access area, synchronizing the source end state change bitmap to the target end, selecting a page to be transferred based on the mark of the page bitmap in the state change bitmap, incrementally transferring the selected page based on the mark of the data block bitmap, and marking the transferred area in the target end state change bitmap; Monitor the source end status change, transfer the new change area to the target end and update the target end status change bitmap, compare the source end and target end status change bitmap, and when the status change bitmap is Figure 1 When the migration is complete and the incremental transfer is complete, a migration completion marker is generated.

[0059] For example, the task descriptor in the stream computing process contains state location information, which specifies key attributes of the task state data in memory, such as the starting address, length, and state type. For example, the state location information of a stream computing task can be expressed as {start address: 0x8000A000, state length: 2MB, state type: aggregate counter}. Based on this state location information, the state space is divided into multiple page storage units, and the size of each page storage unit is usually set to 4KB, consistent with the memory page size of the operating system. For a 2MB state space, it can be divided into 512 page storage units.

[0060] A state change bitmap is formed by establishing a page bitmap and a data block bitmap to precisely mark areas where state changes occur. The page bitmap is a bit sequence, with each bit corresponding to a page storage unit. When the page content changes, the corresponding position is set to 1, otherwise it is set to 0. For a state space of 512 pages, the page bitmap is 512 bits long. The data block bitmap is more sophisticated, further dividing each page into smaller data blocks, typically 64 bytes. A 4KB page can be divided into 64 data blocks, each corresponding to a bit in the data block bitmap. When the data block content changes, the corresponding position is set to 1. For a page, the data block bitmap is 64 bits long. The entire state change bitmap consists of a page bitmap and multiple data block bitmaps. The data block bitmap is only maintained for pages that have changed to reduce storage overhead.

[0061] For example, when the 10th and 256th pages of a stream computing task change, bits 10 and 256 in the page bitmap are set to 1, and the remaining bits are 0. For the 10th page, assuming only the 3rd, 17th, and 55th data blocks change, bits 3, 17, and 55 in the data block bitmap for that page are set to 1, and the remaining bits are 0. This two-level bitmap structure accurately marks the changed areas in the state space, avoiding the overhead of full data transmission.

[0062] When establishing a remote direct memory access connection, a high-performance network interface, such as an RDMA network card, is used to establish a direct memory access channel between the source and target. The source's state space is set as a remote access region, allowing the target to directly read the source's memory contents. This process involves registering the memory region, obtaining a remote access key, and setting access permissions. For example, a state space with a starting address of 0x8000A000 and a length of 2MB is registered as a remote access region, the remote access key is obtained as 0xAB12CD34, and read-only access permissions are set.

[0063] Synchronize the source's state change bitmap to the target, including the page bitmap and the block bitmap for the changed page. For the previous example, the 512-bit page bitmap and the block bitmaps for pages 10 and 256 need to be synchronized, for a total of approximately 80 bytes of synchronized data. Select the page to be transferred based on the synchronized page bitmap flags. For bits set to 1, such as bits 10 and 256, the corresponding pages 10 and 256 are selected for transfer.

[0064] Incremental transmission is performed on selected pages according to the block bitmap markers. For page 10, only the 3rd, 17th, and 55th blocks are transferred, resulting in a total of 3 x 64 bytes = 192 bytes, significantly less than the 4KB of the entire page. For page 256, assuming 20 blocks have changed, the total data transfer is 20 x 64 bytes = 1280 bytes. The total data transfer for the entire state migration process is approximately 1.5KB, a reduction of over 99.9% compared to the 2MB required for a full transfer.

[0065] After the target receives data, it updates the target's status change bitmap, marking the received pages and data blocks as transmitted in the bitmap. For example, after receiving the changed data block of page 10, bit 10 is marked as transmitted in the target's page bitmap, and bits 3, 17, and 55 are marked as transmitted in the corresponding data block bitmap.

[0066] Monitoring source-side state changes involves capturing state write operations using write protection mechanisms or logging. When a stream computing task continues executing and writes new state data, such as the 25th data block on page 11, the system captures this change and updates the source-side state change bitmap, setting bit 11 of the page bitmap to 1 and bit 25 of the data block bitmap on page 11 to 1. The newly changed area is then transferred to the target side, specifically the 25th data block on page 11 (64 bytes), and the target-side state change bitmap is updated to maintain consistency between the source and target bitmaps.

[0067] The state change bitmaps on the source and target are compared to confirm that all changed data has been transferred. When the bitmaps on both ends are completely consistent and no new state changes have occurred, the incremental transfer is complete. A migration completion marker is generated at this point, containing the state space checksum, the migration completion timestamp, and the state consistency flag. For example, the migration completion marker can be represented as {checksum: 0xE7A9C5D8, completion timestamp: 1627489632, consistency status: fully consistent}. This marker serves as proof of successful state migration and is used in subsequent task switching decisions.

[0068] In this embodiment, an incremental transmission remote direct memory access mechanism based on the task descriptor state location information realizes efficient and accurate state migration. Through the two-level change bitmap structure composed of page bitmap and data block bitmap, the change area is accurately located, the full data transmission is avoided, and the network bandwidth consumption is reduced. Remote direct memory access technology eliminates the multiple memory copies and protocol stack overhead in traditional data transmission, significantly reducing CPU occupancy and transmission delay. The state change monitoring mechanism captures new state writes in real time to ensure data consistency during the migration process and make task switching smoother. It greatly reduces the task interruption time, improves the system's adaptability to resource fluctuations and overall throughput, and is particularly suitable for dynamic time window adjustment scenarios that require frequent state migration.

[0069] Figure 2 This is a heat map comparing the state transition time of an embodiment of the present invention. Figure 2 The figure shows the migration time (in milliseconds) for three different state migration methods (full transfer, differential transfer, and our solution) at different state space sizes (4MB to 32MB) and state change rates (5% to 50%). Full transfer methods (such as Apache Storm's checkpointing mechanism) maintain nearly constant migration time across all change rates because they always transfer the entire state space, regardless of the actual amount of change. With a 32MB state space, migration time can reach as high as 1827-1912ms. Differential transfer methods (such as Apache Samza's incremental checkpointing technique) experience significant increases in migration time with increasing change rate, requiring 1573ms for a 32MB state space and a 50% change rate. Our solution, through precise secondary bitmap marking and RDMA incremental transfer mechanisms, demonstrates significant advantages across all conditions. With a 32MB state space and a 5% change rate, migration completes in just 91ms, a 95.02% reduction compared to full transfer. Even at a high change rate of 50%, migration time is only 621ms, still 67.52% lower than traditional methods. It is particularly noteworthy that when the state space is 16MB and the change rate is 20%, the migration time of this technical solution is 143ms, which is only 15.26% of the full transmission method, demonstrating the efficiency and practical value of this technology in the migration of large-scale stream computing tasks.

[0070] In an optional embodiment, based on the target location and resource capacity information in the migration completion mark, coordinating each computing node to complete resource switching through a distributed task orchestration protocol to achieve dynamic optimization configuration of computing resources includes: Determine the target computing node for resource switching and the available computing resources based on the target location and resource capacity information in the migration completion mark; Each computing node maintains local resource status and exchanges resource status through heartbeat messages. The resource status includes the allocatable computing resources specified by resource capacity information, and the status consistency is verified using version numbers and timestamps. Selecting computing nodes to participate in resource switching based on target location information, constructing a task dependency graph between computing nodes, and generating a task scheduling plan based on the task dependency graph and resource capacity information; Initialize a resource allocation token containing a task scheduling plan. The resource allocation token is passed between computing nodes. The computing node that receives the resource allocation token marks the allocable resources according to the local resource status and updates the resource allocation token. After completing one round of transmission, the resource allocation marks of each computing node are aggregated to generate resource allocation confirmation information. Coordinate each computing node to execute resource switching according to the resource allocation confirmation information, record the operation log of the switching process, and roll back according to the operation log when receiving the switching failure information to realize the dynamic optimization configuration of computing resources.

[0071] For example, the migration completion mark, as the output of the state migration phase, contains key target location and resource capacity information. The target location information describes the network address and node identifier of the target computing node for task migration, for example, {node identifier: node-143, IP address: 192.168.10.143, port: 5721}. The resource capacity information indicates the amount of computing resources available at the target node, including the number of CPU cores, memory capacity, network bandwidth, etc., for example, {CPU: 8 cores, memory: 32GB, network bandwidth: 10Gbps, disk IO: 500MB / s}. Based on this information, the target node for resource switching and the available computing resources can be determined.

[0072] Each compute node maintains its local resource state and exchanges resource state via heartbeat messages, typically sent once per second. Resource state includes allocatable computing resources specified by resource capacity information, such as {Available CPU: 3 cores, Available memory: 12GB, Available bandwidth: 6Gbps, Disk IO utilization: 40%}. To ensure the consistency of resource state, version numbers and timestamps are used for verification. The version number increments with each update to the resource state, and the timestamp records the time of the last update. For example, the version control information for a resource state is {Version number: 235, Timestamp: 1627489750}. Upon receiving a heartbeat message, the compute node compares the version number and timestamp and accepts only the newer version. If a version conflict is detected (i.e., the same version number but different content), the state with the newer timestamp is selected.

[0073] Select computing nodes that participate in resource switching based on target location information. In stream computing scenarios, it is usually necessary to consider upstream and downstream nodes that exchange data with the target node to form a resource switching set. For example, for computing tasks that process user behavior data streams, when the aggregation statistics task needs to be migrated, it may be necessary to adjust the resource configuration of the data filtering task and the result output task at the same time. Construct a task dependency graph between computing nodes. The graph is a directed graph where nodes represent computing tasks and edges represent data flow relationships. For example, for a data processing pipeline consisting of 5 tasks, the task dependency can be expressed as {T1→T2, T1→T3, T2→T4, T3→T4, T4→T5}, where T4 is the aggregation statistics task that needs to be migrated. Generate a task scheduling plan based on the task dependency graph and resource capacity information. The task scheduling scheme specifies the execution nodes and resource allocation for each task, such as {T1: node-141 (2 cores, 8GB), T2: node-142 (1 core, 4GB), T3: node-142 (1 core, 4GB), T4: node-143 (3 cores, 16GB), T5: node-144 (1 core, 4GB)}.

[0074] Initialize a resource allocation token containing the task orchestration plan. This token is a data structure consisting of multiple fields, including the orchestration plan ID, a list of participating nodes, a list of resource requests, a resource allocation flag, and the token state. For example, a resource allocation token might be represented as {Orchestration Plan ID: orch-78213, Participating Nodes: [node-141, node-142, node-143, node-144], Resource Request: {node-143: (3 cores, 16GB)}, Resource Allocation Flag: {}, Token State: Initialized}. Resource allocation tokens are passed between compute nodes in a predetermined order, typically in a round-robin fashion. Each compute node receiving a resource allocation token checks whether the resource request can be satisfied based on its local resource state. If so, it marks the resource available for allocation and updates the token. For example, after receiving the token, node-143 checks its local resource status and confirms that it has 3 CPU cores and 16GB of memory. It updates the token to {Orchestration ID: orch-78213, Participating Nodes: [node-141, node-142, node-143, node-144], Resource Request: {node-143: (3 cores, 16GB)}, Resource Allocation Flag: {node-143: (3 cores, 16GB, Confirmed)}, Token Status: Partially Confirmed}. After a round of token transfer, the resource allocation flags of each node are aggregated to generate a resource allocation confirmation message. If all resource requests are confirmed, the resource allocation confirmation message status is "Fully Confirmed." Otherwise, it is "Partially Confirmed" or "Confirmation Failed."

[0075] Based on the resource allocation confirmation information, resource switching is coordinated across all compute nodes. Resource switching occurs in three phases: preparation, submission, and confirmation. In the preparation phase, each node reserves confirmed resources and prepares the task switching environment. In the submission phase, the source node stops task execution and the target node starts it. In the confirmation phase, the task's running status in the new environment is verified. The entire process records a detailed operation log, including the type, time, parameters, and results of each operation. For example, an operation log might read {Operation ID: op-4532, Operation Type: Resource Reservation, Node: node-143, Resources: (3 cores, 16GB), Time: 1627489800, Result: Success}. When a switch failure is detected, a rollback is performed based on the operation log. The rollback process reverses the operations in the reverse order of the operation log to restore the state before the switch. For example, if a task is found to be unable to run properly during the confirmation phase, the task is rolled back to its execution state on the source node, releasing the reserved resources on the target node.

[0076] In stream computing systems, a typical scenario for resource switching is the change in computing load caused by dynamically adjusting the time window. For example, if the resource contention level on the first compute node increases from 0.3 to 0.7, the system sets the window adjustment factor to 1.2, expanding the processing time window. At this point, some computing tasks need to be migrated to the second compute node, where resource contention is lower. Using the distributed task orchestration protocol described above, the first node releases two CPU cores and 8GB of memory, and the second node allocates these resources to execute the migrated tasks.

[0077] Figure 3 This is a flow chart of the distributed task scheduling protocol according to an embodiment of the present invention. Figure 3 As shown in the figure, the flowchart illustrates the complete process of coordinating compute nodes to complete resource switching through a distributed task orchestration protocol. The process begins with a migration completion marker, which contains the target location and resource capacity information. The system uses this marker to determine the target node for resource switching. Each compute node maintains resource status through heartbeat messages and builds a task dependency graph based on the target location information. After the system initializes a resource allocation token, the token is passed around among the compute nodes in a circular manner. Each node marks the resource available for allocation and updates the token. After a round of transmission, resource allocation confirmation information is generated and coordinated based on this information to execute resource switching. The entire process is divided into three phases: preparation, submission, and confirmation. The system records detailed operation logs, and in the event of a switch failure, a rollback can be performed based on the logs. Ultimately, this achieves dynamic optimization of computing resources, meeting the resource elasticity requirements of stream computing scenarios.

[0078] In this embodiment, a distributed task orchestration protocol based on migration completion marking realizes efficient dynamic optimization configuration of computing resources. By accurately transmitting target location and resource capacity information, the system can quickly determine the target node and allocatable resources for resource switching. Each node exchanges resource status through heartbeat messages and uses version numbers and timestamps for consistency verification, which effectively solves the state synchronization problem in a distributed environment. The construction of the task dependency graph makes resource switching no longer limited to a single task, but takes into account the data flow relationship between tasks to ensure the consistency of the overall computing topology. The ring transmission mechanism of resource allocation tokens reduces the pressure on the central coordination node and improves the scalability and fault tolerance of the system. The three-stage resource switching process and detailed operation logs ensure the atomicity and rollback of resource switching, greatly reduce the risk of switching failure, enable the stream computing system to respond to dynamic load changes more flexibly, and achieve coordinated optimization of time windows and resource allocation.

[0079] A second aspect of an embodiment of the present invention provides a stream computing resource scheduling system based on a dynamic time window, the system comprising: The first unit is used to build a master-slave event time monitoring network. The master node aggregates the event time information collected by each slave node to obtain the global event time deviation rate. The slave node detects the processing delay status of the local computing task in real time and uses the global event time deviation rate and processing delay status as the task feature vector. The second unit is used to establish an adaptive time window scheduling mechanism based on the task feature vector. It obtains the resource contention level of each computing node through a distributed state detector, dynamically adjusts the time window size according to the resource contention level, and outputs the window adjustment coefficient and the task resource contention mark. The third unit is used to build a multi-level task priority queue based on the window adjustment coefficient and the task resource competition flag, prioritize tasks, generate task descriptors, perform state migration using a remote direct memory access mechanism with incremental transmission based on the state location information in the task descriptor, mark the changed state area with a state change bitmap, and generate a migration completion flag; The fourth unit is used to coordinate the computing nodes to complete resource switching through the distributed task scheduling protocol based on the target location and resource capacity information in the migration completion mark, thereby realizing dynamic optimization configuration of computing resources.

[0080] According to a third aspect of an embodiment of the present invention, an electronic device is provided, including: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0081] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0082] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A stream computing resource scheduling method based on dynamic time windows, characterized in that: include: Build a master-slave event time monitoring network. The master node aggregates the event time information collected by each slave node to obtain the global event time deviation rate. The slave node detects the processing delay status of the local computing task in real time and uses the global event time deviation rate and processing delay status as the task feature vector. An adaptive time window scheduling mechanism is established based on the task feature vector. The resource contention level of each computing node is obtained through a distributed state detector. The time window size is dynamically adjusted according to the resource contention level, and the window adjustment coefficient and task resource contention mark are output. A multi-level task priority queue is constructed based on the window adjustment coefficient and task resource competition mark, and the tasks are prioritized and task descriptors are generated. Based on the state location information in the task descriptor, a remote direct memory access mechanism with incremental transmission is used to perform state migration. The changed state area is marked with a state change bitmap, and a migration completion mark is generated. According to the target location and resource capacity information in the migration completion mark, the distributed task scheduling protocol is used to coordinate the computing nodes to complete resource switching and realize the dynamic optimization configuration of computing resources.

2. The method according to claim 1, characterized in that Construct a master-slave event time monitoring network. The master node aggregates the event time information collected by each slave node to obtain the global event time deviation rate. The slave node detects the processing delay status of the local computing task in real time. The global event time deviation rate and processing delay status are used as task feature vectors, including: A master-slave event time monitoring network is constructed in a distributed computing environment, including a master node and multiple slave nodes, wherein the master node sends event time monitoring configuration information including a sampling period and a sampling frequency to each slave node; Each slave node sets a local sampling time window according to the event time monitoring configuration information, records the event arrival timestamp and processing completion timestamp within the sampling time window, and calculates the event time sequence based on the recorded timestamp information; Each slave node performs segmented statistics on the event time series to obtain the event time distribution series. The difference between the event arrival time distribution series and the processing completion time distribution series is used as the local processing delay status. The change rate of the local processing delay status is monitored in real time. When the change rate exceeds the dynamic change threshold calculated based on historical data, a status report containing the local processing delay status is sent to the master node. After the master node receives the status report of each slave node, it calculates the node weight coefficient based on the local processing delay status in the status report, and uses the product of the local processing delay status of each node and the corresponding weight coefficient to obtain the global event time deviation rate. The global event time deviation rate is combined with the local processing delay status of each slave node to construct a task feature vector.

3. The method according to claim 1, characterized in that An adaptive time window scheduling mechanism is established based on the task feature vector. The resource contention level of each computing node is obtained through a distributed state detector. The time window size is dynamically adjusted according to the resource contention level. The output window adjustment coefficient and task resource contention mark include: Deploy a distributed status detector on each computing node to collect processor interrupt frequency, memory page fault exception number and network packet retransmission rate to form a resource contention sequence; Calculate the level deviation value of the resource contention sequence, determine the sampling time when the level deviation value exceeds the statistical benchmark value as the resource contention point, and calculate the resource contention intensity based on the time distribution of the resource contention point; Perform interval correlation calculation on each component in the task feature vector and the resource competition intensity. When the absolute value of the correlation coefficient is greater than the correlation threshold, the corresponding feature component is determined as the resource competition feature, and the product of the resource competition feature and the resource competition intensity is used as the resource competition degree. Calculate a resource contention change value according to a rate of change of the resource contention degree, update an adjustment step size based on the resource contention change value, and output a window adjustment coefficient; The time series of the window adjustment coefficient is decomposed into frequency components, the amplitude within the preset frequency interval is extracted as the steady-state resource competition component, the variance of the remaining frequency interval is extracted as the disturbance resource competition component, and the task resource competition mark is generated based on the ratio of the steady-state resource competition component to the disturbance resource competition component.

4. The method according to claim 3, characterized in that The resource contention change value is calculated according to the change rate of the resource contention degree, and the adjustment step size is updated based on the resource contention change value. The output window adjustment coefficient includes: Acquire time series data of resource contention levels, construct a multi-level sliding window group, calculate the rate of change of resource contention levels based on the multi-level sliding window group, perform adaptive weight assignment on the rate of change of resource contention levels, and construct a resource contention change feature matrix by multiplying the assigned weights by the corresponding rate of change; Calculating the projection overlap of the resource competition change characteristic matrix at adjacent moments, and when the projection overlap exceeds a stability determination threshold, taking the product of the norm of the resource competition change characteristic matrix and the projection overlap as the cumulative amount of resource competition change; Based on the directionality and amplitude of the cumulative amount of resource competition change, a nonlinear compression function is used to generate a resource competition change value, and a dynamic attenuation function is constructed according to the projection overlap. The product of the dynamic attenuation function and the reference smoothing coefficient is used as the adjustment step coefficient; Multiply the current adjustment step size by the exponential function of the adjustment step size coefficient to obtain an updated adjustment step size, use the product of the updated adjustment step size and the resource competition change value as the window adjustment amount, perform a nonlinear combination of the benchmark coefficient and the window adjustment amount to obtain the window adjustment coefficient, and perform adaptive boundary restrictions on the window adjustment coefficient to output the final window adjustment coefficient.

5. The method according to claim 1, wherein A multi-level task priority queue is constructed based on the window adjustment coefficient and task resource competition mark, and the tasks are prioritized. The generated task descriptors include: Generate a priority mapping interval according to the window adjustment coefficient, where the upper bound of the priority mapping interval is determined by the inverse of the window adjustment coefficient; calculate the mapping ratio corresponding to each task resource competition mark according to the distribution of the task resource competition mark within the priority mapping interval, and use the product of the mapping ratio and the task resource competition mark as the priority score of the task; Priority scores are graded according to preset priority grading intervals, tasks with the same grading results are grouped into priority sub-queues, and a multi-level task priority queue containing multiple priority sub-queues is constructed; A priority buffer threshold is set in the multi-level task priority queue, the final queue position of the task is determined according to the difference between the task priority score and the average priority score of the adjacent priority sub-queues, and a task descriptor is generated according to the final queue position of the task in the multi-level task priority queue.

6. The method according to claim 1, characterized in that Based on the state location information in the task descriptor, the state migration is performed using the remote direct memory access mechanism of incremental transmission. The changed state area is marked by the state change bitmap, and the migration completion mark is generated, including: Divide the state space into multiple page storage units based on the state location information in the task descriptor, establish a page bitmap and a data block bitmap to form a state change bitmap, and mark the changed state area through the state change bitmap, wherein the page bitmap marks the changed page storage unit and the data block bitmap marks the specific changed area; Establishing a remote direct memory access connection, setting the source end state space as a remote access area, synchronizing the source end state change bitmap to the target end, selecting a page to be transferred based on the mark of the page bitmap in the state change bitmap, incrementally transferring the selected page based on the mark of the data block bitmap, and marking the transferred area in the target end state change bitmap; Monitor state changes on the source side, transfer new changed areas to the target side, and update the target side's state change bitmap. Compare the state change bitmaps on the source and target sides. When the state change bitmaps are consistent and the incremental transfer is complete, generate a migration completion marker.

7. The method according to claim 1, characterized in that Based on the target location and resource capacity information in the migration completion marker, the distributed task orchestration protocol coordinates the computing nodes to complete resource switching, achieving dynamic optimization of computing resources. This includes: Determine the target computing node for resource switching and the available computing resources based on the target location and resource capacity information in the migration completion mark; Each computing node maintains local resource status and exchanges resource status through heartbeat messages. The resource status includes the allocatable computing resources specified by resource capacity information, and the status consistency is verified using version numbers and timestamps. Selecting computing nodes to participate in resource switching based on target location information, constructing a task dependency graph between computing nodes, and generating a task scheduling plan based on the task dependency graph and resource capacity information; Initialize a resource allocation token containing a task scheduling plan. The resource allocation token is passed between computing nodes. The computing node that receives the resource allocation token marks the allocable resources according to the local resource status and updates the resource allocation token. After completing one round of transmission, the resource allocation marks of each computing node are aggregated to generate resource allocation confirmation information. Coordinate each computing node to execute resource switching according to the resource allocation confirmation information, record the operation log of the switching process, and roll back according to the operation log when receiving the switching failure information to realize the dynamic optimization configuration of computing resources.

8. A stream computing resource scheduling system based on dynamic time windows, used to implement the method according to any one of claims 1 to 7, characterized in that: include: The first unit is used to build a master-slave event time monitoring network. The master node aggregates the event time information collected by each slave node to obtain the global event time deviation rate. The slave node detects the processing delay status of the local computing task in real time and uses the global event time deviation rate and processing delay status as the task feature vector. The second unit is used to establish an adaptive time window scheduling mechanism based on the task feature vector. It obtains the resource contention level of each computing node through a distributed state detector, dynamically adjusts the time window size according to the resource contention level, and outputs the window adjustment coefficient and the task resource contention mark. The third unit is used to build a multi-level task priority queue based on the window adjustment coefficient and the task resource competition flag, prioritize tasks, generate task descriptors, perform state migration using a remote direct memory access mechanism with incremental transmission based on the state location information in the task descriptor, mark the changed state area with a state change bitmap, and generate a migration completion flag; The fourth unit is used to coordinate the computing nodes to complete resource switching through the distributed task scheduling protocol based on the target location and resource capacity information in the migration completion mark, thereby realizing dynamic optimization configuration of computing resources.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images