Method and system for stream computing resource scheduling based on dynamic time window

Abstract

The application provides a dynamic time window-based stream computing resource scheduling method and system, relates to the field of stream computing resource scheduling, and comprises the following steps: constructing a master-slave event time monitoring network to obtain a task characteristic vector; establishing an adaptive time window scheduling mechanism to dynamically adjust the window size; constructing a multi-level task priority queue and adopting an incremental transmission remote direct memory access mechanism to perform state migration; and finally, coordinating the computing nodes to complete resource switching through a distributed task arrangement protocol. The application can effectively reduce the competition for computing resources, improve the stream computing task processing efficiency, realize dynamic optimization of the resources, and enhance the system throughput.

Description

TECHNICAL FIELD

[0001] The present application relates to stream computing resource scheduling technology, and in particular to a stream computing resource scheduling method and system based on a dynamic time window. BACKGROUND

[0002] With the continuous expansion of stream computing application scenarios, data stream processing puts forward higher requirements on real-time performance and resource utilization efficiency. The existing stream computing framework adopts a static resource allocation strategy, and the resources are isolated between computing nodes, resulting in low overall resource utilization and the inability to adapt to dynamic load changes.

[0003] In the prior art, the resource scheduling of stream computing tasks mainly relies on a global back pressure detection mechanism, and adjustment cannot be triggered until the back pressure signal is propagated throughout the processing link, resulting in a large response delay. At the same time, due to the lack of fine-grained event time monitoring and state migration mechanisms, data processing delays and state consistency problems are likely to occur during task reallocation.

[0004] The current fault recovery strategy of the stream computing framework mainly uses global restart, which causes all task instances to shut down and restart, resulting in long recovery time and affecting service availability. Therefore, there is an urgent need for a stream computing resource scheduling method based on a dynamic time window to achieve fine-grained dynamic adjustment of computing resources and rapid fault recovery, and to improve the overall performance and reliability of the system. SUMMARY

[0005] The embodiments of the present application provide a stream computing resource scheduling method and system based on a dynamic time window, which can solve the problems in the prior art.

[0006] In a first aspect, the embodiments of the present application provide a stream computing resource scheduling method based on a dynamic time window, comprising:

[0007] A master-slave event time monitoring network is constructed, the master node aggregates the event time information collected by each slave node to obtain a global event time deviation rate, and the slave node detects the processing delay state of the local computing task in real time, and the global event time deviation rate and the processing delay state are used as a task feature vector;

[0008] An adaptive time window scheduling mechanism is established based on the task feature vector, the resource competition degree of each computing node is obtained through a distributed state detector, the size of the time window is dynamically adjusted according to the resource competition degree, and a window adjustment coefficient and a task resource competition marker are outputted;

[0009] According to the window adjustment coefficient and the task resource competition mark, a multi-level task priority queue is constructed, the tasks are prioritized, a task descriptor is generated, based on the state position information in the task descriptor, a remote direct memory access mechanism with incremental transmission is used for state migration, a state change bitmap is used to mark the changed state area, and a migration completion mark is generated;

[0010] According to the target position and resource capacity information in the migration completion mark, the distributed task arrangement protocol is used to coordinate each computing node to complete resource switching, so as to realize dynamic optimization configuration of computing resources.

[0011] In an optional embodiment,

[0012] A master-slave event time monitoring network is constructed, the master node aggregates the event time information collected by each slave node to obtain a global event time deviation rate, and each slave node detects the processing delay state of the local computing task in real time, and the global event time deviation rate and the processing delay state are included as a task feature vector:

[0013] A master-slave event time monitoring network is constructed in a distributed computing environment, which includes a master node and a plurality of slave nodes, and the master node issues event time monitoring configuration information including a sampling period and a sampling frequency to each slave node;

[0014] Each slave node sets a local sampling time window according to the event time monitoring configuration information, records event arrival timestamps and processing completion timestamps in the sampling time window, and calculates an event time sequence based on the recorded timestamp information;

[0015] Each slave node performs segmented statistics on the event time sequence to obtain an event time distribution sequence, takes the difference between the event arrival time distribution sequence and the processing completion time distribution sequence as a local processing delay state, and monitors the change rate of the local processing delay state in real time, and when the change rate exceeds a dynamic change threshold calculated based on historical data, sends a state report including the local processing delay state to the master node;

[0016] After receiving the state reports of each slave node, the master node calculates a node weight coefficient according to the local processing delay state in the state report, obtains a global event time deviation rate by multiplying the local processing delay state of each node by the corresponding weight coefficient, and combines the global event time deviation rate with the local processing delay state of each slave node to construct a task feature vector.

[0017] In an optional embodiment,

[0018] An adaptive time window scheduling mechanism is established based on a task feature vector, a resource competition degree of each computing node is acquired through a distributed state detector, a time window size is dynamically adjusted according to the resource competition degree, and a window adjustment coefficient and a task resource competition marker are output, including:

[0019] A distributed state detector is deployed on each computing node, which collects processor interrupt frequency, memory page fault exception number and network packet retransmission rate to form a resource competition sequence;

[0020] A hierarchical deviation value of the resource competition sequence is calculated, a sampling time point with a hierarchical deviation value exceeding a statistical reference value is determined as a resource competition point, and a resource competition intensity is calculated based on a time distribution of the resource competition point;

[0021] Interval correlation calculation is performed on each component in the task feature vector and the resource competition intensity, when the absolute value of the correlation coefficient is greater than a correlation threshold, the corresponding feature component is determined as a resource competition feature, and the product of the resource competition feature and the resource competition intensity is taken as the resource competition degree;

[0022] A resource competition change value is calculated according to the change rate of the resource competition degree, an adjustment step is updated based on the resource competition change value, and a window adjustment coefficient is output;

[0023] The time sequence of the window adjustment coefficient is decomposed into frequency components, the amplitude in a preset frequency interval is extracted as a steady-state resource competition component, the variance of the remaining frequency interval is extracted as a disturbance resource competition component, and a task resource competition marker is generated based on the ratio of the steady-state resource competition component and the disturbance resource competition component.

[0024] In an optional embodiment,

[0025] A resource competition change value is calculated according to the change rate of the resource competition degree, an adjustment step is updated based on the resource competition change value, and a window adjustment coefficient is output, including:

[0026] Time sequence data of the resource competition degree is acquired, a multi-level sliding window group is constructed, the change rate of the resource competition degree is calculated based on the multi-level sliding window group, adaptive weight allocation is performed on the change rate of the resource competition degree, and the product of the allocated weight and the corresponding change rate is constructed into a resource competition change feature matrix;

[0027] The projection overlap degree of adjacent time points of the resource competition change feature matrix is calculated, when the projection overlap degree exceeds a stability judgment threshold, the product of the norm of the resource competition change feature matrix and the projection overlap degree is taken as a resource competition change accumulation;

[0028] A nonlinear compression function is used to generate a resource competition change value based on the direction and amplitude of the resource competition change accumulation, and a dynamic attenuation function is constructed according to the projection overlap degree, and a product of the dynamic attenuation function and a reference smoothing coefficient is used as an adjustment step coefficient;

[0029] An updated adjustment step is obtained by multiplying the current adjustment step and an exponential function of the adjustment step coefficient, a product of the updated adjustment step and the resource competition change value is used as a window adjustment amount, a window adjustment coefficient is obtained by nonlinearly combining the reference coefficient and the window adjustment amount, and a final window adjustment coefficient is output by adaptively limiting the window adjustment coefficient.

[0030] In an optional embodiment,

[0031] A multi-level task priority queue is constructed according to the window adjustment coefficient and the task resource competition marker, and the tasks are prioritized to generate a task descriptor, including:

[0032] A priority mapping interval is generated according to the window adjustment coefficient, an upper bound of the priority mapping interval is determined by an inverse of the window adjustment coefficient, a mapping ratio corresponding to each task resource competition marker is calculated according to a distribution of the task resource competition marker in the priority mapping interval, and a product of the mapping ratio and the task resource competition marker is used as a priority score of the task.

[0033] The priority scores are classified according to a preset priority classification interval, tasks with the same classification result are grouped into a priority sub-queue, and a multi-level task priority queue including multiple priority sub-queues is constructed.

[0034] A priority buffer threshold is set in the multi-level task priority queue, a final queue position of a task is determined according to a difference between the priority score of the task and an average priority score of adjacent priority sub-queues, and a task descriptor is generated according to the final queue position of the task in the multi-level task priority queue.

[0035] In an optional embodiment,

[0036] Based on state position information in the task descriptor, a remote direct memory access mechanism with incremental transmission is used for state migration, a state change bitmap is used to mark a changed state area, and a migration completion marker is generated, including:

[0037] Based on the state position information in the task descriptor, a state space is divided into multiple page storage units, a page bitmap and a data block bitmap are established to form a state change bitmap, and the state change bitmap is used to mark a changed state area, wherein the page bitmap marks a changed page storage unit, and the data block bitmap marks a specific changed area.

[0038] establishing a remote direct memory access connection, setting a source end state space as a remote access area, synchronizing a source end state change bitmap to a target end, selecting a page to be transmitted according to a mark of a page bitmap in the state change bitmap, performing incremental transmission on the selected page according to a mark of a data block bitmap, and marking a transmitted area in a target end state change bitmap;

[0039] monitoring source end state changes, transmitting a new changed area to the target end and updating the target end state change bitmap, comparing the source end and target end state change bitmaps, marking a state change bit when the state change bit is changed and the incremental transmission is completed, and generating a migration completion mark. Figure One

[0040] In an optional embodiment,

[0041] According to the target position and resource capacity information in the migration completion mark, the distributed task scheduling protocol is used to coordinate each computing node to complete resource switching, and the dynamic optimization configuration of computing resources is realized, including:

[0042] According to the target position and resource capacity information in the migration completion mark, the target computing node for resource switching and the allocatable computing resources are determined.

[0043] Each computing node maintains a local resource state and exchanges resource states through heartbeat messages, the resource state contains allocatable computing resources specified by resource capacity information, and version number and time stamp are used for state consistency verification.

[0044] Based on the target position information, the computing nodes participating in resource switching are selected, a task dependency graph between the computing nodes is constructed, and a task scheduling scheme is generated according to the task dependency graph and the resource capacity information.

[0045] The resource allocation token containing the task scheduling scheme is initialized, the resource allocation token is passed between the computing nodes, the computing node receiving the resource allocation token marks the allocatable resources according to the local resource state and updates the resource allocation token, and after completing a round of passing, the resource allocation marks of each computing node are summarized to generate resource allocation confirmation information.

[0046] According to the resource allocation confirmation information, the resource switching of each computing node is coordinated, the operation log of the switching process is recorded, when the switching failure information is received, the operation log is rolled back, and the dynamic optimization configuration of computing resources is realized.

[0047] In a second aspect of the embodiment of the application, a stream computing resource scheduling system based on a dynamic time window is provided, including:

[0048] ​The first unit is configured to construct a master-slave event time monitoring network, a master node aggregates event time information collected by each slave node to obtain a global event time deviation rate, and each slave node detects a processing delay state of a local computing task in real time, and the global event time deviation rate and the processing delay state are taken as a task feature vector;

[0049] The second unit is configured to establish an adaptive time window scheduling mechanism based on the task feature vector, to obtain a resource competition degree of each computing node through a distributed state detector, to dynamically adjust a time window size according to the resource competition degree, and to output a window adjustment coefficient and a task resource competition marker;

[0050] The third unit is configured to construct a multi-level task priority queue according to the window adjustment coefficient and the task resource competition marker, to perform priority classification on the task, to generate a task descriptor, to perform state migration through a remote direct memory access mechanism of incremental transmission based on state position information in the task descriptor, to mark a changed state area through a state change bitmap, and to generate a migration completion marker.

[0051] The fourth unit is configured to coordinate each computing node to complete resource switching and to realize dynamic optimization configuration of computing resources according to target position and resource capacity information in the migration completion marker through a distributed task arrangement protocol.

[0052] In a third aspect, an electronic device is provided, including:

[0053] a processor;

[0054] a memory for storing processor-executable instructions;

[0055] The processor is configured to invoke the instructions stored in the memory to execute the method described above.

[0056] In a fourth aspect, a computer-readable storage medium is provided, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the method described above.

[0057] In the embodiment, by constructing a master-slave event time monitoring network, the global event time deviation rate and processing delay state are obtained in real time, combined with the resource competition degree obtained by the distributed state detector, the time window size can be accurately and dynamically adjusted, the adaptability of the stream computing system to data stream fluctuations is improved, the processing delay is reduced, and the system stability is enhanced. The multi-level task priority queue is used for hierarchical management of computing tasks, and the state position information based on the task descriptor is used to realize the incremental transmission of the remote direct memory access mechanism, which greatly reduces the data transmission amount in the state migration process, reduces the network bandwidth occupation, speeds up the state migration speed, and improves the resource scheduling efficiency. Through the distributed task arrangement protocol, each computing node is coordinated to complete resource switching, realizing dynamic optimization configuration of computing resources, effectively solving the problems of uneven resource allocation, large task processing delay and low system throughput in traditional stream computing systems, improving the overall computing performance and resource utilization, and making the system better cope with peak load and burst traffic. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 A flowchart of a stream computing resource scheduling method based on a dynamic time window according to an embodiment of the application is shown in

[0059] Figure 2 A state migration time comparison thermodynamic diagram according to an embodiment of the application is shown in

[0060] Figure 3 A distributed task arrangement protocol flowchart according to an embodiment of the application is shown in DETAILED DESCRIPTION

[0061] To make the purpose, technical scheme and advantages of the embodiments of the application clearer, the technical scheme in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application.

[0062] The technical scheme of the application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in some embodiments.

[0063] Figure 1 A flowchart of a stream computing resource scheduling method based on a dynamic time window according to an embodiment of the application is shown in Figure 1 The method comprises:

[0064] A master-slave event time monitoring network is constructed, the master node aggregates event time information collected by each slave node to obtain a global event time deviation rate, and each slave node detects a processing delay state of a local computing task in real time, and the global event time deviation rate and the processing delay state are taken as a task feature vector;

[0065] An adaptive time window scheduling mechanism is established based on the task feature vector, a resource competition degree of each computing node is obtained by a distributed state detector, the time window size is dynamically adjusted according to the resource competition degree, and a window adjustment coefficient and a task resource competition marker are output;

[0066] A multi-level task priority queue is constructed according to the window adjustment coefficient and the task resource competition marker, the tasks are prioritized, a task descriptor is generated, state migration is performed based on state position information in the task descriptor by using an incremental transmission remote direct memory access mechanism, a state change bitmap is marked to change a state area, and a migration completion marker is generated;

[0067] According to target position and resource capacity information in the migration completion marker, resource switching is coordinated by each computing node through a distributed task arrangement protocol to realize dynamic optimization of computing resources.

[0068] In an optional implementation, a master-slave event time monitoring network is constructed, the master node aggregates event time information collected by each slave node to obtain a global event time deviation rate, and each slave node detects a processing delay state of a local computing task in real time, and the global event time deviation rate and the processing delay state are taken as a task feature vector including:

[0069] A master-slave event time monitoring network is constructed in a distributed computing environment, which includes a master node and a plurality of slave nodes, and the master node issues event time monitoring configuration information including a sampling period and a sampling frequency to each slave node;

[0070] Each slave node sets a local sampling time window according to the event time monitoring configuration information, records event arrival timestamps and processing completion timestamps in the sampling time window, and calculates an event time sequence based on the recorded timestamp information;

[0071] Each slave node performs segmented statistics on the event time sequence to obtain an event time distribution sequence, takes a difference between the event arrival time distribution sequence and the processing completion time distribution sequence as a local processing delay state, and monitors a change rate of the local processing delay state in real time, and when the change rate exceeds a dynamic change threshold calculated based on historical data, sends a state report including the local processing delay state to the master node;

[0072] After receiving the state reports of the slave nodes, the master node calculates node weight coefficients according to the local processing delay states in the state reports, obtains a global event time deviation rate by multiplying the local processing delay states of the nodes by the corresponding weight coefficients, and combines the global event time deviation rate with the local processing delay states of the slave nodes to construct a task feature vector.

[0073] In a distributed computing environment, the construction process of the master-slave event time monitoring network includes configuring a master node and multiple slave nodes. The master node serves as a central coordinator and is responsible for issuing event time monitoring configuration information and aggregating data reported by the slave nodes. The slave nodes are responsible for locally collecting event time information and calculating processing delay states.

[0074] The event time monitoring configuration information issued by the master node to the slave nodes mainly includes two key parameters: a sampling period and a sampling frequency. The sampling period defines a complete monitoring time window, for example, set to 300 seconds; the sampling frequency defines the frequency of timestamp recording within the period, for example, set to record once every 10 seconds. These configuration information is sent to each slave node through a network transmission protocol such as TCP / IP.

[0075] After receiving the configuration information, the slave node immediately sets a sampling time window locally. For example, a slave node receives a configuration with a sampling period of 300 seconds and a sampling frequency of 10 seconds / time, and then creates a 300-second time window and records the arrival timestamp and processing completion timestamp of each event once every 10 seconds within the window. The recorded timestamps have millisecond-level precision, for example, the arrival timestamp of event A is 1598452800000 and the processing completion timestamp is 1598452803500, indicating a processing delay of 3.5 seconds.

[0076] Within the sampling time window, the slave node continuously records the timestamp information of all events to form an event time sequence. The sequence contains multiple time point records, each of which includes an event identifier, an arrival timestamp, and a processing completion timestamp. For example, within a 300-second sampling time window, 30 groups of timestamp data may be recorded to form a complete event time sequence.

[0077] The slave node performs segmented statistical processing on the event time sequence, dividing the sampling time window into multiple time periods, for example, dividing the 300-second window into 10 30-second time periods. Within each time period, the distribution of event arrival times and the distribution of processing completion times are counted, and the event arrival frequency and the processing completion frequency are calculated. For example, within a 30-second time period, 15 event arrivals and 12 event processing completions are recorded, resulting in an event arrival frequency of 0.5 per second and a processing completion frequency of 0.4 per second.

[0078] The local processing delay state is calculated by the slave node by comparing the event arrival time distribution sequence with the processing completion time distribution sequence. The specific calculation method is to take the difference of the two distribution sequences to obtain a time sequence representing the processing delay. For example, if the event arrival frequency is 0.5 per second and the processing completion frequency is 0.4 per second in a certain time period, the processing delay state value of the time period is 0.1 per second, indicating that there is a backlog of tasks.

[0079] The slave node monitors the change rate of the local processing delay state in real time, calculates the difference between the current processing delay state and the processing delay state of the previous time period to obtain the change rate. At the same time, a dynamic change threshold is established based on historical data. The threshold is determined by analyzing the change range of the processing delay state in the past certain time, for example, taking the average value of the change rate in the past 10 sampling periods plus twice the standard deviation as the dynamic change threshold. For example, if the calculated threshold is 0.05 per second. When the monitored processing delay state change rate exceeds the dynamic change threshold, the slave node immediately sends a state report to the master node. For example, if the processing delay state of the current time period is 0.15 per second, the previous time period is 0.08 per second, and the change rate is 0.07 per second, which exceeds the threshold of 0.05 per second, the state report is triggered. The state report contains node identifier, current processing delay state, time stamp and other information.

[0080] After receiving the state reports of each slave node, the master node calculates the node weight coefficient according to the processing delay state in the report. The weight calculation takes into account the processing capacity and current load of the node. The node with strong processing capacity and light load is assigned a higher weight. If the processing delay state value of slave node A is low, indicating that its processing capacity is strong or load is light, it may be assigned a weight of 0.4; while the processing delay state values of nodes B and C are high, they may be assigned weights of 0.3 and 0.3 respectively.

[0081] The master node calculates the global event time deviation rate by multiplying the local processing delay state of each node by the corresponding weight coefficient. Assuming that the processing delay states of nodes A, B, and C are 0.05, 0.15, and 0.2 per second, respectively, and the weight coefficients are 0.4, 0.3, and 0.3, respectively, the global event time deviation rate is 0.05 x 0.4 + 0.15 x 0.3 + 0.2 x 0.3 = 0.125 per second, representing the event processing delay state of the entire system. Finally, the master node combines the calculated global event time deviation rate with the local processing delay states of each slave node to construct a task feature vector. This vector contains the global event time deviation rate and the local processing delay states of each slave node, and can be represented as [0.125, 0.05, 0.15, 0.2], where the first element is the global deviation rate and the last three elements are the local processing delay states of nodes A, B, and C, respectively. This task feature vector can be used for subsequent task scheduling, load balancing, or anomaly detection operations.

[0082] In this embodiment, the master-slave event time monitoring network can real-time monitor the event processing state in the distributed system and accurately capture the global time deviation rate. The slave nodes record event timestamp information through a sampling time window and calculate the processing delay state, and only report to the master node when the change rate exceeds the dynamic threshold, greatly reducing the network communication overhead. The master node dynamically calculates the node weight according to the report of each slave node and synthesizes the global event time deviation rate, so that the system can adapt to the time deviation under different load conditions. The constructed task feature vector integrates global time deviation and local processing delay information, providing accurate decision basis for subsequent task scheduling and load balancing. This architecture performs well in high concurrency environment, can effectively identify system bottlenecks and predict performance anomalies, improve the overall stability and reliability of distributed computing environment, and reduce the resource consumption of time synchronization and monitoring.

[0083] In an alternative embodiment, an adaptive time window scheduling mechanism is established based on the task feature vector, the resource competition degree of each computing node is obtained through a distributed state detector, the time window size is dynamically adjusted according to the resource competition degree, and the window adjustment coefficient and the task resource competition marker are output, including:

[0084] A distributed state detector is deployed on each computing node, which collects processor interrupt frequency, memory page fault exception number, and network packet retransmission rate to form a resource competition sequence;

[0085] The hierarchical deviation value of the resource competition sequence is calculated, the sampling time point where the hierarchical deviation value exceeds the statistical reference value is determined as the resource competition point, and the resource competition intensity is calculated based on the time distribution of the resource competition point;

[0086] An interval correlation calculation is performed on each component in the task feature vector and the resource competition intensity, and when the absolute value of the correlation coefficient is greater than a correlation threshold, the corresponding feature component is determined as a resource competition feature, and the product of the resource competition feature and the resource competition intensity is taken as a resource competition degree;

[0087] A resource competition change value is calculated according to a change rate of the resource competition degree, an adjustment step is updated based on the resource competition change value, and a window adjustment coefficient is output;

[0088] A time sequence of the window adjustment coefficient is subjected to frequency component decomposition, an amplitude in a preset frequency interval is extracted as a steady-state resource competition component, a variance of a remaining frequency interval is extracted as a disturbance resource competition component, and a task resource competition marker is generated based on a ratio of the steady-state resource competition component and the disturbance resource competition component.

[0089] In actual implementation, each computing node deploys a distributed state detector to collect bottom-layer system indicators such as processor interrupt frequency, memory page fault exception number, and network packet retransmission rate. The state detector collects system states every 100 milliseconds, records the collected data in a time sequence form, and forms a resource competition sequence. For example, the processor interrupt frequencies collected by a certain node within 10 seconds are [2300, 2350, 2800, 3100, 3400, 3450, 3200, 2900, 2700, 2500] times per second, the memory page fault exception numbers are [45, 48, 52, 78, 95, 105, 98, 76, 60, 52] times per second, and the network packet retransmission rates are [0.02, 0.025, 0.03, 0.045, 0.055, 0.057, 0.048, 0.035, 0.028, 0.023].

[0090] The hierarchical deviation value calculation of the resource competition sequence adopts a sliding window method, and the window size is set to 3 sampling points. For each sampling point, the average value of the difference from the front and rear sampling points is calculated, and then divided by the standard deviation in the window to obtain the hierarchical deviation value. Taking the processor interrupt frequency as an example, the hierarchical deviation value of the fifth sampling point (3400) is calculated as the average value of the difference from the adjacent points (3100 and 3450) (3400-(3100+3450) / 2) divided by the standard deviation of the three points. When the hierarchical deviation value exceeds a statistical reference value (usually set to 1.5), the sampling time is marked as a resource competition point. In the example data, the fourth, fifth, and sixth sampling points are marked as resource competition points. The resource competition intensity is calculated according to the time distribution of the resource competition points, and is quantified by the density and duration of the competition points. The resource competition intensity calculation method is to divide the number of competition points by the total number of sampling points, and then multiply by the average deviation value of the competition points. In the example, the resource competition intensity is (3 / 10) x 1.8 = 0.54, indicating a moderate resource competition.

[0091] The interval correlation of each component in the task feature vector and the resource competition intensity is calculated using the sliding window Pearson correlation coefficient calculation method. The window size is set to 20 sampling points, and each time it is slid by 5 sampling points. When the absolute value of the correlation coefficient is greater than the correlation threshold (set to 0.65), the corresponding feature component is determined as the resource competition feature. For example, the correlation coefficient of the global event time deviation rate and the resource competition intensity is 0.78, and the correlation coefficient of the local processing delay state is 0.71, both of which are determined as resource competition features. The resource competition degree is calculated by the product of the resource competition feature and the resource competition intensity, for example, the global event time deviation rate is 0.032, and the corresponding resource competition degree is 0.032x0.54=0.017.

[0092] The change rate of the resource competition degree is calculated by the difference between the current value and the previous time value divided by the time interval. If the current resource competition degree is 0.017, the previous time is 0.012, and the time interval is 100 milliseconds, then the change rate is (0.017-0.012) / 0.1=0.05. The resource competition change value is the ratio of the change rate to the historical average change rate, for example, the historical average change rate is 0.03, then the resource competition change value is 0.05 / 0.03=1.67.

[0093] The window adjustment coefficient is calculated based on the resource competition change value, using a piecewise linear mapping function. When the resource competition change value is less than 0.8, the window adjustment coefficient decreases linearly, with a value range of [0.6, 0.9]; when the resource competition change value is between [0.8, 1.2], the window adjustment coefficient remains unchanged, which is 1.0; when the resource competition change value is greater than 1.2, the window adjustment coefficient increases linearly, with a value range of [1.1, 1.5]. For the resource competition change value 1.67 in the example, the calculated window adjustment coefficient is 1.3.

[0094] The time series of the window adjustment coefficient is decomposed into frequency components using the sliding window Fourier transform method, and the window size is 64 sampling points. The preset frequency interval is [0, 0.2] Hz, and the average value of the amplitude in this interval is extracted as the steady-state resource competition component, for example, the calculation result is 0.85. The variance of the remaining frequency interval [0.2, 5] Hz is calculated as the disturbance resource competition component, for example, the calculation result is 0.17. The ratio of the steady-state resource competition component and the disturbance resource competition component is 0.85 / 0.17=5. When the ratio is greater than the preset threshold 3, the task resource competition label is generated as "stable competition"; when the ratio is between [1, 3], the label is "fluctuating competition"; when the ratio is less than 1, the label is "burst competition". In the example, the task resource competition label is "stable competition".

[0095] Based on the calculated window adjustment coefficient and the task resource competition marker, the stream computing system can dynamically adjust the execution time window size of the task. For the window adjustment coefficient 1.3 and the "stable competition" marker, the system expands the current time window size from 500 milliseconds to 650 milliseconds (500x1.3) to adapt to the steadily growing resource competition environment. This dynamic adjustment enables the system to better balance the computing load in resource-intensive situations and improve overall task processing efficiency.

[0096] In this embodiment, the resource competition state in the distributed environment can be perceived in real time, and the quantitative evaluation of the resource competition degree is realized by accurately capturing key indicators such as processor interrupts, memory page faults, and network retransmissions. The scheme correlates the task feature vector with the resource competition intensity and performs correlation analysis, dynamically identifies the key factors affecting system performance, and provides accurate basis for time window adjustment. Through frequency component decomposition of the window adjustment coefficient, the system can distinguish different competition modes such as stable competition, fluctuating competition, and sudden competition, and adjust the time window size accordingly to effectively cope with different load scenarios. This adaptive scheduling mechanism significantly improves the throughput of the system in high-load environments, reduces task processing delay, and optimizes resource utilization.

[0097] In an optional implementation, a resource competition change value is calculated according to the rate of change of the resource competition degree, an adjustment step is updated based on the resource competition change value, and the window adjustment coefficient is outputted, including:

[0098] Time series data of the resource competition degree is obtained, a multi-level sliding window group is constructed, the rate of change of the resource competition degree is calculated based on the multi-level sliding window group, adaptive weight allocation is performed on the rate of change of the resource competition degree, and the product of the allocated weight and the corresponding rate of change is constructed into a resource competition change feature matrix;

[0099] The projection overlap degree of adjacent time points of the resource competition change feature matrix is calculated, and when the projection overlap degree exceeds a stability judgment threshold, the product of the norm of the resource competition change feature matrix and the projection overlap degree is taken as a resource competition change accumulation;

[0100] Based on the directionality and amplitude size of the resource competition change accumulation, a nonlinear compression function is used to generate a resource competition change value, and a dynamic attenuation function is constructed according to the projection overlap degree, and the product of the dynamic attenuation function and a reference smoothing coefficient is taken as an adjustment step coefficient;

[0101] The current adjustment step is multiplied by an exponential function of the adjustment step coefficient to obtain an updated adjustment step, the product of the updated adjustment step and the resource competition change value is taken as a window adjustment amount, the reference coefficient is nonlinearly combined with the window adjustment amount to obtain a window adjustment coefficient, and the window adjustment coefficient is adaptively limited to output a final window adjustment coefficient.

[0102] Exemplarily, the application first provides a method for calculating a resource competition change value according to the rate of change of the resource competition degree, updating an adjustment step based on the resource competition change value, and outputting a window adjustment coefficient. The specific implementation is as follows:

[0103] In a computer system, time series data of the resource competition degree is obtained, which can be the change data of resource indicators such as CPU usage, memory occupancy, network bandwidth utilization over time. For example, the system CPU occupancy rate can be collected every 100 milliseconds to obtain a data sequence [45%, 48%, 52%, 53%, 51%, 49%...]. A multi-level sliding window group is constructed, including at least three sliding windows with different time spans, such as a short window (5 sampling points), a medium window (15 sampling points), and a long window (30 sampling points). The rate of change of the resource competition degree is calculated for the data in each window, and the rate of change can be obtained by dividing the difference between the latest value in the window and the starting value of the window by the window time span. For example, the short window change rate is (51%-45%) / 5=1.2% / point, the medium window change rate is 0.8% / point, and the long window change rate is 0.5% / point.

[0104] The calculated multi-level change rates are adaptively weighted, and the weight distribution is based on the stability and timeliness of each window. Stability is measured by the degree of fluctuation of the data in the window, and the smaller the fluctuation, the greater the weight; the timeliness of the short window weight base value is higher. Assuming that the allocated weights are short window 0.5, medium window 0.3, and long window 0.2. The product of the allocated weights and the corresponding change rates is constructed into a resource competition change feature matrix, and the matrix dimension is window number x time point number. For example, the feature vector at time t is [0.5x1.2%, 0.3x0.8%, 0.2x0.5%] = [0.6%, 0.24%, 0.1%].

[0105] The projection overlap degree of the computing resource competition change feature matrix at adjacent time instants represents the similarity of the directions of the feature vectors. Assuming that the feature vectors at time instants t and t+1 are [0.6%, 0.24%, 0.1%] and [0.65%, 0.25%, 0.11%] respectively, the overlap degree is calculated by the vector inner product divided by the vector modulus product, and the value is 0.98. When the projection overlap degree exceeds a preset stability judgment threshold (such as 0.9), it is indicated that the resource competition trend is relatively stable, and the product of the norm of the resource competition change feature matrix and the projection overlap degree is taken as the resource competition change accumulation. The norm can be the Euclidean norm of the vector, for example, the norm of [0.65%, 0.25%, 0.11%] is about 0.7%, and the change accumulation is 0.7% x 0.98 = 0.686%.

[0106] Based on the direction and amplitude size of the resource competition change accumulation, a nonlinear compression function is used to generate a resource competition change value. When the accumulation is positive, it indicates that the resource competition is intensified, and when it is negative, it indicates that the competition is alleviated. The nonlinear compression function can be a hyperbolic tangent-based transformation, which ensures that the output value is within a reasonable range. Assuming that the compressed resource competition change value is 0.42. At the same time, a dynamic attenuation function is constructed according to the projection overlap degree. The higher the projection overlap degree, the more stable the system state, and the smaller the attenuation. For example, the attenuation function can be 1-0.2 x (1-projection overlap degree) = 1-0.2 x 0.02 = 0.996. The product of the dynamic attenuation function and the baseline smoothing coefficient (such as 0.8) is taken as the adjustment step coefficient, that is, 0.996 x 0.8 = 0.7968.

[0107] The product of the current adjustment step (assuming 0.05) and the exponential function of the adjustment step coefficient is taken as the updated adjustment step. The exponential function can be a power function of 2, that is, 0.05 x 2 0.7968 ≈0.087. The product of the updated adjustment step and the resource competition change value is taken as the window adjustment amount, that is, 0.087 x 0.42 = 0.03654. The nonlinear combination of the baseline coefficient (for example, 1) and the window adjustment amount is taken as the window adjustment coefficient. The nonlinear combination can be in the form of weighted sum, such as 1+0.03654 = 1.03654, indicating that the resource window should be expanded by about 3.65%.

[0108] The window adjustment coefficient is adaptively limited to ensure that its value is within a reasonable range, such as [0.7, 1.5], to prevent excessive adjustment from causing system instability. Since 1.03654 is within the allowed range, the final output window adjustment coefficient is 1.03654. The system can dynamically adjust the resource allocation window according to this coefficient, expanding the window when the coefficient is greater than 1 and shrinking the window when the coefficient is less than 1, thereby achieving efficient use of resources.

[0109] The multi-level resource competition awareness capability is realized by the dynamic time window scheduling method based on the resource competition degree change rate. The competition change trend of different time scales is captured by the multi-level sliding window, effectively solving the problem that a single time window cannot simultaneously consider sensitivity and stability. The adaptive weight distribution mechanism dynamically adjusts the weight of each window according to the stability of the change rate, improving the identification accuracy of the system to the real resource competition situation. The projection overlap degree calculation ensures the continuity of window adjustment, avoiding the system jitter caused by frequent fluctuations. The nonlinear compression function and dynamic attenuation mechanism enable the system to quickly expand the window in a severe competition environment, and smoothly shrink in a competition relief, greatly improving the completion rate and processing efficiency of the stream computing task.

[0110] In an optional embodiment, a multi-level task priority queue is constructed according to the window adjustment coefficient and the task resource competition marker, the tasks are prioritized, and the task descriptor is generated including:

[0111] A priority mapping interval is generated according to the window adjustment coefficient, the upper bound of the priority mapping interval is determined by the reciprocal of the window adjustment coefficient, and the mapping proportion corresponding to each task resource competition marker is calculated according to the distribution of the task resource competition marker in the priority mapping interval. The product of the mapping proportion and the task resource competition marker is taken as the priority score of the task;

[0112] The priority scores are classified according to the preset priority classification interval, tasks with the same classification result are grouped into a priority sub-queue, and a multi-level task priority queue containing multiple priority sub-queues is constructed;

[0113] A priority buffer threshold is set in the multi-level task priority queue, the final queue position of the task is determined according to the difference between the task priority score and the average priority score of the adjacent priority sub-queue, and the task descriptor is generated according to the final queue position of the task in the multi-level task priority queue.

[0114] In a specific embodiment, the specific implementation method of first constructing a multi-level task priority queue according to the window adjustment coefficient and the task resource competition marker, prioritizing the tasks, and generating the task descriptor is as follows. The window adjustment coefficient is a key indicator to measure the system resource pressure, usually floating between 0.7 and 1.3. A larger window adjustment coefficient indicates that the system is resource-constrained and needs to expand the time window to balance the load; a smaller window adjustment coefficient indicates that the system is resource-abundant and can reduce the time window to improve processing efficiency. The task resource competition marker includes three types of "stable competition", "fluctuating competition" and "burst competition", reflecting the characteristics of the task in resource competition.

[0115] The generation of the priority mapping interval is based on the reciprocal of the window adjustment coefficient, the lower bound of the interval is fixed at 0, and the upper bound is determined by the reciprocal of the window adjustment coefficient. For example, when the window adjustment coefficient is 1.25, the upper bound of the mapping interval is 1 / 1.25 = 0.8; when the window adjustment coefficient is 0.8, the upper bound of the mapping interval is 1 / 0.8 = 1.25. This design ensures that the priority interval is compressed when resources are scarce, and the priority interval is expanded when resources are abundant, making the task priority allocation more sensitive. The mapping ratio is calculated according to the distribution of the task resource competition marker within the priority mapping interval. Usually, the relative positions of "stable competition", "fluctuating competition" and "burst competition" within the interval are set to 0.7, 0.5 and 0.3 respectively. When the priority mapping interval is [0, 0.8], the mapping ratio of "stable competition" is 0.7 x 0.8 = 0.56, the mapping ratio of "fluctuating competition" is 0.5 x 0.8 = 0.4, and the mapping ratio of "burst competition" is 0.3 x 0.8 = 0.24.

[0116] The mapping ratio is multiplied by the quantized value of the task resource competition marker to obtain the priority score of the task. The quantized value of the task resource competition marker is determined according to the competition intensity, for example, "stable competition" is 5, "fluctuating competition" is 3, and "burst competition" is 1. For a "stable competition" task, the priority score is 0.56 x 5 = 2.8; for a "fluctuating competition" task, the priority score is 0.4 x 3 = 1.2; for a "burst competition" task, the priority score is 0.24 x 1 = 0.24.

[0117] The preset priority classification interval is used to classify the priority score, which is usually set to four levels: high priority [2.0, 3.0], medium-high priority [1.0, 2.0), medium-low priority [0.5, 1.0), and low priority [0, 0.5). According to this classification standard, the "stable competition" task with a priority score of 2.8 is classified into the high priority queue, the "fluctuating competition" task with a score of 1.2 is classified into the medium-high priority queue, and the "burst competition" task with a score of 0.24 is classified into the low priority queue. Tasks with the same classification result are grouped into priority sub-queues to construct a multi-level task priority queue.

[0118] A buffer threshold is set in the multi-level task priority queue to smooth the task allocation at the priority boundary. The buffer threshold is usually set to 10% of the boundary value of two adjacent priority intervals, for example, the boundary value of medium-high priority and high priority is 2.0, and the buffer threshold is 0.2. When the task priority score is less than the buffer threshold from the average priority score of the adjacent priority sub-queue, the task is considered to be moved to the adjacent queue. Assuming that the average priority score of the high priority queue is 2.5, a task with a priority score of 1.9, although it should be classified into the medium-high priority queue according to the grading, is kept in the medium-high priority queue because the difference between its priority score and the average score of the high priority queue is |1.9-2.5|=0.6, which is greater than the buffer threshold 0.2; while a task with a priority score of 1.95, the difference between its priority score and the average score of the high priority queue is |1.95-2.5|=0.55, which is still greater than the buffer threshold, is also kept in the medium-high priority queue. This mechanism avoids frequent cross-queue movement of tasks at the priority boundary, improving system stability.

[0119] A task descriptor is generated according to the final queue position of the task in the multi-level task priority queue. The task descriptor contains information such as task ID, priority level, priority score, resource competition flag, execution time window size, etc. For example, a task with a "stable competition" flag and a priority score of 2.8, its descriptor can be represented as {task ID: Task-123456, priority level: high, priority score: 2.8, resource competition flag: stable competition, execution time window: 625ms}. These information will be used for subsequent task scheduling and resource allocation.

[0120] The stream computing system makes resource allocation and scheduling decisions according to the task descriptor. Tasks in the high priority queue get more processor time and memory resources, tasks in the medium priority queue get a medium resource quota, and tasks in the low priority queue are executed when resources are sufficient. For example, on a node with an 8-core processor, high priority tasks can be allocated 4 cores, medium-high priority tasks 2 cores, medium-low and low priority tasks 1 core respectively. When the system load changes, the resource allocation ratio is also adjusted accordingly.

[0121] In the embodiment, the multi-level task priority queue constructed based on the window adjustment coefficient and the task resource competition marker realizes intelligent matching of resource pressure and task characteristics, the priority mapping interval is dynamically adjusted to change the task grading proportion, the system automatically shrinks the priority interval to highlight the key task when the resource is tight, and the interval is expanded to improve the overall throughput when the resource is sufficient. The priority buffer threshold mechanism effectively smooths the jumping behavior of the task at the priority boundary, reduces the frequent migration of the task between queues, and improves the scheduling stability. The multi-dimensional information contained in the task descriptor provides a basis for fine resource allocation, so that the system can make optimal scheduling decisions according to the task characteristics and the current resource status.

[0122] In an optional embodiment, based on the state position information in the task descriptor, a remote direct memory access mechanism with incremental transmission is used for state migration, a state change bitmap is marked to change the state area, and a migration completion marker is generated, including:

[0123] Based on the state position information in the task descriptor, the state space is divided into a plurality of page storage units, a page bitmap and a data block bitmap are established to form a state change bitmap, and the state change area is marked through the state change bitmap, wherein the page bitmap marks the changed page storage unit, and the data block bitmap marks the specific changed area.

[0124] A remote direct memory access connection is established, the source state space is set as a remote access area, the source state change bitmap is synchronized to the target end, the selected page is transmitted according to the marking of the page bitmap in the state change bitmap, the selected page is incrementally transmitted according to the marking of the data block bitmap, and the transmitted area is marked in the target end state change bitmap;

[0125] The source end state change is monitored, the newly changed area is transmitted to the target end and the target end state change bitmap is updated, the source end and the target end state change bitmaps are compared, and when the state change Figure One and the incremental transmission is completed, a migration completion marker is generated.

[0126] Illustratively, the task descriptor in the stream computing process contains state position information, which indicates the starting address, length, and state type of the task state data in the memory, and other key attributes. For example, the state position information of a stream computing task can be represented as {start address: 0x8000A000, state length: 2MB, state type: aggregate counter}. Based on this state position information, the state space is divided into a plurality of page storage units, and the size of each page storage unit is usually set to 4KB, consistent with the memory page size of the operating system. For a 2MB state space, 512 page storage units can be divided.

[0127] The page bitmap and the data block bitmap are used to form the state change bitmap, which is used to mark the changed area accurately. The page bitmap is a bit sequence, and each bit corresponds to a page storage unit. When the content of a page changes, the corresponding bit is set to 1, otherwise it is set to 0. For a state space of 512 pages, the length of the page bitmap is 512 bits. The data block bitmap is more detailed, and each page is further divided into smaller data blocks, usually 64 bytes. A 4KB page can be divided into 64 data blocks, and each data block corresponds to a bit in the data block bitmap. When the content of a data block changes, the corresponding bit is set to 1. For a page, the length of the data block bitmap is 64 bits. The entire state change bitmap is composed of a page bitmap and multiple data block bitmaps. Only the data block bitmap of the changed page is maintained to reduce storage overhead.

[0128] For example, when the 10th and 256th pages of the stream computing task change, the 10th and 256th bits in the page bitmap are set to 1, and the remaining bits are 0. For the 10th page, assume that only the 3rd, 17th and 55th data blocks change, then the 3rd, 17th and 55th bits in the data block bitmap of the page are set to 1, and the remaining bits are 0. Through this two-level bitmap structure, the changed area in the state space is accurately marked, avoiding the overhead of full data transmission.

[0129] When establishing a remote direct memory access connection, a high-performance network interface such as an RDMA network card is used to establish a direct memory access channel between the source and the target. The source state space is set as a remote access area, allowing the target to directly read the source memory content. This process includes registering the memory area, obtaining the remote access key, and setting the access permission. For example, the state space with a starting address of 0x8000A000 and a length of 2MB is registered as a remote access area, the remote access key 0xAB12CD34 is obtained, and the read-only access permission is set.

[0130] The source state change bitmap is synchronized to the target, including the page bitmap and the data block bitmap of the changed pages. For the above example, the page bitmap with a length of 512 bits and the data block bitmap of the 10th and 256th pages need to be synchronized, totaling about 80 bytes of data to be synchronized. According to the synchronization page bitmap mark, the pages to be transmitted are selected. For the bits set to 1, such as the 10th and 256th bits, the corresponding 10th and 256th pages are selected for transmission.

[0131] The selected pages are incrementally transmitted according to the marking of the data block bitmap. For the 10th page, only the 3rd, 17th and 55th data blocks are transmitted, and the amount of transmitted data is 3x64 bytes = 192 bytes, which is much smaller than the entire page of 4KB. For the 256th page, assuming that there are 20 data blocks that have changed, the amount of transmitted data is 20x64 bytes = 1280 bytes. The amount of transmitted data in the entire state migration process is about 1.5KB, which is reduced by more than 99.9% compared with the full amount of transmission of 2MB.

[0132] After receiving the data at the target end, the target end state change bitmap is updated, and the received pages and data blocks are marked as having been transmitted in the bitmap. For example, after receiving the changed data blocks of the 10th page, the 10th bit in the target end page bitmap is marked as having been transmitted, and the 3rd, 17th and 55th bits in the corresponding data block bitmap are marked as having been transmitted.

[0133] Monitoring the source end state change process involves capturing state write operations using a write protection mechanism or a log recording method. When the stream computing task continues to execute and write new state data, such as the 25th data block of the 11th page, the system captures this change, updates the source end state change bitmap, sets the 11th bit of the page bitmap to 1, and sets the 25th bit of the data block bitmap of the 11th page to 1. The new changed area is transmitted to the target end, i.e. the 25th data block (64 bytes) of the 11th page is transmitted, and the target end state change bitmap is updated to maintain consistency between the source end and target end bitmaps.

[0134] The source end and target end state change bitmaps are compared to confirm whether all changed data has been transmitted. When the bitmaps on both ends are completely consistent and no new state changes occur, the incremental transmission is complete. At this time, a migration completion marker is generated, which includes the checksum of the state space, the migration completion timestamp and the state consistency identifier. For example, the migration completion marker can be represented as {checksum: 0xE7A9C5D8, completion timestamp: 1627489632, consistency state: completely consistent}. This marker serves as a certificate of successful state migration and is used for subsequent task switching decisions.

[0135] In this embodiment, an incremental transfer remote direct memory access mechanism based on task descriptor state location information achieves efficient and accurate state migration. A two-level change bitmap structure composed of page bitmaps and data block bitmaps accurately locates the change area, avoiding full data transmission and reducing network bandwidth consumption. Remote direct memory access technology eliminates multiple memory copies and protocol stack overhead in traditional data transmission, significantly reducing CPU usage and transmission latency. A state change monitoring mechanism captures new state writes in real time, ensuring data consistency during migration and making task switching smoother. This significantly reduces task interruption time, improves the system's adaptability to resource fluctuations and overall throughput, and is particularly suitable for dynamic time window adjustment scenarios requiring frequent state migrations.

[0136] Figure 2 This is a heatmap comparing state transition times in an embodiment of the present invention, such as... Figure 2 As shown in the figure, this diagram illustrates the migration times (in milliseconds) of three different state transition methods (full transfer, differential transfer, and the proposed solution) under varying state space sizes (4MB to 32MB) and state change rates (5% to 50%). The full transfer method (such as Apache Storm's checkpointing mechanism) maintains almost constant migration time across all change rates because it always transfers the entire state space, disregarding actual changes; however, it reaches 1827-1912ms in a 32MB state space. The differential transfer method (such as Apache Samza's incremental checkpointing technique) exhibits a significant increase in migration time with increasing change rate, requiring 1573ms in a 32MB state space with a 50% change rate. The proposed solution, through precise two-level bitmap marking and an RDMA incremental transfer mechanism, demonstrates significant advantages under various conditions. It completes the migration in only 91ms in a 32MB state space with a 5% change rate, reducing the time by 95.02% compared to full transfer. Even under a high change rate of 50%, the migration time is only 621ms, still 67.52% lower than traditional methods. Of particular note is that when the state space is 16MB and the change rate is 20%, the migration time of this technical solution is 143ms, which is only 15.26% of that of the full transfer method, demonstrating the efficiency and practical value of this technology in the migration of large-scale streaming computing tasks.

[0137] In one optional implementation, based on the target location and resource capacity information in the migration completion marker, a distributed task orchestration protocol is used to coordinate resource switching among computing nodes, thereby achieving dynamic optimization of computing resource allocation, including:

[0138] Based on the target location and resource capacity information in the migration completion marker, determine the target computing node and allocable computing resources for resource switching;

[0139] Each computing node maintains a local resource state and exchanges resource states through heartbeat messages, the resource state contains allocable computing resources specified by resource capacity information, version number and timestamp are used for state consistency verification;

[0140] Select computing nodes participating in resource switching based on target location information, construct a task dependency graph between computing nodes, and generate a task orchestration scheme according to the task dependency graph and resource capacity information;

[0141] Initialize a resource allocation token containing the task orchestration scheme, the resource allocation token is passed between computing nodes, and the computing node receiving the resource allocation token marks the allocable resources according to the local resource state and updates the resource allocation token, after completing a round of passing, the resource allocation marks of each computing node are summarized to generate resource allocation confirmation information;

[0142] According to the resource allocation confirmation information, coordinate each computing node to perform resource switching, record the operation log of the switching process, when receiving the switching failure information, roll back according to the operation log, realize the dynamic optimization configuration of computing resources.

[0143] Exemplarily, the migration completion mark is the output result of the state migration stage, which contains key target location and resource capacity information. The target location information describes the network address and node identifier of the target computing node of task migration, for example, {node identifier: node-143, IP address: 192.168.10.143, port: 5721}. The resource capacity information indicates the amount of computing resources that the target node can provide, including CPU core number, memory capacity, network bandwidth, etc., for example, {CPU: 8 cores, memory: 32 GB, network bandwidth: 10 Gbps, disk IO: 500 MB / s}. Based on this information, the target node of resource switching and the allocable computing resources can be determined.

[0144] Each computing node maintains a local resource state and exchanges resource states through heartbeat messages, the sending frequency of the heartbeat message is usually 1 time per second. The resource state contains allocable computing resources specified by resource capacity information, such as {available CPU: 3 cores, available memory: 12 GB, available bandwidth: 6 Gbps, disk IO usage: 40%}. In order to ensure the consistency of the resource state, version number and timestamp are used for verification. The version number is incremented with each update of the resource state, and the timestamp records the time point of the last update. For example, the version control information of the resource state is {version number: 235, timestamp: 1627489750}. When receiving the heartbeat message, the computing node compares the version number and the timestamp, and only accepts the updated version. If a version conflict is found, that is, the same version number but different content, the state with the newer timestamp is selected.

[0145] The computing nodes participating in resource switching are selected based on target location information. In a stream computing scenario, the upstream and downstream nodes that exchange data with the target node need to be considered to form a resource switching set. For example, for a computing task processing user behavior data stream, when the aggregation statistics task needs to be migrated, the resource configuration of the data filtering task and the result output task may need to be adjusted at the same time. A task dependency graph between computing nodes is constructed, which is a directed graph, with nodes representing computing tasks and edges representing data flow relationships. For example, for a data processing pipeline consisting of 5 tasks, the task dependency relationship can be represented as {T1→T2, T1→T3, T2→T4, T3→T4, T4→T5}, where T4 is the aggregation statistics task that needs to be migrated. A task orchestration scheme is generated according to the task dependency graph and resource capacity information. The task orchestration scheme specifies the execution node and resource allocation of each task, such as {T1: node-141 (2 cores, 8 GB), T2: node-142 (1 core, 4 GB), T3: node-142 (1 core, 4 GB), T4: node-143 (3 cores, 16 GB), T5: node-144 (1 core, 4 GB)}.

[0146] A resource allocation token containing the task orchestration scheme is initialized, which is a data structure containing multiple fields, mainly including orchestration scheme ID, participating node list, resource request list, resource allocation marker, and token state. For example, a resource allocation token can be represented as {orchestration scheme ID: orch-78213, participating nodes: [node-141, node-142, node-143, node-144], resource requests: {node-143: (3 cores, 16 GB)}, resource allocation marker: {}, token state: initialization}. The resource allocation token is passed between computing nodes in a predetermined order, usually in a ring passing manner. The computing node receiving the resource allocation token checks whether it can meet the resource request according to the local resource state, and if it can, it marks the allocable resource and updates the token. For example, after node-143 receives the token, it checks the local resource state and confirms that it can provide 3-core CPU and 16 GB memory, and updates the token to {orchestration scheme ID: orch-78213, participating nodes: [node-141, node-142, node-143, node-144], resource requests: {node-143: (3 cores, 16 GB)}, resource allocation marker: {node-143: (3 cores, 16 GB, confirmed)}, token state: partially confirmed}. After one round of token passing, the resource allocation markers of each node are summarized to generate resource allocation confirmation information. If all resource requests are confirmed, the state of the resource allocation confirmation information is "all confirmed", otherwise it is "partially confirmed" or "confirmation failed".

[0147] Based on the resource allocation confirmation information, the system coordinates resource switching across computing nodes. Resource switching consists of three phases: preparation, submission, and confirmation. In the preparation phase, each node reserves the confirmed resources and prepares the task switching environment. In the submission phase, the source node stops task execution, and the target node starts the task. In the confirmation phase, the running status of the task in the new environment is verified. The entire process records detailed operation logs, including the type, time, parameters, and result of each operation. For example, the operation log might record: {Operation ID: op-4532, Operation Type: Resource Reservation, Node: node-143, Resource: (3 cores, 16GB), Time: 1627489800, Result: Success}. When a switching failure message is received, a rollback is performed based on the operation logs. The rollback process executes the reverse operations in the reverse order of the operation logs, restoring the state before the switch. For example, if it is found that the task cannot run normally during the confirmation phase, the system rolls back to the task's execution state on the source node, releasing the resources reserved on the target node.

[0148] In stream computing systems, a typical scenario for resource switching is the change in computational load caused by dynamically adjusting the time window. For example, when the resource contention level of the first computing node is detected to increase from 0.3 to 0.7, the system sets the window adjustment factor to 1.2, expanding the processing time window. At this time, some computing tasks need to be migrated to a second computing node with lower resource contention. Through the distributed task orchestration protocol mentioned above, the first node releases 2 CPU cores and 8GB of memory, and the second node allocates these resources to execute the migrated tasks.

[0149] Figure 3 This is a flowchart of the distributed task orchestration protocol according to an embodiment of the present invention, such as... Figure 3 As shown in the flowchart, this process illustrates the complete workflow of coordinating compute nodes to complete resource switching through a distributed task orchestration protocol. The process begins with a migration completion marker, which includes the target location and resource capacity information. Based on this, the system determines the target node for resource switching. Each compute node maintains its resource status through heartbeat messages and constructs a task dependency graph based on the target location information. After the system initializes a resource allocation token, this token is passed in a circular fashion among the compute nodes. Each node marks itself as available for resource allocation and updates its token. After one round of transmission, resource allocation confirmation information is generated and used to coordinate nodes to execute resource switching. The entire process is divided into three stages: preparation, submission, and confirmation. The system records detailed operation logs. When a switch fails, a rollback can be performed based on the logs, ultimately achieving dynamic optimization and configuration of compute resources to meet the resource elasticity requirements of stream computing scenarios.

[0150] In the embodiment, the distributed task scheduling protocol based on the migration completion mark realizes efficient dynamic optimization configuration of computing resources, and through accurate transmission of target position and resource capacity information, the system can quickly determine the target node of resource switching and allocable resources. The nodes exchange resource states through heartbeat messages and use version numbers and time stamps for consistency verification, effectively solving the state synchronization problem in the distributed environment. The construction of the task dependency graph makes the resource switching no longer limited to a single task, but considers the data flow relationship between tasks, ensuring the coherence of the overall computing topology. The ring transmission mechanism of the resource allocation token reduces the pressure on the central coordination node, improves the system scalability and fault tolerance. The three-stage resource switching process and detailed operation log ensure the atomicity and rollback of resource switching, greatly reducing the risk of switching failure, so that the stream computing system can more flexibly cope with dynamic load changes and realize the coordinated optimization of time windows and resource allocation.

[0151] In a second aspect of the embodiment, a stream computing resource scheduling system based on a dynamic time window is provided, and the system comprises:

[0152] A first unit is configured to construct a master-slave event time monitoring network, a master node aggregates event time information collected by each slave node to obtain a global event time deviation rate, and each slave node detects a processing delay state of a local computing task in real time, and the global event time deviation rate and the processing delay state are used as a task feature vector.

[0153] A second unit is configured to establish an adaptive time window scheduling mechanism based on the task feature vector, obtain a resource competition degree of each computing node through a distributed state detector, dynamically adjust a time window size according to the resource competition degree, and output a window adjustment coefficient and a task resource competition mark.

[0154] A third unit is configured to construct a multi-level task priority queue according to the window adjustment coefficient and the task resource competition mark, grade the priority of the task, generate a task descriptor, perform state migration through an incremental transmission remote direct memory access mechanism based on state position information in the task descriptor, mark a changed state area through a state change bitmap, and generate a migration completion mark.

[0155] A fourth unit is configured to complete resource switching of each computing node through a distributed task scheduling protocol based on target position and resource capacity information in the migration completion mark, and realize dynamic optimization configuration of computing resources.

[0156] In a third aspect of the embodiment, an electronic device is provided, and the electronic device comprises:

[0157] A processor;

[0158] A memory for storing processor-executable instructions.

[0159] The processor is configured to invoke instructions stored in the memory to perform the method described above.

[0160] The fourth aspect of the embodiment of the present application provides a computer readable storage medium, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the method described above.

[0161] The present application can be a method, apparatus, system and / or computer program product. The computer program product can include a computer readable storage medium having computer readable program instructions stored therein for implementing various aspects of the present application.

[0162] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A stream computing resource scheduling method based on dynamic time windows, characterized in that, include: A master-slave event time monitoring network is constructed. The master node aggregates the event time information collected by each slave node to obtain the global event time deviation rate. The slave nodes detect the processing delay status of their local computing tasks in real time. The global event time deviation rate and the processing delay status are used as task feature vectors. An adaptive time window scheduling mechanism is established based on task feature vectors. The resource contention level of each computing node is obtained through a distributed state detector. The time window size is dynamically adjusted according to the resource contention level, and the window adjustment coefficient and task resource contention flag are output. A multi-level task priority queue is constructed based on the window adjustment coefficient and task resource contention flag. Tasks are prioritized and task descriptors are generated. Based on the state position information in the task descriptors, a remote direct memory access mechanism with incremental transfer is used for state migration. The changed state area is marked by a state change bitmap and a migration completion flag is generated. Based on the target location and resource capacity information in the migration completion marker, the distributed task orchestration protocol coordinates the resource switching of each computing node to achieve dynamic optimization of computing resources.

2. The method according to claim 1, characterized in that, A master-slave event time monitoring network is constructed. The master node aggregates the event time information collected by each slave node to obtain the global event time deviation rate. The slave nodes detect the processing latency status of their local computing tasks in real time. The global event time deviation rate and processing latency status are used as task feature vectors, including: A master-slave event time monitoring network is constructed in a distributed computing environment, including a master node and multiple slave nodes. The master node sends event time monitoring configuration information, including sampling period and sampling frequency, to each slave node. Each slave node sets a local sampling time window according to the event time monitoring configuration information, records the event arrival timestamp and processing completion timestamp within the sampling time window, and calculates the event time series based on the recorded timestamp information. Each slave node performs segmented statistics on the event time series to obtain the event time distribution sequence. The difference between the event arrival time distribution sequence and the processing completion time distribution sequence is used as the local processing delay status. The change rate of the local processing delay status is monitored in real time. When the change rate exceeds the dynamic change threshold calculated based on historical data, a status report containing the local processing delay status is sent to the master node. After receiving the status reports from each slave node, the master node calculates the node weight coefficient based on the local processing delay status in the status report, and obtains the global event time deviation rate by multiplying the local processing delay status of each node with the corresponding weight coefficient. The global event time deviation rate is then combined with the local processing delay status of each slave node to construct a task feature vector.

3. The method according to claim 1, characterized in that, An adaptive time window scheduling mechanism is established based on task feature vectors. A distributed state detector is used to obtain the resource contention level of each computing node. The time window size is dynamically adjusted according to the resource contention level. The output window adjustment coefficient and task resource contention flag include: Distributed state detectors are deployed on each computing node. These distributed state detectors collect processor interrupt frequency, memory page fault count, and network packet retransmission rate to form a resource contention sequence. Calculate the hierarchical deviation value of the resource competition sequence, determine the sampling time when the hierarchical deviation value exceeds the statistical benchmark value as the resource competition point, and calculate the resource competition intensity based on the time distribution of the resource competition point; The correlation between each component in the task feature vector and the resource competition intensity is calculated. When the absolute value of the correlation coefficient is greater than the correlation threshold, the corresponding feature component is determined as the resource competition feature. The product of the resource competition feature and the resource competition intensity is taken as the degree of resource competition. Calculate the resource competition change value based on the rate of change of the resource competition degree, update the adjustment step size based on the resource competition change value, and output the window adjustment coefficient; The time series of the window adjustment coefficient is decomposed into frequency components. The amplitude within a preset frequency range is extracted as the steady-state resource contention component, and the variance of the remaining frequency range is extracted as the disturbance resource contention component. A task resource contention marker is generated based on the ratio of the steady-state resource contention component and the disturbance resource contention component.

4. The method according to claim 3, characterized in that, Calculate the resource competition change value based on the rate of change in the degree of resource competition, update the adjustment step size based on the resource competition change value, and output the window adjustment coefficient, which includes: Obtain time-series data on the degree of resource competition, construct a multi-level sliding window group, calculate the rate of change of the degree of resource competition based on the multi-level sliding window group, perform adaptive weight allocation on the rate of change of the degree of resource competition, and construct a resource competition change feature matrix by multiplying the allocated weights with the corresponding rate of change. Calculate the projection overlap of the resource competition change feature matrix at adjacent time points. When the projection overlap exceeds the stability judgment threshold, the product of the norm of the resource competition change feature matrix and the projection overlap is taken as the cumulative amount of resource competition change. Based on the directionality and magnitude of the cumulative change in resource competition, a nonlinear compression function is used to generate the change value of resource competition, and a dynamic decay function is constructed according to the projection overlap. The product of the dynamic decay function and the benchmark smoothing coefficient is used as the adjustment step size coefficient. The updated adjustment step size is obtained by multiplying the current adjustment step size by the exponential function of the adjustment step size coefficient. The updated adjustment step size is then multiplied by the resource competition change value as the window adjustment amount. The base coefficient and the window adjustment amount are then nonlinearly combined to obtain the window adjustment coefficient. The window adjustment coefficient is then subjected to adaptive boundary constraints to output the final window adjustment coefficient.

5. The method according to claim 1, characterized in that, A multi-level task priority queue is constructed based on the window adjustment coefficient and task resource contention flag. Tasks are then prioritized and task descriptors are generated, including: A priority mapping interval is generated based on the window adjustment coefficient. The upper bound of the priority mapping interval is determined by the reciprocal of the window adjustment coefficient. Based on the distribution of task resource contention markers within the priority mapping interval, the mapping ratio corresponding to each task resource contention marker is calculated. The product of the mapping ratio and the task resource contention marker is taken as the priority score of the task. Priority scores are graded according to a preset priority grading range. Tasks with the same grading results are grouped into priority sub-queues, and a multi-level task priority queue containing multiple priority sub-queues is constructed. In a multi-level task priority queue, a priority buffer threshold is set. The final queue position of a task is determined based on the difference between its priority score and the average priority score of the adjacent priority sub-queues. A task descriptor is generated based on the final queue position of the task in the multi-level task priority queue.

6. The method according to claim 1, characterized in that, Based on the state location information in the task descriptor, a remote direct memory access mechanism with incremental transfer is used for state transition. A state change bitmap is used to mark the changed state regions, and a transition completion marker is generated, including: Based on the state location information in the task descriptor, the state space is divided into multiple page storage units. A page bitmap and a data block bitmap are established to form a state change bitmap. The state change bitmap marks the changed state areas, wherein the page bitmap marks the changed page storage unit and the data block bitmap marks the specific changed area. Establish a remote direct memory access connection, set the source end state space as a remote access area, synchronize the source end state change bitmap to the target end, select the page to be transmitted according to the mark of the page bitmap in the state change bitmap, perform incremental transmission of the selected page according to the mark of the data block bitmap, and mark the transmitted area in the target end state change bitmap. Monitor the state changes at the source end, transmit the newly changed area to the target end and update the state change bitmap at the target end, compare the state change bitmaps at the source end and the target end, and generate a migration completion marker when the state change bitmaps are consistent and the incremental transmission is completed.

7. The method according to claim 1, characterized in that, Based on the target location and resource capacity information in the migration completion marker, a distributed task orchestration protocol is used to coordinate the resource switching among computing nodes, achieving dynamic optimization of computing resource allocation, including: Based on the target location and resource capacity information in the migration completion marker, determine the target computing node and allocable computing resources for resource switching; Each computing node maintains its local resource status and exchanges resource status through heartbeat messages. The resource status includes allocable computing resources specified by resource capacity information, and uses version number and timestamp to verify status consistency. Based on the target location information, select computing nodes to participate in resource switching, construct a task dependency graph between computing nodes, and generate a task orchestration scheme based on the task dependency graph and resource capacity information. Initialize a resource allocation token containing a task orchestration scheme. The resource allocation token is passed between computing nodes. The computing node that receives the resource allocation token marks the available resources according to its local resource status and updates the resource allocation token. After one round of transmission, the resource allocation marks of each computing node are summarized to generate resource allocation confirmation information. Based on the resource allocation confirmation information, coordinate each computing node to perform resource switching, record the operation log of the switching process, and roll back according to the operation log when a switching failure message is received, so as to realize dynamic optimization and configuration of computing resources.

8. A stream computing resource scheduling system based on dynamic time windows, used to implement the method of any one of claims 1-7, characterized in that, include: The first unit is used to build a master-slave event time monitoring network. The master node summarizes the event time information collected by each slave node to obtain the global event time deviation rate. The slave nodes detect the processing delay status of local computing tasks in real time and use the global event time deviation rate and processing delay status as task feature vectors. The second unit is used to establish an adaptive time window scheduling mechanism based on task feature vectors. It obtains the resource contention level of each computing node through a distributed state detector, dynamically adjusts the time window size according to the resource contention level, and outputs the window adjustment coefficient and task resource contention flag. The third unit is used to construct a multi-level task priority queue based on the window adjustment coefficient and task resource contention flag, classify tasks by priority, generate task descriptors, and perform state migration using an incremental transfer remote direct memory access mechanism based on the state position information in the task descriptors. The changed state area is marked by a state change bitmap, and a migration completion flag is generated. The fourth unit is used to coordinate the resource switching of each computing node through a distributed task orchestration protocol based on the target location and resource capacity information in the migration completion marker, so as to realize the dynamic optimization and allocation of computing resources.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images