Data exchange task scheduling method and system based on resource prediction

Abstract

The invention discloses a data exchange task scheduling method and system based on resource prediction. A resource prediction model is constructed by acquiring a resource state change vector of a computing node in a preset time period, and resource availability in a future scheduling window is predicted; obtaining a to-be-scheduled data exchange task queue, and determining task characteristics of each task; based on the task characteristics and the resource availability prediction result, calculating the dominant demand degree of the task to the resource and the implicit attraction of the resource to the task, and carrying out task and resource matching to obtain a task-resource affinity matrix; and generating a scheduling strategy according to the affinity matrix to realize data exchange task scheduling. By introducing a resource availability prediction mechanism, the resource utilization rate is improved, scheduling conflicts and resource bottlenecks are avoided, and the overall performance of a data exchange platform is enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a data exchange task scheduling method and system based on resource prediction. BACKGROUND

[0002] Data exchange tasks usually need to be completed on distributed computing nodes, including data extraction, conversion, encrypted transmission, parsing storage and other multi-link processing, which has different degrees of occupation demand for computing resources, storage resources and network resources. In order to ensure the real-time and high reliability of data exchange, the platform needs to reasonably schedule and dynamically allocate resources in the case of multi-task concurrent execution.

[0003] The existing data exchange task scheduling method generally makes scheduling decisions based on static or current resource state. When the node appears resource congestion, bandwidth shortage or I / O conflict, the original scheduling strategy is difficult to adjust in time, resulting in an increase in task execution delay, and even scheduling failure. At the same time, the traditional scheduling strategy mainly focuses on the explicit demand of tasks for resources, such as computing demand and bandwidth demand, ignoring the implicit factors such as execution history between nodes and cache data reuse, which have potential influence on scheduling efficiency, and it is difficult to realize the optimal allocation of tasks on nodes.

[0004] In addition, under the multi-node architecture, whether the task is allocated to the node with data cache advantage or execution familiarity will significantly affect the data loading cost and processing speed.

[0005] Therefore, an intelligent scheduling method that can combine future resource availability prediction of computing nodes and use historical execution information to mine the implicit attraction of resources to tasks is urgently needed to improve resource utilization efficiency, reduce task waiting and migration cost, and realize overall performance optimization of data exchange platform. SUMMARY

[0006] In order to solve at least one of the above technical problems, the present application provides a data exchange task scheduling method and system based on resource prediction.

[0007] The first aspect of the present application provides a data exchange task scheduling method based on resource prediction, comprising: obtaining the resource state change vector of each computing node of the data exchange platform in a preset time period, constructing a resource prediction model according to the resource state change vector, predicting the resource availability curve of each computing node in the future scheduling window according to the resource prediction model, and obtaining the resource availability prediction result of each computing node; obtaining the data exchange task queue of the data exchange platform to be scheduled, extracting the task description information of each data exchange task in the data exchange task queue, and determining the task characteristics of each data exchange task according to the task description information; determine explicit demand degrees of the tasks on the resources and implicit attraction degrees of the resources on the tasks according to the task characteristics of each data exchange task and the resource availability prediction results of each computing node; perform task and resource matching on the explicit demand degrees and the implicit attraction degrees, determine affinities between the data exchange tasks and the computing node resources, and construct a task-resource affinity matrix of the data exchange platform; perform task scheduling on a data exchange task queue according to the task-resource affinity matrix, and construct a data exchange task scheduling strategy.

[0008] In the scheme, the resource state change vector of each computing node of the data exchange platform in a preset time period is obtained, a resource prediction model is constructed according to the resource state change vector, and the resource availability curve of each computing node in a future scheduling window is predicted according to the resource prediction model to obtain the resource availability prediction result of each computing node, specifically as follows: The resource state change vector of each computing node of the data exchange task platform in a preset time period is obtained, and the resource state change vector includes CPU utilization change, memory usage change, network bandwidth occupancy change, and disk input / output load change. A resource prediction model is constructed based on a long short-term memory network, the input layer dimension, the hidden layer dimension, and the output layer dimension of the resource prediction model are set, and the mean square error is taken as the loss function of the resource prediction model. The resource state change vector of each computing node is input into the resource prediction model for training, the weight parameters of the resource prediction model are optimized through a back propagation algorithm until the loss function converges, and the trained resource prediction model is obtained. The current resource state vector of each computing node of the data exchange task platform is input into the trained resource prediction model to predict the resource state change of each computing node in a future scheduling window, the resource availability curve of each computing node is determined according to the resource state change, the resource availability curve includes CPU availability curve, memory availability curve, network bandwidth availability curve, and disk I / O availability curve, and the resource availability prediction result is obtained.

[0009] In the scheme, the data exchange task queue to be scheduled by the data exchange platform is obtained, the task description information of each data exchange task in the data exchange task queue is extracted, and the task characteristics of each data exchange task are determined according to the task description information, specifically as follows: The data exchange task queue to be scheduled by the data exchange platform is acquired, a task configuration file of each data exchange task in the data exchange task queue is parsed, task description information in the task configuration file is extracted, and the task description information includes a data source address, a data target address, a data volume size, a task priority identifier, a task deadline timestamp, and a data exchange protocol type of the data exchange task; The task description information is structurally parsed, the data source address field and the data target address field are geographically coordinate-converted to obtain geographic location information of the data exchange task, and a data transmission network path length of the data exchange task is calculated according to the geographic location information; A minimum required transmission rate of the data exchange task is calculated according to the data volume size field and the task deadline timestamp field, and a task priority numerical value is obtained by numerically mapping the task priority identifier field; A communication protocol adopted by the data exchange task is determined according to the data exchange protocol type field, and a protocol overhead coefficient corresponding to the communication protocol is acquired; The data transmission network path length, the minimum required transmission rate, the task priority numerical value, and the protocol overhead coefficient are feature-fused to construct a task feature of each data exchange task.

[0010] In this scheme, the explicit demand degree of a task for a resource and the implicit attraction of a resource to a task are determined according to the task feature of each data exchange task and the resource availability prediction result of each computing node, and specifically: A demand vector of a task for a computing resource is extracted according to the task feature of each data exchange task, the demand vector includes a demand intensity of the task for CPU computing capability, a demand intensity of the task for memory capacity, a demand intensity of the task for network bandwidth, and a demand intensity of the task for disk I / O throughput, and the explicit demand degree of the task for the resource is calculated based on the demand vector; Each computing node is accessed in turn, a node entering an access state is marked as an access node, a task execution history log of the access node is acquired, and a historical execution situation of the access node for each data exchange task in the data exchange task queue in a preset historical period is determined according to the task execution history log; A cache data block identifier generated after execution of a data exchange task with the historical execution situation and a corresponding last access timestamp are acquired, a time interval between a current time and the last access timestamp is calculated, the availability weight of the cache data block is determined according to the time interval, and average cache hit rate data of the access node in a historical same period is simultaneously counted; The resource availability prediction result, the availability weight, and the average cache hit rate are fused and calculated to obtain a cache hit gain coefficient of each data exchange task in each computing node in a future scheduling window. The implicit attractiveness of each compute node's resources to the task is evaluated based on the cache hit gain coefficient.

[0011] In this scheme, the cache hit gain coefficient of each data exchange task in each computing node within the future scheduling window is obtained by fusing the resource availability prediction results, availability weights, and average cache hit rate. Specifically: Based on the resource availability prediction results, extract the memory availability curve and disk I / O availability curve related to cache performance. Perform integration on the memory availability curve and disk I / O availability curve within the future scheduling window to obtain the cache capacity support index of the computing node within the future scheduling window. Read the size of the data block corresponding to the cached data block identifier, multiply the data block size by the availability weight, and obtain the effective availability value of a single cached data block; Traverse all cached data block identifiers generated by data exchange tasks in the compute node, sum up the effective usable value of all cached data blocks, and obtain the total cached data value of the target data exchange task on the access node. The average cache hit rate data and the cache capacity support index are weighted and fused to calculate the predicted cache service capability coefficient of the access node in the future scheduling window. The total cached data value is multiplied by the predicted cache service capacity coefficient, and the calculation result is normalized to map the result to a preset gain coefficient range, so as to obtain the cache hit gain coefficient of each data exchange task in each computing node within the future scheduling window.

[0012] In this solution, the process of matching explicit demand and implicit attractiveness to tasks and resources, determining the affinity between data exchange tasks and computing node resources, and constructing a task-resource affinity matrix for the data exchange platform specifically involves: Each data exchange task in the data exchange task queue is used as a row element of the matrix, and each computing node in the data exchange platform is used as a column element of the matrix. For any data exchange task and any computing node, the explicit demand of the data exchange task for the computing node is multiplied by the implicit attraction of the computing node for the data exchange task to obtain the original affinity score of the task-node pair. The original affinity score is substituted into the S-shaped growth curve function for normalization, and the score is mapped to a preset continuous interval to obtain a standardized task-resource affinity value. All the calculated standardized task-resource affinity values ​​are filled into a two-dimensional matrix according to the row corresponding to the data exchange task and the column corresponding to the computing node, to construct the task-resource affinity matrix.

[0013] In this solution, the step of scheduling data exchange task queues based on the task-resource affinity matrix and constructing a data exchange task scheduling strategy specifically involves: The maximum affinity value corresponding to each data exchange task is extracted based on the task-resource affinity matrix. When the maximum affinity value is greater than the preset affinity threshold, the corresponding data exchange task is scheduled to the computing node pointed to by the maximum affinity value. When a computing node is selected by multiple data exchange tasks at the same time, it is designated as a multi-task computing node. The total resource supply within the future scheduling window is calculated based on the resource availability prediction results of the multi-task computing node. Based on the task characteristics, extract the resource requirement vector of the data exchange task of each selected multi-task computing node, and sum up the resource requirements of all selected tasks for each type of resource to obtain the total resource requirement of the data exchange task of the selected multi-task computing node. When the total resource demand exceeds the total resource supply, the data exchange tasks in the multi-task computing node are sorted from low to high according to the affinity value to the multi-task computing node based on the task-resource affinity matrix. The sorted data exchange tasks are called out in order until the total resource demand of the data exchange tasks in the sorted list is less than the total resource supply of the multi-task computing node. The data exchange tasks that were called out are returned to the queue of tasks to be scheduled, and the affinity value of the task-resource affinity matrix is ​​updated according to the resource status after the data exchange tasks are allocated to each computing node. The updated queue of tasks to be scheduled is rescheduled according to the updated task-resource affinity matrix until all data exchange tasks are successfully scheduled, thus forming the data exchange task scheduling strategy.

[0014] A second aspect of the present invention also provides a data exchange task scheduling system based on resource prediction. The system includes a memory and a processor. The memory includes a data exchange task scheduling method program based on resource prediction. When the processor executes the data exchange task scheduling method program based on resource prediction, it performs the following steps: The resource status change vectors of each computing node in the data exchange platform within a preset time period are obtained. A resource prediction model is constructed based on the resource status change vectors. The resource availability curve of each computing node in the future scheduling window is predicted based on the resource prediction model, and the resource availability prediction result of each computing node is obtained. Obtain the data exchange task queue to be scheduled from the data exchange platform, extract the task description information of each data exchange task in the data exchange task queue, and determine the task characteristics of each data exchange task based on the task description information. The explicit resource requirements of each data exchange task and the implicit attractiveness of resources to each computing node are determined based on the task characteristics of each data exchange task and the resource availability prediction results of each computing node. Task-resource matching is performed on the explicit demand and implicit attractiveness to determine the affinity between data exchange tasks and computing node resources, and a task-resource affinity matrix of the data exchange platform is constructed. Based on the task-resource affinity matrix, the data exchange task queue is scheduled to construct a data exchange task scheduling strategy.

[0015] This invention discloses a data exchange task scheduling method and system based on resource prediction. By acquiring the resource status change vector of computing nodes within a preset time period, a resource prediction model is constructed to predict resource availability within the future scheduling window. A queue of data exchange tasks to be scheduled is obtained, and the task characteristics of each task are determined. Based on the task characteristics and resource availability prediction results, the explicit demand of tasks for resources and the implicit attraction of resources to tasks are calculated, and task-resource matching is performed to obtain a task-resource affinity matrix. A scheduling strategy is generated based on the affinity matrix to realize data exchange task scheduling. This invention improves resource utilization, avoids scheduling conflicts and resource bottlenecks, and enhances the overall performance of the data exchange platform by introducing a resource availability prediction mechanism. Attached Figure Description

[0016] Figure 1 A flowchart of a data exchange task scheduling method based on resource prediction according to the present invention is shown; Figure 2 A flowchart illustrating the resource availability prediction results for each computing node obtained by this invention is shown. Figure 3 The flowchart illustrating the construction of the task-resource affinity matrix according to the present invention is shown; Figure 4 A block diagram of a data exchange task scheduling system based on resource prediction according to the present invention is shown. Detailed Implementation

[0017] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0018] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0019] Figure 1 A flowchart of a data exchange task scheduling method based on resource prediction according to the present invention is shown.

[0020] like Figure 1 As shown, the first aspect of the present invention provides a data exchange task scheduling method based on resource prediction, comprising: S102, obtain the resource status change vector of each computing node in the data exchange platform within a preset time period, construct a resource prediction model based on the resource status change vector, predict the resource availability curve of each computing node in the future scheduling window based on the resource prediction model, and obtain the resource availability prediction result of each computing node. S104, Obtain the data exchange task queue to be scheduled on the data exchange platform, extract the task description information of each data exchange task in the data exchange task queue, and determine the task characteristics of each data exchange task based on the task description information. S106, Based on the task characteristics of each data exchange task and the resource availability prediction results of each computing node, determine the explicit resource requirement of the task and the implicit attractiveness of the resource to the task. S108, perform task-resource matching on the explicit demand and implicit attractiveness, determine the affinity between data exchange tasks and computing node resources, and construct the task-resource affinity matrix of the data exchange platform; S110, Perform task scheduling on the data exchange task queue according to the task-resource affinity matrix, and construct a data exchange task scheduling strategy.

[0021] It should be noted that by acquiring the resource status change vectors of each computing node in the data exchange platform within a preset time period and constructing a resource prediction model, the resource availability trend of each node within the future scheduling window can be grasped in advance. By analyzing the data exchange task queue and extracting task description information to construct task characteristics, the differences in resource requirements of different tasks in terms of computing, storage, and network bandwidth can be accurately characterized. By combining task characteristics and resource availability prediction results, the explicit demand for resources by tasks and the implicit attraction of resources to tasks can be calculated, taking into account both the direct occupation of resources by tasks and potential gain factors such as the historical execution advantages of nodes and cache reuse, so as to achieve a quantitative assessment of the comprehensive correlation between tasks and resources. The explicit demand and implicit attraction are matched and a task-resource affinity matrix is ​​constructed to reflect the optimal fit between each data exchange task and each computing node. Finally, the scheduling strategy is executed based on the task-resource affinity matrix, which can prioritize the allocation of data exchange tasks to nodes with sufficient resources and execution advantages, effectively reducing task waiting time and migration costs, improving resource utilization, and enhancing data exchange processing efficiency.

[0022] Figure 2 A flowchart illustrating the resource availability prediction results for each computing node obtained by this invention is shown.

[0023] According to an embodiment of the present invention, the method of obtaining the resource status change vectors of each computing node in the data exchange platform within a preset time period, constructing a resource prediction model based on the resource status change vectors, and predicting the resource availability curve of each computing node in a future scheduling window based on the resource prediction model, thereby obtaining the resource availability prediction result for each computing node, specifically includes: The resource status change vector of each computing node of the data exchange task platform is obtained within a preset time period. The resource status change vector includes changes in CPU utilization, memory utilization, network bandwidth utilization, and disk input / output load. A resource prediction model is constructed based on a long short-term memory network. The input layer dimension, hidden layer dimension, and output layer dimension of the resource prediction model are set, and the mean squared error is used as the loss function of the resource prediction model. The resource state change vector of each computing node is input into the resource prediction model for training. The weight parameters of the resource prediction model are optimized by the backpropagation algorithm until the loss function converges, and the trained resource prediction model is obtained. The current resource status vector of each computing node in the data exchange task platform is imported into the trained resource prediction model to predict the resource status changes of each computing node in the future scheduling window. Based on the resource status changes, the resource availability curve of each computing node is determined. The resource availability curve includes the CPU availability curve, memory availability curve, network bandwidth availability curve, and disk I / O availability curve, thus obtaining the resource availability prediction result.

[0024] It should be noted that by introducing a Long Short-Term Memory (LSTM) network to construct a resource prediction model to learn and predict the resource state change vectors of each computing node, the dynamic correlation and trend characteristics of multi-dimensional resources such as CPU, memory, network bandwidth, and disk I / O over time can be effectively captured. Based on the current resource state input, the trained model can output availability curves for various resources within the future scheduling window, achieving refined prediction of node resource supply capacity. This allows for the early identification of resource bottleneck nodes and potential load risks before task scheduling, reducing scheduling failures or delays caused by sudden resource shortages.

[0025] According to an embodiment of the present invention, the step of obtaining the data exchange task queue to be scheduled by the data exchange platform, extracting the task description information of each data exchange task in the data exchange task queue, and determining the task characteristics of each data exchange task based on the task description information specifically includes: Obtain the data exchange task queue to be scheduled on the data exchange platform, parse the task configuration file of each data exchange task in the data exchange task queue, and extract the task description information in the task configuration file. The task description information includes the data source address, data target address, data size, task priority identifier, task deadline timestamp, and data exchange protocol type of the data exchange task. The task description information is structured and parsed, and the data source address field and data target address field are converted into geographic coordinates to obtain the geographic location information of the data exchange task. The data transmission network path length of the data exchange task is calculated based on the geographic location information. The minimum required transmission rate for the data exchange task is calculated based on the data size field and the task deadline timestamp field. The task priority identifier field is then numerically mapped to obtain the task priority value. The communication protocol used by the data exchange task is determined based on the data exchange protocol type field, and the protocol overhead coefficient corresponding to the communication protocol is obtained. The data transmission network path length, minimum required transmission rate, task priority value, and protocol overhead coefficient are fused to construct the task characteristics of each data exchange task.

[0026] It should be noted that by parsing and extracting features from the task description information of data exchange tasks, the differences in data source and destination distribution, data volume, time sensitivity, priority requirements, and communication protocol complexity of each task can be comprehensively characterized. This accurately reflects the specific requirements of the task for computing resources, storage resources, and network bandwidth during execution. The task features are used to characterize the resource requirements of the data exchange task for computing nodes.

[0027] According to an embodiment of the present invention, determining the explicit resource requirement of a task and the implicit attractiveness of resources to a task based on the task characteristics of each data exchange task and the resource availability prediction results of each computing node specifically involves: Based on the task characteristics of each data exchange task, a task demand vector for computing resources is extracted. The demand vector includes the task's demand intensity for CPU computing power, memory capacity, network bandwidth, and disk I / O throughput. The explicit demand degree of the task for resources is calculated based on the demand vector. Each computing node is accessed sequentially, and the node that enters the access state is marked as the access node. The task execution history log of the access node is obtained, and the historical execution status of each data exchange task in the data exchange task queue of the access node is determined based on the task execution history log within a preset historical period. Obtain the identifier of the cached data block generated after the execution of the data exchange task with historical execution status and its corresponding last access timestamp, calculate the time interval between the current time and the last access timestamp, determine the availability weight of the cached data block based on the time interval, and at the same time, count the average cache hit rate data of the access node in the same historical time period. Based on the resource availability prediction results, availability weights and average cache hit rates, a fusion calculation is performed to obtain the cache hit gain coefficient of each data exchange task in each computing node within the future scheduling window. The implicit attractiveness of each compute node's resources to the task is evaluated based on the cache hit gain coefficient.

[0028] It's important to note that in a multi-task concurrent environment of a data exchange platform, different tasks have significantly different requirements for computing, storage, and network resources. However, traditional scheduling methods typically rely solely on current resource availability or task queue order, failing to quantify and identify the specific resource requirements of each task. Furthermore, data processing between distributed nodes exhibits historical correlation; some nodes may have cached data or possess experience executing similar tasks, giving them a natural advantage in specific tasks. If the same data is exchanged across different computing nodes, potential cache hit benefits are overlooked, leading to duplicate data loading and resource waste. During data exchange task execution, computing nodes often generate and retain some task-related data caches. When similar or identical data exchange tasks are rescheduled to the same node, historical cached data can be directly utilized, reducing data reloading, transmission, and decoding operations, thereby significantly improving execution efficiency. By analyzing the task execution history logs of nodes, the existence and recent access times of corresponding cached data blocks can be identified. Combining this with time intervals determines the availability of cached data, effectively reflecting its actual potential contribution to future task execution. Meanwhile, by statistically analyzing the average cache hit rate of nodes within similar time periods, the stability and reliability of node caching service capabilities can be further quantified. The cache hit gain coefficient, derived from the fusion of resource availability prediction results and cache availability, comprehensively reflects a node's ability to leverage historical cache to improve task execution speed and reduce resource consumption, thereby achieving a quantitative assessment of a node's execution advantages. Therefore, by utilizing the execution history correlation reflected in cache data, the implicit attractiveness of nodes to tasks can be objectively determined, enabling scheduling strategies to prioritize nodes with cache gain advantages for executing target tasks, thereby reducing task execution overhead and improving overall scheduling efficiency. The longer the time interval, the lower the availability of cached data blocks.

[0029] According to an embodiment of the present invention, the step of performing a fusion calculation based on the resource availability prediction result, availability weight, and average cache hit rate to obtain the cache hit gain coefficient of each data exchange task in each computing node within the future scheduling window is specifically as follows: Based on the resource availability prediction results, extract the memory availability curve and disk I / O availability curve related to cache performance. Perform integration on the memory availability curve and disk I / O availability curve within the future scheduling window to obtain the cache capacity support index of the computing node within the future scheduling window. Read the size of the data block corresponding to the cached data block identifier, multiply the data block size by the availability weight, and obtain the effective availability value of a single cached data block; Traverse all cached data block identifiers generated by data exchange tasks in the compute node, sum up the effective usable value of all cached data blocks, and obtain the total cached data value of the target data exchange task on the access node. The average cache hit rate data and the cache capacity support index are weighted and fused to calculate the predicted cache service capability coefficient of the access node in the future scheduling window. The total cached data value is multiplied by the predicted cache service capacity coefficient, and the calculation result is normalized to map the result to a preset gain coefficient range, so as to obtain the cache hit gain coefficient of each data exchange task in each computing node within the future scheduling window.

[0030] It should be noted that by integrating the memory availability curve and disk I / O availability curve within the future scheduling window, the total cache capacity that a node can sustainably provide throughout the entire window can be quantified. The average cache hit rate data refers to the statistical result of the proportion of successful cache hits for specific types or similar data exchange tasks during historical execution. It reflects the efficiency and stability of the node in fulfilling task requests using cached data blocks over a period of time, i.e., the frequency with which data does not need to be reloaded from remote storage or the network during task execution. By statistically analyzing the ratio of cache hits to total accesses under the same time period or similar task conditions, the average cache hit rate of the node within a specific task type or time window can be obtained.

[0031] Figure 3 The flowchart illustrating the construction of the task-resource affinity matrix according to the present invention is shown.

[0032] According to an embodiment of the present invention, the step of performing task-resource matching on the explicit demand and implicit attractiveness, determining the affinity between data exchange tasks and computing node resources, and constructing a task-resource affinity matrix for the data exchange platform specifically involves: Each data exchange task in the data exchange task queue is used as a row element of the matrix, and each computing node in the data exchange platform is used as a column element of the matrix. For any data exchange task and any computing node, the explicit demand of the data exchange task for the computing node is multiplied by the implicit attraction of the computing node for the data exchange task to obtain the original affinity score of the task-node pair. The original affinity score is substituted into the S-shaped growth curve function for normalization, and the score is mapped to a preset continuous interval to obtain a standardized task-resource affinity value. All the calculated standardized task-resource affinity values ​​are filled into a two-dimensional matrix according to the row corresponding to the data exchange task and the column corresponding to the computing node, to construct the task-resource affinity matrix.

[0033] It should be noted that the task-resource affinity matrix is ​​a two-dimensional matrix used to comprehensively quantify and represent the matching degree between each task and each computing node in the data exchange platform. The rows of the matrix correspond to the data exchange tasks to be scheduled, and the columns correspond to the computing nodes in the platform. Each matrix element is obtained by multiplying the explicit resource requirement of the task on the node by the implicit attraction of the node to the task and then normalizing it using an S-shaped growth curve, reflecting the overall adaptability or affinity of the task on that node.

[0034] According to an embodiment of the present invention, the step of scheduling data exchange task queues based on the task-resource affinity matrix and constructing a data exchange task scheduling strategy specifically includes: The maximum affinity value corresponding to each data exchange task is extracted based on the task-resource affinity matrix. When the maximum affinity value is greater than the preset affinity threshold, the corresponding data exchange task is scheduled to the computing node pointed to by the maximum affinity value. It should be noted that by extracting the maximum affinity value corresponding to each data exchange task based on the task-resource affinity matrix, and prioritizing the scheduling of tasks with affinity values ​​greater than a preset threshold to the most suitable computing node, optimal matching between tasks and nodes can be achieved, thereby improving the data transmission efficiency of data exchange tasks.

[0035] When a computing node is selected by multiple data exchange tasks at the same time, it is designated as a multi-task computing node. The total resource supply within the future scheduling window is calculated based on the resource availability prediction results of the multi-task computing node. Based on the task characteristics, extract the resource requirement vector of the data exchange task of each selected multi-task computing node, and sum up the resource requirements of all selected tasks for each type of resource to obtain the total resource requirement of the data exchange task of the selected multi-task computing node. When the total resource demand exceeds the total resource supply, the data exchange tasks in the multi-task computing node are sorted from low to high according to the affinity value to the multi-task computing node based on the task-resource affinity matrix. The sorted data exchange tasks are called out in order until the total resource demand of the data exchange tasks in the sorted list is less than the total resource supply of the multi-task computing node. The data exchange tasks that were called out are returned to the queue of tasks to be scheduled, and the affinity value of the task-resource affinity matrix is ​​updated according to the resource status after the data exchange tasks are allocated to each computing node. The updated queue of tasks to be scheduled is rescheduled according to the updated task-resource affinity matrix until all data exchange tasks are successfully scheduled, thus forming the data exchange task scheduling strategy.

[0036] It's important to note that during task scheduling of the data exchange task queue based on the task-resource affinity matrix, multiple tasks may simultaneously select the same computing node for scheduling, causing the node to become a multi-task computing node. Directly scheduling all tasks to this node may lead to insufficient resource supply, such as insufficient CPU, memory, network bandwidth, or disk I / O to meet the concurrent demands of all tasks, resulting in task delays, execution failures, or node overload. By dynamically assessing the resource supply and demand of multi-task computing nodes and comparing the total resource supply of the node with the cumulative task demand, the risk of resource overload can be detected in real time. Furthermore, by combining the task-resource affinity matrix, tasks with low affinity are sequentially removed and re-entered into the scheduling queue, achieving dynamic reallocation of tasks among nodes. This ensures that node resources are rationally utilized within future scheduling windows, avoiding overload or resource waste, while balancing task distribution and improving task execution success rate and overall scheduling efficiency. By iteratively updating the affinity matrix and rescheduling, it ensures that all tasks are effectively allocated, achieving an efficient, stable, and predictable task scheduling strategy for the data exchange platform.

[0037] Figure 4 A block diagram of a data exchange task scheduling system based on resource prediction according to the present invention is shown.

[0038] A second aspect of the present invention also provides a data exchange task scheduling system based on resource prediction. The system includes a memory 401, a processor 402, and a communication interface 403. The memory includes a data exchange task scheduling method program based on resource prediction. The communication interface is used for data connection communication between the memory and the processor. When the data exchange task scheduling method program based on resource prediction is executed by the processor, it performs the following steps: The resource status change vectors of each computing node in the data exchange platform within a preset time period are obtained. A resource prediction model is constructed based on the resource status change vectors. The resource availability curve of each computing node in the future scheduling window is predicted based on the resource prediction model, and the resource availability prediction result of each computing node is obtained. Obtain the data exchange task queue to be scheduled from the data exchange platform, extract the task description information of each data exchange task in the data exchange task queue, and determine the task characteristics of each data exchange task based on the task description information. The explicit resource requirements of each data exchange task and the implicit attractiveness of resources to each computing node are determined based on the task characteristics of each data exchange task and the resource availability prediction results of each computing node. Task-resource matching is performed on the explicit demand and implicit attractiveness to determine the affinity between data exchange tasks and computing node resources, and a task-resource affinity matrix of the data exchange platform is constructed. Based on the task-resource affinity matrix, the data exchange task queue is scheduled to construct a data exchange task scheduling strategy.

[0039] This invention discloses a data exchange task scheduling method and system based on resource prediction. By acquiring the resource status change vector of computing nodes within a preset time period, a resource prediction model is constructed to predict resource availability within the future scheduling window. A queue of data exchange tasks to be scheduled is obtained, and the task characteristics of each task are determined. Based on the task characteristics and resource availability prediction results, the explicit demand of tasks for resources and the implicit attraction of resources to tasks are calculated, and task-resource matching is performed to obtain a task-resource affinity matrix. A scheduling strategy is generated based on the affinity matrix to realize data exchange task scheduling. This invention improves resource utilization, avoids scheduling conflicts and resource bottlenecks, and enhances the overall performance of the data exchange platform by introducing a resource availability prediction mechanism.

[0040] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0041] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0042] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0043] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0044] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0045] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for resource prediction based data exchange task scheduling, characterized in that, The method comprises the following steps: obtaining resource state change vectors of each computing node of the data exchange platform within a preset time period, constructing a resource prediction model according to the resource state change vectors, predicting resource availability curves of each computing node within a future scheduling window according to the resource prediction model, and obtaining resource availability prediction results of each computing node; obtaining a data exchange task queue to be scheduled by the data exchange platform, extracting task description information of each data exchange task in the data exchange task queue, and determining task characteristics of each data exchange task according to the task description information; determining explicit demand degrees of tasks for resources and implicit attraction degrees of resources for tasks according to the task characteristics of each data exchange task and the resource availability prediction results of each computing node; performing task and resource matching on the explicit demand degrees and the implicit attraction degrees, determining affinities between data exchange tasks and computing node resources, and constructing a task-resource affinity matrix of the data exchange platform; performing task scheduling on the data exchange task queue according to the task-resource affinity matrix, and constructing a data exchange task scheduling strategy. 2.The method of claim 1, wherein, The method comprises the following steps: obtaining resource state change vectors of each computing node of the data exchange platform within a preset time period, constructing a resource prediction model according to the resource state change vectors, predicting resource availability curves of each computing node within a future scheduling window according to the resource prediction model, and obtaining resource availability prediction results of each computing node, specifically: obtaining resource state change vectors of each computing node of the data exchange task platform within a preset time period, wherein the resource state change vectors include CPU utilization change, memory usage change, network bandwidth occupancy change, and disk input / output load change; constructing a resource prediction model based on a long short-term memory network, setting input layer dimensions, hidden layer dimensions, and output layer dimensions of the resource prediction model, and setting a mean square error as a loss function of the resource prediction model; inputting the resource state change vectors of each computing node into the resource prediction model for training, optimizing weight parameters of the resource prediction model through a back propagation algorithm until the loss function converges, and obtaining a trained resource prediction model; 3.The method of claim 1, wherein, inputting current resource state vectors of each computing node of the data exchange task platform into the trained resource prediction model, predicting resource state changes of each computing node within a future scheduling window, determining resource availability curves of each computing node according to the resource state changes, wherein the resource availability curves include CPU availability curves, memory availability curves, network bandwidth availability curves, and disk I / O availability curves, and obtaining resource availability prediction results. The method comprises the following steps: obtaining resource state change vectors of each computing node of the data exchange platform within a preset time period, constructing a resource prediction model according to the resource state change vectors, predicting resource availability curves of each computing node within a future scheduling window according to the resource prediction model, and obtaining resource availability prediction results of each computing node, specifically: Obtaining a data exchange task queue to be scheduled by a data exchange platform, parsing a task configuration file of each data exchange task in the data exchange task queue, extracting task description information in the task configuration file, the task description information including a data source address, a data target address, a data volume size, a task priority identifier, a task deadline timestamp, and a data exchange protocol type of the data exchange task; Structurally analyzing the task description information, performing geographic coordinate conversion on the data source address field and the data target address field to obtain geographic location information of the data exchange task, and calculating a data transmission network path length of the data exchange task according to the geographic location information; Calculating a minimum required transmission rate of the data exchange task according to the data volume size field and the task deadline timestamp field, and obtaining a task priority value by numerically mapping the task priority identifier field; Determining a communication protocol adopted by the data exchange task according to the data exchange protocol type field, and obtaining a protocol overhead coefficient corresponding to the communication protocol; Feature fusion is performed on the data transmission network path length, the minimum required transmission rate, the task priority value, and the protocol overhead coefficient to construct a task feature of each data exchange task.

4. The method of claim 1, wherein, The explicit demand degree of the task for the resource and the implicit attraction of the resource for the task are determined according to the task feature of each data exchange task and the resource availability prediction result of each computing node, specifically as follows: A demand vector of the task for the computing resource is extracted according to the task feature of each data exchange task, the demand vector including a demand intensity of the task for CPU computing capability, a demand intensity of the task for memory capacity, a demand intensity of the task for network bandwidth, and a demand intensity of the task for disk I / O throughput, and the explicit demand degree of the task for the resource is calculated based on the demand vector; Each computing node is accessed in turn, a node entering an access state is marked as an access node, a task execution history log of the access node is obtained, and a historical execution situation of the access node for each data exchange task in the data exchange task queue in a preset historical period is determined according to the task execution history log; A cache data block identifier generated after execution of the data exchange task with the historical execution situation and a corresponding last access timestamp are obtained, a time interval between a current time and the last access timestamp is calculated, the availability weight of the cache data block is determined according to the time interval, and average cache hit rate data of the access node in the historical same period is simultaneously counted; The resource availability prediction result, the availability weight, and the average cache hit rate are fused and calculated to obtain a cache hit gain coefficient of each data exchange task in each computing node in a future scheduling window; The implicit attraction of the resource for the task is evaluated according to the cache hit gain coefficient.

5. The method of claim 4, wherein, The resource availability prediction result, the availability weight, and the average cache hit rate are fused and calculated to obtain a cache hit gain coefficient of each data exchange task in each computing node in a future scheduling window, specifically as follows: According to the resource availability prediction result, a memory availability curve and a disk I / O availability curve related to cache performance are extracted, and the memory availability curve and the disk I / O availability curve are integrated in a future scheduling window to obtain a cache capacity support index of the computing node in the future scheduling window; The data block size corresponding to the cache data block identifier is read, the data block size is multiplied by the availability weight to obtain the effective availability value of a single cache data block; All cache data block identifiers generated by data exchange tasks in the computing node are traversed, and the effective availability values of all cache data blocks are accumulated to obtain the total cache data value of the target data exchange task on the access node; The average cache hit rate data and the cache capacity support index are weighted and fused to calculate a predicted cache service capability coefficient of the access node in the future scheduling window; The total cache data value and the predicted cache service capability coefficient are multiplied, and the calculation result is normalized to map the result to a preset gain coefficient interval to obtain a cache hit gain coefficient of each data exchange task in each computing node in the future scheduling window.

6. The method of claim 1, wherein, The explicit demand degree and the implicit attraction are matched with tasks and resources to determine the affinity between the data exchange task and the computing node resource, and a task-resource affinity matrix of the data exchange platform is constructed, specifically: Each data exchange task in the data exchange task queue is taken as a row element of the matrix, and each computing node in the data exchange platform is taken as a column element of the matrix; For any one data exchange task and any one computing node, the explicit demand degree of the data exchange task to the computing node and the implicit attraction of the computing node to the data exchange task are multiplied to obtain an original affinity score of the task-node pair; The original affinity score is substituted into an S-shaped growth curve function for normalization processing to map the score to a preset continuous interval to obtain a standardized task-resource affinity value; All calculated standardized task-resource affinity values are filled into a two-dimensional matrix in a manner that rows correspond to data exchange tasks and columns correspond to computing nodes to construct the task-resource affinity matrix.

7. The method of claim 1, wherein, According to the task-resource affinity matrix, the data exchange task queue is scheduled to construct a data exchange task scheduling strategy, specifically: According to the task-resource affinity matrix, the maximum affinity value corresponding to each data exchange task is extracted, and when the maximum affinity value is greater than a preset affinity threshold, the corresponding data exchange task is scheduled to the computing node pointed to by the maximum affinity value; When a computing node is simultaneously selected by multiple data exchange tasks, the computing node is designated as a multi-task computing node, and the total resource supply in the future scheduling window is calculated according to the resource availability prediction result of the multi-task computing node; According to the task characteristics, a resource demand vector of the data exchange task of each selected multi-task computing node is extracted, and the demand amounts of all selected tasks for various resources are accumulated to obtain the total resource demand amount of the data exchange task of the selected multi-task computing node; When the total resource demand exceeds the total resource supply, the data exchange tasks in the multi-task computing node are sorted according to the affinity values from low to high according to the task-resource affinity matrix, and the sorted data exchange tasks are called out in order until the total resource demand of the data exchange tasks in the sorting list is less than the total resource supply of the multi-task computing node; The called-out data exchange tasks are reclassified into the task queue to be scheduled, and the affinity values of the task-resource affinity matrix are updated according to the resource state of each computing node after the data exchange tasks are allocated; The updated task queue to be scheduled is rescheduled according to the updated task-resource affinity matrix until all data exchange tasks are successfully scheduled, forming a data exchange task scheduling strategy.

8. A resource prediction based data exchange task scheduling system, characterized in that, The resource prediction-based data exchange task scheduling system includes a storage and a processor, the storage includes a resource prediction-based data exchange task scheduling method program, and the resource prediction-based data exchange task scheduling method program is executed by the processor to realize the following steps: Obtain the resource state change vector of each computing node of the data exchange platform in a preset time period, construct a resource prediction model according to the resource state change vector, predict the resource availability curve of each computing node in the future scheduling window according to the resource prediction model, and obtain the resource availability prediction result of each computing node; Obtain the data exchange task queue to be scheduled of the data exchange platform, extract the task description information of each data exchange task in the data exchange task queue, and determine the task characteristics of each data exchange task according to the task description information; Determine the explicit demand degree of the task to the resource and the implicit attraction of the resource to the task according to the task characteristics of each data exchange task and the resource availability prediction result of each computing node; Match the explicit demand degree and the implicit attraction to determine the affinity between the data exchange task and the computing node resource, and construct the task-resource affinity matrix of the data exchange platform; Schedule the data exchange task queue according to the task-resource affinity matrix to construct a data exchange task scheduling strategy.

Images