A task dynamic allocation system and method for cloud platform data management

Abstract

The application discloses a kind of task dynamic allocation system and method for cloud platform data management, it is related to cloud table task allocation technical field, setting label generation mechanism and initial task allocation mechanism are used for the initial matching of task and computing node;Setting node running state and task priority analysis mechanism are used for monitoring the resource tension of computing node and obtaining the adjustment priority of task;Setting task allocation dynamic adjustment mechanism is used to detect whether there is idle computing node in computing node;Setting computing node running condition analysis mechanism is used to obtain the best dynamic adjustment standby node;Avoid the problem that overall task progress is seriously stagnated due to the stagnation of key task or the processing efficiency is too low, obtain the best dynamic adjustment standby node with preemption condition in standby node, improve resource utilization and overall task execution efficiency.

Description

Technical Field

[0001] This invention relates to the field of gimbal task allocation technology, specifically a dynamic task allocation system and method for cloud platform data management. Background Technology

[0002] With the rapid development of cloud computing technology, the data processing tasks carried by cloud platforms are becoming increasingly complex, and the number of tasks and the scale of computing nodes are continuously expanding. This places higher demands on the rationality, stability, and fault tolerance of task allocation. Currently, task allocation technologies in cloud platform data management generally have significant shortcomings: fault tolerance methods are relatively passive, lacking the ability to predict potential failures. Most rely on post-event retries and migrations, failing to anticipate anomalies such as impending computing node crashes, disk storage exhaustion, or network congestion. These shortcomings directly lead to frequent task interruptions, data loss, and redundant calculations during cloud platform operation. When a computing node suddenly crashes, tasks running on that node are forced to stop, and without timely backups, data loss occurs. Upon restarting, the tasks must be re-executed. This leads to redundant calculations; when disk storage is nearing full capacity or network congestion occurs, task execution efficiency drops significantly, and tasks may even freeze or crash, affecting not only the overall task completion progress but also causing serious waste of cloud platform computing, storage, and network resources; existing technologies lack proper handling techniques for critical tasks under resource constraints in platform task allocation, typically employing a crude task preemption mechanism where high-priority tasks directly preempt low-priority tasks, which easily leads to cascading failures; after low-priority tasks are repeatedly interrupted, recalculation and rollback operations are required, increasing redundant computation and potentially causing problems such as incomplete data writing and inconsistent task states, leading to numerous retries, further consuming system resources and reducing execution efficiency. Summary of the Invention

[0003] The purpose of this invention is to provide a dynamic task allocation system and method for cloud platform data management, so as to solve the problems raised in the prior art.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a method for dynamic task allocation in cloud platform data management, the method comprising the following: A tag generation mechanism is set up to analyze the received gimbal tasks and gimbal computing node operating parameters, and to tag the gimbal tasks with computing feature tags and the computing nodes with capability tags. An initial task allocation mechanism is set up to analyze the matching values ​​between tasks and computing nodes, and to perform initial matching of tasks and computing nodes based on the matching values ​​between each task and each computing node. The node running status and task priority analysis mechanism is set up to monitor the resource stress of computing nodes in real time after the initial task allocation is completed, and obtain the task adjustment priority by analyzing the execution status of the tasks to be adjusted. The task allocation dynamic adjustment mechanism is set up to detect the running status of computing nodes, detect whether there are idle computing nodes, and trigger the computing node running status analysis mechanism when there are no idle computing nodes that are suitable for the task to be adjusted. The compute node operation analysis mechanism is set up to analyze and obtain dynamically adjusted backup nodes that are suitable for high-priority tasks that need to be adjusted in compute nodes running low-priority tasks. After obtaining the dynamically adjusted backup nodes, the task preemption control mechanism is triggered to analyze the optimal dynamically adjusted backup nodes.

[0005] Furthermore, the tag generation mechanism is used to generate computational feature tags for tasks and capability tags for computing nodes; obtain the tasks to be assigned and related information from the cloud platform data management, extract the computational features of each task, and assign a unique computational feature tag to each task; the computational features include the task's computing power requirements, task type, task priority, task dependencies, estimated task execution time, and data processing scale; obtain the hardware parameters, software configuration, and operating status information of all computing nodes in the cloud platform, extract the capability features of each computing node, and assign a unique capability tag to each computing node; the capability features include node computing power, memory capacity, storage capacity, network bandwidth, current load, and historical execution efficiency.

[0006] Furthermore, the reward function analysis mechanism is used to analyze the matching reward value between tasks and computing nodes; it collects the adaptation degree δ1 between task computing power requirements and node computing power, the matching degree δ2 between task computing feature tags and node capability tags, the overall task completion time optimization coefficient δ3, and the resource utilization coefficient δ4. The values ​​of the above four parameters are all in the range of [0, 1]. It sets the task computing power requirement adaptation coefficient γ1, the bidirectional tag matching degree coefficient γ2, the overall task completion time optimization coefficient γ3, and the resource utilization coefficient γ4, and γ1+γ1+γ1+γ1=1. It merges the four parameters to obtain the matching value between tasks and computing nodes; and matches each task allocation value with the computing node with the highest value according to the obtained matching value between tasks and computing nodes, thereby completing the initial task allocation.

[0007] Furthermore, the node running status and task priority analysis mechanism is used to monitor the running computing nodes in real time after the initial task allocation is completed. By collecting the CPU utilization, memory utilization and network bandwidth occupancy of the running computing nodes in real time, the resource utilization of the computing nodes is analyzed, and the resource utilization of the computing nodes is used to determine whether there is resource shortage. After determining that there is resource shortage, the value of the task to be adjusted, the execution time and the remaining execution time are combined to obtain the adjustment priority of the task.

[0008] Furthermore, the CPU utilization of the computing node collected in real time is recorded as r1, the memory utilization as r2, and the network bandwidth occupancy as r3. The resource stress of the computing node is obtained by weighted fusion of the three parameters, and a resource stress threshold α is set. The resource stress is then calculated according to the following formula: Where k1 represents the weight of CPU utilization; k2 represents the weight of memory utilization; and k3 represents the weight of network bandwidth utilization. The resource stress of the real-time monitored computing nodes is compared and analyzed with the set thresholds. The results are as follows: If P≤α, it means that the resource stress is within the threshold range, and no dynamic adjustment will be made to the task allocation; If P>α, it means that the resource stress has exceeded the threshold, so the task allocation will be dynamically adjusted, triggering the task adjustment priority analysis mechanism. When resource shortages are detected, the priority of tasks that need to be adjusted and replaced with computing nodes are dynamically adjusted and calculated in a timely manner. High-priority tasks are acquired in a timely manner and available computing nodes are found for high-priority critical tasks, which ensures the smooth execution of critical tasks and avoids the problem of severe stagnation of the overall task progress due to the stagnation of critical tasks or low processing efficiency. For all tasks in execution, a dynamic adjustment priority analysis for task reallocation is performed. The task value u1, execution time u2, and remaining execution time u3 of each task in execution are collected. The dynamic adjustment priority for task reallocation is calculated according to the following formula: ; Where φ represents the dynamic adjustment priority of task reallocation; Set a dynamic adjustment priority threshold φ' for task reallocation, and classify the tasks to be adjusted into different levels. The classification results are as follows: If φ≥φ', it means that the dynamic adjustment priority of task reallocation is high, and the corresponding task will be marked as a high-priority task. If φ < φ', it means that the dynamic adjustment priority of task reallocation is low, and the corresponding task will be marked as a low-priority task.

[0009] Further, the dynamic task allocation adjustment mechanism is configured to detect the operating status of computing nodes, detect whether there is an idle computing node among the computing nodes, and analyze whether the matching degree between the idle computing node and the task to be adjusted meets the standard, so as to determine whether there is an idle computing node adapted to the task to be adjusted. If there is no idle computing node adapted to the task to be adjusted, the computing node operating condition analysis mechanism is triggered.

[0010] Further, for dynamic adjustment of task reallocation for the analyzed high-priority tasks, first, the operating status of computing nodes is detected, and whether there is an idle computing node is analyzed. If there is an idle computing node, the matching degree between the idle computing node and each high-priority task to be adjusted is analyzed, a high matching degree threshold M0 is set, the matching degree between the idle computing node and the task to be adjusted is judged, and the analysis results are as follows: If M≥M0, it indicates that the matching degree between the idle computing node and the task to be adjusted is high, and the high-priority task to be adjusted is allocated to the idle node; this step first detects whether there is a computing node that can match the task to be adjusted among the idle computing nodes, which improves the utilization rate of idle resources; If M<M0, it indicates that the matching degree between the idle computing node and the task to be adjusted is low, and the computing node operating condition analysis mechanism is triggered; If there is no idle computing node, the computing node operating condition analysis mechanism is triggered directly.

[0011] Further, the computing node operating condition analysis mechanism is configured to, when there is no idle computing node and there is a high-priority task to be adjusted, analyze dynamically adjusted spare nodes adapted to the high-priority task to be adjusted among computing nodes running low-priority tasks, and trigger the task preemption control mechanism after obtaining the dynamically adjusted spare nodes. The task preemption control mechanism is configured to obtain an optimal dynamically adjusted spare node from the obtained dynamically adjusted spare nodes when a high-priority task preempts the computing node resources of a low-priority task.

[0012] Further, the computing node operating condition analysis mechanism is configured to analyze the matching degree between the operating computing nodes and the high-priority task, mark the computing nodes with high matching degree as candidate computing nodes, analyze the dynamic adjustment priority φ of the tasks running on the candidate computing nodes, and obtain dynamically adjusted spare computing nodes according to the analysis results; the analysis results are as follows: If φ<φ', the node is marked as a dynamically adjusted spare node, which is used for dynamic adjustment and reserve allocation of resources for the high-priority task to be adjusted; If φ≥φ', the node is not marked as a dynamically adjusted spare computing node; The dynamically adjusted backup nodes are denoted as {b1, b2, ..., bm} in descending order of their matching degree with the task to be adjusted, where m represents the total number of dynamically adjusted backup nodes. After acquiring the dynamically adjusted backup nodes, a task preemption control mechanism is triggered. This mechanism detects and controls the preemption of node computing resources by high-priority tasks from low-priority tasks. First, the running status of tasks on the dynamically adjusted backup computing node b1 is analyzed. The number of times a low-priority task running on the dynamically adjusted backup node b1 has been preempted within the past duration Δd is recorded as n. The remaining execution time Δt of the running low-priority task is analyzed. A maximum threshold of n0 is set for the number of times a low-priority task has been preempted within duration Δd, and a minimum threshold of Δt0 is set for the remaining execution time of the task. Real-time data is compared and analyzed with the set thresholds. If n>n0 or Δt≥Δt0, it means that the standby computing node b1 no longer meets the preemption conditions. In this case, b1 is removed from the standby nodes, and the analysis continues to determine whether the next dynamically adjusted standby node meets the preemption conditions. If n≤n0 and Δt<Δt0, it means that the standby computing node b1 is ready to be preempted. In this case, b1 is selected as the optimal dynamically adjusted standby node, and low-priority tasks in b1 are migrated to other computing nodes with lower loads, reserving resources for high-priority tasks. The new execution status of low-priority tasks in computing nodes is analyzed, and computing nodes containing low-priority tasks with high execution progress or those that have been preempted are excluded. The optimal dynamically adjusted standby node with preemption is obtained from all standby nodes, which improves resource utilization and overall task execution efficiency. If no computing node with preemption conditions is obtained after analyzing m dynamically adjusted standby nodes, the low-priority task with the shortest remaining execution time among the m standby nodes is selected as the computing node to be preempted, and the waiting time is set to the same as the remaining execution time. The computing node will be preempted after the low-priority task in the computing node to be preempted is completed.

[0013] A dynamic task allocation system for cloud platform data management includes a task tag generation module, an initial task allocation module, a node running status and task priority analysis module, a dynamic task allocation adjustment module, and a computing node running status analysis module. The task tag generation module is used to analyze the received gimbal tasks and gimbal computing node operating parameters, and to tag the gimbal tasks with computing feature tags and the computing nodes with capability tags. The initial task allocation module is used to analyze the matching value between tasks and computing nodes, and to perform initial matching of tasks and computing nodes based on the matching value between each task and each computing node. The node running status and task priority analysis module is used to monitor the resource stress of computing nodes in real time after the initial task allocation is completed, and to obtain the task adjustment priority by analyzing the execution status of the tasks to be adjusted. The task allocation dynamic adjustment module is used to detect the running status of computing nodes, detect whether there are idle computing nodes, and trigger the computing node running status analysis mechanism when there are no idle computing nodes that are compatible with the task to be adjusted. The computing node operation analysis module is used to analyze and obtain dynamically adjusted backup nodes that are compatible with the high-priority tasks to be adjusted in computing nodes where low-priority tasks are running, and after obtaining the dynamically adjusted backup nodes, trigger the task preemption control mechanism to analyze the optimal dynamically adjusted backup nodes.

[0014] Compared with the prior art, the beneficial effects of the present invention are: This invention utilizes a node operation status and task priority analysis mechanism to monitor the resource stress of computing nodes in real time. Upon detecting resource stress, it dynamically adjusts the priority of tasks requiring node replacement, promptly identifying high-priority tasks and finding available computing nodes for them. This ensures the smooth execution of critical tasks and avoids significant overall task progress stagnation caused by critical task stalls or low processing efficiency. Furthermore, the invention employs a dynamic task allocation adjustment mechanism to detect and analyze computing nodes. First, it checks if any idle computing nodes can match the tasks requiring adjustment, improving the utilization of idle resources. Finally, it includes a computing node operation status analysis mechanism to address situations where there are no idle computing nodes and high-priority tasks requiring adjustment exist. In the case of high-priority tasks, other computing nodes executing the task are analyzed to obtain dynamically adjusted backup nodes that are compatible with the high-priority computing nodes to be adjusted. After obtaining the backup nodes, a task preemption control mechanism is triggered to analyze the operation of the dynamically adjusted backup nodes. The analysis examines the execution status of low-priority tasks in the computing nodes, excluding computing nodes containing low-priority tasks with high execution progress or those that have been preempted. The best dynamically adjusted backup node with preemption conditions is obtained from all backup nodes, which improves resource utilization and overall task execution efficiency. This avoids the problem of high-priority tasks directly preempting computing nodes of low-priority tasks, which would cause low-priority tasks to be repeatedly interrupted, resulting in a large number of retries, further consuming system resources and reducing execution efficiency. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating a method for dynamic task allocation in cloud platform data management according to the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] like Figure 1 As shown, the present invention provides a technical solution, a method for dynamic task allocation in cloud platform data management, the method comprising the following: A tag generation mechanism is set up to analyze the received gimbal tasks and gimbal computing node operating parameters, and to tag the gimbal tasks with computing feature tags and the computing nodes with capability tags. An initial task allocation mechanism is set up to analyze the matching values ​​between tasks and computing nodes, and to perform initial matching of tasks and computing nodes based on the matching values ​​between each task and each computing node. The node running status and task priority analysis mechanism is set up to monitor the resource stress of computing nodes in real time after the initial task allocation is completed, and obtain the task adjustment priority by analyzing the execution status of the tasks to be adjusted. The task allocation dynamic adjustment mechanism is set up to detect the running status of computing nodes, detect whether there are idle computing nodes, and trigger the computing node running status analysis mechanism when there are no idle computing nodes that are suitable for the task to be adjusted. The compute node operation analysis mechanism is set up to analyze and obtain dynamically adjusted backup nodes that are suitable for high-priority tasks that need to be adjusted in compute nodes running low-priority tasks. After obtaining the dynamically adjusted backup nodes, the task preemption control mechanism is triggered to analyze the optimal dynamically adjusted backup nodes.

[0018] The tag generation mechanism is used to generate computational feature tags for tasks and capability tags for computing nodes; it obtains the tasks to be assigned and related information from the cloud platform's data management, extracts the computational features of each task, and assigns a unique computational feature tag to each task; the computational features include the task's computing power requirements, task type, task priority, task dependencies, estimated task execution time, and data processing scale; it obtains the hardware parameters, software configuration, and operating status information of all computing nodes in the cloud platform, extracts the capability features of each computing node, and assigns a unique capability tag to each computing node; the capability features include node computing power, memory capacity, storage capacity, network bandwidth, current load, and historical execution efficiency.

[0019] A reward function analysis mechanism is set up to analyze the matching reward value between tasks and computing nodes. The following parameters are collected: the adaptability δ1 between task computing power requirements and node computing power, the matching degree δ2 between task computing feature tags and node capability tags, the overall task completion time optimization coefficient δ3, and the resource utilization coefficient δ4. The values ​​of the above four parameters are all in the range of [0, 1]. The task computing power requirement adaptability coefficient γ1, the bidirectional tag matching degree coefficient γ2, the overall task completion time optimization coefficient γ3, and the resource utilization coefficient γ4 are set, and γ1+γ1+γ1+γ1=1. The four parameters are fused to obtain the matching value between tasks and computing nodes. Based on the obtained matching value between tasks and computing nodes, each task is assigned to the computing node with the highest value, thereby completing the initial task allocation.

[0020] The node running status and task priority analysis mechanism is used to monitor the running computing nodes in real time after the initial task allocation is completed. It analyzes the resource utilization of the computing nodes by collecting the CPU utilization, memory utilization and network bandwidth occupancy of the running computing nodes in real time, and judges whether there is resource pressure on the nodes based on the resource utilization. If resource pressure is determined, the value of the task to be adjusted, the execution time and the remaining execution time are combined to obtain the adjustment priority of the task.

[0021] The CPU utilization of the computing node collected in real time is recorded as r1, the memory utilization as r2, and the network bandwidth occupancy as r3. The resource stress P of the computing node is obtained by weighted fusion of the three parameters, and a resource stress threshold α is set. The resource stress is calculated according to the following formula: Where k1 represents the weight of CPU utilization; k2 represents the weight of memory utilization; and k3 represents the weight of network bandwidth utilization. The resource stress of the real-time monitored computing nodes is compared and analyzed with the set thresholds. The results are as follows: If P≤α, it means that the resource stress is within the threshold range, and no dynamic adjustment will be made to the task allocation; If P>α, it means that the resource stress has exceeded the threshold, so the task allocation will be dynamically adjusted, triggering the task adjustment priority analysis mechanism. For all tasks in execution, a dynamic adjustment priority analysis for task reallocation is performed. The task value u1, execution time u2, and remaining execution time u3 of each task in execution are collected. The dynamic adjustment priority for task reallocation is calculated according to the following formula: ; Where φ represents the dynamic adjustment priority of task reallocation; Set a dynamic adjustment priority threshold φ' for task reallocation, and classify the tasks to be adjusted into different levels. The classification results are as follows: If φ ≥ φ', it indicates that the dynamic adjustment priority of task reallocation is high, and the corresponding task is marked as a high-priority task; If φ < φ', it indicates that the dynamic adjustment priority of task reallocation is low, and the corresponding task is marked as a low-priority task.

[0022] The dynamic task allocation adjustment mechanism is used to detect the running status of computing nodes, detect whether there are idle computing nodes in the computing nodes, and analyze whether the matching degree between the idle computing nodes and the tasks to be adjusted meets the standard, so as to determine whether there are idle computing nodes适配 for the tasks to be adjusted. If there is no idle computing node适配 for the task to be adjusted, the computing node operation analysis mechanism is triggered.

[0023] For the dynamic adjustment of task reallocation for the high-priority tasks obtained through analysis, first, the running status of computing nodes is detected to analyze whether there are idle computing nodes. If there are idle computing nodes, the matching degree between the idle computing nodes and each high-priority task to be adjusted is analyzed, and a high matching degree threshold M0 is set to determine the matching degree between the idle computing node and the task to be adjusted. The analysis results are as follows: If M ≥ M0, it indicates that the matching degree between the idle computing node and the task to be adjusted is high, and the high-priority task to be adjusted is allocated to the idle node; If M < M0, it indicates that the matching degree between the idle computing node and the task to be adjusted is low, and the computing node operation analysis mechanism is triggered; If there is no idle computing node, the computing node operation analysis mechanism is triggered directly.

[0024] The computing node operation analysis mechanism is used to, when there is no idle computing node and there is a high-priority task to be adjusted, analyze the dynamically adjusted standby nodes适配 for the high-priority computing node to be adjusted among the computing nodes running low-priority tasks, and trigger the task preemption control mechanism after obtaining the dynamically adjusted standby nodes. The task preemption control mechanism is used to obtain the optimal dynamically adjusted standby node from the obtained dynamically adjusted standby nodes when the high-priority task preempts the computing resources of the node where the low-priority task is located.

[0025] The computing node operation analysis mechanism is used to analyze the matching degree between the running computing node and the high-priority task, mark the computing node with high matching degree as a candidate computing node, analyze the dynamic adjustment priority φ of the task running in the candidate computing node, and obtain the dynamically adjusted standby computing node according to the analysis result. The analysis results are as follows: If φ < φ', the node is marked as a dynamically adjusted standby node, which is used for dynamic adjustment and reserve resources for the high-priority task to be adjusted; If φ≥φ', then the node will not mark the computing node as a dynamically reserved computing node; The dynamically adjusted backup nodes are denoted as {b1, b2, ..., bm} in descending order of their matching degree with the task to be adjusted, where m represents the total number of dynamically adjusted backup nodes. After acquiring the dynamically adjusted backup nodes, a task preemption control mechanism is triggered. This mechanism detects and controls the preemption of node computing resources by high-priority tasks from low-priority tasks. First, the running status of tasks on the dynamically adjusted backup computing node b1 is analyzed. The number of times a low-priority task running on the dynamically adjusted backup node b1 has been preempted within the past duration Δd is recorded as n. The remaining execution time Δt of the running low-priority tasks is also analyzed. A maximum threshold of n0 is set for the number of times a low-priority task has been preempted within duration Δd, and a minimum threshold of Δt0 is set for the remaining execution time of the task. Real-time data is then compared and analyzed with the set thresholds. If n>n0 or Δt≥Δt0, it means that the standby computing node b1 no longer meets the preemption conditions. In this case, b1 is removed from the standby nodes, and the analysis continues to determine whether the next dynamically adjusted standby node meets the preemption conditions. If n≤n0 and Δt<Δt0, it means that the standby computing node b1 is ready to be preempted. In this case, b1 will be the best dynamically adjusted standby node, and the low-priority tasks in b1 will be migrated to other computing nodes with lower loads, reserving resources for high-priority tasks. If no computing node with preemption conditions is obtained after analyzing m dynamically adjusted standby nodes, the low-priority task with the shortest remaining execution time among the m standby nodes is selected as the computing node to be preempted, and the waiting time is set to the same as the remaining execution time. The computing node will be preempted after the low-priority task in the computing node to be preempted is completed.

[0026] A dynamic task allocation system for cloud platform data management includes a task tag generation module, an initial task allocation module, a node running status and task priority analysis module, a dynamic task allocation adjustment module, and a computing node running status analysis module. The task tag generation module is used to analyze the received gimbal tasks and gimbal computing node operating parameters, and to tag the gimbal tasks with computing feature tags and the computing nodes with capability tags. The initial task allocation module is used to analyze the matching values ​​between tasks and computing nodes, and to perform initial matching of tasks and computing nodes based on the matching values ​​between each task and each computing node. The node running status and task priority analysis module is used to monitor the resource stress of computing nodes in real time after the initial task allocation is completed, and to obtain the adjustment priority of tasks by analyzing the execution status of tasks to be adjusted. The task allocation dynamic adjustment module is configured to detect the operating status of computing nodes, detect whether there is an idle computing node among the computing nodes, and trigger a computing node operation analysis mechanism when there is no idle computing node adapted to the task to be adjusted; The computing node operation analysis module is configured to analyze and obtain dynamically adjusted backup nodes adapted to the high-priority task to be adjusted from the computing nodes running low-priority tasks, and after obtaining the dynamically adjusted backup nodes, trigger a task preemption control mechanism to analyze the optimal dynamically adjusted backup node.

[0027] Embodiment 1: Collect the adaptation degree of task computing power requirement to node computing power δ1=0.8, the matching degree between task calculation feature labels and node capability labels δ2=0.9, the overall task completion time optimization coefficient δ3=0.8 and the resource utilization coefficient δ4=0.7; set the task computing power requirement adaptation coefficient γ1=0.3, the two-way label matching degree coefficient γ2=0.3, the overall task completion time optimization coefficient γ3=0.2 and the resource utilization coefficient γ4=0.2, and obtain the matching value M=0.81 between the task and the computing node; set the high matching degree threshold M0=0.75, then M>M0, which indicates that the matching degree between the idle computing node and the task to be adjusted is high, and the high-priority task to be adjusted is allocated to the idle node; Embodiment 2: Collect the task value u1=0.9, the executed time u2=3h and the remaining execution time u3=1h of the task being executed, substitute the values into the formula to calculate and obtain φ=0.8132; set the dynamic adjustment priority threshold φ'=0.7 for task reallocation, then φ≥φ', which indicates that the dynamic adjustment priority of task reallocation is high, and the corresponding task is marked as a high-priority task; detect that within the past duration Δd=1h, the number of preemptions of the low-priority task running in the dynamically adjusted backup node b1 is recorded as n=0, and analyze the remaining execution duration Δt=3h of the running low-priority task, set the maximum threshold for the number of preemptions of a low-priority task within the duration Δd as n0=2; set the minimum threshold for the remaining task execution duration Δt0=2, then if n<n0 and Δt<Δt0, it indicates that the backup computing node b1 meets the preemption condition, so b1 is taken as the optimal dynamically adjusted backup node, the low-priority task in b1 is migrated to other computing nodes with lower load, and resources are reserved for the high-priority task.

[0028] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A method for dynamic task allocation in cloud platform data management, characterized in that: The method includes the following: A tag generation mechanism is set up to analyze the received gimbal tasks and gimbal computing node operating parameters, and to tag the gimbal tasks with computing feature tags and the computing nodes with capability tags. An initial task allocation mechanism is set up to analyze the matching values ​​between tasks and computing nodes, and to perform initial matching of tasks and computing nodes based on the matching values ​​between each task and each computing node. The node running status and task priority analysis mechanism is set up to monitor the resource stress of computing nodes in real time after the initial task allocation is completed, and obtain the task adjustment priority by analyzing the execution status of the tasks to be adjusted. The task allocation dynamic adjustment mechanism is set up to detect the running status of computing nodes, detect whether there are idle computing nodes, and trigger the computing node running status analysis mechanism when there are no idle computing nodes that are suitable for the task to be adjusted. The compute node operation analysis mechanism is set up to analyze and obtain dynamically adjusted backup nodes that are suitable for high-priority tasks that need to be adjusted in compute nodes running low-priority tasks. After obtaining the dynamically adjusted backup nodes, the task preemption control mechanism is triggered to analyze the optimal dynamically adjusted backup nodes.

2. The method for dynamic task allocation for cloud platform data management according to claim 1, characterized in that: The node running status and task priority analysis mechanism is used to monitor the running computing nodes in real time after the initial task allocation is completed. By collecting the CPU utilization, memory utilization and network bandwidth occupancy of the running computing nodes in real time, the resource utilization of the computing nodes is analyzed, and the resource utilization of the computing nodes is used to determine whether there is resource shortage in the nodes. Once resource constraints are identified, the value, execution time, and remaining execution time of the tasks to be adjusted are combined to determine the task adjustment priority.

3. The method for dynamic task allocation for cloud platform data management according to claim 2, characterized in that: The CPU utilization of the computing node collected in real time is recorded as r1, the memory utilization as r2, and the network bandwidth occupancy as r3. These three parameters are weighted and fused to obtain the resource stress P of the computing node. A resource stress threshold α is set, and the real-time monitored resource stress of the computing node is compared and analyzed with the set threshold. The analysis results are as follows: If P≤α, it means that the resource stress is within the threshold range, and no dynamic adjustment will be made to the task allocation; If P>α, it means that the resource stress has exceeded the threshold, so the task allocation will be dynamically adjusted, triggering the task adjustment priority analysis mechanism. For all tasks in execution, a dynamic adjustment priority analysis for task reallocation is performed. The task value u1, execution time u2, and remaining execution time u3 of each task in execution are collected. The dynamic adjustment priority for task reallocation is calculated according to the following formula: ； Where φ represents the dynamic adjustment priority of task reallocation; Set a dynamic adjustment priority threshold φ' for task reallocation, and classify the tasks to be adjusted into different levels. The classification results are as follows: If φ≥φ', it means that the dynamic adjustment priority of task reallocation is high, and the corresponding task will be marked as a high-priority task. If φ < φ', it means that the dynamic adjustment priority of task reallocation is low, and the corresponding task will be marked as a low-priority task.

4. The method for dynamic task allocation for cloud platform data management according to claim 1, characterized in that: The dynamic task allocation adjustment mechanism is configured to detect the operating status of computing nodes, detect whether there is an idle computing node among the computing nodes, and analyze whether the matching degree between the idle computing node and the task to be adjusted meets the standard, so as to determine whether there is an idle computing node adapted to the task to be adjusted. If there is no idle computing node adapted to the task to be adjusted, a computing node operating condition analysis mechanism is triggered.

5. The method for dynamic task allocation for cloud platform data management according to claim 4, characterized in that: For dynamic adjustment of task reallocation for the analyzed high-priority tasks, first, the operating status of computing nodes is detected to analyze whether there is an idle computing node. If there is an idle computing node, the matching degree between the idle computing node and each high-priority task to be adjusted is analyzed, a high matching degree threshold M0 is set, and the matching degree between the idle computing node and the task to be adjusted is determined. The analysis results are as follows: If M≥M0, it indicates that the matching degree between the idle computing node and the task to be adjusted is high, and the high-priority task to be adjusted is allocated to the idle node; If M<M0, it indicates that the matching degree between the idle computing node and the task to be adjusted is low, and the computing node operating condition analysis mechanism is triggered; If there is no idle computing node, the computing node operating condition analysis mechanism is triggered directly.

6. The method for dynamic task allocation for cloud platform data management according to claim 3, characterized in that: The computing node operating condition analysis mechanism is configured to, when there is no idle computing node and there is a high-priority task to be adjusted, analyze a dynamic adjustment standby node adapted to the high-priority task to be adjusted among computing nodes executing low-priority tasks, and trigger a task preemption control mechanism after obtaining the dynamic adjustment standby node. The task preemption control mechanism is configured to obtain an optimal dynamic adjustment standby node from the obtained dynamic adjustment standby nodes when a high-priority task preempts the node computing resources of a low-priority task.

7. The method for dynamic task allocation for cloud platform data management according to claim 6, characterized in that: The computing node operating condition analysis mechanism is configured to analyze the matching degree between an operating computing node and a high-priority task, mark a computing node with high matching degree as a candidate computing node, analyze the dynamic adjustment priority φ of the task running in the candidate computing node, and obtain a dynamic adjustment standby computing node according to the analysis result. The analysis results are as follows: If φ<φ', the node is marked as a dynamic adjustment standby node for dynamic adjustment and reserve allocation of resources for the high-priority task to be adjusted; If φ≥φ', the node is not marked as a dynamic adjustment standby computing node; The dynamically adjusted backup nodes are denoted as {b1, b2, ..., bm} in descending order of their matching degree with the task to be adjusted, where m represents the total number of dynamically adjusted backup nodes. After acquiring the dynamically adjusted backup nodes, a task preemption control mechanism is triggered. This mechanism detects and controls the preemption of node computing resources by high-priority tasks from low-priority tasks. First, the running status of tasks on the dynamically adjusted backup computing node b1 is analyzed. The number of times a low-priority task running on the dynamically adjusted backup node b1 has been preempted within the past duration Δd is recorded as n. The remaining execution time Δt of the running low-priority task is analyzed. A maximum threshold of n0 is set for the number of times a low-priority task has been preempted within duration Δd, and a minimum threshold of Δt0 is set for the remaining execution time of the task. Real-time data is compared and analyzed with the set thresholds. If n>n0 or Δt≥Δt0, it means that the standby computing node b1 no longer meets the preemption conditions. In this case, b1 is removed from the standby nodes, and the analysis continues to determine whether the next dynamically adjusted standby node meets the preemption conditions. If n≤n0 and Δt<Δt0, it means that the standby computing node b1 is ready to be preempted. In this case, b1 will be the best dynamically adjusted standby node, and the low-priority tasks in b1 will be migrated to other computing nodes with lower loads, reserving resources for high-priority tasks. If no computing node with preemption conditions is obtained after analyzing m dynamically adjusted standby nodes, the low-priority task with the shortest remaining execution time among the m standby nodes is selected as the computing node to be preempted, and the waiting time is set to the same as the remaining execution time. The computing node will be preempted after the low-priority task in the computing node to be preempted is completed.

8. The method for dynamic task allocation for cloud platform data management according to claim 1, characterized in that: The tag generation mechanism is used to generate computational feature tags for tasks and capability tags for computing nodes; it obtains the tasks to be assigned and related information from the cloud platform's data management, extracts the computational features of each task, and assigns a unique computational feature tag to each task; the computational features include the task's computing power requirements, task type, task priority, task dependencies, estimated task execution time, and data processing scale; it obtains the hardware parameters, software configuration, and operating status information of all computing nodes in the cloud platform, extracts the capability features of each computing node, and assigns a unique capability tag to each computing node; the capability features include node computing power, memory capacity, storage capacity, network bandwidth, current load, and historical execution efficiency.

9. A method for dynamic task allocation for cloud platform data management according to claim 1, characterized in that: An initial task allocation mechanism is set up to analyze the matching value between tasks and computing nodes. By comprehensively integrating and analyzing the adaptability of task computing power requirements and node computing power, the matching degree of task computing feature tags and node capability tags, the overall task completion time optimization coefficient and resource utilization coefficient, the matching value between tasks and computing nodes is obtained. Based on the obtained matching reward value between tasks and computing nodes, each task allocation value is matched with the computing node with the highest value, thereby completing the initial task allocation.

10. A dynamic task allocation system for cloud platform data management, applied to the dynamic task allocation method for cloud platform data management as described in any one of claims 1-9, characterized in that: The system includes a task tag generation module, an initial task allocation module, a node running status and task priority analysis module, a task allocation dynamic adjustment module, and a computing node running status analysis module. The task tag generation module is used to analyze the received gimbal tasks and gimbal computing node operating parameters, and to tag the gimbal tasks with computing feature tags and the computing nodes with capability tags. The initial task allocation module is used to analyze the matching value between tasks and computing nodes, and to perform initial matching of tasks and computing nodes based on the matching value between each task and each computing node. The node running status and task priority analysis module is used to monitor the resource stress of computing nodes in real time after the initial task allocation is completed, and to obtain the task adjustment priority by analyzing the execution status of the tasks to be adjusted. The task allocation dynamic adjustment module is used to detect the running status of computing nodes, detect whether there are idle computing nodes, and trigger the computing node running status analysis mechanism when there are no idle computing nodes that are compatible with the task to be adjusted. The computing node operation analysis module is used to analyze and obtain dynamically adjusted backup nodes that are compatible with the high-priority tasks to be adjusted in computing nodes where low-priority tasks are running, and after obtaining the dynamically adjusted backup nodes, trigger the task preemption control mechanism to analyze the optimal dynamically adjusted backup nodes.

Images