Task processing method and device for computing power node, equipment and medium

Abstract

The invention discloses a task processing method and device for a computing power node, equipment and a medium, and relates to the field of computing power nodes and task scheduling. The task processing method of the computing power node comprises the following steps: acquiring a task set, and analyzing a data dependency relationship and a control dependency relationship between tasks; based on the data dependency relationship and the control dependency relationship, a task dependency graph is constructed, and the task dependency graph represents the dependency direction between tasks through directed edges; sorting the tasks to be executed under the constraint of the task dependency graph to generate a task execution sequence; calculating an adaptive value of a task and a node resource based on the CPU utilization rate, the memory occupancy rate and the GPU utilization rate of the computing power node acquired in real time; and dynamically distributing the tasks to the computing power node with the highest adaptive value according to the task sequence of the task execution sequence. In this way, the task processing efficiency of the computing power node can be improved.

Description

Technical Field

[0001] This application relates to the field of task scheduling for computing power nodes, and in particular to a task processing method, apparatus, device and medium for computing power nodes. Background Technology

[0002] In the field of task scheduling for computing nodes, existing technologies suffer from the technical bottleneck of fragmented dependency relationships. Relying on neural networks to implicitly learn task dependencies cannot distinguish data dependencies, such as output-input relationships, and control dependencies, such as status code detection. Adding new dependencies requires retraining the model, and it cannot detect circular dependencies in real time. Dynamic dependency management is prone to failure, resulting in unclear dependency relationships and low task scheduling efficiency. Summary of the Invention

[0003] This application mainly provides a task processing method, apparatus, device and medium for computing nodes to solve the problem of low task scheduling efficiency caused by the failure of dynamic dependency management.

[0004] To address the aforementioned technical problems, this application provides a task processing method for computing power nodes, comprising: acquiring a task set and parsing the data dependencies and control dependencies between tasks; constructing a task dependency graph based on the data dependencies and control dependencies, wherein the task dependency graph uses directed edges to represent the dependency directions between tasks; sorting the tasks to be executed under the constraints of the task dependency graph to generate a task execution sequence; calculating the suitability value between tasks and node resources based on real-time collected CPU utilization, memory usage, and GPU utilization of the computing power nodes; and dynamically allocating tasks to the idle resource nodes with the highest suitability values ​​according to the task order of the task execution sequence.

[0005] In some embodiments, constructing a task dependency graph based on the data dependencies and the control dependencies includes: establishing partial directed edges of the task dependency graph based on the data dependencies between input data and output data of each task; and establishing the remaining directed edges of the task dependency graph based on the control dependencies between status codes and execution detection requirements of each task.

[0006] In some embodiments, the step of sorting under the constraints of the task dependency graph to generate a task execution sequence includes: real-time detection of the in-degree value of each task in the real-time task dependency graph, and adding the task to the candidate execution queue in response to the in-degree value of the task being zero; and arranging the tasks in the candidate execution queue in descending order according to a priority function to form the task execution sequence.

[0007] In some embodiments, the priority function is calculated using the following formula:

[0008] Among them, P static For static priority, Uc is the CPU load rate of the target node, α and β are weighting coefficients, and e is a constant; P static The value of α ranges from 0 to 10, and the sum of α and β is 1.

[0009] In some embodiments, calculating the adaptation value between the task and node resources based on the real-time collected CPU utilization, memory occupancy, and GPU utilization of the computing power node includes: mapping the CPU utilization, memory occupancy, and GPU utilization of the computing power node to a unit interval; configuring weight coefficients for the CPU utilization, memory occupancy, and GPU utilization of the computing power node according to the task type; and calculating the adaptation value between the task and node resources based on the weighted CPU utilization, weighted memory occupancy, and weighted GPU utilization of the computing power node.

[0010] In some embodiments, the formula for the fit value between computing tasks and node resources is:

[0011] Among them, S j The overall adaptation value between task and node resources, where i is the ordinal number of the summation calculation, cpu is the CPU utilization of the computing node, mem is the memory usage, gpu is the GPU utilization, and ω is the overall resource utilization. i For dynamic weights, (1-ΔR) i () represents the single resource matching degree.

[0012] In some embodiments, when multiple tasks have an in-degree of zero at the same time in the task execution sequence, the execution order of the tasks is selected in descending order of task priority.

[0013] To address the aforementioned technical problems, another technical solution adopted in this application is: providing a task processing device for a computing power node, comprising: a parsing module for acquiring a task set and parsing the data dependencies and control dependencies between tasks; a construction module for constructing a task dependency graph based on the data dependencies and control dependencies, wherein the task dependency graph represents the dependency direction between tasks through directed edges; a sorting module for sorting the tasks to be executed under the constraints of the task dependency graph to generate a task execution sequence; a calculation module for calculating the fit value between tasks and node resources based on real-time collected CPU utilization, memory usage, and GPU utilization of the computing power node; and an allocation module for dynamically allocating tasks to the idle resource node with the highest fit value according to the task order of the task execution sequence.

[0014] This application also provides a computer device, the computer device comprising: a memory and at least one processor, the memory storing instructions; the at least one processor calling the instructions in the memory to cause the computer device to execute the task processing method of the computing node as described above.

[0015] This application also provides a computer-readable storage medium storing instructions that, when executed by a processor, implement the task processing method of the computing node as described above.

[0016] The beneficial effects of this application are as follows: Unlike existing technologies, this application discloses a task processing method, apparatus, device, and medium for computing power nodes. It avoids invalid task execution through a dual mechanism of explicit management of control dependencies and real-time resource constraint decision-making. It obtains a task set, parses the data and control dependencies between tasks, and constructs a task dependency graph based on these dependencies. The task dependency graph uses directed edges to represent the dependency directions between tasks, eliminating implicit dependency modeling delays and avoiding scheduling inefficiencies caused by dynamic dependency management failures. Under the constraints of the task dependency graph, the tasks to be executed are sorted to generate a task execution sequence, overcoming the bottleneck of blind resource sorting. Based on real-time collected CPU utilization, memory usage, and GPU utilization of the computing power node, it calculates the matching value between tasks and node resources, considering the real-time resource status of the computing power node to avoid mismatch between resource allocation and task requirements. Tasks are dynamically allocated to the idle resource node with the highest matching value according to the task order of the task execution sequence, overcoming the bottleneck of task execution sequence generation. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein: Figure 1 This is a flowchart illustrating an embodiment of the task processing method for computing nodes provided in this application; Figure 2 Is it like this? Figure 1 The flowchart of step 20 of the method shown is a schematic diagram of one embodiment; Figure 3 Is it like this? Figure 1 The flowchart of step 30 of the method shown is a schematic diagram of an embodiment; Figure 4 Is it like this? Figure 1 The flowchart of step 40 of the method shown is a schematic diagram of an embodiment. Figure 5This is a schematic diagram of the structure of an embodiment of the task processing device for the computing power node provided in this application; Figure 6 This is a schematic diagram of the structure of a computer device according to an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0019] The terms "first," "second," and "third" used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0020] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0021] See Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the task processing method for a computing node provided in this application. The task processing method for the computing node includes the following steps: 10: Obtain the task set and resolve the data and control dependencies between tasks.

[0022] Collect a set of task units to be processed from the task scheduling system or batch tasks submitted by users. Each task contains metadata such as execution logic, input / output data description, and execution conditions.

[0023] Data dependency resolution results in a data dependency (A→B) when the output data of task A is the input of task B. By analyzing the descriptions of task input / output data (such as file paths, variable names, database tables, etc.), a data flow graph between tasks can be established. For example, if task B needs to read the result.csv generated by task A, then A must be executed before B.

[0024] The resolution result of control dependencies is that the execution status (success / failure) of task C determines whether task D is executed, and checks the status code dependency of the task (such as if taskC.status==SUCCESS) or explicit triggering conditions. For example, if the configuration of task D declares depends_on: [taskC], then a control dependency edge (C→D) is formed.

[0025] By statically analyzing task descriptions (such as input / output data declarations) or dynamically capturing task status codes, we can accurately identify the data flow between tasks (such as A's output serving as B's input) and control logic (such as task D being triggered only after task C succeeds). This lays the foundation for building an unambiguous task dependency graph and avoids logical errors caused by missing or conflicting dependencies during task execution, such as data pollution issues caused by executing subsequent tasks before completing preceding tasks.

[0026] Optionally, step 10 extracts dependency declarations through static code analysis or task description files (such as YAML / JSON). During dynamic runtime, implicit dependencies need to be captured, circular dependencies need to be detected, and exceptions need to be thrown.

[0027] The task set is a collection of computational task units to be scheduled, containing multiple independent or interdependent tasks. The task set includes a unique identifier, data dependency definitions, and control dependency definitions.

[0028] Data dependencies are hard constraints on data flow between tasks, used to ensure the correctness of task execution order.

[0029] Controlling dependencies are soft constraints on the execution logic between tasks, determining the order in which tasks are executed. They are used to build a task execution order model, ensuring that tasks are executed correctly according to their dependencies and improving system scheduling efficiency.

[0030] By precisely analyzing two types of key dependencies between tasks, the core contradiction between logical correctness and execution efficiency in distributed task scheduling is resolved. Specifically, by separating data dependencies (mandatory order) and control dependencies (conditional triggering), independent subsets of tasks without dependencies can be identified, providing opportunities for parallel execution in subsequent dynamic scheduling. This maximizes resource utilization and shortens the overall completion time of the task set while satisfying dependency constraints.

[0031] The dependency resolution phase can detect circular dependencies (such as task A depending on B, and B depending on A), avoid system deadlock through abnormal interruption or dependency reconstruction, effectively protect the directed acyclic graph (DAG) property of the task dependency graph, and ensure that tasks can always generate a legal execution sequence.

[0032] 20: Based on data dependencies and control dependencies, construct a task dependency graph. The task dependency graph uses directed edges to represent the dependency direction between tasks.

[0033] Each node in the task dependency graph represents a task, and edges represent dependencies. This ensures that tasks are executed in the correct order, avoiding logical errors and data corruption. A topological sorting algorithm is used to sort the graph, determining the task execution order, improving scheduling efficiency, and ensuring stable system operation.

[0034] The construction of the task dependency graph further includes: generating a first directed graph; analyzing whether each operation generates data using a data dependency function; analyzing the control dependencies between operations using a control dependency function; and generating a data dependency graph and a control dependency graph based on the analysis results.

[0035] Furthermore, step 20 also includes the following steps: 21: Based on the input data dependencies and output data dependencies between tasks, establish partial directed edges of the task dependency graph.

[0036] Parse the input data declarations (such as file paths, variable names, or database tables) for each task in the task set to identify the flow of output data to input data. For example, if task B must wait for task A to generate a result.csv file before it can execute, a directed edge (A→B) is established from A to B. Data dependencies typically involve RAW (read-after-write) types, meaning that data producers (writes) must execute before consumers (reads).

[0037] Building a task dependency graph based on data dependencies can ensure the continuity of data flow and avoid execution errors (such as data pollution) caused by missing or overwritten data.

[0038] 22: Based on the control dependency relationship between the status codes and execution detection requirements of each task, establish the remaining directed edges of the task dependency graph.

[0039] Analyze the status detection requirements in the task metadata (e.g., depends_on: taskC.status == SUCCESS). When the status of task C determines whether task D is executed, establish a directed edge (C→D) from C to D. For example, in a project management scenario, the test task depends on the completion status of the development task.

[0040] Building task dependency graphs based on control dependencies can support conditional branching logic (such as retrying or skipping on failure), enhancing the flexibility and fault tolerance of task scheduling.

[0041] The directed edges (arrows) of the task dependency graph explicitly indicate the order in which tasks are executed. For example, A→B means that B must start after A is completed. This feature simplifies the scheduling logic and allows task sequence generation to be directly implemented based on graph topology sorting.

[0042] The dependency graph must be a directed acyclic graph, meaning there are no circular paths (such as A→B→A). This characteristic ensures that tasks can always generate feasible execution sequences, avoiding deadlocks or infinite waiting.

[0043] Task nodes without direct dependencies (without edge connections) can be executed in parallel to maximize resource utilization.

[0044] Among them, the longest path (critical path) in the graph determines the shortest completion time of the entire task set, and optimizing the scheduling strategy can shorten the delay.

[0045] By constructing a task dependency graph based on data and control dependencies, and representing the dependency direction between tasks through directed edges in the graph, the system ensures the orderliness and efficiency of task execution. A topological sorting algorithm is used to traverse the dependency graph, generating an acyclic execution sequence to prevent task conflicts. Through precise identification of data and control flows, the system can process independent tasks in parallel when resources are sufficient, further improving execution efficiency.

[0046] 30: Sort the tasks to be executed under the constraints of the task dependency graph to generate a task execution sequence.

[0047] Based on the Directed Acyclic Graph (DAG) property of the task dependency graph, topological sorting is achieved using in-degree detection. The in-degree of each task is monitored in real time. When the in-degree of a task drops to zero, the candidate queue is sorted according to a priority function to form the final sequence.

[0048] A task execution sequence is a linear task queue that satisfies the conditions of dependency compliance, dynamic adjustability, and parallel optimization. Dependency compliance means that if there is a dependency edge A→B, then A must precede B in the sequence; dynamic adjustability means that execution nodes can be adjusted based on real-time resource adaptation values; and parallel optimization means that tasks without dependencies can be executed concurrently.

[0049] During the sorting process, the execution order is dynamically adjusted, including: When invoking the current operation, the execution order of its successor operations is checked to ensure consistency. Specifically, if the in-degree of subsequent tasks is not updated synchronously, the sequence is recalculated and adjusted to ensure dependency consistency. Through a real-time resource monitoring module, task priorities are dynamically adjusted, resource allocation is optimized, and the overall processing cycle is further shortened.

[0050] If inconsistencies are found, the order of subsequent operations is corrected. Dependency consistency ensures task execution continuity and avoids resource waste. Real-time monitoring and dynamic adjustments complement each other, improving system response speed and ensuring efficient task completion. Precise sorting combined with parallel optimization maximizes resource utilization.

[0051] The integrity of dependencies is verified by checking the number of predecessor operations to ensure logical consistency. Changes in dependencies are monitored in real time, and the execution sequence is dynamically updated to ensure tasks are completed in order. Dependency edges are accurately identified to avoid execution conflicts and improve system stability.

[0052] Furthermore, step 30 includes the following steps: 31: Real-time detection of the in-degree value of each task in the real-time task dependency graph, and in response to a task having an in-degree value of zero, adding the task to the candidate execution queue.

[0053] Task status changes are monitored through event listening or polling. Upon completion of each task, the in-degree value of its successor task is updated. When the in-degree value of a task reaches zero, it is immediately added to the candidate queue. For example, after task A is completed, the in-degree of its only successor task B changes from 1 to 0, and B enters the candidate queue.

[0054] The in-degree value refers to the number of directed edges in the task dependency graph pointing to a certain task, that is, the total number of predecessor tasks that the task depends on. For example, if task B has two incoming edges (A→B, C→B), then its in-degree value is 2. The in-degree drops to zero, indicating that all predecessor tasks have been completed and the current task is ready to execute.

[0055] Specifically, the real-time in-degree detection mechanism monitors task status changes through event listening or polling, and updates the in-degree value of each subsequent task after each task is completed.

[0056] When the in-degree of a task reaches zero, it is immediately added to the candidate queue. For example, after task A is completed, the in-degree of its only successor task B changes from 1 to 0, and B enters the candidate queue.

[0057] The candidate execution queue is a temporary container used to store tasks that currently meet the execution conditions (in-degree of zero), serving as an intermediate buffer between topological sorting and final execution. The candidate execution queue continuously receives new ready tasks and outputs them to the execution sequence after reordering them by priority.

[0058] By monitoring in-degree values ​​to track task state changes, the execution order can be dynamically adjusted to ensure efficient task flow. Whenever a task's in-degree reaches zero, the system immediately triggers a state update, moving the task from the candidate queue to the execution sequence, ensuring no dependency conflicts.

[0059] 32: Sort the tasks in the candidate execution queue in descending order according to the priority function to form a task execution sequence.

[0060] A priority function is applied to all tasks in the candidate queue, scores are calculated, and tasks are sorted in descending order of score to form a temporary execution sequence. Reordering is triggered when a new task is added to the queue. Resource changes (such as GPU release) may cause updates to the priority function return value, requiring real-time adjustments to the order. In-degree detection ensures strict fulfillment of dependencies and avoids logical errors. Combined with priority sorting, it optimizes resource allocation and task throughput on the basis of compliance. By using a candidate queue as a buffer layer, it decouples dependency detection and execution scheduling, improves system response speed, and achieves technical synergy.

[0061] The priority function is used to quantify the urgency of tasks, and is typically dynamically generated based on task attributes such as deadlines, resource requirements, and historical execution durations. Descending priority ensures that higher-priority tasks are executed first; for example, deadline priority means tasks closer to their deadline have higher priority, and resource utilization priority means tasks requiring less GPU memory have higher priority.

[0062] Specifically, the formula for calculating the priority function is:

[0063] Among them, P static For static priority, Uc is the CPU load rate of the target computing node, α and β are weighting coefficients, and e is a constant. P static The value of α ranges from 0 to 10, and the sum of α and β is 1.

[0064] Optionally, when multiple tasks in the task execution sequence have an in-degree of zero at the same time, the execution order of the tasks is selected in descending order of task priority.

[0065] When all the preceding tasks of a task are completed, its in-degree value becomes zero, and the system adds it to the candidate queue. If multiple tasks simultaneously meet this condition (e.g., tasks B and C both have their in-degrees become zero after task A is completed), then the priority sorting phase begins. Each task calculates its priority value according to preset rules, and the candidate queue is arranged in descending order of priority to ensure that high-priority tasks enter the execution sequence first.

[0066] Topology sorting ensures that the basic execution order satisfies the DAG constraint, while priority strategy only affects the order between tasks that have no direct dependency relationship. Executing high-priority tasks first can reduce critical path latency, prevent low-priority tasks from blocking high-value tasks, and improve system throughput.

[0067] 40: The matching value between computing tasks and node resources is calculated based on the real-time collected CPU utilization, memory usage, and GPU utilization of computing nodes.

[0068] CPU utilization indicates the busyness of the current node's processing tasks, memory usage reflects the tightness of available memory, and GPU utilization reflects the load on the graphics processing unit.

[0069] The fit value reflects how well a task matches the current resource status; a higher value indicates a stronger fit. The system dynamically adjusts task allocation based on the fit value, prioritizing tasks with high fit values ​​to resource-rich nodes to maximize resource utilization, reduce task waiting time, and improve overall execution efficiency.

[0070] Furthermore, step 40 also includes the following steps: 41: Map the CPU utilization, memory usage, and GPU utilization of computing nodes to a unit interval.

[0071] A threshold truncation method is used to map CPU utilization of computing nodes, linear normalization is used to map memory usage, and dynamic normalization is used to map GPU utilization. After mapping, the values ​​of each indicator are between 0 and 1, facilitating unified calculation. Based on the mapped indicator values, combined with task priority and resource adaptation values, the system comprehensively evaluates the task execution order to ensure resource allocation strategies are implemented.

[0072] 42: Configure the weighting coefficients for CPU utilization, memory usage, and GPU utilization of computing nodes according to the task type.

[0073] The weight coefficients of various parameters of the computing power node are configured according to the user type classification.

[0074] Specifically, in computationally intensive scenarios such as model training, the weights are configured as CPU 0.2 / Memory 0.1 / GPU 0.7; in memory-intensive scenarios such as graph databases, the weights are configured as CPU 0.3 / Memory 0.6 / GPU 0.1; and in hybrid tasks, adaptive weights are used, such as automatically reducing the CPU weight when GPU utilization exceeds 80%.

[0075] Through dynamic weight adjustment, the system accurately matches tasks and node resources, optimizes execution paths, reduces resource conflicts, and improves task completion efficiency. Resource utilization is significantly improved in various scenarios, task waiting time is greatly shortened, and overall system performance is significantly enhanced.

[0076] 43: Calculate the matching value between task and node resources based on the weighted CPU utilization, weighted memory usage, and weighted GPU utilization of the computing power node.

[0077] Through weighted calculation, the system generates a comprehensive fit value that accurately reflects the matching degree between tasks and node resources. Tasks with high fit values ​​are scheduled first to ensure efficient resource utilization, further shortening task processing cycles and improving overall system performance. The fit values ​​for each scenario are dynamically adjusted to optimize task execution order, significantly enhancing resource utilization and task processing efficiency.

[0078] Specifically, the formula for calculating the fit value between the task and node resources is:

[0079] Among them, S j The overall adaptation value between task and node resources, where i is the ordinal number of the summation calculation, cpu is the CPU utilization of the computing node, mem is the memory usage, gpu is the GPU utilization, and ω is the overall resource utilization. i For dynamic weights, (1-ΔR) i () represents the single resource matching degree.

[0080] 50: Dynamically allocate tasks to the computing power node with the highest suitability value according to the task execution sequence.

[0081] Maintain an ordered queue of tasks to be executed, sorted by submission time or preset priority, and retrieve the next task to be assigned from the head of the queue each time. Traverse idle nodes and select the node with the highest current fit value. If multiple nodes have the same fit value, prioritize the node with a more balanced load.

[0082] The system dynamically adjusts task allocation based on the overall compatibility value, ensuring that tasks with high compatibility values ​​are executed first, maximizing resource utilization. By monitoring resource status in real time and dynamically updating the task execution sequence, the system further optimizes task scheduling, improves overall task processing efficiency, and achieves precise matching of resources and tasks.

[0083] Optionally, the task processing method based on computing power nodes provided in this application further includes critical path analysis, including: Analyze the preceding and succeeding operations of a task. Identify bottleneck tasks on the critical path, prioritize scheduling critical path tasks, and ensure efficient progress of the overall task flow.

[0084] If the sets have an intersection, the intersection operation is performed. Through critical path analysis, the system accurately identifies task bottlenecks, optimizes scheduling strategies, ensures that critical tasks are executed with priority, further shortens the overall task completion time, and improves system operating efficiency.

[0085] Critical path localization is achieved through intersection operations. By analyzing critical paths, the system accurately identifies task bottlenecks, optimizes scheduling strategies, ensures the priority execution of critical tasks, further shortens overall task completion time, and improves system efficiency. Critical path localization based on intersection operations dynamically adjusts resource allocation, enabling efficient task flow coordination and significantly improving overall system performance and resource utilization.

[0086] By combining the critical path and task dependency graph, an optimal execution order is generated to optimize the process. The system first identifies task nodes on the critical path, such as T_1, T_2, T_3, T_4, and T_5, and dynamically adjusts the execution order based on dependencies and fit values. T_1 is scheduled first, followed by T_2 and T_3 in sequence, ensuring that T_4 and T_5 follow closely behind, achieving precise matching of resources and tasks, further optimizing the task flow, and improving the overall system performance.

[0087] The task processing method based on computing power nodes in the embodiments of the present invention has been described above. The task processing device based on computing power nodes in the embodiments of the present invention will be described below. Please refer to [link / reference]. Figure 5 One embodiment of the task processing device for computing nodes in this invention includes: The parsing module 401 is used to obtain the task set and parse the data dependencies and control dependencies between tasks.

[0088] Module 402 is used to construct a task dependency graph based on data dependencies and control dependencies. The task dependency graph uses directed edges to represent the dependency direction between tasks.

[0089] The sorting module 403 is used to sort the tasks to be executed under the constraints of the task dependency graph and generate a task execution sequence.

[0090] The computing module 404 is used to calculate the matching value between the task and the node resources based on the real-time collected CPU utilization, memory usage and GPU utilization of the computing nodes.

[0091] The allocation module 405 is used to dynamically allocate tasks to the computing power node with the highest matching value according to the task order of the task execution sequence.

[0092] above Figure 5 The feature extraction device in the embodiments of the present invention will be described in detail from the perspective of modular functional entities. The computer device in the embodiments of the present invention will be described in detail from the perspective of hardware processing.

[0093] Figure 6This is a schematic diagram of a computer device 500 provided in an embodiment of the present invention. The computer device 500 can vary significantly due to different configurations or performance characteristics. It may include one or more central processing units (CPUs) 510 (e.g., one or more processors) and a memory 520, and one or more storage media 530 (e.g., one or more mass storage devices) for storing application programs 533 or data 532. The memory 520 and storage media 530 may be temporary or persistent storage. The program stored in the storage media 530 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the computer device 500. Furthermore, the processor 510 may be configured to communicate with the storage media 530 and execute the series of instruction operations in the storage media 530 on the computer device 500.

[0094] Computer device 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input / output interfaces 560, and / or one or more operating systems 531, such as Windows Server, MacOSX, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 6 The computer device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0095] The present invention also provides a computer device, the computer device including a memory and a processor, the memory storing computer-readable instructions, which, when executed by the processor, cause the processor to perform the steps of the task processing method of the computing node in the above embodiments.

[0096] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when the instructions are executed on a computer, cause the computer to perform the steps of the task processing method of the computing node.

[0097] Unlike existing technologies, the task processing method for computing nodes in this application systematically analyzes task dependencies and combines them with a resource-aware scheduling mechanism to intelligently optimize the execution order, significantly improving resource utilization and processing efficiency. This method is applicable to various hardware configurations, ensuring efficient task execution and optimizing computing node performance. Through the collection and preprocessing of task sets, the system can accurately identify task requirements, normalize resource usage, and further optimize scheduling strategies, ensuring stable operation of computing nodes under high load environments and comprehensively improving the reliability and response speed of task processing.

[0098] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0099] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0100] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A task processing method for a computing node, characterized in that, include: Obtain the task set and resolve the data and control dependencies between tasks; Based on the data dependency relationship and the control dependency relationship, a task dependency graph is constructed, wherein the task dependency graph represents the dependency direction between tasks through directed edges; Under the constraints of the task dependency graph, the tasks to be executed are sorted to generate a task execution sequence; Map the CPU utilization, memory usage, and GPU utilization of computing nodes to a unit interval; Configure the weighting coefficients for the CPU utilization, memory usage, and GPU utilization of the computing node according to the task type; The adaptation value between the task and node resources is calculated based on the weighted CPU utilization, weighted memory usage, and weighted GPU utilization of the computing power node; the adaptation value is used to reflect the matching degree between the task and node resources. The formula for calculating the fit value between the task and node resources is: Among them, S j The overall adaptation value between task and node resources, where i is the ordinal number of the summation calculation, cpu is the CPU utilization of the computing node, mem is the memory usage, gpu is the GPU utilization, and ω is the overall resource utilization. i For dynamic weights, (1-ΔR) i () represents the single resource matching degree; Tasks are dynamically assigned to the computing power node with the highest matching value according to the task execution sequence.

2. The task processing method for computing nodes according to claim 1, characterized in that, The construction of the task dependency graph based on the data dependency relationship and the control dependency relationship includes: Based on the data dependencies between input data and output data of each task, a partial directed edge is established in the task dependency graph. Based on the control dependency relationship between the status codes and execution detection requirements of each task, the remaining directed edges of the task dependency graph are established.

3. The task processing method for computing nodes according to claim 1, characterized in that, The step of sorting under the constraints of the task dependency graph to generate a task execution sequence includes: The system detects the in-degree value of each task in the real-time task dependency graph and adds the task to the candidate execution queue when the in-degree value of the task is zero. The tasks in the candidate execution queue are arranged in descending order according to a priority function to form the task execution sequence.

4. The task processing method for computing nodes according to claim 3, characterized in that, The formula for calculating the priority function is: Among them, P static For static priority, Uc is the CPU load rate of the target computing node, α and β are weighting coefficients, and e is a constant; P static The value of α ranges from 0 to 10, and the sum of α and β is 1.

5. The task processing method for computing nodes according to claim 1, characterized in that, In the task execution sequence, when the in-degree of multiple tasks is zero at the same time, the execution order of the tasks is selected in descending order of task priority.

6. A task processing device for a computing node, characterized in that, include: The parsing module is used to obtain the task set and parse the data dependencies and control dependencies between the tasks; The construction module is used to construct a task dependency graph based on the data dependency relationship and the control dependency relationship, wherein the task dependency graph represents the dependency direction between tasks through directed edges; The sorting module is used to sort the tasks to be executed under the constraints of the task dependency graph and generate a task execution sequence. The computing module is used to calculate the matching value between tasks and node resources based on the real-time collected CPU utilization, memory usage, and GPU utilization of computing nodes. The computing module is further used to map the CPU utilization, memory usage, and GPU utilization of the computing power node to a unit interval; According to the task type, the weight coefficients of the CPU utilization, memory occupancy, and GPU utilization of the computing power node are configured respectively; the adaptation value between the task and the node resources is calculated based on the weighted CPU utilization, weighted memory occupancy, and weighted GPU utilization of the computing power node; the adaptation value is used to reflect the matching degree between the task and the node resources. The formula for calculating the fit value between the task and node resources is: Among them, S j The overall adaptation value between task and node resources, where i is the ordinal number of the summation calculation, cpu is the CPU utilization of the computing node, mem is the memory usage, gpu is the GPU utilization, and ω is the overall resource utilization. i For dynamic weights, (1-ΔR) i () represents the single resource matching degree; The allocation module is used to dynamically allocate tasks to the computing power node with the highest adaptation value according to the task order of the task execution sequence.

7. A computer device, characterized in that, The computer device includes: a memory and at least one processor, wherein the memory stores instructions; The at least one processor invokes the instructions in the memory to cause the computer device to execute the task processing method of the computing node as described in any one of claims 1-5.

8. A computer-readable storage medium storing instructions thereon, characterized in that, When the instruction is executed by the processor, it implements the task processing method of the computing node as described in any one of claims 1-5.

Images