Heterogeneous task processing methods, apparatus, systems and computer-readable storage media

Abstract

This invention discloses a heterogeneous task processing method, apparatus, system, and computer-readable storage medium. The method includes real-time monitoring of the target task's runtime indicator vector, which includes processor computational demand indicators, I / O demand indicators, and data relevance indicators. Based on the target task's task type, corresponding feature weights are assigned to each dimension of the indicator vector, generating a weighted comprehensive feature vector. A composite sharding strategy is determined based on the comprehensive feature vector and a pre-defined sharding strategy library. The target task is split into multiple sharded tasks according to the composite sharding strategy, and a virtual computing device is created across physical nodes for each sharded task. The virtual computing device integrates at least two types of heterogeneous hardware resources. The execution results of each sharded task are progressively merged in a layered manner, outputting the task integration result. This invention overcomes the limitations of static resource allocation, supports multi-mode sharding, and achieves progressive merging.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a heterogeneous task processing method, apparatus, system, and computer-readable storage medium. Background Technology

[0002] With the rapid development of cloud computing and distributed computing technologies, containerization technology has become the mainstream method for modern application deployment due to its advantages such as lightweight nature, strong portability, and good resource isolation. Container orchestration systems, especially open-source platforms represented by Kubernetes, undertake the core task of scheduling and managing containerized workloads. However, when dealing with increasingly complex and diverse computing tasks, especially when involving high-performance computing, artificial intelligence training and inference, big data resource services, databases, and cloud database services, existing container scheduling technologies have the following shortcomings, making it difficult to meet the needs of fine-grained resource management and high-performance execution: 1) Static resource allocation strategies lack dynamic adaptability; 2) Coarse-grained resource allocation cannot meet the needs of heterogeneous collaboration; 3) Low task merging efficiency. Summary of the Invention

[0003] This invention aims to address at least one of the technical problems existing in the prior art. To this end, this invention proposes a heterogeneous task processing method, apparatus, system, and computer-readable storage medium, which can overcome the limitations of static resource allocation, support multi-mode fragmentation, and achieve progressive merging.

[0004] In a first aspect, embodiments of the present invention provide a heterogeneous task processing method, comprising: The system monitors the metrics vectors during the execution of the target task in real time. These metrics vectors include processor computing requirements metrics, I / O requirements metrics, and data relevance metrics. Based on the task type of the target task, assign corresponding feature weights to each dimension of the indicator vector and generate a weighted comprehensive feature vector. Based on the comprehensive feature vector and the preset sharding strategy library, a composite sharding strategy is determined; According to the composite sharding strategy, the target task is split into multiple shard tasks, and a virtual computing device is created across physical nodes for each shard task. The virtual computing device integrates at least two heterogeneous hardware resources. The execution results of each of the aforementioned fragmented tasks are merged in a hierarchical and progressive manner, and the integrated task result is output.

[0005] According to some embodiments of the present invention, the real-time monitoring of the indicator vector during the execution of the target task includes: Real-time monitoring of processor runtime, I / O operations per second, and data block size during the execution of the target task; Based on the processor's runtime, determine the processor's computational requirements. The I / O requirement metrics are determined based on the number of I / O operations per second and the data block size.

[0006] According to some embodiments of the present invention, determining the processor computing requirements based on the processor runtime includes: The processor's computational requirements are determined by weighting the ratio of the processor's runtime to the theoretical minimum processor runtime. And / or, The step of determining the IO requirement metrics based on the number of I / O operations per second and the data block size includes: The product of the number of I / O operations per second and the data block size is normalized to determine the I / O requirement index.

[0007] According to some embodiments of the present invention, the real-time monitoring of the indicator vector during the execution of the target task includes: Based on the parent task to which the target task is located, a directed acyclic graph of tasks is constructed. The directed acyclic graph of tasks contains multiple logical sub-task nodes, and the directed edges of each logical sub-task node are used to represent the data dependencies between the logical sub-task nodes. Based on the directed acyclic graph of the task, determine the data correlation index between the target task and its upstream and downstream logical subtasks.

[0008] According to some embodiments of the present invention, determining the data correlation index between the target task and its upstream and downstream logical subtasks based on the directed acyclic graph of the task includes: Based on the directed acyclic graph of the task, all direct downstream subtasks of the target task are determined, and the normalized downstream influence degree is determined based on the ratio of the total number of input dependencies of the direct downstream subtasks to the total number of input dependencies of the parent task. Based on the directed acyclic graph of the task, all direct upstream subtasks of the target task are determined, and the normalized upstream dependency is determined based on the ratio of the total number of output dependencies of the direct upstream subtasks to the total number of output dependencies of the parent task. Based on the downstream influence and upstream dependence, determine the data correlation index between the target task and the logical subtasks in the upstream and downstream.

[0009] According to some embodiments of the present invention, creating a virtual computing device across physical nodes for each of the sharded tasks includes: Broadcast the sharding descriptor corresponding to the sharding task to all target physical nodes participating in task sharding; After receiving the fragment descriptor, the target physical node forms a unified address space based on the address mapping information in the fragment descriptor to create a virtual computing device across physical nodes.

[0010] According to some embodiments of the present invention, the step of performing a hierarchical and progressive merging of the execution results of each of the segmented tasks and outputting a task integration result includes: In the case of completing a single fragmentation task, a CRC check operation is performed on the completed fragmentation task; If a preset number of the sharded tasks are completed, a cross-shard vector clock comparison operation is performed on the completed sharded tasks to perform progressive task merging. Before downstream processing begins, a business semantic verification operation based on write-ahead logs is performed on the merged sharded tasks.

[0011] In a second aspect, an embodiment of the present invention provides a heterogeneous task processing device, comprising: The indicator monitoring module is used to monitor the indicator vectors during the execution of the target task in real time. The indicator vectors include processor computing demand indicators, I / O demand indicators, and data correlation indicators. The vector generation module is used to assign corresponding feature weights to each dimension of the indicator vector according to the task type of the target task, and generate a weighted comprehensive feature vector. The strategy determination module is used to determine a composite sharding strategy based on the comprehensive feature vector and a preset sharding strategy library. The task sharding module is used to split the target task into multiple sharded tasks according to the composite sharding strategy, and to create a virtual computing device across physical nodes for each sharded task, wherein the virtual computing device integrates at least two heterogeneous hardware resources. The merge output module is used to perform hierarchical and progressive merging of the execution results of each of the fragmented tasks and output the task integration result.

[0012] Thirdly, embodiments of the present invention provide a heterogeneous task processing system, including a processor and a memory, wherein the memory stores a computer program, and the processor runs the computer program to implement the above-described heterogeneous task processing method.

[0013] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed, implements the above-described heterogeneous task processing method.

[0014] The embodiments of the present invention have at least the following beneficial effects: By monitoring the indicator vectors of the target task in real time and assigning corresponding feature weights to each dimension of the indicator vectors according to the task type, a weighted comprehensive feature vector is generated. Based on the comprehensive feature vector and a preset sharding strategy library, a composite sharding strategy is determined. Dynamic sharding decisions can be made by combining the indicator vectors of the task in time and the task type, realizing multi-mode task sharding. This allows for the creation of suitable virtual computing devices across physical nodes for each sharded task. By performing hierarchical and progressive merging of the execution results of each sharded task, the efficiency of task merging can be improved. This overcomes the limitations of static resource allocation, supports multi-mode sharding, and enables progressive merging.

[0015] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0016] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is a flowchart illustrating the steps of the heterogeneous task processing method according to an embodiment of the present invention. Figure 2 This is a schematic block diagram of a heterogeneous task processing device according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the heterogeneous task processing system according to an embodiment of the present invention. Detailed Implementation

[0017] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0018] In the description of this invention, "several" means one or more, "multiple" means two or more, "greater than," "less than," "exceeding," etc. are understood to exclude the stated number, and "above," "below," "within," etc. are understood to include the stated number. If "first," "second," etc. are used in the description, they are only for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance or implicitly indicating the number of indicated technical features or the order of the indicated technical features.

[0019] With the rapid development of cloud computing and distributed computing technologies, containerization technology has become the mainstream method for modern application deployment due to its advantages such as lightweight nature, strong portability, and good resource isolation. Container orchestration systems, especially open-source platforms represented by Kubernetes, undertake the core task of scheduling and managing containerized workloads. However, when dealing with increasingly complex and diverse computing tasks, especially those involving high-performance computing, artificial intelligence training and inference, big data resource services, databases, and cloud database services, existing container scheduling technologies have the following shortcomings: 1) Static resource allocation strategy, lacking dynamic adaptability: Traditional container schedulers (such as Kubernetes' default scheduling strategy) usually rely on preset static priority rules to schedule tasks. When making scheduling decisions, traditional container schedulers often cannot perceive or adapt to the dynamic behavior changes of applications during runtime and the real-time switching of task types, resulting in a disconnect between resource allocation and actual needs, which may lead to resource bottlenecks or waste, and reduce the overall efficiency and responsiveness of the system.

[0020] 2) Coarse-grained resource allocation cannot meet the needs of heterogeneous collaboration: Traditional container schedulers are too coarse in their resource allocation, usually reserving and allocating resources on a node-by-node basis (e.g., selecting nodes with specific hardware types based on node labels). This seriously ignores the complexity and differences of heterogeneous resources within a node, such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), TPU (Tensor Processing Unit), and high-speed network cards (such as those supporting RDMA protocol). Therefore, they cannot perform fine-grained, collaborative joint allocation and scheduling of heterogeneous resources.

[0021] 3) Inefficiency in the task merging phase: In distributed processing scenarios, a large task is often divided into multiple subtasks that are executed in parallel. Ultimately, the results of these subtasks need to be merged. Traditional container schedulers often lack effective coordination and optimization of the subtask execution progress. The result merging process typically requires waiting for all subtasks to complete, and any subtask that executes the slowest can become a bottleneck in the task merging process. During this period, the resources of completed tasks are idle, creating a window of resource utilization and increasing overall processing latency, severely impacting end-to-end processing efficiency.

[0022] Therefore, please refer to Figure 1This embodiment discloses a heterogeneous task processing method, including steps S100 to S500. It should be noted that the numbering of the steps in this embodiment is only for ease of review and understanding, and not to limit the execution order of the steps. The content of each step is described in detail below: S100, Real-time monitoring of the target task's runtime metric vectors, including processor computing requirements, I / O requirements, and data relevance metrics; For example, the target task refers to the task to be processed. In this embodiment, an index vector is used to quantify the running status of the target task. The index vector includes a processor computing demand index (CD), an I / O demand index (IR), and a data dependency index (DD). The index vector can be represented as [CD, IR, DD]. The processor computing demand index is used to quantify the degree to which the task depends on the processor's computing power, where the processor can be a central processing unit (CPU) or a graphics processing unit (GPU). The I / O demand index is used to reflect the task's sensitivity to storage bandwidth, while the data dependency index is used to measure the strength of data coupling between subtasks.

[0023] For ease of understanding, this embodiment uses the distributed training task of an artificial intelligence model as an example. The training task of an artificial intelligence model, as a large task, can be decomposed into multiple sub-tasks through logical partitioning, such as data loading (task s1), forward propagation inference of the first layer of the neural network (task s2), forward propagation inference of the second layer of the neural network (task s3), and backpropagation and parameter update (task s4). Any one of tasks s1, s2, s3, and s4 can be used as the target task in this embodiment.

[0024] Different subtasks have different hardware resource requirements during operation. For example, task s1 mainly consumes I / O (input / output) bandwidth, while tasks s1, s2, and s3 are highly dependent on the processor's computing power. Furthermore, there are certain data dependencies between the subtasks. For instance, tasks s2 and s3 are independent in data flow and can be processed in parallel, while the input data of task s4 depends on the output results of tasks s2 and s3. By monitoring the indicator vectors of the target task in real time, the current running status of the target task can be dynamically evaluated, providing a basis for subsequent decision-making on task segmentation, thereby achieving dynamic scheduling based on hardware resources and task topology.

[0025] S200. Based on the task type of the target task, assign corresponding feature weights to each dimension of the indicator vector and generate a weighted comprehensive feature vector. For example, to improve the rationality of subsequent sharding decisions, this embodiment provides auxiliary judgment criteria for sharding decisions based on the task type of the target task. The task type can be scientific computing or ETL (Extraction, Transformation, Loading), etc. Scientific computing refers to tasks that use computers to numerically solve complex mathematical problems proposed in scientific research or engineering technology; typical scenarios include numerical simulation, biological system modeling, or artificial intelligence model training. ETL describes data processing tasks that extract, transform, and load data from the source to the destination. Different task types have different hardware resource requirements. Therefore, this embodiment assigns corresponding feature weights to each dimension of the indicator vector according to the task type of the target task. For example, scientific computing tasks are highly dependent on processor computing power, so a higher feature weight can be assigned to the processor computing requirement (CD) indicator, such as 0.6~0.8, while the feature weights of the I / O requirement (IR) indicator and the data relevance (DD) indicator are assigned lower values, and the sum of the feature weights of CD, IR, and DD equals 1. As another example, ETL tasks mainly consume I / O bandwidth, so a higher feature weight can be assigned to the I / O requirement (IR) indicator, such as 0.5~0.7, while the processor computing requirement (CD) indicator and the data relevance (DD) indicator are assigned lower feature weights. In this way, each dimension of the indicator vector can be weighted and adjusted to generate a comprehensive feature vector. The comprehensive feature vector can more accurately and reasonably reflect the actual hardware resource requirements of the target task, thus providing a reliable basis for subsequent dynamic task partitioning decisions.

[0026] S300. Based on the comprehensive feature vector and the preset sharding strategy library, determine the composite sharding strategy; Exemplarily, the sharding policy library is configured with multiple sharding policies and records the sharding basis for different sharding policies. For example, assuming that the comprehensive feature vector is represented as [CD, IR, DD], when CD > 0.7 and IR < 0.3, the composite sharding policy is determined to be mathematical sharding, and the GPU node is the preferred allocation node; also, for example, when 0.3 < IR < 0.6 and DD > 0.8, the composite sharding policy is determined to be data sharding, and the node that supports the NVMe (Non-Volatile Memory Express) protocol is preferentially allocated; again, for example, when CD < 0.4 and IR > 0.8, the composite sharding policy adopts time sharding, and the node with high-bandwidth network is preferentially allocated. It should be noted that the threshold values in the above examples are only example thresholds and can be specifically configured according to different actual application requirements. By integrating different sharding policies such as different mathematical sharding (applicable to matrix operations), data sharding (applicable to ETL tasks), and time sharding (applicable to stream processing) into the same scheduling framework, multi-mode task sharding can be achieved. Moreover, through the sharding policy library, custom extension of sharding rules can be realized, which can be adapted to different computing frameworks, such as Spark or Flink, etc., which is beneficial to enhancing the scalability of sharding policies and framework compatibility. [[ID=!]]

[0027] S400. Split the target task into multiple sharding tasks according to the composite sharding policy, and create a virtual computing device across physical nodes for each sharding task. The virtual computing device integrates at least two heterogeneous hardware resources; Exemplarily, the execution node of the target task can be a certain node of the distributed system. During the running of the target task, this execution node enters the full-load state. Splitting the target task into multiple sharding tasks according to the composite sharding policy can reduce the load of this execution node and improve the execution efficiency of the target task. Among them, the task types of the sharding tasks formed after splitting can be the same as or different from the target task, that is, the requirements for hardware resources of different sharding tasks may vary. For example, for the above task s1, during the data loading process, the task can be split into sharding tasks such as data reading, data calculation (preprocessing), and data caching. The data reading and data caching tasks mainly consume I / O bandwidth, while the data calculation task mainly depends on the processor computing power. According to the load status of the hardware resources of different physical nodes in the distributed system, create a virtual computing device across physical nodes for each sharding task. The virtual computing device integrates at least two heterogeneous hardware resources, such as CPU, GPU, FPGA, TPU, or high-speed network card (such as supporting the RDMA protocol), etc., so as to balance the resource idle rate of different nodes and improve the task processing efficiency. [[ID=!]]

[0028] S500. Perform hierarchical progressive merging on the execution results of each sharding task and output the task integration result.

[0029] For example, in related technologies, the execution results are merged only after all the sharded tasks have been completed, which results in a bottleneck effect. In contrast, this embodiment performs a hierarchical and progressive merging of the execution results of each sharded task. The sharded tasks can be merged after a preset number of sharded tasks have been completed, and the results of tasks that have completed shard merging are allowed to enter the next processing stage in advance, which can avoid the bottleneck effect and improve task processing efficiency.

[0030] Therefore, this embodiment monitors the indicator vector of the target task in real time, assigns corresponding feature weights to each dimension of the indicator vector according to the task type of the target task, and generates a weighted comprehensive feature vector. Based on the comprehensive feature vector and the preset sharding strategy library, a composite sharding strategy is determined. Dynamic sharding decisions can be made by combining the indicator vector of the task in runtime and the task type to realize multi-mode task sharding, thereby creating a suitable virtual computing device across physical nodes for each sharded task. By performing hierarchical and progressive merging of the execution results of each sharded task, the efficiency of task merging can be improved. This can overcome the limitations of static resource allocation, support multi-mode sharding, and realize progressive merging.

[0031] In step S100, the indicator vectors during the execution of the target task are monitored in real time, including: Real-time monitoring of processor runtime, I / O operations per second, and data block size during target task execution; Determine the processor's computing requirements based on its runtime. Determine IO demand metrics based on the number of I / O operations per second and the data block size.

[0032] For example, during the execution of the target task, the hardware resources of the distributed system nodes executing the target task can be monitored in real time. This includes, for instance, collecting data such as CPU utilization, processor runtime, disk I / O parameters, and memory access characteristics through a monitoring agent. Processor computing requirements are determined based on processor runtime, and I / O requirements are determined based on the number of I / O operations per second and data block size. Combining this with hardware resource utilization analysis allows for a reliable basis for task sharding.

[0033] In some application examples, the above steps involve determining processor computational requirements based on processor runtime, including: The processor computing requirements are determined by weighting the ratio of the processor's runtime to the theoretical minimum processor runtime.

[0034] For example, processor runtime is the real-time data collected on the execution time of a processor (CPU or GPU) to the target task, while the theoretical minimum processor runtime is used to characterize the minimum runtime required for the processor to execute the target task to the current progress under ideal conditions. The ratio of processor runtime to theoretical minimum processor runtime can quantify the target task's dependence on processor computing power. By weighting the ratio of the two through weighting factors, adaptive adjustments can be made according to different application scenarios, which is beneficial to improving the reliability of engineering applications.

[0035] In some application examples, the above steps involve determining IO requirement metrics based on the number of I / O operations per second and the data block size, including: The product of the number of I / O operations per second and the data block size is normalized to determine the I / O requirement metrics.

[0036] For example, for I / O operations, the I / O read / write rate can be characterized by the number of I / O operations per second, while the data block size can reflect the amount of data involved in a read / write request. By normalizing the product between the number of I / O operations per second and the data block size, the IO demand index can be obtained, which can facilitate the quantitative comparison of the IO demand index and thus reflect the sensitivity of the target task to storage bandwidth.

[0037] In some application examples, step S100, which involves real-time monitoring of the target task's runtime metric vectors, also includes: Based on the parent task to which the target task is located, construct a directed acyclic graph of tasks. The directed acyclic graph of tasks contains multiple logical subtask nodes, and the directed edges of each logical subtask node are used to represent the data dependencies between logical subtask nodes. Based on the directed acyclic graph of tasks, determine the data correlation indicators between the target task and its upstream and downstream logical subtasks.

[0038] For example, as described above, the AI ​​model training task is decomposed into multiple subtasks through logical partitioning, and each subtask can serve as the target task in this embodiment. Therefore, the AI ​​model training task is the parent task of the target task. Based on the parent task containing the target task, a directed acyclic graph (DAG) is constructed. Each subtask of the parent task can be considered a logical subtask node; for example, task s1 is the first node, task s2 is the second node, task s3 is the third node, and so on. The directed edges of each logical subtask node represent the data dependencies between them. For instance, if there is a data dependency between task s1 and tasks s2 and s3, but no data dependency between task s2 and s3, then in the DAG, there are directed edges between the first node and the second and third nodes, but no directed edge between the second and third nodes. The DAG allows for analysis of the data coupling strength between subtasks. By determining the data correlation index between the target task and its upstream and downstream logical subtasks, the data coupling strength between tasks can be quantified.

[0039] In some application examples, the above steps—determining data correlation metrics between the target task and its upstream and downstream logical subtasks based on the directed acyclic graph of tasks—include: Based on the directed acyclic graph of the task, all direct downstream subtasks of the target task are identified, and the normalized downstream influence degree is determined based on the ratio of the total number of input dependencies of the direct downstream subtasks to the total number of input dependencies of the parent task. Based on the directed acyclic graph of the task, all directly upstream subtasks of the target task are identified, and the normalized upstream dependency is determined based on the ratio of the total number of output dependencies of the directly upstream subtasks to the total number of output dependencies of the parent task. Based on downstream impact and upstream dependence, determine the data correlation indicators between the target task and the logical subtasks in the upstream and downstream.

[0040] For example, for a logical subtask node corresponding to a target task, the downstream subtask nodes directly connected to that node can be determined based on the task's directed acyclic graph (DAG). This identifies the set of direct downstream subtasks of the target task. Counting the number of direct downstream subtasks determines the total number of input dependencies for each direct downstream subtask. Counting all input dependencies in the DAG determines the total number of input dependencies for the parent task. The normalized downstream influence is determined based on the ratio of the total number of input dependencies for direct downstream subtasks to the total number of input dependencies for the parent task. Similarly, through statistical analysis, the total number of output dependencies for direct upstream subtasks of the target task and the total number of output dependencies for the parent task can be determined. The normalized upstream dependency is then determined based on the ratio of these two values. Based on the downstream influence and upstream dependency, a data correlation index between the target task and its upstream and downstream logical subtasks is determined. For example, in some applications, the larger value between downstream influence and upstream dependency is taken, and then weighted according to the node attributes of the logical sub-task nodes corresponding to the target task to obtain a data relevance index. For instance, if the logical sub-task node corresponding to the target task is a master node, it is assigned a high weight (e.g., 1.2); otherwise, a low weight is assigned (e.g., 1.0). Alternatively, in other applications, a weighted average of downstream influence and upstream dependency is taken, and then weighted again according to the node attributes of the logical sub-task nodes corresponding to the target task to obtain a data relevance index, which can balance the impact of downstream influence and upstream dependency.

[0041] In step S400 above, creating a virtual computing device across physical nodes for each sharded task includes: Broadcast the sharding descriptor corresponding to the sharding task to all target physical nodes participating in the task sharding; After receiving the fragment descriptor, the target physical node forms a unified address space based on the address mapping information in the fragment descriptor in order to create a virtual computing device across physical nodes.

[0042] For example, a sharding descriptor describes key information such as the attributes, dependencies, resource requirements, and execution context of a task shard, facilitating the correct allocation, migration, monitoring, and recovery of task shards. The sharding descriptor corresponding to the sharded task is broadcast to all target physical nodes participating in the task sharding. Target physical nodes refer to nodes in the distributed system. Upon receiving the sharding descriptor, for example, for a node configured with an FPGA, the FPGA triggers a local hardware interrupt via a doorbell interrupt. After all target physical nodes participating in the task sharding have completed the hardware interrupt triggering, a unified virtual address space is established based on the address mapping information in the sharding descriptor. This allows heterogeneous hardware resources distributed across multiple physical nodes to appear as a single addressable virtual computing device. These heterogeneous hardware resources can be CPUs, GPUs, FPGAs, TPUs, or high-speed network cards (such as those supporting RDMA protocols). Compared to traditional tag-based node selection schemes, this embodiment can integrate hardware resources across physical nodes, enabling fine-grained collaboration of heterogeneous hardware resources, which is beneficial for adapting to increasingly complex and diverse computing tasks.

[0043] In some application examples, step S500 involves performing a hierarchical, progressive merging of the execution results of each shard task, outputting the task integration result, including: When a single sharding task is completed, a CRC (Cyclic Redundancy Check) operation is performed on the completed sharding task. If a preset number of sharding tasks are completed, cross-shard vector clock comparison operations are performed on the completed sharding tasks to perform progressive task merging. Before downstream processing begins, perform business semantic verification operations based on write-ahead logs on the merged sharded tasks.

[0044] For example, in practical applications, the number of sharding tasks is usually multiple, and it is difficult to ensure that all sharding tasks can be completed at the same time. To reduce the impact of the "bottleneck" effect, this embodiment adopts a hierarchical progressive merging approach to integrate the output results of all sharding tasks. "Hierarchical" refers to dividing the verification into multiple layers, where CRC verification is layer L1, vector clock alignment is layer L2, and business semantic verification based on rain boot logs is layer L3. Since the completion time of each sharding task may vary, performing CRC verification on a single sharding task after its completion does not require waiting for other sharding tasks, thus reducing the load pressure of centralized verification of all sharding tasks. As time progresses, the number of completed sharding tasks gradually increases. After completing a preset number of sharding tasks, cross-shard vector clock alignment is performed on the completed sharding tasks for progressive task merging, without waiting for all sharding tasks to complete before merging, which improves the efficiency of task result merging. The threshold for the number of sharding tasks to be merged can be specifically configured according to different application scenarios. When the execution results of the sharding tasks have been merged to a certain extent, such as reaching 50%, downstream processing can begin without waiting for all sharding tasks to be merged, which improves data processing efficiency. Before downstream processing begins, a business semantic verification operation based on write-ahead logs is performed on the merged sharding tasks. This write-ahead log verification records the intermediate calculation results in a traceable and rollback-capable manner before formally merging them into the final output, and uses a verification mechanism to ensure the integrity and consistency of the merging process, thus improving the fault tolerance of the merge. When sharding tasks are merged, only the key shards with high data correlation need to be recalculated.

[0045] Please refer to Figure 2 Based on the above technical concept, this embodiment provides a heterogeneous task processing device, comprising: The indicator monitoring module 110 is used to monitor the indicator vectors during the execution of the target task in real time. The indicator vectors include processor computing demand indicators, IO demand indicators and data correlation indicators. The vector generation module 120 is used to assign corresponding feature weights to each dimension of the indicator vector according to the task type of the target task, and generate a weighted comprehensive feature vector. The strategy determination module 130 is used to determine a composite sharding strategy based on the comprehensive feature vector and the preset sharding strategy library. The task sharding module 140 is used to split the target task into multiple sharded tasks according to the composite sharding strategy, and to create a virtual computing device across physical nodes for each sharded task. The virtual computing device integrates at least two heterogeneous hardware resources. The merge output module 150 is used to perform hierarchical and progressive merging of the execution results of each segment task and output the integrated task result.

[0046] The inventive concept of this heterogeneous task processing device embodiment is the same as that of the heterogeneous task processing method embodiment described above. Content not covered in this heterogeneous task processing device embodiment can be referred to in the heterogeneous task processing method embodiment described above, and will not be repeated here. This embodiment monitors the indicator vector during the execution of the target task in real time, and assigns corresponding feature weights to each dimension of the indicator vector according to the task type of the target task, generating a weighted comprehensive feature vector. Based on the comprehensive feature vector and a preset sharding strategy library, a composite sharding strategy is determined. Dynamic sharding decisions can be made by combining the indicator vector during task execution and the task type, achieving multi-mode task sharding, thereby creating a suitable virtual computing device across physical nodes for each sharded task. By performing hierarchical and progressive merging of the execution results of each sharded task, the efficiency of task merging is improved. This overcomes the limitations of static resource allocation, supports multi-mode sharding, and achieves progressive merging.

[0047] Please refer to Figure 3 This embodiment provides a heterogeneous task processing system, including a processor 210 and a memory 220. The memory 220 stores a computer program, and the processor 210 executes the computer program to implement the above-mentioned heterogeneous task processing method. Details of the heterogeneous task processing method can be found above and will not be repeated here. This embodiment monitors the indicator vector of the target task in real time, assigns corresponding feature weights to each dimension of the indicator vector according to the task type of the target task, and generates a weighted comprehensive feature vector. Based on the comprehensive feature vector and a preset sharding strategy library, a composite sharding strategy is determined. Dynamic sharding decisions can be made by combining the indicator vector and task type during task execution, achieving multi-mode task sharding, thereby creating a suitable virtual computing device across physical nodes for each sharded task. By performing hierarchical and progressive merging of the execution results of each sharded task, the efficiency of task merging is improved. This overcomes the limitations of static resource allocation, supports multi-mode sharding, and achieves progressive merging.

[0048] This embodiment provides a computer-readable storage medium storing a computer program. When the computer program is run, it implements the heterogeneous task processing method described above. Details of the heterogeneous task processing method can be found above and will not be repeated here. This embodiment monitors the indicator vector of the target task in real time and assigns corresponding feature weights to each dimension of the indicator vector according to the task type of the target task, generating a weighted comprehensive feature vector. Based on the comprehensive feature vector and a preset sharding strategy library, a composite sharding strategy is determined. Dynamic sharding decisions can be made by combining the indicator vector and task type during task execution, achieving multi-mode task sharding. This creates suitable virtual computing devices across physical nodes for each sharded task. By progressively merging the execution results of each sharded task in a layered manner, task merging efficiency is improved. This overcomes the limitations of static resource allocation, supports multi-mode sharding, and achieves progressive merging.

[0049] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A heterogeneous task processing method, characterized in that, include: The system monitors the metrics vectors during the execution of the target task in real time. These metrics vectors include processor computing requirements metrics, I / O requirements metrics, and data relevance metrics. Based on the task type of the target task, assign corresponding feature weights to each dimension of the indicator vector and generate a weighted comprehensive feature vector. Based on the comprehensive feature vector and the preset sharding strategy library, a composite sharding strategy is determined; According to the composite sharding strategy, the target task is split into multiple shard tasks, and a virtual computing device is created across physical nodes for each shard task. The virtual computing device integrates at least two heterogeneous hardware resources. The execution results of each of the aforementioned fragmented tasks are merged in a hierarchical and progressive manner, and the integrated task result is output.

2. The heterogeneous task processing method according to claim 1, characterized in that, The real-time monitoring index vector for the target task during runtime includes: Real-time monitoring of processor runtime, I / O operations per second, and data block size during the execution of the target task; Based on the processor's runtime, determine the processor's computational requirements. The I / O requirement metrics are determined based on the number of I / O operations per second and the data block size.

3. The heterogeneous task processing method according to claim 2, characterized in that, The step of determining the processor computing requirements based on the processor's runtime includes: The processor's computational requirements are determined by weighting the ratio of the processor's runtime to the theoretical minimum processor runtime. And / or, The step of determining the IO requirement metrics based on the number of I / O operations per second and the data block size includes: The product of the number of I / O operations per second and the data block size is normalized to determine the I / O requirement index.

4. The heterogeneous task processing method according to claim 1, 2, or 3, characterized in that, The real-time monitoring index vector for the target task during runtime includes: Based on the parent task to which the target task is located, a directed acyclic graph of tasks is constructed. The directed acyclic graph of tasks contains multiple logical sub-task nodes, and the directed edges of each logical sub-task node are used to represent the data dependencies between the logical sub-task nodes. Based on the directed acyclic graph of the task, determine the data correlation index between the target task and its upstream and downstream logical subtasks.

5. The heterogeneous task processing method according to claim 4, characterized in that, The step of determining the data correlation indicators between the target task and its upstream and downstream logical subtasks based on the directed acyclic graph of the task includes: Based on the directed acyclic graph of the task, all direct downstream subtasks of the target task are determined, and the normalized downstream influence degree is determined based on the ratio of the total number of input dependencies of the direct downstream subtasks to the total number of input dependencies of the parent task. Based on the directed acyclic graph of the task, all direct upstream subtasks of the target task are determined, and the normalized upstream dependency is determined based on the ratio of the total number of output dependencies of the direct upstream subtasks to the total number of output dependencies of the parent task. Based on the downstream influence and upstream dependence, determine the data correlation index between the target task and the logical subtasks in the upstream and downstream.

6. The heterogeneous task processing method according to claim 1, characterized in that, The creation of virtual computing devices across physical nodes for each of the sharded tasks includes: Broadcast the sharding descriptor corresponding to the sharding task to all target physical nodes participating in task sharding; After receiving the fragment descriptor, the target physical node forms a unified address space based on the address mapping information in the fragment descriptor to create a virtual computing device across physical nodes.

7. The heterogeneous task processing method according to claim 1, characterized in that, The step of progressively merging the execution results of each of the segmented tasks and outputting the task integration result includes: In the case of completing a single fragmentation task, a CRC check operation is performed on the completed fragmentation task; If a preset number of the sharded tasks are completed, a cross-shard vector clock comparison operation is performed on the completed sharded tasks to perform progressive task merging. Before downstream processing begins, a business semantic verification operation based on write-ahead logs is performed on the merged sharded tasks.

8. A heterogeneous task processing device, characterized in that, include: The indicator monitoring module is used to monitor the indicator vectors during the execution of the target task in real time. The indicator vectors include processor computing demand indicators, I / O demand indicators, and data correlation indicators. The vector generation module is used to assign corresponding feature weights to each dimension of the indicator vector according to the task type of the target task, and generate a weighted comprehensive feature vector. The strategy determination module is used to determine a composite sharding strategy based on the comprehensive feature vector and a preset sharding strategy library. The task sharding module is used to split the target task into multiple sharded tasks according to the composite sharding strategy, and to create a virtual computing device across physical nodes for each sharded task, wherein the virtual computing device integrates at least two heterogeneous hardware resources. The merge output module is used to perform hierarchical and progressive merging of the execution results of each of the fragmented tasks and output the task integration result.

9. A heterogeneous task processing system, comprising a processor and a memory, wherein the memory stores a computer program, characterized in that, When the processor runs the computer program, it is used to implement the heterogeneous task processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is run, it implements the heterogeneous task processing method as described in any one of claims 1 to 7.

Images