Fine-grained multi-operator parallel scheduling method and system based on heterogeneous data flow architecture

Abstract

The application relates to a fine-grained multi-operator parallel scheduling method and system based on a heterogeneous data flow architecture. The method comprises the following steps: establishing an operator topology graph according to operators related to a task scheduling process obtained through decomposition, and constructing an operator data flow graph; analyzing data dependency between operators in a task according to the operator topology graph, grouping the operators based on the data dependency and runtime characteristics of the operators, and maintaining a corresponding scheduling queue for each operator group; scheduling parallel operator groups according to the scheduling queue, and allocating computing resources to the parallel operators based on the runtime characteristics; dynamically reconstructing an original operator data flow graph of the parallel operator groups according to the operator data flow graph; after the PE data flow array parallelly executes the operators, checking whether each scheduling queue is empty, if yes, ending the operator scheduling, otherwise, continuing to schedule the parallel operator groups according to the scheduling queue. The application can reduce access to SPM and realize efficient multi-operator parallel scheduling.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data flow, in particular to a fine-grained multi-operator parallel scheduling method and system based on a heterogeneous data flow architecture. BACKGROUND

[0002] In the field of modern computing, technology is gradually changing from general-purpose computing to special-purpose computing, and the integration of big data, graph computing and artificial intelligence has become a mainstream trend. This fusion computing has the characteristics of data-intensive, complex interweaving of control and storage access, which brings challenges to computer systems with single data flow or control flow architecture. An important problem is that in multiple computing modes, the mixed execution efficiency of operators is not high, which is manifested as fixed scheduling strategy, single execution order, and insufficient utilization of parallelism between operators.

[0003] Data flow architecture creates a data flow graph between processing units through a data-driven mechanism, aiming to reduce storage access and synchronization overhead. This architecture performs well in deep learning and scientific computing because it can effectively improve data parallelism and resource utilization. The concept of data flow originated from data flow computers and has evolved from instruction-level parallelism to thread-level parallelism, with representative examples including the Codelet model and the EARTH data flow computer.

[0004] In the execution of a data flow program, a computing task is decomposed into multiple small tasks, which are organized into a data flow graph (directed acyclic graph), where nodes represent computing tasks and edges represent task dependency relationships. In the current implementation of data flow architecture, the task scheduling capability is limited, and the system generally executes tasks sequentially, lacking the ability to dynamically adjust the order.

[0005] To solve these problems, the existing technology provides a new type of heterogeneous data flow architecture DFU. This architecture is based on RISCV core and PE data flow array, including: PE data flow array for data flow computing, RISCV core for controlling array operation, DFU_Host host responsible for data storage, SPM for data caching, and register stack for storing operands and operation codes. The architecture creates a data flow graph for operators and deploys it on the PE data flow array, input data is transmitted from the RISCV core's DRAM to the shared SPM through DMA, and the PE data flow array extracts data from the SPM for processing, achieving high parallelism and significantly reducing memory access time.

[0006] However, the current PE data flow array only supports data flow execution of a single operator, and lacks corresponding scheduling strategies and runtime support for fine-grained parallelism of multiple operators. In a multi-task environment, tasks can be divided into multiple different operators, but the existing data flow accelerator cannot effectively select and schedule appropriate operators for parallel execution. When multiple operators are executed in parallel, in order to improve data reuse and reduce memory access, a suitable data flow graph needs to be constructed for deployment. However, how to construct a new data flow graph from a single operator environment to support parallel operators is a challenge.

[0007] At present, fine-grained multi-operator parallelism can be achieved by analyzing the data flow relationship between different operators and manually constructing a data flow graph. However, in a multi-task environment, the number of combinations of different operators is huge, and it is almost impossible to manually configure all fine-grained multi-operator parallel data flow graphs. Therefore, how to design a fine-grained multi-operator parallel scheduling method and runtime system on a new heterogeneous data flow architecture to dynamically select appropriate operators for parallel execution and dynamically generate a suitable data flow graph is a problem to be solved.

[0008] In addition, on the one hand, there are differences in the understanding of those skilled in the art; on the other hand, the applicant has studied a large number of literatures and patents when making the present application, but due to the limited space, all details and contents have not been listed in detail, which does not mean that the present application does not have these prior art characteristics, on the contrary, the present application has all the characteristics of the prior art, and the applicant reserves the right to add relevant prior art in the background art. SUMMARY

[0009] The task scheduling capability of the current data flow architecture is limited, and the system usually executes tasks in sequence, lacking the ability to dynamically adjust the sequence. In view of this defect, the prior art proposes a new heterogeneous data flow architecture DFU (Device Firmware Upgrade). The architecture is constructed based on RISCV cores and PE data flow arrays, and includes a PE data flow array, a RISCV core, a DFU_Host main memory, an SPM, and a register heap. The architecture creates a data flow graph for the operator and deploys it on the PE data flow array, the input data is transmitted from the DRAM of the RISCV core to the shared SPM through DMA, and the PE data flow array extracts data from the SPM for processing, thereby realizing high parallelism and greatly reducing the memory access time. However, the current PE data flow array only supports single-operator data flow execution, and lacks corresponding scheduling strategies and runtime support for fine-grained parallelism of multiple operators. In a multi-task environment, tasks can be divided into multiple different operators, but the existing data flow accelerator cannot effectively select and schedule suitable operators for parallel execution. When multiple operators are executed in parallel, in order to improve data reuse and reduce memory access times, a suitable data flow graph needs to be constructed for deployment. However, how to construct a new data flow graph from a single-operator environment to support parallel operators is a difficult problem that has not been solved.

[0010] Currently, fine-grained multi-operator parallelism can be achieved by analyzing the data flow relationship between different operators and manually constructing a data flow graph. However, in a multi-task environment, the number of combinations of different operators is large, and it is almost impossible to manually configure all fine-grained multi-operator parallel data flow graphs. Therefore, how to design a fine-grained multi-operator parallel scheduling method and runtime system on the new heterogeneous data flow architecture to dynamically select suitable operators for parallel execution and dynamically generate suitable data flow graphs is a problem that needs to be solved.

[0011] In view of the deficiencies of the prior art, the present application provides, from a first aspect, a fine-grained multi-operator parallel scheduling method based on a heterogeneous data flow architecture, the method comprising: establishing an operator topology graph according to the operators related to the task scheduling process obtained by decomposition, and constructing an operator data flow graph; analyzing the data dependency relationship between the operators in the task according to the operator topology graph, grouping the operators based on the data dependency relationship and the runtime characteristics of the operators, and maintaining a corresponding scheduling queue for each operator group; scheduling the parallel operator groups according to the scheduling queue, and allocating computing resources to the parallel operators to be parallelized within the parallel group based on the runtime characteristics; dynamically reconstructing the original operator data flow graph of the parallel operator group according to the operator data flow graph; when the PE data flow array parallel execution operator ends, checking whether each scheduling queue is empty, if yes, ending the operator scheduling, otherwise, continuing to schedule the parallel operator groups according to the scheduling queue.

[0012] The application combines to-be-scheduled operators into three different types of operator groups by analyzing data dependencies between operators, dynamically reconstructs a multi-operator parallel subtask data flow graph according to the categories of the parallel scheduling operators, reduces the number of accesses to DRAM by the operators, reduces the access to SPM through data flow between PEs, and realizes efficient multi-operator parallel scheduling.

[0013] The application dynamically allocates PE data flow arrays to multi-operators in an operator group according to the structure of the PE data flow array on the DFU accelerator, balances the load of PE computing resources, improves the utilization of PEs, ensures efficient multi-operator parallelism, and reduces idle time of computing resources.

[0014] According to a preferred embodiment, the method for constructing an operator data flow graph on a PE data flow array includes code segment division of the decomposed operators, generation of subtasks corresponding to each code segment, and sequential execution of the subtasks by the PE data flow array. This can automatically finely decompose and map the operators to the PE data flow array for execution, simplifies the complexity of manually constructing a data flow graph, enables the system to flexibly cope with a large number of operator combinations in a multi-task environment, and improves execution efficiency and system scalability.

[0015] According to a preferred embodiment, the method for grouping operators based on data dependency relationships and runtime characteristics of the operators includes dividing the operators into data-dependent operator groups, operator groups processing the same data, and operator groups processing different data; wherein, in an operator topology graph, the dependency relationship between each operator and its successor is identified, if an operator has only one successor in the operator topology graph, the operator and the successor are combined into a data-dependent operator group, and if the capacity and the number of registers of the SPM are sufficient, the operator and the successor are merged. This can effectively organize and manage the dependency relationship between the operators, reduce data transmission overhead, improve the parallelism of operator execution and resource utilization, and thus improve overall computing performance.

[0016] According to a preferred embodiment, the method for grouping operators based on data dependency relationships and runtime characteristics of the operators further includes at least one storage operation on the result data generated by the current operator if the operator has multiple successors in the operator topology graph, and combination of all the successors into a data-sharing operator group to avoid repeated execution of the current operator and repeated loading of the result data generated by the current operator. This reduces repeated calculation and transmission of data, optimizes the use efficiency of system resources, reduces waste of memory and computing resources, and improves the overall execution speed and efficiency of the system.

[0017] According to a preferred embodiment, the method of grouping operators based on data dependency and runtime characteristics of operators further comprises: when selecting operators in the data sharing operator group, dynamically updating the head pointer and tail pointer of the data sharing operator ordered list based on a greedy method, keeping the average PE utilization rate of the operator group close to the average PE utilization rate of the current operator group in the data sharing operator ordered list, so as to avoid the PE utilization rate of the parallel operator group being too low. In this way, the efficiency of PE use is optimized, and it is ensured that the system can still operate efficiently under high load.

[0018] According to a preferred embodiment, the method of allocating computing resources to parallel operators comprises: calculating the computing resources of each operator according to the proportion of the CAL function component occupancy of each operator in parallel scheduling; and allocating a corresponding number of processing units to the operators in the operator group according to the calculated computing resources. In this way, the computing resources can be reasonably allocated, the number of processing units can be dynamically adjusted according to the actual needs of the operators, resource waste can be avoided, and the overall computing efficiency and resource utilization of the system can be improved.

[0019] The present application provides, from a second aspect, a fine-grained multi-operator parallel scheduling system based on a heterogeneous data flow architecture, comprising a processor configured to: establish an operator topology graph according to the operators related to the task scheduling process obtained by decomposition, and construct an operator data flow graph; analyze the data dependency between operators in the task according to the operator topology graph, group the operators based on the data dependency and the runtime characteristics of the operators, and maintain a corresponding scheduling queue for each operator group; schedule the parallel operator groups according to the scheduling queue, and allocate computing resources to the parallel operators to be executed in parallel within the parallel group based on the runtime characteristics; dynamically reconstruct the original operator data flow graph of the parallel operator group according to the operator data flow graph; when the parallel execution of the operators ends, check whether each scheduling queue is empty, if yes, end the operator scheduling, otherwise, continue to schedule the parallel operator groups according to the scheduling queue.

[0020] By automatically establishing the operator topology graph and constructing the operator data flow graph through the processor, the data dependency between operators can be accurately analyzed, and the operators can be grouped and scheduled based on these dependencies and runtime characteristics. This efficient scheduling mechanism can make full use of hardware resources, especially the parallel computing capability of processing elements (PEs), dynamically adjust computing resource allocation and data flow graph reconstruction, improve the overall processing performance and resource utilization of the system, and reduce resource idling and computing bottlenecks.

[0021] According to a preferred embodiment, the processor is further configured to, when performing the step of grouping the operators based on the data dependency and the runtime characteristics of the operators, divide the operators into operator groups with data dependency, operator groups processing the same data, and operator groups processing different data; wherein, in the operator topology graph, the dependency of each operator and its successor is identified, if the operator has only one successor in the operator topology graph, the operator and the successor are combined into a data-dependent operator group, and if the capacity and the number of registers of the SPM are sufficient, the operator and the successor are merged.

[0022] When grouping the operators, the processor can identify the dependencies in the operator topology graph and combine the operators into different types of operator groups according to these dependencies. This grouping method optimizes the allocation of hardware resources for operator execution by fully utilizing the capacity of the SPM and the number of registers, optimizes resource utilization, reduces data transmission overhead and waiting time, and improves the execution efficiency and response speed of the system.

[0023] According to a preferred embodiment, the processor is further configured to, when performing the step of grouping the operators based on the data dependency and the runtime characteristics of the operators, if the operator has multiple successors in the operator topology graph, perform at least one storage operation on the result data generated by the current operator, and combine all the successor operators into a data-sharing operator group to avoid repeated execution of the current operator and repeated loading of the result data generated by the current operator.

[0024] When processing operators with multiple successors, the processor stores the result data and combines the successor operators into a data-sharing operator group. This method effectively avoids repeated computation and repeated loading of data, optimizes data storage and transmission paths, improves the utilization efficiency of storage resources and computing resources, reduces the waste of hardware resources and the overall processing time of the system, and improves the performance and efficiency of parallel computing.

[0025] According to a preferred embodiment, the processor is further configured to, when performing the step of grouping the operators based on the data dependency and the runtime characteristics of the operators, when selecting operators in the data-sharing operator group, dynamically update the head pointer and the tail pointer of the data-sharing operator ordered list based on the greedy method, keep the average PE utilization rate of the operator group close to the average PE utilization rate of all operators in the data-sharing operator ordered list, and avoid the PE utilization rate of the parallel operator group being too low.

[0026] The processor adopts a greedy algorithm to dynamically update the head pointer and tail pointer of the data sharing operator ordered list, so as to keep the average PE utilization rate of the operator group close to the average PE utilization rate of all operators in the ordered list. Through this dynamic adjustment, the processor can balance the allocation of hardware computing resources, avoid the overload or idling of some PEs, optimize the utilization rate of the overall PEs, and ensure that the system can still run efficiently and stably in a high-load and complex task environment. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 is a flowchart of the fine-grained multi-operator parallel scheduling method based on the heterogeneous data flow architecture provided by the present application;

[0028] Figure 2 is a hardware structure schematic diagram of the data flow accelerator based on the heterogeneous data flow architecture provided by the present application;

[0029] Figure 3 is a multi-operator hierarchical partitioning schematic diagram of the heterogeneous data flow architecture provided by the present application;

[0030] Figure 4 is a multi-operator grouping schematic diagram provided by the present application;

[0031] Figure 5 is a schematic diagram of a new data flow graph formed by splicing different operators subtasks provided by the present application;

[0032] Figure 6 is an operator topology diagram of the Unet network provided by the present application;

[0033] Figure 7 is a data flow diagram of the activation function Relu and the maximum pooling layer MaxPool of the Unet network provided by the present application;

[0034] Figure 8 is an operator group classification schematic diagram of the Unet network provided by the present application;

[0035] Figure 9 is a reconstruction and scheduling schematic diagram of the operators of the four maximum pooling layers MaxPool of the data sharing of the Unet network provided by the present application. DETAILED DESCRIPTION

[0036] The following will be described in detail with reference to the accompanying drawings.

[0037] Heterogeneous data flow architecture: a computing architecture that combines different types of processing units (such as general-purpose processors and special-purpose processors), which can achieve efficient parallel computing through a data flow graph.

[0038] Multi-Operator Parallelism: A parallel computing method that allows multiple operations (or "operators") to be executed simultaneously to improve computational efficiency and resource utilization. In heterogeneous dataflow architectures, multi-operator parallelism can fully utilize different types of computing resources to improve performance.

[0039] Task Scheduling: In computing tasks, task scheduling refers to the process of allocating computing resources (such as processing units) to execute tasks. Effective task scheduling algorithms can optimize resource utilization, improve computational efficiency and performance.

[0040] Operator Topology Graph: A graph structure representing the data dependency relationship between multiple operators. Each node represents an operator, and the edge represents the data dependency relationship. By analyzing the operator topology graph, the execution order of the task and the parallel opportunity can be determined.

[0041] PE Dataflow Array: An array structure composed of multiple processing units (PEs) specifically designed for dataflow computing. Each PE is connected through data paths, enabling efficient execution of dataflow tasks.

[0042] Operator Dataflow Graph: In dataflow computing, a graph structure representing the execution order of operators and data transfer relationships. By constructing an operator dataflow graph, operators can be efficiently scheduled and executed on a PE dataflow array.

[0043] Data Dependency Relationship: In computing tasks, the output of a certain operator serves as the input of other operators, forming a data dependency relationship.

[0044] Scheduling Queue: In the task scheduling process, a queue of information is used to maintain the to-be-executed operators. According to different operator characteristics and data dependency relationships, the scheduling queue can be divided into multiple types, such as data-dependent queue, data-independent queue, etc.

[0045] Starting Operator List: In the task scheduling process, a set of operators to be scheduled in the initial stage. The starting operator is usually an operator with an in-degree of 0 in the topology graph, i.e., it does not depend on the input data of other operators.

[0046] SPM (Scratch-Pad Memory): A cache module used to temporarily store data used in the computing process. The size of SPM is an important factor affecting the efficiency of operator execution.

[0047] Average PE Utilization: The average usage rate of processing units (PEs), representing the workload of PEs within a certain period of time. High average PE utilization means that computing resources are fully utilized.

[0048] Dynamic data flow graph refactoring: During computation, the data flow graph is adjusted and optimized based on actual runtime conditions. Dynamic data flow graph refactoring can improve computational efficiency and resource utilization, and adapt to changing task requirements.

[0049] The meanings of the variables involved in this invention are shown in Table 1.

[0050] Table 1: Meaning of each variable

[0051] Variable Reference Meaning MEM i ]] Peak memory occupancy of operator i when processing a unit data block OCC cali ]]> Occupancy of CAL functional units of operator i at runtime UTL i ]]> PE utilization of operator i at runtime REG i ]]> Number of registers required by operator i at runtime to process a unit data block list1 Starting operator list list2 Ordered list of data sharing operators MAXP Maximum number of parallel operators Par r ]] Number of parallel operators in the current operator group memSpm Size of total memory occupancy of the operator group regNum Current number of required registers of the operator group ULT mean ]]> Average PE utilization of all operators in list2 ULT cur ]] Average PE utilization of the current operator group Resoucei Operator i Number of allocated PE computing resources OCC caltot ]] Total CAL functional unit occupancy of the operator group OCC i ]] Operator i CAL function component occupancy

[0052] Example 1

[0053] In existing technologies, current PE dataflow arrays can only support dataflow execution of a single operator. They lack corresponding scheduling strategies and runtime support for fine-grained parallelism of multiple operators. For example, in a multi-task environment, tasks can be divided into multiple different operators, but current dataflow accelerators cannot select suitable operators for parallel execution. When multiple operators are executed in parallel, to improve data reuse and reduce memory accesses, a suitable dataflow graph needs to be constructed for deployment. This raises the question of how to construct a new dataflow graph based on the dataflow graph of parallel operators in a single-operator environment. Currently, fine-grained multi-operator parallelism can be achieved by analyzing the data flow relationships between different operators and manually constructing a dataflow graph using these operators. However, in a multi-task environment, the large number of combinations of different operators makes manually configuring all fine-grained multi-operator parallel dataflow graphs almost impossible. Therefore, designing a fine-grained multi-operator parallel scheduling method and runtime system on a novel heterogeneous dataflow architecture to dynamically select suitable operators for parallel execution and dynamically generate suitable dataflow graphs is a problem that urgently needs to be solved.

[0054] The present invention discloses a fine-grained multi-operator parallel scheduling system based on a heterogeneous data flow architecture, comprising a processor for executing a fine-grained multi-operator parallel scheduling method based on a heterogeneous data flow architecture.

[0055] Preferably, such as Figure 2 As shown, the processor includes a Host with a RISC-CV core, DMA, DRAM, a Memory Controller, an SPM, and a PE data stream array.

[0056] The Host with RISCV core is connected with DMA through data transfer port, realizing data transfer between Host and memory. DMA is connected with storage controller through memory data port, managing data transfer between memory and other components. Storage controller is connected with SPM through cache port, controlling data read and write operation to ensure storage efficiency. SPM is connected with PE data stream array through data port, providing data storage support for calculation. Storage controller is also connected with DMA to optimize data transfer management.

[0057] Host is the master part in processor, responsible for operating system and application execution. RISCV core is an open-source instruction set architecture (ISA) that instructs how processor executes programs.

[0058] Specifically, Host is equipped with RISCV core, including data transfer port. DMA includes memory data port and manages memory data transfer through this port. Storage controller has cache port for connecting with SPM to optimize data read and write speed. SPM (data cache unit) includes data port, which is used to connect with PE data stream array to provide data storage support for calculation.

[0059] According to the connection between transfer port and DMA memory data port, data transfer between Host and DMA is realized through PCIe or AXI high-speed data bus. DMA is connected with storage controller through memory data port, using AXI bus or similar high-speed interface for data transfer. Storage controller is connected with SPM through cache port, using DDR interface or LPDDR interface to ensure efficient data read and write operation. At the same time, storage controller is also connected with DMA through AXI bus to optimize data transfer management. SPM's data port is connected with PE data stream array through dedicated high-speed data channel (such as AXI or custom parallel bus), ensuring high-throughput data transfer and supporting complex calculation task execution.

[0060] DMA (Direct Memory Access) allows different modules to directly read and write memory without going through CPU. This improves data transfer efficiency and reduces CPU burden.

[0061] DRAM (Dynamic Random-Access Memory) is used to store data that needs to be accessed quickly. DRAM needs to be refreshed regularly to maintain data.

[0062] Storage controller manages data read and write operations between processor and memory. It is responsible for arranging data access sequence and optimizing memory performance.

[0063] SPM (Scratchpad Memory) is a small-capacity high-speed memory used for temporary storage of data, usually used to accelerate specific computing tasks and reduce access to main memory. In this invention, SPM refers to the data cache shared by all PEs of the PE dataflow array.

[0064] PE (Processing Element) dataflow array: responsible for performing computationally intensive tasks. Each processing unit (PE) inside can independently perform calculations, making data processing more efficient.

[0065] CAL functional components refer to functional components private to each PE, used to perform calculation operations.

[0066] In the PE dataflow array, the time required for data flow between PEs is much less than the access time of PEs to SPM, and data flow between PEs and functional components within PEs can be performed through the constructed dataflow graph, thereby improving data reuse and reducing access to SPM cache.

[0067] As shown in Figure 1 The fine-grained multi-operator parallel scheduling method based on the heterogeneous dataflow architecture of the present invention includes the following steps, which are executed by the Host.

[0068] S1: According to the operators related to the task scheduling process obtained by decomposition, an operator topology graph is established, and an operator dataflow graph is constructed.

[0069] S2: According to the data dependency relationship between operators in the task, the operators are grouped based on the data dependency relationship and the runtime characteristics of the operators, and a corresponding scheduling queue is maintained for each operator group.

[0070] S3: Schedule the parallel operator groups according to the scheduling queue, and allocate computing resources to the parallel operators to be parallelized within the parallel group based on the runtime characteristics.

[0071] S4: Dynamically reconstruct the original operator dataflow graph of the parallel operator group according to the operator dataflow graph.

[0072] S5: When the PE dataflow array parallel execution of the operator ends, check whether each scheduling queue is empty, if yes, end the operator scheduling, otherwise, continue to schedule the parallel operator groups according to the scheduling queue.

[0073] The present invention combines the to-be-scheduled operators into three different types of operator groups by analyzing the data dependency between operators, dynamically reconstructs the subtask dataflow graph of multi-operator parallel according to the category of parallel scheduling operators, reduces the access frequency of operators to DRAM, reduces the access to SPM through data flow between PEs, and realizes efficient parallel scheduling of multi-operators.

[0074] The application dynamically allocates the PE data stream array in the multi-operator group according to the structure of the PE data stream array on the DFU accelerator, balances the load of the PE computing resources, improves the utilization of the PE, ensures efficient parallelism of the multi-operator, and reduces the idle time of the computing resources.

[0075] The in-degree indeg and the out-degree outdeg of the operators in the operator topology graph must be maintained when a series of operations in step S1 are performed, so as to classify the subsequent operator groups.

[0076] The steps of S1 are as follows.

[0077] S11: The operator topology graph is analyzed in advance by breadth-first search (BFS) on the RISCV core of the Host, and is stored on the DRAM, and is not transmitted to the SPM of the PE data stream array.

[0078] S12: The method for constructing the operator data flow graph on the PE data stream array includes: code segment division of the decomposed operators; generation of subtasks corresponding to each code segment; and sequential execution of the subtasks by the PE data stream array.

[0079] The data flow graph refers to the operations performed by each PE in the execution process of the operator on the PE data stream array, and the data flow relationship between the PEs.

[0080] The operator on the PE data stream array is executed by deploying the data flow graph on the PE data stream array. Due to hardware limitations, the operator execution process needs to be divided into multiple data flow graphs, and each data flow graph is called a subtask. In order to deploy the operator on the PE data stream array, the operator needs to be code segment divided, different subtasks are generated for different code segments, and the subtasks are deployed on the PE data stream array and sequentially executed, as shown in Figure 3 .

[0081] As shown in Figure 3 , the total load of the operator includes the sequentially executed operator Subkernel0 and the operator Subkernel1. The operator Subkernel0 is divided into Task0, Task1, Task2 and Task3. These tasks Task0, Task1, Task2 and Task3 are scheduled in parallel to form an operator group. The operator group includes Task0, Task1 and Task2. The operator group is constructed into a data flow graph and deployed on the PE data stream array.

[0082] The operator Subkernel1 is divided into Task0, Task1, Task2 and Task3. Task0, Task1, Task2 and Task3 are scheduled in parallel to form two independent parallel operator groups. Each operator group includes Task0, Task1 and Task2. In each operator group, Task0, Task1 and Task2 are executed in sequence. The parallel operator groups Task0, Task1 and Task2 are deployed on the PE data flow array to ensure efficient execution of tasks and optimal utilization of resources.

[0083] This can automatically and granularly decompose and map operators to the PE data flow array for execution, simplifying the complexity of manually constructing a data flow graph, enabling the system to flexibly cope with a large number of operator combinations in a multi-task environment, improving execution efficiency and system scalability.

[0084] The steps of S2 are described as follows.

[0085] S21: Divide the operators into operator groups with data dependencies, operator groups processing the same data, and operator groups processing different data.

[0086] The Host in the data flow accelerator traverses all generated operator topologies and extracts operators with an in-degree of 0 in the operator topology. If the out-degree of the operator is 0, it is added to the data-independent queue PQ3, otherwise it is added to the start operator list list1.

[0087] The Host in the data flow accelerator analyzes the runtime characteristics of different operators. The peak memory occupancy (MEM i ) of the operator i when processing a unit of data block, the occupancy rate (OCC cali ) of the CAL functional component of the operator i at runtime, the PE utilization rate (UTL i ) of the operator i at runtime, and the number of registers (REG i ) required by each operator i to process a unit of data block at runtime.

[0088] Various runtime characteristics of the operator can be obtained by feeding back data through the PE data flow array in the data flow accelerator. The total memory occupancy of the parallel scheduled operator group and the number of registers currently required by the operator group are:

[0089] memSpm=MEM1+MEM2+…+MEM Parr .

[0090] regNum=REG1+REG2+…+REG Parr .

[0091] wherein, Parr represents the number of parallel operators in the current operator group, memSpm represents the size of the total memory occupancy of the operator group, regNum represents the number of current required registers of the operator group, MEM i represents the peak memory occupancy when the operator i processes a unit data block, REG i represents the number of registers required when the operator i processes a unit data block.

[0092] The total memory amount MEM tot required by the parallel scheduled operator group should not exceed the size of SPM, and the total number of registers REG tot required by the parallel scheduled operator group should not exceed the maximum number of registers of the PE data flow array.

[0093] According to the operator topology, the data dependency relationship between operators in the task is analyzed. The RISCV core in the Host divides the operators into three types of operator groups according to the operator runtime characteristics: operator groups with data dependency, operator groups processing the same data, and operator groups processing different data, and maintains a corresponding scheduling queue for each type of operator group: a data dependency queue PQ1, a data sharing queue PQ2, and a data independent queue PQ3, which are used to store the operator groups with data dependency, the operator groups processing the same data, and the operator groups processing different data obtained when the task is decomposed. The three data structures of the data dependency queue PQ1, the data sharing queue PQ2, and the data independent queue PQ3 are stored on the DRAM.

[0094] Due to the data flow characteristics of the PE data flow array, the operator groups with data dependency and the operator groups with data sharing have a data flow relationship, which can effectively reduce the cache access times of the SPM shared by the PEs. The operator groups without data dependency do not have data flow, and cannot take advantage of the data flow between the PEs. At the same time, the operator groups with data sharing need to store the intermediate results of multiple operators, increasing the demand for memory capacity, while the operator groups with data dependency do not, so the priority of the three queues is: data dependency queue > data sharing queue > data independent queue. In the scheduling process, the operators are scheduled in parallel from the queues according to the priority.

[0095] S22: In the operator topology, the dependency relationship between each operator and its successor is identified.

[0096] The maximum number of parallel operators in the data dependent operator group is set to MAXP. The operators in the list1 of starting operators are taken out one by one. The number of parallel operators in the current operator group Parr=0, the number of current required registers regNum=0, and the size of the current required SPM memSpm=0 are set.

[0097] For example, Figure 3As shown, if the operator has only one successor operator in the operator topological graph, the operator and the successor operator are combined into a data dependent operator group. If the capacity of the SPM and the number of registers are sufficient, the operator and the successor operator are merged. In this way, the dependency relationship between operators can be effectively organized and managed, the data transmission overhead is reduced, the parallel degree of operator execution and resource utilization are improved, and thus the overall computing performance is improved.

[0098] Because only the final result is stored when the data dependent operator group is executed, other successor operators cannot obtain the result data generated by the current operator execution, causing repeated execution of the current operator. For the case of multiple successor operators, the result data generated by the current operator is first stored once, and all successor operators are combined into a data sharing operator group, to avoid repeated execution of the current operator and repeated loading of the result data generated by the current operator. As shown. Figure 2

[0099] The specific operation is as follows.

[0100] If there is only one successor operator, the current required number of registers of the operator group is calculated as follows:

[0101] regNum = regNum + REG i (i = 0, 1, …, i).

[0102] In the above formula, REG i represents the i-th operator that joins the operator group. The principle of the formula is to constantly find nodes with an in-degree of 0 and an out-degree of 1, and add them to regNum, thereby obtaining the total number of registers required by the operator group.

[0103] If the average number of registers required by the operator (regNum + (Parr-1) / (Parr-1)) is greater than the number of registers of the PE data flow array, the operator is added to the start operator list list1, otherwise, the operator is added to the operator group, and the successors of the operator are continuously checked until the maximum number of registers of the PE data flow array is exceeded or the number of parallel operators Parr in the current operator group is equal to the maximum number of parallel operators MAXP or the operator has multiple successors. If the checked operator has multiple successors, the operator group is added to the data dependent queue PQ1, and step S23 is implemented for the current operator, otherwise, step S24 is implemented.

[0104] ​S23: If the operator has multiple successor operators in the operator topology graph, at least one storage operation is performed on the result data generated by the current operator, and all successor operators are combined into a data sharing operator group to avoid repeated execution of the current operator and repeated loading of the result data generated by the current operator. In this way, repeated calculation and transmission of data are reduced, the use efficiency of system resources is optimized, the waste of memory and computing resources is reduced, and the overall execution speed and efficiency of the system are improved.

[0105] Specifically, if the current operator has multiple successor operators, all the successor operators are added to the data sharing operator ordered list list2, and the head pointer ptr1 and the tail pointer ptr2 and the average PE utilization rate ULT of all operators in the data sharing operator ordered list list2 are maintained. mean and the average PE utilization rate ULT of the current operator group cur The head pointer ptr1 points to the head of the data sharing operator ordered list list2, and the tail pointer ptr2 points to the tail of the data sharing operator ordered list list2.

[0106] If ULT cur > ULT mean , the operator pointed to by the head pointer ptr1 is added to the operator group, and ULT cur , regNum, memSpm and Parr are updated. The head pointer ptr1 points to the next operator.

[0107] If ULT cur ≤ ULT mean , the operator pointed to by the tail pointer ptr2 is added to the operator group, and the average PE utilization rate ULT cur of the current operator group, the number of registers currently required by the operator group regNum, the total memory occupied by the operator group memSpm and the number of parallel operators in the current operator group Parr are updated; the tail pointer ptr2 points to the previous operator until the number of registers currently required by the operator group regNum exceeds the maximum number of registers of the PE data volume array, or the number of parallel operators in the current operator group Parr is equal to the maximum number of parallel operators MAXP, or the total memory occupied by the operator group memSpm is greater than the SPM capacity, or the position pointed to by the head pointer ptr1 is behind the tail pointer ptr2. The data sharing operator ordered list list2 is emptied, and step S24 is implemented.

[0108] During runtime, the PE data flow array maintains the data sharing operator ordered list list2 and the head pointer ptr1 and the tail pointer ptr2 in the list. The data sharing operator ordered list list2 is arranged in descending order according to the PE utilization rate (UTL i ) of the operator i during runtime.

[0109] When selecting operators in the data sharing operator group, the first pointer ptr1 and the tail pointer ptr2 of the data sharing operator ordered list list2 are dynamically updated based on the greedy method, so as to keep the average PE utilization rate of the operator group close to the average PE utilization rate of all operators in the data sharing operator ordered list list2 (ULT mean ), so as to avoid the PE utilization rate of the parallel operator group being too low. In this way, the PE usage efficiency is optimized, and it is ensured that the system can still operate efficiently under high load.

[0110] S24: Extract the operators in the starting operator list list1, and perform step S22 or S23 according to the number of successor operators, until the starting operator list list1 is empty.

[0111] S3: Schedule the parallel operator group according to the scheduling queue, and allocate the calculation resources for the parallel operators to be parallelized in the parallel group based on the runtime characteristics. This step is as follows.

[0112] S31: Calculate the calculation resources of each operator according to the proportion of the CAL functional component occupancy rate of each operator in the parallel scheduling.

[0113] S32: Allocate the corresponding number of PEs to the operators in the operator group according to the calculated calculation resources. In this way, the calculation resources can be reasonably allocated, the number of processing elements (PEs) can be dynamically adjusted according to the actual demand of the operators, resource waste can be avoided, and the overall calculation efficiency and resource utilization of the system can be improved.

[0114] Specifically, the parallel operator group is scheduled according to the three obtained scheduling queues. For the PE data flow array, there are 16 PE calculation resources thereon.

[0115] Because the number of operations required to be performed by different operators is different, the time required by the operators to process data is different, so that the operators with a small number of operations to be performed are executed in advance and end, and the operators with a large number of operations to be performed are waiting for execution. The PE calculation resources are idle. At the same time, because the PE utilization rate is related to the occupancy rate of the CAL functional component when the operator is executed, in order to reduce the waste of calculation resources and ensure the high efficiency of multi-operator parallelization, the PE calculation resources will be allocated according to the proportion of the CAL functional component occupancy rate of each operator in the parallel scheduling.

[0116] The calculation resource allocation formula for parallel scheduling of each operator is as follows:

[0117] OCC caltot =OCC1+OCC2+…+OCC Parr .

[0118] Resouce i =max(OCC i / OCCcaltot *16,1)。

[0119] Wherein, P arr represents the number of parallel operators in the operator group, OCC caltot represents the total CAL functional component occupancy of the operator group, OCC i represents the CAL functional component occupancy of the operator i, Resouce i represents the number of PE computing resources allocated to the operator i, and 16 is the maximum number of PEs in the PE data stream array.

[0120] According to the calculated computing resources of each operator, a corresponding number of PEs is allocated to the operators in the operator group in sequence. Since the data flow distance between PEs has the optimal substructure characteristics when the data flow graph is constructed, in order to make the data flow distance between PEs as close as possible, a greedy strategy is adopted to allocate PE computing resources to operators during allocation. In order to facilitate programming and subsequent dynamic reconstruction of the data flow graph, the method will allocate computing resources to each parallel operator in sequence according to the PE number.

[0121] The application calculates the number of PE computing resources Resouce i allocated to each operator i by calculating the proportion of the CAL functional component occupancy OCC caltot of each parallel scheduled operator to the total CAL functional component occupancy OCC i of the operator group. In order to ensure that each operator can be allocated PE computing resources, the minimum value of the number of PE computing resources Resouce i allocated to the operator i is set to 1, that is, at least one PE is used to execute the operator. 16 is the total number of PE computing resources of the PE data stream array.

[0122] S4: The data flow graph dynamic reconstruction step of the original operator data flow graph of the parallel operator group according to the operator data flow graph is as follows.

[0123] Since multiple operators are parallel, the corresponding subtasks of each operator need to be spliced, which will involve changes in the data flow between PEs. Therefore, for PE computing resources executing the same operator, the data flow between PEs does not change.

[0124] According to the PE computing resources allocated to each operator in step S3, each operator original data flow graph is mapped to a new data flow graph in sequence. For PE computing resources executing different operators, dynamic reconstruction is performed according to the scheduled operator group category.

[0125] When scheduling a data-independent operator group, no data flow relationship is constructed between PE computing resources. When scheduling a data-dependent operator group or a data-dependent operator group, a data flow relationship is constructed between PE computing resources, such as Figure 4As shown. At the same time, since the computational resources occupied by each operator in the entire PE data flow array before the data flow graph is stitched together will be scaled down after the data flow graph is stitched together, it is necessary to dynamically adjust the number of instructions executed by the operator on each PE computational resource according to the number of PE computational resources in order to produce the correct results.

[0126] like Figure 4 As shown, the topology diagrams of operators 1 to 5 are illustrated.

[0127] In the first operator topology graph, a data dependency group is constructed: the result data generated by operator 1 is directly passed to the subsequent operator 2 without being stored.

[0128] In the second operator topology, in order to execute operator 3, operator 1 needs to be re-executed, and operator 3 is executed repeatedly.

[0129] In the third operator topology graph, the result data of operator 1 is extracted first. A data sharing group is established: operator 1 does not need to be executed repeatedly. The number of accesses to the three operators changes from 3 to 1.

[0130] S5: After the parallel execution of the operator by the PE data stream array is completed, check whether all three queues are empty. If so, end the operator scheduling; otherwise, return to step S3.

[0131] Specifically, if the current scheduling is a data dependency operator group, and the total data generated by the operator group is less than the SPM capacity, the subsequent data dependency operator group is obtained through step S3, inserted at the head of the data dependency operator group queue, and the process returns to step S3. Otherwise, the process directly returns to step S3 until all queues are empty.

[0132] The data generated by the operator group can be stored directly in the SPM without being stored in DRAM. If there are subsequent operators, the operators can be directly scheduled to perform calculations using the data in the SPM without having to load data from DRAM again.

[0133] If all three operator group scheduling queues dynamically maintained by the runtime system are empty, it means that all nodes in the operator topology graph have been executed, the task execution is complete, and the operation of the PE data stream array ends.

[0134] like Figure 5 As shown, when the data stream accelerator performs this step, in the case of sequential execution of individual operators, operator subtask00 is divided into subtask01 and subtask02. In this figure, operator subtask01 is divided into 4 subtasks: subtask10, subtask11, subtask12, and subtask13.

[0135] In the case of multi-operator parallel execution, the operator subtask00 is divided into subtask01 and subtask02; the subtask10 operator 2 is divided into 3 subtasks, and in the original scheme, the data flow accelerator will schedule the operators one by one and execute the subtasks divided by them in turn. The instruction templates of the subtask00 and subtask10 operators are combined, the subtask11 is combined, and different operator groups are deployed on the data flow accelerator. Among them, the instruction template of the newsubtask0 is deployed on the PE data flow array in the data flow accelerator.

[0136] In the proposed multi-operator parallel execution method, the application provides a subtask dynamic splicing method, which splices the same subtask of different operators into a new subtask, thereby forming a new data flow graph and deploying it on a data flow accelerator, so that multi-operator parallelism is realized at the subtask granularity.

[0137] Embodiment 2

[0138] This embodiment provides a specific application case to realize the soft and hard combined multi-operator fine-grained parallel scheduling. In market application, the system can be applied to efficiently schedule tasks on the DFU accelerator, schedule the operators into three types of operator groups, fully utilize the characteristics of the data flow of the DFU accelerator, reduce the access overhead of the SPM on the DFU, and improve the utilization rate of the PE computing resources.

[0139] The implementation steps of the application are described taking the Unet network as an example.

[0140] First, the Unet network is analyzed to obtain the operator topology graph between the multi-operators in the task execution process.

[0141] Figure 6 The operator topology graph in the deep learning model is shown; in particular, the whole process of data flow and processing in a deep learning model is shown. A part of the Unet network starts with the activation function ReLU, which introduces nonlinearity into the network by setting all negative values to zero, thereby allowing the Unet network to capture more complex features and patterns. The reason why only a part of the Unet network is scheduled in this place is that: (1) the overall computation graph of the Unet network is very large, and it cannot be displayed in a limited space, or the nodes in the computation graph will be very small after being displayed, which is difficult to see; (2) the purpose of this place is to schedule the multi-operators of a part of the Unet network to ensure that the scheduling method is feasible and correct.

[0142] Next, the data flows through four MaxPool layers, which successively reduce the spatial dimensions of the feature maps while preserving key information, helping to improve the computational efficiency of the model and reduce the risk of overfitting.

[0143] Subsequently, the data stream enters four successive Conv layers, which extract features from the input data using convolutional kernels and map these features to a new feature space. Following the Conv layers, the Unet network includes two Shape operations, which can be used to adjust the dimensions of the feature maps to ensure they match the input requirements of subsequent layers. This is followed by a Slice operation, which can segment the feature maps into multiple parts for independent processing or to extract specific features.

[0144] These segmented features are then merged through a Concat operation, which helps integrate features from different layers or different parts, enhancing the feature fusion capabilities of the Unet network. After that, the model includes three successive Resize operations, which can be used to adjust the size of the feature maps to a specific output size or to adapt to the input requirements of subsequent layers. Finally, all the adjusted feature maps are merged through a final Concat operation, providing a comprehensive feature representation for the final decision or classification of the Unet network.

[0145] Overall, this operator topology diagram illustrates the series of processing steps that data goes through in a deep learning model from input to output, covering multiple key operations such as activation, pooling, convolution, shape adjustment, slicing, concatenation, and resizing. These operations work together to extract and fuse features, providing a foundation for completing complex classification and recognition tasks.

[0146] Figure 7 The data flow diagram of the activation function Relu and the MaxPool layer in the Unet network is shown. In the activation function layer Relu, the first row of PEs is assigned to perform the Relu activation function calculation, and the remaining PEs are idle. In the MaxPool layer, the first and second rows of PEs are assigned to perform the pooling operation, and the third and fourth rows of PEs are idle.

[0147] As Figure 8 shown, the operator groups are classified according to the data flow diagram on the configured PE data flow array, such as in the Unet network, the activation function Relu and the MaxPool layer can be constructed into an operator group that shares data, and other parts of the Unet network calculation can be constructed into data-dependent operator groups and data-independent operator groups, and the corresponding priority queues are maintained.

[0148] The PE data stream array dynamically allocates computing resources for different operator groups according to the operator group priority queue. In order to fully utilize the PE computing resources, the PE computing resource allocation calculation is performed. The multi-operator parallel data stream graph is dynamically reconstructed at the subtask level, and the data stream scheduling and multi-operator parallelism are performed. This case takes the data shared operator group composed of the data output by the operator of the activation function Relu and the operators of the four max pooling layers MaxPool as an example. The operator group is taken out from the corresponding priority queue, the memory address of the data output by the activation function Relu and the data stream graph of the operators of the four max pooling layers MaxPool are obtained through the stored data structure, the data stream graph of the activation function Relu and the max pooling layers MaxPool is reconstructed, and then the scheduling of the operators of the four max pooling layers MaxPool after reconstruction is performed. As shown in Figure 9 Figure 9 In the SPM extracts the result data of the activation function Relu. These data are then fed into the operators of the four max pooling layers MaxPool formed after reconstruction. The operator of the first max pooling layer MaxPool1 formed after reconstruction is responsible for performing the first max pooling operation; the operator of the second max pooling layer MaxPool2 further reduces the spatial size of the feature map on the basis of the first pooling; the operator of the third max pooling layer MaxPool3 continues to perform the pooling operation to extract more abstract features; and the operator of the fourth max pooling layer MaxPool4 provides highly abstract feature representation for the subsequent layers of the network. By reconstructing and scheduling these operators, it can be ensured that each PE is fully allocated and utilized, thereby improving the computing efficiency of the entire system. The data stream graph reconstruction and scheduling of other operator groups are similar to this process.

[0149] It should be noted that the above specific embodiments are exemplary, and those skilled in the art can come up with various solutions under the inspiration of the disclosure of the present application, and these solutions also belong to the disclosed range of the present application and fall within the protection scope of the present application. Those skilled in the art should understand that the specification and drawings of the present application are illustrative and do not constitute a limitation on the claims. The protection scope of the present application is defined by the claims and their equivalents. The specification of the present application contains multiple inventive concepts, such as "preferably", "according to one preferred embodiment", which means that the corresponding paragraph discloses an independent concept, and the applicant reserves the right to file a divisional application according to each inventive concept.​

Claims

1. A fine-grained multi-operator parallel scheduling method based on a heterogeneous dataflow architecture, characterized in that, The method includes: Based on the operators related to the task scheduling process obtained from the decomposition, an operator topology graph is established, and an operator data flow graph is constructed. Based on the operator topology graph, the data dependencies between operators in the task are analyzed. Based on the data dependencies and the runtime characteristics of the operators, the operators are grouped and a corresponding scheduling queue is maintained for each operator group. The operators are divided into groups of operators with data dependencies, groups of operators that process the same data, and groups of operators that process different data. In the operator topology graph, the dependency relationship between each operator and its successor operator is identified. If an operator has only one successor operator in the operator topology graph, the operator and its successor operator are combined into a data dependency operator group. If the capacity and number of registers of the SPM are sufficient, the operator and its successor operator are merged. If an operator has multiple successor operators in the operator topology graph, the result data generated by the current operator is stored at least once, and all successor operators are combined into a data-sharing operator group to avoid repeated execution of the current operator and repeated loading of the result data generated by the current operator. The parallel operator group is scheduled according to the scheduling queue, and computing resources are allocated to the parallel operators to be parallelized within the parallel group based on the runtime characteristics. The original operator data flow graph of the parallel operator group is dynamically reconstructed based on the operator data flow graph. After the PE data stream array finishes executing the operators in parallel, check whether each of the scheduling queues is empty. If it is, the operator scheduling ends; otherwise, continue to schedule the parallel operator group according to the scheduling queues.

2. The fine-grained multi-operator parallel scheduling method based on heterogeneous data flow architecture according to claim 1, characterized in that, Methods for constructing operator dataflow graphs on PE dataflow arrays include: The decomposed operators are divided into code segments; Generate subtasks corresponding to each code segment; The PE data stream array executes subtasks sequentially.

3. The fine-grained multi-operator parallel scheduling method based on heterogeneous data stream architecture according to claim 2, characterized in that, The method for grouping operators based on the data dependencies and runtime characteristics of the operators also includes: When selecting operators in the data-sharing operator group, a greedy method is used to dynamically update the first and last pointers of the ordered list of data-sharing operators, keeping the average PE utilization of the operator group close to the average PE utilization of all operators in the ordered list of data-sharing operators, so as to avoid the PE utilization of the parallel operator group being too low.

4. The fine-grained multi-operator parallel scheduling method based on heterogeneous data flow architecture according to claim 3, characterized in that, The method for allocating computing resources to the parallel operator includes: The computing resources of each operator are calculated according to the proportion of the CAL functional component occupancy rate of each operator in parallel scheduling. Based on the calculated computing resources, allocate a corresponding number of processing units to the operators in the operator group.

5. A fine-grained multi-operator parallel scheduling system based on a heterogeneous dataflow architecture, comprising a processor, characterized in that, The processor is configured to: Based on the operators related to the task scheduling process obtained from the decomposition, an operator topology graph is established, and an operator data flow graph is constructed. Based on the operator topology graph, the data dependencies between operators in the task are analyzed. Based on the data dependencies and the runtime characteristics of the operators, the operators are grouped and a corresponding scheduling queue is maintained for each operator group. The operators are divided into groups of operators with data dependencies, groups of operators that process the same data, and groups of operators that process different data. In the operator topology graph, the dependency relationship between each operator and its successor operator is identified. If an operator has only one successor operator in the operator topology graph, the operator and its successor operator are combined into a data dependency operator group. If the capacity and number of registers of the SPM are sufficient, the operator and its successor operator are merged. If an operator has multiple successor operators in the operator topology graph, the result data generated by the current operator is stored at least once, and all successor operators are combined into a data-sharing operator group to avoid repeated execution of the current operator and repeated loading of the result data generated by the current operator. The parallel operator group is scheduled according to the scheduling queue, and computing resources are allocated to the parallel operators to be parallelized within the parallel group based on the runtime characteristics. The original operator data flow graph of the parallel operator group is dynamically reconstructed based on the operator data flow graph. After the parallel execution of an operator ends, check whether each of the scheduling queues is empty. If it is, the operator scheduling ends; otherwise, continue to schedule the parallel operator group according to the scheduling queues.

6. The fine-grained multi-operator parallel scheduling system based on a heterogeneous data stream architecture according to claim 5, characterized in that, The processor is also configured to: In the step of grouping operators based on the data dependencies and the runtime characteristics of the operators, when selecting operators in the data-sharing operator group, the first and last pointers of the ordered list of data-sharing operators are dynamically updated based on a greedy method to keep the average PE utilization of the operator group close to the average PE utilization of all operators in the ordered list of data-sharing operators, so as to avoid the PE utilization of the parallel operator group being too low.

Images