Intelligent multi-operator cross-scheduling method and system

Abstract

The application relates to an intelligent multi-operator cross scheduling method and system, which comprises the following steps: collecting the runtime characteristics of operators, determining the number of subkernels that should be divided for each operator, and constructing an execution flow graph in units of subkernels to clearly show the dependency relationship between the subkernels; according to the operator priority PR and the scheduling strategy, a subkernel node without a preceding dependency in the execution flow graph is selected for scheduling; corresponding operator instructions are loaded from an instruction cache unit to an instruction storage unit to ensure that all necessary instructions have been loaded; the PE array processing unit executes the subkernel and updates the execution flow graph, and the iteration is performed until the task is completed. The application performs more fine-grained subkernel scheduling through data cutting, the subkernels can be dynamically switched and cross-scheduled, and the flexibility of task execution is improved.

Description

Technical Field

[0001] This invention relates to the field of runtime technology in the computer field, and to an intelligent multi-operator cross-scheduling method and system, particularly to a multi-operator cross-scheduling method and runtime system for heterogeneous data flow architectures. Background Technology

[0002] Currently, computing technology is gradually shifting from general-purpose computing to domain-specific computing, with the convergence of big data, graph computing, and artificial intelligence becoming a significant development trend. However, this multi-domain convergence computing is characterized by data-, control-, and memory-intensive spatiotemporal interweaving, posing numerous challenges to existing computer systems, especially those with single data-flow or control-flow architectures. One core issue is the low efficiency of mixed operator execution under multi-computing models, primarily due to the rigidity of multi-operator scheduling strategies, the uniformity of operator execution order, and the insufficient exploitation of inter-operator concurrency.

[0003] Unlike the traditional von Neumann architecture, dataflow architecture constructs a data flow graph (DFG) in a data-driven manner to reduce memory access overhead and synchronization latency. This architecture performs exceptionally well in fields such as deep learning and scientific computing, effectively improving data parallelism and resource utilization. For example, Google's TPU (Tensor Processing Unit) employs a dataflow architecture, allowing for a significant reduction in data transfer time when handling a large number of parallel operations, thereby increasing the speed of model training. Furthermore, NVIDIA's CUDA platform also utilizes the dataflow concept, enabling graphics processing units (GPUs) to achieve efficient parallel computing when handling complex image rendering and machine learning tasks. The concept of dataflow originated from dataflow computers, and its development has evolved from instruction-level parallelism to thread-level parallelism. Important research works include the Codelet model and the EARTH dataflow computer, which laid the theoretical and practical foundation for the development of dataflow computing.

[0004] During the execution of a dataflow program, the computational problem is decomposed into multiple smaller tasks, which form a dataflow graph in the form of a directed acyclic graph (DAG). In this graph, nodes represent computational tasks, and edges represent dependencies between tasks. A concrete example is TensorFlow, which constructs a dataflow graph as a network containing multiple operations during neural network training. Each operation (such as matrix multiplication, activation functions, etc.) is considered a node in the graph, and data flows between these nodes. Task scheduling in dataflow execution models is usually somewhere between static and dynamic. However, due to the limitations of current dataflow architectures, existing systems often adopt a sequential execution approach when executing multiple tasks, lacking the flexibility and ability to dynamically adjust the task execution order. For example, in some deep learning frameworks, when processing complex models, there may be situations where certain computation results are pending. Such sequential execution leads to a waste of computational resources and reduces overall computational efficiency.

[0005] For example, CN114881376A discloses a genetic crossover method and an item scheduling method based on an active block crossover operator. The genetic crossover method based on the active block crossover operator includes: determining a parent selection block in the parent generation's activity list using multiple crossover points; determining the positions of the multiple crossover points according to the item scheduling problem; determining a parent selection block in the parent generation's activity list using multiple crossover points; replacing the parent selection block with blocks to generate some offspring; inserting the activities to be assigned into the activity list of some offspring to generate an offspring scheduling scheme. This invention mainly relies on the genetic crossover operator and item scheduling method, lacking optimization for parallel computing. Furthermore, this invention mainly depends on pre-determined crossover points and genetic algorithms during the scheduling process, lacking dynamic priority management; it lacks relevant mechanisms to optimize instruction loading and cache management, which may result in low resource utilization in high-concurrency environments. This invention also lacks explicit execution flow graph management, which may make the scheduling process more complex and difficult to manage.

[0006] Therefore, how to adjust the execution order of tasks during the execution of a dataflow program in order to avoid wasting computing resources and reducing overall computing efficiency is a problem that exists in the existing technology but has not yet been solved.

[0007] Furthermore, on the one hand, there are differences in understanding among those skilled in the art; on the other hand, the applicant studied a large number of documents and patents when making this invention, but due to space limitations, not all details and contents were listed in detail. However, this does not mean that the present invention does not possess the features of these prior art. On the contrary, the present invention already possesses all the features of the prior art, and the applicant reserves the right to add relevant prior art to the background art. Summary of the Invention

[0008] During the execution of a dataflow program, a computational problem is broken down into multiple smaller tasks. These smaller tasks are organized into a dataflow graph (directed acyclic graph), where nodes represent computational tasks and edges represent dependencies between tasks. In the dataflow execution model, task scheduling lies between static and dynamic approaches. However, due to the limitations of current dataflow architectures, existing systems typically employ sequential execution when executing multiple tasks, lacking the ability to dynamically adjust the task execution order.

[0009] To address the shortcomings of existing technologies, this invention provides an intelligent multi-operator cross-scheduling method from a first aspect. The method includes: collecting runtime characteristics of operators, determining the number of subkernels that each operator should be divided into, and constructing an execution flow graph based on subkernels, clarifying the dependencies between subkernels; selecting subkernel nodes without prerequisite dependencies in the execution flow graph for scheduling based on operator priority (PR) and scheduling strategy; loading the corresponding operator instructions from the instruction cache unit to the instruction storage unit, ensuring that all necessary instructions have been loaded; executing the subkernels in the PE array processing unit and updating the execution flow graph, iterating until the task ends. This invention achieves finer-grained subkernel scheduling through data segmentation, allowing for dynamic switching and cross-scheduling between subkernels, thus improving the flexibility of task execution.

[0010] This invention divides the input data of operators into blocks while maintaining the same computational instructions; that is, each operator corresponds to one kernel. Each kernel further divides its input data into multiple subkernels. Subkernels of the same operator are of the same size but have different data, sharing the same instructions. In this way, the scheduling granularity is reduced from kernel to subkernel, and when multiple operators are executed, multiple identical or different subkernels can be dynamically scheduled to participate in the computation, achieving a mixed scheduling execution of operators in both interleaved and sequential manner.

[0011] According to a preferred embodiment, the step of clarifying the dependencies between subkernels includes: constructing dependency edges based on the dependencies between subkernels when a node corresponds to a subkernel. By clarifying the dependencies between subkernels, an accurate task execution flow graph can be constructed. This enables the present invention to execute tasks in the correct order, avoid data conflicts and resource contention, and improve the accuracy and efficiency of execution.

[0012] According to a preferred embodiment, the method further includes updating the residency coefficient of operator instructions within the instruction cache unit. The residency coefficient is an indicator that measures the importance and frequency of use of instructions in the cache. By periodically updating the residency coefficient, instruction cache can be managed more intelligently, ensuring that frequently used instructions are preferentially retained, reducing cache misses, and improving system execution efficiency.

[0013] According to a preferred embodiment, the step of selecting subkernel nodes without prerequisite dependencies in the execution flow graph for scheduling based on operator priority (PR) and scheduling policy includes: dividing the subkernel load among operators; and implementing cross-switching of multiple operators using a hybrid scheduling method that combines sequential execution mode and cross-execution mode. Dividing the subkernel load and the hybrid scheduling policy allow the system to switch between different execution modes, making full use of computing resources. Cross-switching helps reduce the waiting time of a single operator execution, improves the overall system throughput and response speed, and enables parallel processing of multiple tasks.

[0014] According to a preferred embodiment, the step of selecting a Subkernel node without prerequisite dependencies in the execution flow graph includes: selecting at least one Subkernel node from the ready queue to add to the run queue, provided that the peak memory usage of the total Subkernels does not exceed the cache capacity of the data cache unit. By ensuring that the memory usage of Subkernels does not exceed the cache capacity, the system can avoid task interruptions or delays due to insufficient memory. This step helps optimize memory management, improve the utilization of the data cache unit, and ensure smooth task execution.

[0015] According to a preferred embodiment, the step of loading the corresponding operator instruction from the instruction cache unit to the instruction storage unit includes: if the first instruction of the current subkernel and the second instruction remaining in the instruction storage unit belong to the same operator, and the two subkernels have the same data size and execute independently, the two subkernels share the same set of instructions, and there is no need to load them from the instruction cache unit again. By identifying and sharing instructions of the same operator, the system can reduce unnecessary instruction loading operations, saving time and resources. This step improves the efficiency of instruction loading and reduces system overhead and latency.

[0016] According to a preferred embodiment, the step of loading the corresponding operator instruction from the instruction cache unit to the instruction storage unit includes: if the first instruction of the current subkernel and the second instruction remaining in the instruction storage unit belong to different operators, then the instruction cache unit is accessed; if a cache hit occurs, the corresponding operator instruction is loaded from the instruction cache unit into the instruction storage unit; if a cache miss occurs and the instruction cache unit is not full, the instruction storage unit retrieves the operator instruction from the main memory unit and copies the operator instruction into the instruction cache unit; if a cache miss occurs and the instruction cache unit is full, after the instruction storage unit retrieves the operator instruction from the main memory unit, it determines which instruction in the instruction cache unit should be swapped out based on the residency coefficient. This step details the instruction loading logic, including the handling methods for cache hits and misses. By optimizing instruction loading and cache management, the system can utilize the instruction cache more efficiently, reduce loading latency, and improve task execution efficiency. Determining the instruction to be swapped out based on the residency coefficient further optimizes cache utilization.

[0017] The instruction cache and its accompanying dynamic caching mechanism of this invention enable the instruction storage unit to prioritize accessing the instruction cache unit when retrieving operator instructions, which can effectively reduce the number of instruction transfers between the main memory unit and the acceleration platform. Furthermore, the dynamic caching mechanism comprehensively considers operator priority and runtime characteristics, thereby improving the instruction cache hit rate.

[0018] The present invention provides an intelligent multi-operator cross-scheduling system from a second aspect, comprising a host and a PE array processing unit.

[0019] The host is used to collect runtime characteristics of operators, determine the number of subkernels that each operator should be divided into, and construct an execution flow graph based on subkernels, clarifying the dependencies between subkernels; select subkernel nodes without prerequisite dependencies in the execution flow graph for scheduling according to operator priority (PR) and scheduling policy; load the corresponding operator instructions from the instruction cache unit to the instruction storage unit to ensure that all necessary instructions have been loaded; the PE array processing unit executes the subkernels and updates the execution flow graph, iterating until the task ends.

[0020] Instructions for different operators are independent of each other. Therefore, when scheduling subkernels of different operators to the acceleration array for processing, the corresponding instructions must be loaded from the main memory unit on the dataflow architecture into the instruction memory unit (CBUF). This process inevitably incurs transmission overhead, and especially as the number of operators increases, frequent subkernel switching will significantly impact the overall system performance. Therefore, optimizing the loading and updating of operator instructions becomes one of the keys to improving runtime system speed and performance. Through these optimizations, this invention will be able to manage computing resources more efficiently to meet the ever-increasing computing demands. This not only improves computing efficiency but also provides greater flexibility and scalability for future applications, further promoting the development of domain-specific computing.

[0021] According to a preferred implementation, the step of the host clarifying the dependencies between subkernels includes: constructing dependency edges based on the dependencies between subkernels when a node corresponds to a subkernel.

[0022] According to a preferred embodiment, the step of the host selecting a Subkernel node without prior dependencies in the execution flow graph for scheduling includes: dividing the Subkernel load among operators; and implementing cross-switching of multiple operators using a hybrid scheduling method of sequential and cross-execution modes. To address this issue, this paper proposes a dynamic operator switching method under a novel heterogeneous dataflow architecture. This method specifically performs block processing on the input data of operators to ensure the consistency of computational instructions. Specifically, each operator corresponds to a Kernel, and the Kernel can be further divided into multiple Subkernels. These Subkernels are consistent in size but differ in data, and they share the same instruction set. This scheme reduces the scheduling granularity from Kernel to Subkernel, enabling dynamic scheduling of multiple identical or different Subkernels for computation during multi-operator execution, thereby achieving a hybrid scheduling of cross-switching and sequential operations for operators. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the overall framework of a heterogeneous data flow architecture accelerator according to a preferred embodiment of the present invention;

[0024] Figure 2 This is a schematic diagram of the hierarchical division of operator operation provided by the present invention;

[0025] Figure 3 This is a flowchart of the multi-operator cross-scheduling process for heterogeneous data flow architecture provided by the present invention;

[0026] Figure 4This is a schematic diagram of the operator Subkernel partitioning provided by the present invention;

[0027] Figure 5 This is a schematic diagram of the execution of the Subkernel computation flow graph provided by the present invention;

[0028] Figure 6 This is a schematic diagram of task execution and instruction acquisition provided by the present invention. Detailed Implementation

[0029] The following is a detailed explanation with reference to the accompanying drawings.

[0030] This invention provides explanations and interpretations for some terms and nouns.

[0031] A subkernel is a part of an operator. An operator is based on input data, divided into multiple independent loads, each of which is a subkernel. Although each subkernel processes different data, they perform the same function. This partitioning improves computational efficiency and parallel processing capabilities.

[0032] Instruction Buffer: This is the area that stores instructions, located at one end of the acceleration array, between the main memory and the instruction buffer (CBUF). It can store instructions for multiple operators so that they can be quickly provided to the instruction buffer (CBUF) and further computation units when needed.

[0033] CBUF: Instruction Memory Unit, is a region used to store the instructions for a single operator. The PE array (PE array processing unit) directly fetches instructions from the instruction memory unit CBUF to execute the specified computational task. The role of the instruction memory unit CBUF is to ensure that the PE array can continuously and efficiently fetch and execute the required instructions.

[0034] A Processing Element (PE) array is an array composed of multiple processing units, each containing functional components that perform computational tasks. The PE array executes instructions fetched from the instruction memory (CBUF) to perform the actual computation and processing. Through parallel processing, the PE array can significantly improve computational efficiency and performance.

[0035] In computer science, a runtime system refers to the underlying software and hardware environment that provides support during program execution. It helps manage and control the program's execution process, providing functions such as memory management, thread and parallelism management, exception handling, input / output (I / O) management, and dynamic loading. It provides the execution environment on which programs run, including hardware abstractions and operating system functions, allowing programs to avoid directly handling low-level details. Different programming languages ​​have their own runtime systems, such as Java's JVM (Java Virtual Machine) and Python's interpreter. These systems improve program execution efficiency through just-in-time (JIT) compilation, memory pool management, and other optimizations, while providing security and isolation mechanisms. For example, Java's JVM provides a cross-platform execution environment, memory management, garbage collection, thread management, and exception handling, while Python's interpreter provides dynamic type checking, memory management, dynamic module loading, and exception handling. The operating system itself can also be considered a runtime system, providing basic services such as process management, memory management, and I / O operations.

[0036] Table 1 is a list of the referential meanings of the variables in this invention.

[0037] Table 1: List of the referential meanings of variables.

[0038]

[0039] During the execution of a dataflow program, a computational problem is broken down into multiple smaller tasks. These smaller tasks are organized into a dataflow graph (directed acyclic graph), where nodes represent computational tasks and edges represent dependencies between tasks. In the dataflow execution model, task scheduling lies between static and dynamic approaches. However, due to the limitations of current dataflow architectures, existing systems typically employ sequential execution when executing multiple tasks, lacking the ability to dynamically adjust the task execution order.

[0040] To address the aforementioned issues, this invention provides a heterogeneous data stream architecture accelerator, as well as an intelligent multi-operator cross-scheduling method and system.

[0041] Example 1

[0042] This embodiment provides a heterogeneous data stream architecture accelerator, the framework of which is as follows: Figure 1 As shown.

[0043] The heterogeneous dataflow architecture accelerator, also known as the acceleration platform, includes a main memory unit, an instruction storage unit (CBUF), an instruction cache unit, a data cache unit (SPM), and a PE array processing unit. Figure 1 The example uses a 4x4 PE array processing unit. That is to say, Figure 1Within the PE array processing unit, the processing units are arranged in a 4x4 array.

[0044] Preferably, the main memory unit, instruction storage unit (CBUF), instruction cache unit, data cache unit (SPM), and PE array processing unit can be integrated into a dedicated processor, dedicated integrated chip, or server for this invention.

[0045] like Figure 1 As shown, the host (HOST) is connected to the instruction cache unit and the data cache unit (SPM), respectively. The instruction cache unit is connected to the instruction storage unit (CBUF). The instruction storage unit (CBUF) is connected to the PE array processing unit. The data cache unit (SPM) is connected to the PE array processing unit. In other words, the host communicates with the acceleration platform via wired or wireless means.

[0046] Specifically, the host includes two main ports: a host input port and a host output port. The host input port is used to transmit instructions to the instruction buffer unit. The host output port is used to transmit data to the data buffer unit.

[0047] The instruction cache unit comprises three main ports: a first port (input port), a second port (output port), and a third port (bidirectional port, if applicable). The first port (input port) receives instructions from the host. The second port (output port) transmits instructions to the instruction memory unit (CBUF), with the connection achieved via low-latency built-in wires or a dedicated high-speed interface. The third port (bidirectional port, if applicable) performs bidirectional data transfer with the main memory unit, with the connection achieved via a high-bandwidth DDR interface.

[0048] The data buffer unit (SPM) includes three ports: a fourth port (input port), a fifth port (output port), and a sixth port (bidirectional port). The fourth port (input port) is used to receive data from the host. The fifth port (output port) is used to transfer data to the PE array processing unit. The sixth port (bidirectional port) is used for reading and writing data with the main memory unit.

[0049] The instruction cache unit (CBUF) then includes two ports: port seven (input port) and port eight (output port). Port seven (input port) is used to receive instructions from the instruction cache unit. Port eight (output port) is used to transfer instructions to the PE array processing unit.

[0050] The PE array processing unit includes three ports: port 9 (input port), port 10 (input port), and port 11 (output port). Port 9 (input port) is used to receive instructions from the instruction storage unit (CBUF). Port 10 (input port) is used to receive data from the data cache unit (SPM). Port 11 (output port) is used to return the calculation results to the data cache unit (SPM) or other storage units; the connection is achieved through a high-bandwidth data transmission line.

[0051] The main memory unit includes a twelfth port (bidirectional port).

[0052] The connection relationship between the host and each module is as follows: The host input port is connected to the first port (input port) of the instruction cache unit via a high-speed transmission line (such as PCIe) for transmitting instructions. Similarly, the host output port is connected to the fourth port (input port) of the data cache unit via a high-speed transmission line for transmitting data.

[0053] The second port (output port) of the instruction cache unit is connected to the seventh port (input port) of the instruction memory unit (CBUF) via a low-latency built-in wire or a dedicated high-speed interface. The third port (bidirectional port, if applicable) of the instruction cache unit is connected to the twelfth port (bidirectional port) of the main memory unit via a high-bandwidth DDR interface to enable bidirectional data transfer.

[0054] The fifth port (output port) of the data cache unit SPM is connected to the tenth port (input port) of the PE array processing unit via a high-bandwidth data transmission line. The sixth port (bidirectional port) of the data cache unit SPM is connected to the twelfth port (bidirectional port) of the main memory unit via a high-bandwidth DDR interface, and can also interact with the eleventh port (output port) of the PE array processing unit to realize data reading, writing and transmission.

[0055] The eighth port (output port) of the instruction storage unit (CBUF) is connected to the ninth port (input port) of the PE array processing unit via a high-bandwidth data transmission line.

[0056] The eleventh port (output port) of the PE array processing unit is connected to the sixth port (bidirectional port) of the data cache unit or other storage units via a high-bandwidth data transmission line.

[0057] The sixth port of the data cache unit (SPM) can be connected to the twelfth port of the main memory unit, or it can interact with the eleventh port of the PE array processing unit through its bidirectional capability. If the sixth port is connected to the twelfth port of the main memory unit, it is mainly used for data exchange with the main memory. If the eleventh port is connected to the sixth port, it means that the data output of the PE array processing unit can be directly written to the main memory (through the data cache unit as an intermediary).

[0058] Preferably, in the parallel connection scenario, port 6 is simultaneously connected to ports 12 and 11 to achieve bidirectional data transmission. In this mode, port 6 is configured to manage multiple data channels, thereby handling data streams from both directions. In the serial connection scenario, data from port 11 first passes through a data buffer unit and then enters port 12 through port 6. In this mode, the data buffer unit acts as an intermediary, managing the data transmission order and bandwidth control.

[0059] When the main memory unit, instruction memory unit (CBUF), instruction cache unit, data cache unit (SPM), and PE array processing unit are integrated into a single dedicated processor or dedicated circuit chip, the dedicated processor or dedicated circuit chip includes three ports: an instruction input port, a data input port, and a data output port. The host input port is connected to the instruction input port, sending instructions to its internal instruction cache unit. The host output port is connected to the data input port, sending data to its internal data cache unit. The data output port sends the processed data from the processor or dedicated circuit chip back to the host (HOST) or other external systems. Internally, the processor or dedicated circuit chip uses internal connection ports to exchange data and instructions between its various modules (such as the PE array processing unit, instruction memory unit (CBUF), and SPM).

[0060] When the host (HOST) is wirelessly connected to the processor or dedicated integrated chip (DICJ), the host, processor, and DICJ each integrate a wireless communication module. This wireless communication module may be, for example, Wi-Fi or Bluetooth. The host and processor or DICJ pair and connect via the wireless communication module and wireless protocol, establishing a wireless communication link. The host wirelessly transmits commands to the processor or DICJ's second wireless communication module via the first wireless communication module. The second wireless communication module transmits the commands to the command input port via a data transmission interface, and these commands are stored in the command cache unit within the processor or DICJ. The host also transmits data to the processor or DICJ's second wireless communication module via the first wireless communication module. The second wireless communication module transmits the data to the data input port via a data transmission interface, and this data is stored in the data cache unit within the processor or DICJ. The processor or DICJ sends the processed data to the second wireless communication module via a data output port. The second wireless communication module then transmits the processed data to the host's first communication module via a wireless signal. The first communication module then sends the data back to the host or other external systems.

[0061] The host is the system's control center, responsible for executing the main control logic of the program. It includes the Central Processing Unit (CPU) and main memory (RAM).

[0062] The main memory unit stores program data and instructions, and is the system's primary storage space. It provides high-speed data access to the host and other units. The main memory unit is connected to the instruction cache unit and the data cache unit (SPM), respectively, to transfer the data for the instructions and operations to be executed. The main memory unit is a processor core based on the RISC-V instruction set architecture (ISA). It can be a standalone processor or part of a multi-core processor. The main memory unit, or RAM, fetches program instructions and performs calculations, data transfers, and other operations.

[0063] The instruction cache unit stores instructions fetched from main memory for fast access and reduced latency. It is responsible for loading instructions from main memory and passing them to the instruction store unit (CBUF). The CBUF temporarily stores instructions and is responsible for distributing instructions from the instruction cache unit to the PE array processing units. The data cache unit (SPM) stores data fetched from main memory for fast access and reduced latency. It is responsible for loading data from main memory and passing it to the processing units. The PE array processing unit typically contains multiple parallel processing units for efficient execution of computational tasks.

[0064] Instruction cache and data cache units are typically integrated within a heterogeneous dataflow architecture accelerator and directly transmit instructions and data to the PE array. The PE array processing unit can be a CPU used for accelerated computation.

[0065] The operating principle of the heterogeneous data flow architecture accelerator is as follows:

[0066] First, the host loads programs and data from external storage into main memory. Next, the host loads instructions and data into the instruction cache unit and data cache unit (SPM) respectively to improve access speed. The instruction cache unit retrieves instructions from main memory and passes them to the instruction storage unit (CBUF), while the data cache unit (SPM) retrieves data from main memory and stores it, ready to provide it to the PE array processing unit. Subsequently, the instruction storage unit (CBUF) sends the cached instructions to the PE array processing unit. The PE array processing unit, based on the instructions received from the instruction storage unit (CBUF), retrieves the corresponding data from the data cache unit (SPM) for processing.

[0067] During the parallel processing phase, the PE array processing units execute computational tasks in parallel, thereby improving computational efficiency. Each PE array processing unit can process multiple instructions and data simultaneously, enhancing the overall system performance. After processing, the PE array processing unit returns the result data to main memory or feeds it back to the host via the data cache unit (SPM). The host then performs subsequent operations based on the processing results, such as further computation or outputting the final result. This architecture, which separates instruction and data caching, achieves higher parallel processing capabilities and efficiency, ensuring the system can quickly respond to and process complex computational tasks.

[0068] like Figure 1 As shown, the preset instruction cache unit is located between the main memory unit and the instruction storage unit (CBUF) to cache instructions for multiple operators. Before starting the PE array processing unit for accelerated processing, the operator's instructions need to be loaded from the instruction storage unit (CBUF) into the PE array processing unit's internal dedicated instruction RAM for execution. Without an instruction cache, each time a new operator is scheduled for processing, the RISC-V core controls the main memory unit and the instruction storage unit (CBUF) to transfer instructions, and the time overhead of this process affects system performance. The instruction cache design reduces this situation, concentrating instruction transfer between the instruction cache unit and the instruction storage unit (CBUF). Since both hardware components are located on the acceleration array side, the additional time generated by this process is far less than the time caused by frequent communication between the main memory unit and the acceleration array side.

[0069] Example 2

[0070] This embodiment is a further improvement on embodiment 1, and repeated content will not be described again.

[0071] like Figure 2 As shown, the operator performs operations at three levels.

[0072] (1) Total load partitioning of operators: The operator is divided into multiple subkernels based on the data. The subkernels are executed sequentially. The cross-execution proposed in this invention is based on sequential execution and can switch between multiple operators without waiting for all subkernels of the previous operator to finish before scheduling the next operator to participate in the operation.

[0073] (2) Concurrent scheduling: Tasks are executed in parallel, which can be data concurrent or task concurrent.

[0074] (3) Task mapping: Subtasks are executed sequentially, and fine-grained instructions are mapped to different PE array processing units for execution.

[0075] like Figure 3 As shown, this invention proposes a multi-operator cross-scheduling method for heterogeneous data flow architectures, which includes the following steps:

[0076] Collect runtime characteristics of operators, determine the number of subkernels that each operator should be divided into, and construct an execution flow graph based on subkernels, clarifying the dependencies between subkernels; select subkernel nodes without prerequisite dependencies in the execution flow graph for scheduling based on operator priority (PR) and scheduling policy; load the corresponding operator instructions from the instruction cache unit to the instruction storage unit, ensuring that all necessary instructions have been loaded; execute the subkernels in the PE array processing unit and update the execution flow graph, iterating until the task ends.

[0077] By refining the above steps, we can obtain steps S1 to S7.

[0078] S1: Collect the runtime characteristics of operators and determine the number of subkernels that each operator should be divided into.

[0079] S2: Construct the execution flow graph of the task in units of Subkernel, with nodes corresponding to Subkernels, and construct dependency edges based on the dependency relationships between Subkernels.

[0080] S3: Select a Subkernel node with no prior dependencies in the execution flow graph for scheduling based on the operator priority PR and the scheduling policy.

[0081] S4: Before scheduling the Subkernel to the PE array processing unit, load the corresponding operator instructions from the instruction cache unit to the instruction storage unit.

[0082] S5: Update the residency coefficient of operator instructions in the instruction cache unit.

[0083] S6: The PE array processing unit retrieves instructions from the instruction storage unit and determines whether any operator instructions have not been loaded during this execution. If so, the operator instructions are loaded again; otherwise, the Subkernel is executed.

[0084] S7: The PE array processing unit starts and begins executing the Subkernel. After execution, it updates the execution flow graph and determines whether the termination condition is met. If so, the task execution ends; otherwise, it iterates until the task ends.

[0085] Each step is explained in detail.

[0086] Step S1: Collect runtime characteristics of operators and determine the number K of subkernels that each operator should be divided into. i .

[0087] The ideal number of Subkernels is calculated using the following formula:

[0088]

[0089] Among them, DS i SPM represents the peak memory usage from the start to the end of operator i's execution. size This represents the SPM cache capacity. The operator's input data is divided into K... i One batch, namely K i There are K subkernels. The initial number of subkernels for each operator is K. i The size should be as small as possible to avoid the overhead of frequent instruction loading. The formula for calculating the input data size of each subkernel is as follows:

[0090]

[0091] Among them, INPUT i Batch is the total input data for operator i. i This represents the input data for each subkernel of operator i. For example... Figure 4 A diagram illustrating the number of Subkernels divided by three different operators.

[0092] Operator 1 is divided into Subkernel0, Subkernel1, Subkernel2 and Subkernel3.

[0093] Operator 2 is divided into Subkernel0, Subkernel1 and Subkernel2.

[0094] Operator 3 is divided into Subkernel0 and Subkernel1.

[0095] In step S1, by collecting the runtime characteristics of the operators, it is determined how to divide the operators into smaller processing units (subkernels). This improves the flexibility of task execution, allowing tasks to be scheduled more finely, thereby optimizing resource utilization.

[0096] Step S2: Construct the task execution flow graph on a per-subkernel basis, with each node corresponding to a subkernel. If there are dependencies between subkernels, dependency edges need to be constructed.

[0097] like Figure 5 As shown, in the constructed execution flow graph, after the three operators are divided into subkernels, dependency edges are constructed between subkernels with data dependencies. That is, after the previous subkernel finishes execution, its output is used as the input of the next subkernel, thereby activating the execution of the next subkernel. This data dependency may exist between different operators or within the same operator. According to... Figure 5 The constructed execution flow graph is maintained by the RISC-V core using three important data structures: the Ready Queue, the Execution Queue, and the Pending Queue. During queue initialization, all nodes (subkernels) with an in-degree of 0 in the execution flow graph are added to the Ready Queue. The remaining nodes (subkernels) that depend on the execution results of other subkernels are inserted into the Pending Queue. The system then selects nodes from the Ready Queue to add to the Execution Queue. Figure 5 In this process, Subkernel nodes without prior dependencies can be simultaneously scheduled onto PE array processing units.

[0098] Step S2 establishes the organizational structure for task execution by constructing an execution flow graph by clarifying the dependencies between subkernels. This step lays the foundation for dynamic switching and cross-scheduling of subkernels, ensuring the effectiveness of scheduling and the correct execution order of tasks.

[0099] Step S3: Taking into account both operator priority (PR) and scheduling strategy, select one or more Subkernel nodes (without prior dependencies) on the execution flow graph for scheduling each time.

[0100] Selecting nodes without prerequisite dependencies, i.e., selecting one or more Subkernel nodes from the ready queue to add to the running queue, requires that the peak memory usage of all Subkernels does not exceed the SPM cache capacity, with the following constraints:

[0101]

[0102] Specifically, the switching between operators is controlled by a scheduling policy and is performed at the subkernel granularity. After the switching, multiple operators exhibit a mixed execution mode of sequential and interleaved execution, as described in the reference. Figure 5 The scheduling process is as follows:

[0103] (1) First scheduling: The ready queue contains operator 1-Subkernel0, operator 2-Subkernel0, operator 2-Subkernel1, and operator 2-Subkernel2. According to the scheduling strategy, operator 1-Subkernel0 and operator 2-Subkernel0 are finally selected and scheduled to run on the PE array processing unit. Figure 5 In order to keep the display simple, 1-Subkernel0 is displayed as 1-0, and the other operators are displayed in the same way.

[0104] (2) Update the execution flow graph. The ready queue contains operator 1-Subkernel1, operator 2-Subkernel1, and operator 2-Subkernel2. According to the scheduling policy, operator 2-Subkernel1 and operator 2-Subkernel2 are selected and scheduled to run on the PE array processing unit. Operator 2-Subkernel1 and operator 2-Subkernel2 belong to the same operator as operator 2-Subkernel0 in the previous scheduling. This scheduling is a sequential execution.

[0105] (3) Update the execution flow graph. The ready queue contains operator 1-Subkernel1, operator 3-Subkernel0, and operator 3-Subkernel1. Select operator 1-Subkernel1 and operator 3-Subkernel0. Operator 1-Subkernel1 and operator 3-Subkernel0 are different operators from the previously scheduled operator 2-Subkernel1 and operator 2-Subkernel2. This scheduling involves a crossover switch.

[0106] Step S3 selects eligible subkernels for scheduling, considering operator priorities and scheduling strategies. This step improves the efficiency and flexibility of task scheduling, and dynamic scheduling helps to fully utilize computing resources.

[0107] Step S4: Before scheduling the Subkernel to the PE array processing unit, load the corresponding operator instructions from the instruction cache unit to the instruction storage unit CBUF.

[0108] like Figure 6 As shown, when the instruction storage unit CBUF retrieves instructions from the Subkernel, it needs to determine whether the corresponding instructions already exist in the instruction storage unit CBUF and the instruction cache unit.

[0109] (1) If the instructions in the current subkernel and the instructions left in the instruction cache unit (CBUF) belong to the same operator, and the two subkernels have the same data size and execute independently, the two subkernels share the same set of instructions and do not need to be loaded from the instruction cache unit again. It should be noted that since the input data of each subkernel is of the same size after the operator is divided into subkernels (if the input data can be divided into K equal parts), the execution is independent, and the address of the data transferred to the data cache unit (SPM) is the same each time, the offset address of the instruction is also consistent. Under this condition, the instruction does not need to be retransmitted.

[0110] (2) If the instructions remaining in the current Subkernel and the instruction cache (CBUF) belong to different operators, the instruction cache is accessed. If a cache hit occurs, the corresponding operator instruction is loaded from the instruction cache into the instruction cache (CBUF). If a cache miss occurs and the instruction cache is not full, the instruction cache (CBUF) retrieves the operator instruction from main memory and copies it into the instruction cache. If a cache miss occurs and the instruction cache is full, after the instruction cache (CBUF) retrieves the operator instruction from main memory, it determines which instruction in the instruction cache should be swapped out based on the resident coefficient.

[0111] By designing an instruction cache and a corresponding dynamic caching mechanism, this invention enables the instruction storage unit (CBUF) to prioritize accessing the instruction cache unit when retrieving operator instructions. This effectively reduces the number of instruction transfers between RISC-V or the host and the acceleration platform. Furthermore, the dynamic caching mechanism comprehensively considers operator priority and runtime characteristics, thereby improving the instruction cache hit rate.

[0112] Step S4 preloads the necessary operator instructions to prepare for the upcoming execution of the subkernel. This step reduces the number of instruction transfers between the host with the RISC-V core and the acceleration platform, improving instruction loading efficiency.

[0113] Step S5: Update the residency coefficients of operator instructions in the instruction cache unit. Specifically, after loading the instruction, recalculate the residency coefficients corresponding to the operator instructions in the instruction cache.

[0114] Resident coefficient R′ t The calculation formula is as follows:

[0115]

[0116] Among them, R t K represents the residency coefficient of operators that resided in the instruction cache before the update. i N represents the number of subkernels into which operator i is divided. i N represents the number of subkernels that have been executed by operator i, and N represents the total number of subkernels for the operator. t Represents the total number of all executed subkernels, PR i Let be the priority weight value of operator i, C represent the influence value of the operator's subkernel on the next scheduling of the same operator's subkernel after the operator's subkernel is scheduled this time, and β is the influence coefficient of the operator priority.

[0117] The higher the residency coefficient, the greater the likelihood that the operator will be scheduled again. Therefore, when the instruction cache is full and the instruction fetch from the instruction memory location (CBUF) fails, the residency coefficient is used to determine which operator instruction in the cache should be evicted.

[0118] Step S5 adjusts the instruction caching strategy and dynamically updates the instruction residency information in the cache. This step can improve the instruction cache hit rate, making the instruction loading process more efficient.

[0119] Step S6: The PE array processing unit retrieves instructions from the instruction storage unit CBUF and determines whether any operator instructions have not been loaded for this execution. If so, it jumps to S4; otherwise, it executes S7.

[0120] If multiple subkernels are simultaneously scheduled to a PE array processing unit, instructions need to be loaded sequentially onto the corresponding PE array processing units. For example, if Subkernel0 and Subkernel1 can be scheduled to a PE array processing unit simultaneously, and processing units 0-7 of the PE array processing unit are responsible for executing Subkernel0, while processing units 8-15 are responsible for executing Subkernel1, then the instructions for the two subkernels need to be retrieved sequentially from the instruction memory unit (CBUF) and loaded into the instruction RAM inside the corresponding PE array processing unit.

[0121] Step S6 ensures that the PE array processing unit has all the necessary instructions before executing the Subkernel. This step guarantees the continuity and reliability of Subkernel execution, preventing interruptions.

[0122] Step S7: The PE array processing unit starts and begins executing the Subkernel. After execution, it updates the execution flow graph and determines whether the termination condition is met. If yes, the task execution ends; otherwise, it iterates until the task ends. That is, if yes, the task execution ends; otherwise, it jumps to step S3 until the task ends.

[0123] The task execution ends when the ready queue is empty, meaning all subkernels have finished executing.

[0124] The present invention also provides a scheduling system or apparatus for heterogeneous data flow architecture, the system including a scheduling module and an instruction cache unit.

[0125] The main memory unit or processor with a RISC-V core is configured as a scheduling module for global task scheduling. During runtime, the system analyzes the peak memory usage of each operator to be run, while also considering the maximum capacity of the SPM (System Processing Unit) cache to determine the ideal number of subkernels for each operator. The scheduling module then constructs a computation flow graph based on subkernels, establishing dependency edges between subkernels with data dependencies. During task scheduling, one or more subkernels from the ready queue are selected for accelerated processing by the PE (Processing Array) unit according to the scheduling policy.

[0126] The instruction cache unit is used to fetch the execution instructions required by the subkernel. Before execution by the PE array processing unit, the subkernel instructions need to be loaded from the instruction storage unit (CBUF) into the instruction RAM inside the PE array processing unit. Since the execution instructions change each time the subkernel switches, the operator instructions inside the instruction storage unit (CBUF) need to be updated. The instruction cache can cache instructions for multiple operators and acts as an intermediate instruction buffer between the main memory unit and the instruction storage unit (CBUF). Each instruction fetch synchronously updates the resident coefficients of each operator instruction in the instruction buffer.

[0127] Example 3

[0128] This embodiment is an explanation of the application cases of Embodiments 1 and 2, and the repeated content will not be repeated.

[0129] An intelligent multi-operator cross-scheduling system, applied to a multi-user single-server processing environment, can dynamically switch the task response order based on user task priorities (some user tasks require guaranteed service quality but are not concerned about response time; while others are concerned with both response time and ensuring that service quality is not too poor) and runtime status, thus guaranteeing the average service quality of tasks. Specifically, multiple users send task requests at the same or different times. After receiving the task requests, the server begins to sort the requests and processes the tasks in the task queue sequentially. Different users' tasks have different response priorities, and the processing process may be affected by factors such as network congestion, requiring dynamic adjustment of the execution order between tasks to adapt to the real-time operating environment.

[0130] The intelligent multi-operator cross-scheduling system is applied to this multi-task processing environment, and the relevant steps are as follows:

[0131] The runtime characteristics of each task are collected (for simplicity, each task can be considered to contain only a single operator) to determine the number of subkernels that should be allocated to each task. Since each user is independent, the task requests submitted by each user are also processed independently, and there is no dependency between the subkernels allocated to tasks of different users. The subkernels of a user's task are selected for scheduling based on the user priority (PR) and scheduling policy. During this process, the user's response priority may change (e.g., a user's task suddenly needs to be completed within a specified time), or the task may be affected by changes in the external environment and need to adjust its execution priority (a user's task requires a strict network environment to ensure service quality and does not care about the task delivery time, so it may need to be temporarily interrupted when the network is congested, and then completed with high quality in a stable network environment). Therefore, each time a subkernel is scheduled, the scheduling priority is re-determined based on some status parameters of each task to ensure the average service quality in a multi-task environment. High-priority task instructions are loaded from the instruction cache unit to the instruction storage unit (CBUF) to ensure that all necessary instructions have been loaded. The subkernel is executed in the PE array processing unit, iterating until the task ends.

[0132] By refining the above steps, we can obtain steps S101 to S107:

[0133] S101: Collect runtime characteristics of each user task (network stability, user response priority, etc.) to determine the number of subkernels that should be allocated to each task.

[0134] S102: Different user tasks are independent of each other. The execution flow graph of the task is constructed in units of subkernel. In this application, there is no dependency between subkernels of different user tasks. The sequential execution relationship only exists within the subkernels of a single user task.

[0135] S103: Recalculate the user's task priority (PR) and select a Subkernel node with no prerequisites in the execution flow graph for scheduling based on the scheduling policy. Switching between different tasks depends on whether the recalculation of each user's task priority has changed.

[0136] S104: Before scheduling the Subkernel to the PE array processing unit, load the corresponding task instructions from the instruction cache unit to the instruction storage unit CBUF.

[0137] S105: Update the residency coefficient of task instructions within the instruction cache unit.

[0138] S106: The PE array processing unit retrieves instructions from the instruction storage unit CBUF and determines whether any operator instructions have not been loaded during this execution. If so, the operator instructions are loaded again; otherwise, the Subkernel is executed.

[0139] S107: The PE array processing unit starts and begins executing the Subkernel. After execution, it updates the execution flow graph and determines whether the termination condition is met. If so, the task execution ends; otherwise, it iterates until the task ends.

[0140] For example, intelligent multi-operator cross-scheduling systems can be applied to the automated production management of smart factories.

[0141] In a smart factory, various automated devices (such as robots, sensors, and machine tools) need to operate in coordination to achieve efficient production management and product quality monitoring. By employing PE array processing units and optimized task scheduling strategies, production line efficiency can be improved, downtime reduced, and resource utilization optimized.

[0142] S201: Collect status information for each automated device in real time, including uptime, failure rate, and maintenance cycle. Simultaneously, assess the priority of each production task based on the production plan and order delivery time to determine the number of subkernels each task should be divided into. For example, the manufacturing process of a product may be divided into multiple subkernels, such as raw material preparation, processing, assembly, and quality inspection.

[0143] S202: Execution flow graph is constructed based on subkernels. Different production tasks are independent of each other, but within a single task, there are sequential relationships between subkernels. For example, in a manufacturing task, raw material preparation must be completed before processing.

[0144] S203: Based on the actual operation of the production line and order delivery time, recalculate the task priority (PR) in real time, and select a subkernel without prerequisite dependencies for scheduling according to the scheduling strategy. For example, if raw material preparation is complete, the processing step can begin.

[0145] S204: Before scheduling the Subkernel to the PE array processing unit, the task instructions are loaded from the instruction cache unit to the instruction storage unit CBUF to preload the operation instructions for the processing steps.

[0146] S205: Update the residency coefficient of task instructions within the instruction cache unit to improve future loading efficiency.

[0147] S206: The PE array processing unit retrieves instructions from the instruction storage unit CBUF and determines whether there are any unloaded operator instructions. If so, it loads them again; otherwise, it continues executing the Subkernel. For example, executing the processing steps of product A.

[0148] S207: The PE array processing unit initiates the execution of a subkernel and updates the execution flow graph upon completion, checking if the task has ended. If not, it continues scheduling the next subkernel, such as from processing to assembly, and then to quality inspection, until the task is completed. In this way, the production management system can efficiently coordinate equipment operation, optimize production processes, reduce downtime, and ensure product quality, meeting the needs of diverse and rapid delivery.

[0149] Through the steps described above, the automated production management system of a smart factory can efficiently coordinate the operation of various devices, optimize production processes, reduce downtime, and ensure product quality. Simultaneously, it adjusts task priorities and optimizes task scheduling in real time, improving the flexibility and resource utilization efficiency of the production line and meeting diverse and rapid delivery needs.

[0150] It should be noted that the specific embodiments described above are exemplary. Those skilled in the art can devise various solutions inspired by the disclosure of this invention, and these solutions all fall within the scope of this invention and its protection. Those skilled in the art should understand that this specification and its accompanying drawings are illustrative and not intended to limit the scope of the claims. The scope of protection of this invention is defined by the claims and their equivalents. This specification contains multiple inventive concepts; phrases such as "preferredly" or "according to a preferred embodiment" indicate that the corresponding paragraph discloses an independent concept. The applicant reserves the right to file divisional applications based on each inventive concept.

Claims

1. An intelligent multi-operator cross-scheduling method, characterized in that, The method includes: Collect runtime characteristics of operators, determine the number of subkernels that each operator should be divided into, and construct an execution flow graph with subkernels as the unit, clarifying the dependencies between subkernels; Based on the operator priority (PR) and scheduling policy, select the subkernel node in the execution flow graph that has no prior dependencies for scheduling; The corresponding operator instructions are loaded from the instruction cache unit to the instruction storage unit, ensuring that all necessary instructions have been loaded. Specifically, if the first instruction of the current subkernel and the second instruction left in the instruction cache belong to the same operator, and the two subkernels have the same data size and execute independently, the two subkernels share the same set of instructions and do not need to be loaded from the instruction cache again. If the first instruction of the current subkernel and the second instruction remaining in the instruction storage unit belong to different operators, then the instruction cache unit is accessed. If a cache hit occurs, the corresponding operator instruction is loaded from the instruction cache unit into the instruction storage unit. If a cache miss occurs and the instruction cache unit is not full, the instruction storage unit will retrieve the operator instruction from the main memory unit and copy the operator instruction into the instruction cache unit. If a cache miss occurs and the instruction cache unit is full, after the instruction storage unit retrieves the operator instruction from the main memory unit, it will determine which instruction in the instruction cache unit to swap out based on the resident coefficient. The Subkernel is executed in the PE array processing unit, and the execution flow graph is updated. This process is repeated iteratively until the task ends.

2. The intelligent multi-operator cross-scheduling method according to claim 1, characterized in that, The steps to clarify the dependencies between Subkernels include: When nodes correspond to Subkernels, dependency edges are constructed based on the dependency relationships between Subkernels.

3. The intelligent multi-operator cross-scheduling method according to claim 2, characterized in that, The method further includes updating the residency coefficient of the operator instruction within the instruction cache unit.

4. The intelligent multi-operator cross-scheduling method according to claim 3, characterized in that, The steps for selecting and scheduling Subkernel nodes without prerequisite dependencies in the execution flow graph based on operator priority (PR) and scheduling policy include: Divide the Subkernel load among operators; The cross-switching of multiple operators is achieved by using a hybrid scheduling method that combines sequential execution mode and cross-execution mode.

5. The intelligent multi-operator cross-scheduling method according to claim 4, characterized in that, The steps for selecting a Subkernel node in the execution flow graph that has no preceding dependencies include: If the peak memory usage of the total Subkernel does not exceed the cache capacity of the data cache unit, at least one Subkernel node in the ready queue is selected to be added to the running queue.

6. An intelligent multi-operator cross-scheduling system, characterized in that, include: The host machine collects runtime characteristics of operators, determines the number of subkernels that each operator should be divided into, and constructs an execution flow graph based on subkernels, clarifying the dependencies between subkernels; it selects subkernel nodes without prerequisite dependencies in the execution flow graph for scheduling based on operator priority (PR) and scheduling policy; and it loads the corresponding operator instructions from the instruction cache unit to the instruction storage unit, ensuring that all necessary instructions have been loaded. Specifically, if the first instruction of the current subkernel and the second instruction left in the instruction cache belong to the same operator, and the two subkernels have the same data size and execute independently, the two subkernels share the same set of instructions and do not need to be loaded from the instruction cache again. If the first instruction of the current subkernel and the second instruction remaining in the instruction storage unit belong to different operators, then the instruction cache unit is accessed. If a cache hit occurs, the corresponding operator instruction is loaded from the instruction cache unit into the instruction storage unit. If a cache miss occurs and the instruction cache unit is not full, the instruction storage unit will retrieve the operator instruction from the main memory unit and copy the operator instruction into the instruction cache unit. If a cache miss occurs and the instruction cache unit is full, after the instruction storage unit retrieves the operator instruction from the main memory unit, it will determine which instruction in the instruction cache unit to swap out based on the resident coefficient. PE array processing unit: Executes the Subkernel and updates the execution flow graph, iterating until the task ends.

7. The multi-operator cross-scheduling system according to claim 6, characterized in that, The steps for the host to clarify the dependencies between subkernels include: When nodes correspond to Subkernels, dependency edges are constructed based on the dependency relationships between Subkernels.

8. The multi-operator cross-scheduling system according to claim 7, characterized in that, The step of the host selecting and scheduling a Subkernel node with no preceding dependencies in the execution flow graph includes: Divide the Subkernel load among operators; The cross-switching of multiple operators is achieved by using a hybrid scheduling method that combines sequential execution mode and cross-execution mode.

Images