Task mapping method and device for neural network processor, computer device, storage medium and program product

Abstract

The present disclosure provides a task mapping method and device for a neural network processor, a computer device, a storage medium and a program product, relating to the technical field of computer and artificial intelligence, and the implementation scheme is: obtaining kernel description information associated with a target task; dividing an iteration space into at least one target block; determining a read set and a write set associated with each target block based on the kernel description information, constructing a dependency relationship between the at least one target block; determining a calculation operation to be used by each target block, and a direct memory access operation matched with the calculation operation; and determining a mapping result of the target task based on the dependency relationship and the determined calculation operation and direct memory access operation, including a command stream, a synchronization event pair conforming to the dependency relationship between the at least one target block is inserted in the command stream, which can improve the portability, schedulability and performance predictability of task mapping.

Description

Technical Field

[0001] This disclosure relates to the fields of computer and artificial intelligence (AI) technology, particularly to the fields of neural network processors, parallel task processing, compilation and task scheduling, and especially to a task mapping method, apparatus, computer device, computer-readable storage medium and computer program product for a neural network processor. Background Technology

[0002] With the development of AI technology, such as the continuous expansion of AI models based on neural networks, dedicated hardware devices for matrix or vector operations have emerged in the hardware field, such as Neural Processing Units (NPUs). However, devices from different manufacturers vary significantly, often requiring developers to spend considerable effort adapting to these differences. This may involve writing separate kernels for each type of device to implement task mapping, or relying on the manufacturer's compiler for automatic mapping. Although the industry has begun to explore more general task mapping methods, most are based on thread block semantics, such as mapping a large number of parallel threads to a single kernel. Because this thread-level abstraction often mismatches with hardware queue primitives, such methods still struggle to meet practical needs, necessitating a more efficient task mapping mechanism. Summary of the Invention

[0003] This disclosure provides a task mapping method, apparatus, computer device, computer-readable storage medium, and computer program product for a neural network processor.

[0004] According to one aspect of this disclosure, a task mapping method for a neural network processor is provided, comprising: acquiring kernel description information associated with a target task to be processed by the neural network processor, the kernel description information indicating an iteration space, the iteration space including a plurality of iteration points required to process the target task; dividing the iteration space into at least one target block, each target block including at least one target iteration point from the plurality of iteration points, wherein the address intervals corresponding to the data access windows of each of the at least one target iteration point are continuous or overlapping, and the size of each of the at least one target block is subject to a first constraint associated with the on-chip memory capacity of the neural network processor and an alignment requirement with the computational operations of the neural network processor. The process involves: determining the common constraints of the associated second constraint and the third constraint associated with the direct memory access operations of the neural network processor; constructing at least one dependency relationship between target blocks by determining the read and write sets associated with each target block based on kernel description information; determining the computational operations to be used by each target block and the direct memory access operations matching the computational operations; and determining the mapping result of the target task to be processed by the neural network processor based on the dependency relationship between at least one target block and the determined computational operations and direct memory access operations, wherein the mapping result includes a command stream, in which synchronization event pairs conforming to the dependency relationship between at least one target block are inserted.

[0005] In some embodiments, at least one target block includes a base block and a boundary block. The base blocks are arranged continuously in the iteration space with the same size, and the boundary blocks are smaller than the size of the base blocks in at least one dimension of the iteration space, so as to be padded to the size of the base blocks or set with predicate access rules.

[0006] In some embodiments, the kernel description information includes a static index expression, wherein constructing a dependency relationship between at least one target block by determining the read set and write set associated with each target block based on the kernel description information includes: determining the read address range and write address range of each of at least one target iteration point included in each target block based on the static index expression; merging the read address range and write address range of each of the at least one target iteration point to obtain the read set and write set associated with each target block; and establishing a dependency relationship between the second target block and the first target block in response to determining that the write set of the first target block and the read set of the second target block intersect.

[0007] In some embodiments, dependencies are constructed as a dependency graph, wherein dependency edges in the dependency graph, which represent dependencies, are used to provide corresponding landing points for the insertion of synchronous event pairs.

[0008] In some embodiments, the dependency edges include any one of a first type, a second type, or a third type, wherein the first type is associated with the same queue in the same neural network processor, the second type is associated with different queues in the same neural network processor, and the third type is associated with different neural network processors.

[0009] In some embodiments, the synchronization event pair includes a wait-trigger event, wherein the wait-trigger event has an event synchronization primitive corresponding to a first type, a second type, and a third type, and the event synchronization primitive includes a hardware event, a doorbell, a semaphore, or an interconnect message.

[0010] In some embodiments, determining the computational operation to be used for each target block, and the direct memory access operation matching the computational operation, includes: generating the computational operation by determining the hardware operator operation instruction matching each target block; and generating the direct memory access operation by determining the transport configuration information for direct memory access corresponding to the computational operation.

[0011] In some embodiments, determining the mapping result of the target task to be processed by the neural network processor based on the dependencies between at least one target block and the determined computational operations and direct memory access operations includes: arranging the computational operations and direct memory access operations corresponding to each of the at least one target block into at least one queue according to the dependencies to obtain at least one command stream.

[0012] According to another aspect of this disclosure, a task mapping apparatus for a neural network processor is provided, comprising: an information acquisition module configured to acquire kernel description information associated with a target task to be processed by the neural network processor, the kernel description information indicating an iteration space including a plurality of iteration points required to process the target task; and a block partitioning module configured to partition the iteration space into at least one target block, each target block including at least one target iteration point from a plurality of iteration points, wherein the address intervals corresponding to the data access windows of each of the at least one target iteration point are continuous or overlapping, and the size of each of the at least one target block is subject to a first constraint associated with the on-chip memory capacity of the neural network processor and an alignment requirement associated with the computational operations of the neural network processor. The second constraint and the third constraint associated with the direct memory access operation of the neural network processor are common constraints; a dependency building module is configured to construct a dependency relationship between at least one target block by determining the read set and write set associated with each target block based on kernel description information; an operation determination module is configured to determine the computational operation to be used by each target block and the direct memory access operation matching the computational operation; and a mapping determination module is configured to determine the mapping result of the target task to be processed by the neural network processor based on the dependency relationship between at least one target block and the determined computational operation and direct memory access operation, wherein the mapping result includes a command stream, in which synchronization event pairs that conform to the dependency relationship between at least one target block are inserted.

[0013] According to another aspect of this disclosure, a computer device is provided, comprising: at least one processor; and a memory having a computer program stored thereon, wherein when executed by the at least one processor, the computer program causes the at least one processor to perform the methods provided above in this disclosure.

[0014] According to another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, causes the processor to perform the methods provided above in this disclosure.

[0015] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, causes the processor to perform the methods provided above in this disclosure.

[0016] According to one or more embodiments of this disclosure, abstract parallel task descriptions can be reliably mapped to asynchronous hardware operations and inter-device synchronization events executable by a neural network processor, improving the portability, schedulability, and performance predictability of the method, thereby enhancing the universal adaptability of task mapping to different NPU devices.

[0017] These and other aspects of this disclosure will be apparent from the embodiments described below, and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0018] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of this disclosure. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0019] Figure 1 This is a schematic diagram illustrating an NPU applying a task mapping method according to an embodiment of the present disclosure.

[0020] Figure 2 This is a flowchart illustrating a task mapping method for an NPU according to an embodiment of the present disclosure.

[0021] Figure 3 This is a schematic diagram illustrating the basic blocks and boundary blocks in the target block according to an embodiment of the present disclosure.

[0022] Figure 4 This is a schematic diagram illustrating the process of determining the read set and write set of the target block according to an embodiment of the present disclosure.

[0023] Figure 5 This is a schematic diagram illustrating the dependencies for characterizing target blocks according to embodiments of the present disclosure.

[0024] Figure 6 This is a block diagram illustrating a task mapping apparatus for an NPU according to an embodiment of the present disclosure.

[0025] Figure 7 An example computer device is shown in which any of the embodiments described herein may be implemented. Detailed Implementation

[0026] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0027] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0028] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. As used herein, the term "multiple" means two or more, and the term "based on" should be interpreted as "at least partially based on". Furthermore, the terms "and / or" and "at least one of..." cover any one of the listed items and all possible combinations thereof.

[0029] In related technologies, for hardware devices like NPUs, asynchronous operation queues, data transport, and event synchronization are typically used as the main execution primitives. Computation and transport are separated, and the fusion and pipelining of operators depend on the progression of explicit events. In this context, traditional thread-block semantic-based mapping suffers from inherent problems due to the mismatch between thread-level abstraction and hardware queue primitives. This makes it difficult for compilers to consistently generate efficient command flows, impacting not only performance predictability but also resulting in suboptimal portability and schedulability.

[0030] To this end, embodiments of this disclosure provide a general task mapping method for NPUs that can reliably map abstract parallel task descriptions (such as CUDA-like kernels) to asynchronous hardware operations and inter-device synchronization events executable by the NPU, thereby improving the portability, schedulability, and performance predictability of the method.

[0031] Before describing in detail the task mapping method for an NPU according to embodiments of this disclosure, an NPU to which the method is applied will be briefly described for ease of understanding.

[0032] Figure 1 This is a schematic diagram illustrating an NPU applying a task mapping method according to an embodiment of the present disclosure.

[0033] like Figure 1As shown, the first NPU 100 may include a first computing engine 101, a first data transfer engine 102, a first on-chip event management unit 103, and a first on-chip memory unit 104. The second NPU 200 may have the same structure as the first NPU 100, including a second computing engine 201, a second data transfer engine 202, a second on-chip event management unit 203, and a second on-chip memory unit 204. Here, the description focuses on the first NPU 100 as an example. It is understood that although... Figure 1 The description uses two NPUs as an example, but in practice, it is not limited to this.

[0034] The first computing engine 101 can perform computational operations, such as matrix operations or vector operations. The first data transfer engine 102 can perform Direct Memory Access (DMA) operations, i.e., data transfer, such as transferring data from external memory to the first on-chip memory unit 104, or transferring data from the first on-chip memory unit 104 to external memory. The first data transfer engine 102 does not participate in computational operations, and therefore can work independently and in parallel with the first computing engine 101. The first on-chip event management unit 103 can be used to manage hardware events on the first NPU 100, such as the completion of computational operations or DMA operations, and can also be used to manage event dependencies across NPU devices (such as the first NPU 100 and the second NPU 200). The first on-chip memory unit 104 can be a high-speed storage unit inside the first NPU 100, typically with small capacity but high bandwidth, and can be used to store data for the current computational operation.

[0035] In the example, the first transfer engine 102 can transfer the required data from external memory to the first on-chip memory unit 104, whereby the first computing engine 101 can read the data from the first on-chip memory unit 104 to perform computing operations and write the results to the first on-chip memory unit 104. The first transfer engine 102 can then continue to transfer the results to external memory.

[0036] Figure 2 This is a flowchart illustrating a task mapping method for an NPU according to an embodiment of the present disclosure.

[0037] like Figure 2 As shown, the task mapping method for the NPU includes steps S201 to S205.

[0038] In step S201, kernel description information associated with the target task to be processed by the NPU is obtained. The kernel description information indicates an iteration space, which includes multiple iteration points required to process the target task.

[0039] In the example, kernel description information may include a CUDA-like kernel (often referred to as a "CUDA-like kernel") and its metadata. A CUDA-like kernel can be executed concurrently by multiple execution instances to perform a target task (such as matrix or vector operations). Each execution instance can correspond to an iteration point in the iteration space, and the data addresses that the execution instance needs to access are determined by predefined static index expressions. Metadata may include grid dimensions indicating thread blocks and threads, shared / global memory access patterns indicating data access, and optional operator fusion hints indicating optimized computation methods.

[0040] In step S202, the iteration space is divided into at least one target block, each target block including at least one target iteration point from the plurality of iteration points, wherein the address ranges corresponding to the data access windows of each of the at least one target iteration point are continuous or overlapping. The size of each of the at least one target block is subject to a common constraint: a first constraint associated with the on-chip memory capacity of the NPU, a second constraint associated with the alignment requirements of the NPU's computational operations, and a third constraint associated with the NPU's DMA operations.

[0041] In the example, each target block can be viewed as an aggregation of several target iteration points. The several target iteration points that can be aggregated have a specific correlation, meaning that the address ranges corresponding to the data access windows of each of these target iteration points are continuous or overlapping. That is, the target block uses the data access window with strong locality as the aggregation criterion. Here, the data access window of a target iteration point can refer to the range of data that needs to be accessed when traversing to that target iteration point, and this data range can be related to the corresponding address space. Therefore, in the embodiments of this disclosure, it is required that the address ranges corresponding to the data access windows of at least one target iteration point included in each target block are continuous or overlapping, which means that the at least one target iteration point can share a single DMA transfer. Taking the convolution operation on the input feature map to obtain the output feature map as an example, a point in the output feature map can correspond to an iteration point in the iteration space, and the coverage area of ​​the convolution kernel on the input feature map can correspond to the data access window. Several points in the output feature map can be aggregated into a target block based on whether the address ranges corresponding to the data access windows are continuous or overlapping.

[0042] In the example, the size of the target block can refer to the number of target iteration points included in that block across all dimensions. For example, a target block size of (4, 4) could represent 4 target iteration points in the height direction and 4 target iteration points in the width direction. Therefore, the larger the target block size, the more target iteration points there are, and the larger the data volume. Regarding the first constraint, since the on-chip memory capacity of the NPU can refer to the on-chip memory units in the NPU (such as those combined with...), Figure 1 The first constraint takes into account the inherent limitations of the NPU in terms of memory, specifically the memory capacity of the first on-chip memory unit (104). This prevents the data size corresponding to a target block from exceeding the capacity of the on-chip memory, as the NPU's computational engine needs to read data from the on-chip memory unit. For the second constraint, the alignment requirement of the NPU's computational operations can refer to the granularity of data processing by the NPU's computational engine. For example, matrix operations require input blocks such as 4×4 or 8×8, and vector operations require inputs that are integer multiples of 16 or 32 elements. Therefore, the second constraint takes into account the inherent limitations of the NPU in terms of computation. Similarly, the third constraint is related to the NPU's DMA operations, i.e., the granularity of DMA transactions. For example, if a DMA operation moves data in units of 32 bytes, then the data size corresponding to a target block needs to be an integer multiple of the DMA transaction granularity. Therefore, the third constraint takes into account the inherent limitations of the NPU in terms of data movement.

[0043] In step S203, the read set and write set associated with each target block are determined based on kernel description information, and at least one dependency relationship between target blocks is constructed.

[0044] In the example, if a dependency exists between two target blocks, it means that the computation of one target block requires the computation result of the other target block, and therefore the computation of the other target block must be completed first. In the embodiments of this disclosure, this dependency is constructed based on the logical relationship between reads and writes. That is, if the read set of one target block overlaps with the write set of another target block, it means that a dependency exists between the two target blocks, meaning that the write set of the other target block must be completed before the first target block can read from the read set.

[0045] In the example, since the static index expression in the kernel description information can provide an access formula for deriving the address access range based on the coordinates of the iteration points, it can be used to derive the address access range for each target iteration point. Furthermore, by merging the address access ranges of all target iteration points within each target block, the read set and write set corresponding to each target block can be obtained, and both the read set and write set can include one or more address ranges.

[0046] In step S204, the computational operation to be used for each target block and the DMA operation that matches the computational operation are determined.

[0047] In the example, when the NPU supports multi-level queues or has an independent replication engine, compute operations and DMA operations can be assigned to their respective queues to form a pipeline.

[0048] In step S205, based on the dependencies between at least one target block and the determined computational and DMA operations, a mapping result for the target task to be processed by the NPU is determined. This mapping result includes a command stream, into which pairs of synchronization events conforming to the dependencies between the at least one target block are inserted.

[0049] In the example, after dividing the target blocks into individual blocks via step S202 and determining the dependencies between the target blocks via step S203, intermediate representation information of the target blocks can be obtained. Then, this intermediate representation information is further downgraded into a command flow for the NPU devices via steps S204 and S205. Steps S204 and S205 can be executed by a runtime scheduler, which can determine the command flow by referring to a topology description of the NPU devices. This topology description may include the number of NPU devices, the interconnect bandwidth / latency between the NPU devices, the number of queues for a single NPU device, and the event primitive capabilities of the NPU devices. The command flow may include asynchronous hardware operations and event operations. Asynchronous hardware operations may include computational operations and DMA operations performed by the NPU. Event operations may include synchronous event pairs to indicate waiting and triggering within the same queue in the same NPU, between different queues in the same NPU, or across NPU devices. Accordingly, the synchronous event pairs need to conform to the dependencies between the target blocks, thus interspersed among the computational / DMA operations in the form of event instrumentation.

[0050] Therefore, the task mapping method according to embodiments of this disclosure provides a general task mapping method for NPUs. By introducing target-block-based programming semantics, the kernel's iteration space is explicitly divided into multiple target blocks, and target blocks are used as basic scheduling units to construct target block-level dependencies. Based on this, each target block is mapped to a set of asynchronous hardware operations (such as matrix multiplication / convolution instructions, vector operations, DMA transfers, on-chip padding / transpose, etc.), and event operations are inserted based on dependencies. This enables scheduling in scenarios within the same queue of the same NPU, in different queues of the same NPU, and between different NPU devices. This method is adaptable to both CUDA-like kernel expressions and common NPU execution modes combining command queues, events, and explicit transfers. It allows for parallelism control, memory usage constraints, and pipelining optimization at compile and runtime using unified intermediate representation information, thereby improving the portability, schedulability, and performance predictability of the task mapping method, and ultimately enhancing the general adaptability of task mapping to different NPU devices.

[0051] The following describes in further detail various aspects of the task mapping method for an NPU according to embodiments of the present disclosure.

[0052] Figure 3 This is a schematic diagram illustrating the basic blocks and boundary blocks in the target block according to an embodiment of the present disclosure.

[0053] In some embodiments, the at least one target block may include a basic block and boundary blocks, such as Figure 3 The diagram shows a first basic block 311 and a second basic block 312, as well as a first boundary block 321 and a second boundary block 322. The basic blocks can be arranged continuously with the same size in the iteration space, such as... Figure 3 As shown, the first basic block 311 and the second basic block 312 can have the same size and be arranged continuously in the iteration space 300. The boundary block has a smaller size than the basic block in at least one dimension of the iteration space, such as... Figure 3 As shown, the dimensions of the first boundary block 321 and the second boundary block 322 in the width dimension are smaller than the dimensions of the first basic block 311 and the second basic block 312. It is understandable that... Figure 3 The illustration uses a two-dimensional iteration space as an example, but the scope of this disclosure is not limited to this, and the iteration space can also have more dimensions.

[0054] In this case, for the first boundary block 321 or the second boundary block 322, it can be padded to the size of the first basic block 311 or the second basic block 312, or it can be processed by setting predicated access rules. The padding operation can include filling in values ​​such as zero to make the size of the boundary block reach the size of the basic block. Predicated access is a processing mechanism that determines whether to execute an operation by adding a condition (i.e., a predicated access rule) to each operation without padding the data. For example, assuming the size of the basic block is 4×4 and the size of the boundary block is 3×3, the predicated access rule can be set to calculate the first three points in the first to third rows and skip the fourth point, while skipping all four points in the fourth row.

[0055] Therefore, in the task mapping method according to the embodiments of this disclosure, since both basic blocks and boundary blocks are fully considered in the process of dividing the target blocks, and optional processing methods are set for the special forms of boundary blocks, it can more flexibly adapt to the hardware capabilities of different NPUs. For example, it can cover special forms even when the hardware supports or does not support branch prediction, thereby ensuring the accuracy and effectiveness of the target block division and improving the overall implementation effect of the method.

[0056] In some embodiments, the above combination Figure 2Step S203, which involves determining the read set and write set associated with each target block based on kernel description information and constructing a dependency relationship between at least one target block, may include: determining the read address range and write address range of each target iteration point included in each target block based on a static index expression; merging the read address range and write address range of each target iteration point to obtain the read set and write set associated with each target block; and establishing a dependency relationship between the second target block and the first target block in response to determining that the write set of the first target block and the read set of the second target block intersect.

[0057] Figure 4 This is a schematic diagram illustrating the process of determining the read set and write set of the target block according to an embodiment of the present disclosure.

[0058] like Figure 4 As shown, the following example illustrates the process: the first target block 410 includes the first target iteration point 411 and the second target iteration point 412, and the second target block 420 includes the third target iteration point 421 and the fourth target iteration point 422.

[0059] As mentioned earlier, since static index expressions can provide access formulas for deriving address access ranges based on iteration point coordinates, they can be used to derive the address access range for each target iteration point, namely the read address range and the write address range.

[0060] For the first target block 410, the read address range 1 and write address range 1 of the first target iteration point 411, and the read address range 2 and write address range 2 of the second target iteration point 412 can first be determined based on the static index expression. Then, read address range 1 and read address range 2 can be merged, and write address range 1 and write address range 2 can also be merged. In this example, merging multiple address ranges may include taking the union of these multiple address ranges. Assuming that in this example, merging read address range 1 and read address range 2 yields read set 1, and merging write address range 1 and write address range 2 yields write set 1. Similarly, for the second target block 420, since the operation process is similar, it will not be described again here. Assuming that read set 2 and write set 2 of the second target block 420 are determined via the read address ranges and write address ranges of the third target iteration point 421 and the fourth target iteration point 422 respectively.

[0061] In this case, it can be further determined whether there is an intersection between the read set 1 of the first target block 410 and the read set 2 of the second target block 420, and whether there is an intersection between the write set 1 of the first target block 410 and the write set 2 of the second target block 420. Assuming that in this example, it is determined that there is an intersection between the write set 1 of the first target block 410 and the read set 2 of the second target block 420, it can be determined that the second target block 420 depends on the first target block 410; that is, the write operation of the first target block 410 should be performed first, followed by the read operation of the second target block 420.

[0062] Therefore, by constructing the dependency relationship between target blocks based on the logical relationship of reading and writing, the target blocks with predecessor dependencies can be identified from all target blocks. This allows target blocks without dependencies to be executed in parallel without synchronization, thereby avoiding global synchronization idle time and improving parallel efficiency and resource utilization.

[0063] Figure 5 This is a schematic diagram illustrating the dependencies for characterizing target blocks according to embodiments of the present disclosure.

[0064] In some embodiments, dependencies can be constructed as a dependency graph. The dependency edges in the dependency graph, which represent dependencies, can be used to provide corresponding endpoints for the insertion of synchronous event pairs.

[0065] like Figure 5 As shown, target block 2 depends on target block 1, and this dependency is represented by dependency edge 500. In this example, the dependency graph can be in the form of a DAG (Directed Acyclic Graph). Accordingly, nodes in the DAG can be used to represent target blocks, and edges (i.e., dependency edge 500) can be used to represent dependencies. As previously mentioned, synchronization event pairs can be used to indicate waiting and triggering within the same queue in the same NPU, between different queues in the same NPU, or across NPU devices. Therefore, the synchronization event pair needs to conform to the dependencies between the target blocks, thus being interspersed among the various compute / DMA operations in the form of event instrumentation. Accordingly, in this DAG, dependency edge 500 can be used to provide the appropriate landing point for the insertion of the synchronization event pair.

[0066] By using dependency graphs, a bridge can be built from logical dependencies to physical synchronization, providing a clear graph structure basis for event instrumentation. This makes the insertion position of synchronization events precise and controllable, which helps to reliably generate command flows for task mapping.

[0067] In some embodiments, dependency edge 500 may include any one of a first type, a second type, or a third type. The first type may be associated with the same queue within the same NPU. The second type may be associated with different queues within the same NPU. The third type may be associated with different NPUs.

[0068] In some embodiments, the synchronization event pair may include a wait-signal event. The wait-signal event may have event synchronization primitives corresponding to the first, second, and third types, respectively. Event synchronization primitives may include hardware events, doorbells, semaphores, or interconnect messages.

[0069] In the example, a wait-trigger event can involve a process where, if one target chunk depends on another target chunk, the first target chunk waits for the second target chunk to complete before triggering its own processing, thus becoming a synchronization event pair. When implementing wait-trigger events in different scenarios, they can be adapted to different hardware mechanisms; therefore, event synchronization primitives can include hardware events, doorbells, semaphores, or interconnect messages. Hardware events can support the first and second types for intra-device use. Semaphores can be used to support the second type for cross-queue counting synchronization within the same device. Doorbells and interconnect messages can be used for the third type for cross-device notification or synchronization.

[0070] By classifying dependency edges according to the differences in the hardware location where dependencies occur, different synchronization mechanisms and overheads can be adapted to achieve fine-grained scheduling optimization. Simultaneously, through the polymorphism of event primitives, a single dependency description can be used to adapt to multiple hardware environments, thus helping to adaptively select the optimal hardware synchronization primitive and achieve a balance between overhead and performance.

[0071] In some embodiments, the above combination Figure 2 Step S204, namely the step of determining the computation operation to be used for each target block and the DMA operation matching the computation operation, may include: generating a computation operation by determining the hardware operator operation instruction matching each target block; and generating a DMA operation by determining the transfer configuration information for DMA corresponding to the computation operation.

[0072] In the example, available hardware operator instructions can be selected for each target block, such as matrix multiplication / convolution instructions, vector operations, and fusion operator combinations. The transport configuration information used for DMA can include parameters such as source / destination addresses, stride, alignment, rearrangement / transpose, etc.

[0073] Therefore, by downgrading the logical description of the target block to a physical command flow executable by the NPU, a bridge can be built between software abstraction and hardware primitives, thereby achieving a precise mapping from abstract target blocks to specific hardware instructions, which helps to enable efficient collaboration between computing and data handling.

[0074] In some embodiments, the above combination Figure 2 Step S205, which is the step of determining the mapping result of the target task to be processed by the NPU based on the dependency relationship between at least one target block and the determined computational operations and DMA operations, may include: arranging the computational operations and DMA operations corresponding to each of the at least one target block into at least one queue according to the dependency relationship to obtain at least one command stream.

[0075] In the example, when the NPU comprises a single queue, computation operations and DMA operations can be mixed within that single queue. Therefore, this queue can correspond to a command stream, with operations within the queue executed sequentially, and dependencies guaranteed by the order within the queue. For example, the command stream of this queue might include computation operations for target block A, DMA write-back operations for target block A, computation operations for target block B (dependent on target block A), and DMA write-back operations for target block B. When the NPU supports multi-level queues, computation operations and DMA operations can be assigned to their respective queues. Similarly, for cross-device dependencies, runtime synchronization is achieved through the transmission of appropriate signals, allowing data transfer and computation to proceed in parallel within queues on different devices.

[0076] By transforming the dependencies at the target block level into a sequence of hardware-executable command streams, the final mapping from logical target block scheduling to physical hardware execution can be achieved, thereby enabling the NPU to run efficiently in an event-driven manner and achieve coordination between asynchronous parallel execution and precise synchronization.

[0077] According to embodiments of this disclosure, a task mapping apparatus for an NPU is also provided.

[0078] Figure 6 This is a block diagram illustrating a task mapping apparatus for an NPU according to an embodiment of the present disclosure.

[0079] like Figure 6 As shown, in some embodiments, the task mapping device 600 for the NPU includes an information acquisition module 601, a block partitioning module 602, a dependency building module 603, a hardware operation determination module 604, and a mapping determination module 605.

[0080] The information acquisition module 601 is configured to acquire kernel description information associated with a target task to be processed by the NPU, the kernel description information indicating an iteration space that includes multiple iteration points required to process the target task.

[0081] The block partitioning module 602 is configured to divide the iteration space into at least one target block, each target block including at least one target iteration point from the plurality of iteration points, wherein the address ranges corresponding to the data access windows of the at least one target iteration point are continuous or overlapping, and the size of the at least one target block is subject to the common constraints of a first constraint associated with the on-chip memory capacity of the NPU, a second constraint associated with the alignment requirements of the NPU's computational operations, and a third constraint associated with the NPU's DMA operations.

[0082] The dependency building module 603 is configured to build dependencies between the at least one target block by determining the read and write sets associated with each target block based on kernel description information.

[0083] The hardware operation determination module 604 is configured to determine the computational operation to be used for each target block, as well as the DMA operation that matches the computational operation.

[0084] The mapping determination module 605 is configured to determine the mapping result of the target task to be processed by the NPU based on the dependency relationship between the at least one target block and the determined computational and DMA operations, wherein the mapping result includes a command stream, and the command stream is inserted with synchronization event pairs that conform to the dependency relationship between the at least one target block.

[0085] The operations of the aforementioned information acquisition module 601, block partitioning module 602, dependency construction module 603, hardware operation determination module 604, and mapping determination module 605 can be combined. Figure 2 The operations of steps S201 to S205 are the same, so the details of each aspect will not be repeated here.

[0086] In some embodiments, the dependency building module 603 may include: an address range determination module 6031, configured to determine, based on a static index expression, the read address range and write address range of each of the at least one target iteration points included in each target block; a set determination module 6032, configured to merge the read address range and write address range of each of the at least one target iteration point to obtain a read set and a write set associated with each target block; and a relationship establishment module 6033, configured to establish a dependency relationship between the second target block and the first target block in response to determining that the write set of the first target block and the read set of the second target block intersect.

[0087] In some embodiments, the hardware operation determination module 604 may include: a calculation determination module 6041, configured to generate a calculation operation by determining a hardware operator operation instruction that matches each target block; and a transport determination module 6042, configured to generate a DMA operation by determining transport configuration information for DMA corresponding to the calculation operation.

[0088] In some embodiments, the mapping determination module 605 may include a command stream generation module 6051, configured to arrange the computation operations and DMA operations corresponding to each of the at least one target block into at least one queue according to their dependencies to obtain at least one command stream.

[0089] According to one aspect of this disclosure, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory. The processor is configured to execute the computer program to implement the steps of any of the method embodiments described above.

[0090] According to one aspect of this disclosure, a non-transitory computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of any of the method embodiments described above.

[0091] According to one aspect of this disclosure, a computer program product is also provided, which includes a computer program that, when executed by a processor, implements the steps of any of the method embodiments described above.

[0092] Figure 7 An example computer device 700 is shown in which any of the embodiments described herein may be implemented. The computer device 700 may be used to implement one or more components of the systems and methods described above. The computer device 700 may include a bus 702 or other communication mechanism for communicating information, and one or more processors 704 coupled to the bus 702 for processing information. The processor 704 may be, for example, one or more general-purpose microprocessors.

[0093] Computer device 700 may also include main memory 706, such as random access memory (RAM), cache, and / or other dynamic storage devices, coupled to bus 702, for storing information and instructions to be executed by processor 704. Main memory 706 may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 704. Such instructions, when stored in a storage medium accessible to processor 704, can make computer device 700 a special-purpose machine customized to perform the operations specified in the instructions. Main memory 706 may include non-volatile media and / or volatile media. Non-volatile media may include, for example, optical discs or magnetic disks. Volatile media may include dynamic memory. Common media formats may include, for example, floppy disks, collapsible disks, hard disks, solid-state drives, magnetic tapes or any other magnetic data storage media, CD-ROMs (read-only optical disc drives), any other optical data storage media, any physical media with a perforated arrangement, RAM (random access memory), DRAM (dynamic random access memory), PROM (programmable read-only memory) and EPROM (erasable programmable read-only memory), FLASH-EPROM (fast erase programmable read-only memory), NVRAM (non-volatile random access memory), any other memory chips or tape cartridges, or network versions of the above.

[0094] Computer device 700 may implement the techniques described herein using custom hardwired logic, one or more ASICs (Application-Specific Integrated Circuits) or FPGAs (Field-Programmable Gate Arrays), firmware, and / or program logic, which, when combined with computer device 700, enable computer device 700 to become a special-purpose machine or to be programmed therein. According to one embodiment, the techniques herein are executed by computer device 700 in response to processor 704 executing one or more sequences of one or more instructions contained in main memory 706. Such instructions may be read into main memory 706 from another storage medium, such as storage device 708. Executing the sequence of instructions contained in main memory 706 causes processor 704 to perform the processing steps described herein. For example, the processes / methods disclosed herein may be implemented by computer program instructions stored in main memory 706. When these instructions are executed by processor 704, they may perform the steps shown in the corresponding figures and as described above. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0095] Computer device 700 also includes a network interface 710 coupled to bus 702. Network interface 710 can provide bidirectional data communication coupled to one or more network links connected to one or more networks. As another example, network interface 710 can be a local area network (LAN) card to provide data communication connectivity with a compatible LAN (or a WAN component communicating with a WAN (wide area network)). Wireless links can also be implemented.

[0096] The performance of certain operations can be distributed across processors, not just residing within a single machine, but deployed across many machines. In some exemplary embodiments, the processor or the processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other exemplary embodiments, the processor or the processor-implemented engine may be distributed across many geographic locations.

[0097] Each process, method, and algorithm described in the preceding sections can be embodied in a code module executed by one or more computer systems or computer processors including computer hardware, and can be fully or partially automated by them. These processes and algorithms can be implemented, in part or in whole, in a specific application circuit.

[0098] When the functions disclosed herein are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Specific technical solutions (all or part) disclosed herein, or aspects contributing to the prior art, can be embodied in the form of a software product. This software product can be stored in a storage medium and includes instructions to cause a computer device (which may be a personal computer, server, network device, etc.) to perform all or part of the steps of the methods described in the embodiments of this application. The storage medium may include a flash drive, a portable hard drive, ROM, RAM, a magnetic disk, an optical disk, another medium suitable for storing program code, or any combination thereof.

[0099] The embodiments disclosed herein can be implemented via a cloud platform, server, or group of servers that interact with a client. The client can be a terminal device or a client registered by a user on the platform, wherein the terminal device can be a mobile terminal, a personal computer (PC), or any device that can install platform applications.

[0100] The various features and processes described above can be used independently or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Furthermore, certain method or process blocks may be omitted in some embodiments. The methods and processes described herein are not limited to any particular order, and associated blocks or states may be executed in other suitable orders. For example, described blocks or states may be executed in a non-specifically disclosed order, or multiple blocks or states may be combined in a single block or state. Exemplary blocks or states may be executed serially, in parallel, or otherwise. Blocks or states may be added to or removed from the disclosed exemplary embodiments. The exemplary systems and components described herein may be configured differently from those described. For example, elements may be added, removed, or rearranged compared to the disclosed exemplary embodiments.

[0101] The various operations of the exemplary methods described herein can be performed at least in part by an algorithm. An algorithm may consist of program code or instructions stored in memory (such as the non-transitory computer-readable storage medium described above). Such an algorithm may include a machine learning algorithm. In some embodiments, the machine learning algorithm may not be explicitly programmed into the computer to perform the function, but may learn from training data to obtain a predictive model for performing that function.

[0102] The various operations of the exemplary methods described herein can be performed at least in part by one or more processors, which are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute the engine of a processor implementation whose operation is to perform one or more of the operations or functions described herein.

[0103] Similarly, the methods described herein can be implemented at least partially by a processor, where a specific processor or one or more processors are examples of hardware. For example, at least some operations of the methods can be performed by one or more processors or an engine implemented by a processor. Furthermore, one or more processors can also run in a “cloud computing” environment or as “Software as a Service” (SaaS) to support the execution of the relevant operations. For example, at least some operations can be performed by a group of computers (as an example of a machine including processors), which can be accessed via a network (e.g., the Internet) and through one or more appropriate interfaces (e.g., application programming interfaces (APIs)).

[0104] The performance of certain operations can be distributed across processors, not just residing within a single machine, but deployed across many machines. In some exemplary embodiments, the processor or the processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other exemplary embodiments, the processor or the processor-implemented engine may be distributed across many geographic locations.

[0105] In this specification, multiple instances may implement components, operations, or structures described as a single instance. Although individual operations of one or more methods are described and illustrated as independent operations, one or more individual operations may be performed concurrently, and these operations are not required to be performed in the order shown. Structures and functionalities presented as independent components in the example configuration may be implemented as combined structures or components. Similarly, structures and functionalities presented as individual components may be implemented as independent components. These and other variations, modifications, additions, and improvements are all within the scope of this document.

[0106] As used herein, “or” is inclusive rather than exclusive unless explicitly stated or indicated by context. Furthermore, “and” is both common and individual unless explicitly stated or indicated by context. Moreover, multiple instances may be provided for the resources, operations, or structures described herein as a single example. Furthermore, the boundaries between various resources, operations, engines, and data stores are somewhat arbitrary, and specific operations are illustrated within the context of a particular illustrative configuration. The allocation of other functionalities is conceivable and may fall within the scope of various embodiments of this disclosure. Generally, structures and functionalities presented as independent resources in example configurations may be implemented as combined structures or resources. Similarly, structures and functionalities presented as individual resources may be implemented as independent resources. These and other variations, modifications, additions, and improvements are all within the scope of embodiments of this disclosure. Therefore, this specification and accompanying drawings should be viewed in an illustrative rather than restrictive sense.

[0107] The terms “comprising” or “including” are used to indicate the presence of a subsequently stated feature, but do not preclude the addition of other features. Conditional language, in particular, such as “may,” “can,” or “may,” unless specifically stated or otherwise understood in the context of use, is generally intended to express that certain embodiments include certain features, elements, and / or steps, while other embodiments do not. Therefore, such conditional language generally does not imply that a feature, element, and / or step is necessary in any way for one or more embodiments, or that one or more embodiments must include logic that, with or without user input or prompting, determines whether such features, elements, and / or steps are included in any particular embodiment, or whether they are to be performed in any particular embodiment.

Claims

1. A task mapping method for a neural network processor, characterized in that, The method includes: Obtain kernel description information associated with a target task to be processed by a neural network processor, the kernel description information indicating an iteration space including multiple iteration points required to process the target task; The iteration space is divided into at least one target block, each target block including at least one target iteration point from the plurality of iteration points, wherein the address ranges corresponding to the data access windows of each of the at least one target iteration point are continuous or overlapping, and the size of each of the at least one target block is subject to the common constraints of a first constraint associated with the on-chip memory capacity of the neural network processor, a second constraint associated with the alignment requirements of the computational operations of the neural network processor, and a third constraint associated with the direct memory access operations of the neural network processor. By determining the read set and write set associated with each target block based on the kernel description information, the dependency relationship between corresponding two target blocks in the at least one target block is constructed; Determine the computational operations to be used for each target block, and the direct memory access operations that match the computational operations; and Based on the dependencies and the determined computational operations and direct memory access operations, a mapping result for the target task to be processed by the neural network processor is determined, wherein the mapping result includes a command stream, and the command stream contains synchronization event pairs that conform to the dependencies.

2. The method according to claim 1, characterized in that, The at least one target block includes a basic block and a boundary block. The basic blocks are arranged continuously with the same size in the iteration space. The boundary blocks are smaller than the size of the basic blocks in at least one dimension of the iteration space, so that they can be padded to the size of the basic blocks or set with predicate access rules.

3. The method according to claim 1 or 2, characterized in that, The kernel description information includes static index expressions, wherein the step of determining the read set and write set associated with each target block based on the kernel description information, and constructing the dependency relationship between corresponding two target blocks in the at least one target block, includes: Based on the static index expression, the read address range and write address range of each of the at least one target iteration point included in each target block are determined; The read address range and write address range of each of the at least one target iteration point are merged to obtain the read set and write set associated with each target block; and In response to determining that the write set of the first target block and the read set of the second target block intersect, a dependency relationship is established whereby the second target block depends on the first target block.

4. The method according to claim 1 or 2, characterized in that, The dependencies are constructed in the form of a dependency graph, wherein the dependency edges in the dependency graph that represent the dependencies are used to provide corresponding landing points for the insertion of the synchronization event pairs.

5. The method according to claim 4, characterized in that, The dependency edges include any one of a first type, a second type, or a third type, wherein the first type is associated with the same queue in the same neural network processor, the second type is associated with different queues in the same neural network processor, and the third type is associated with different neural network processors.

6. The method according to claim 5, characterized in that, The synchronization event pair includes wait-trigger events, wherein the wait-trigger events have event synchronization primitives corresponding to the first type, the second type and the third type respectively, and the event synchronization primitives include hardware events, doorbells, semaphores or interconnect messages.

7. The method according to claim 1 or 2, characterized in that, The determination of the computational operation to be used for each target block, and the direct memory access operation matching the computational operation, includes: The computational operation is generated by determining the hardware operator operation instructions that match each target block; and The direct memory access operation is generated by determining the transport configuration information for direct memory access corresponding to the computation operation.

8. The method according to claim 1 or 2, characterized in that, The process of determining the mapping result of the target task to be processed by the neural network processor based on the dependencies and the determined computational operations and direct memory access operations includes: The computational operations and direct memory access operations corresponding to each of the at least one target block are arranged into at least one queue according to the dependency relationship to obtain at least one command stream.

9. A task mapping device for a neural network processor, characterized in that, The device includes: The information acquisition module is configured to acquire kernel description information associated with a target task to be processed by a neural network processor, the kernel description information indicating an iteration space, the iteration space including multiple iteration points required to process the target task; The block partitioning module is configured to divide the iteration space into at least one target block, each target block including at least one target iteration point among the plurality of iteration points, wherein the address intervals corresponding to the data access windows of each of the at least one target iteration point are continuous or overlapping, and the size of each of the at least one target block is subject to the common constraints of a first constraint associated with the on-chip memory capacity of the neural network processor, a second constraint associated with the alignment requirements of the computational operations of the neural network processor, and a third constraint associated with the direct memory access operations of the neural network processor. The dependency building module is configured to determine the read set and write set associated with each target block based on the kernel description information, and to build the dependency relationship between corresponding two target blocks in the at least one target block; An operation determination module is configured to determine the computational operation to be used for each target block, and the direct memory access operation matching the computational operation; and The mapping determination module is configured to determine the mapping result of the target task to be processed by the neural network processor based on the dependency relationship and the determined computational operation and the direct memory access operation, wherein the mapping result includes a command stream, and the command stream is inserted with synchronization event pairs that conform to the dependency relationship.

10. A computer device, characterized in that, The computer device includes: At least one processor; A memory having a computer program stored thereon, wherein, when executed by the at least one processor, the computer program causes the at least one processor to perform the method of any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1-8.

12. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1-8.

Images