Methods, products, and media for use in ai accelerator chips

Abstract

Disclosed are a method, product, and medium for use in an AI accelerator chip, the method comprising: statically dividing a plurality of physical buffers in a shared memory of the AI accelerator chip according to a preset number of pipeline stages, and locking a base address of each of the plurality of physical buffers; parsing computation and memory allocation logic of an operator block of an AI model, and inserting a synchronization token on one or more intermediate representation nodes of the AI model based on a result of the parsing, the synchronization token inserted on any one of the intermediate representation nodes containing information about a synchronization dependency relationship in a physical pipeline in which a hardware engine of the AI accelerator chip performs a hardware operation represented by the intermediate representation node; and generating a hardware instruction stream based on the intermediate representation of the AI model, and interleaving and rearranging instruction blocks in the hardware instruction stream based on the synchronization token inserted on the one or more intermediate representation nodes.

Description

Technical Field

[0001] This disclosure relates to the field of computers, and more specifically to a method, computer program product, and computer-readable storage medium for use in an artificial intelligence (AI) accelerator chip. Background Technology

[0002] Existing high-performance operators primarily run on general-purpose graphics processing units (GPUs). Because general-purpose GPUs emphasize versatility, the scheduling strategies and execution behaviors of the hardware scheduler are not transparent to the software layer, making it difficult for developers to have fine-grained control over the timing of data transfer and computation execution. Summary of the Invention

[0003] This disclosure provides a method for use in an AI accelerator chip, comprising: statically partitioning multiple physical buffers in the shared memory of the AI ​​accelerator chip according to a preset pipeline number, and locking the base address of each of the multiple physical buffers; parsing the computation and memory allocation logic of operator blocks of an AI model to be run on the AI ​​accelerator chip, and inserting synchronization tokens on one or more intermediate representation nodes of the AI ​​model based on the parsing results, wherein the synchronization token inserted on any intermediate representation node of the AI ​​model contains information about the synchronization dependency of the hardware operation represented by the intermediate representation node in the physical pipeline executed by the hardware engine in the AI ​​accelerator chip; and generating a hardware instruction stream based on the intermediate representation of the AI ​​model, and interleaving and rearranging the instruction blocks in the hardware instruction stream based on the synchronization tokens inserted on one or more intermediate representation nodes of the AI ​​model, wherein the hardware engine in the AI ​​accelerator chip works in cooperation with the multiple physical buffers to realize asynchronous execution of the physical pipeline executed by the hardware engine in the AI ​​accelerator chip based on the hardware instruction stream with its instruction blocks interleaved and rearranged. Attached Figure Description

[0004] When read in conjunction with the accompanying drawings, various aspects of this disclosure are best understood through the following detailed description. It should be noted that, in accordance with standard practice in the art, the features are not drawn to scale. In fact, for clarity of discussion, the dimensions of the features may be arbitrarily increased or decreased.

[0005] Figure 1 A schematic flowchart of a method used in an AI accelerator chip according to an embodiment of the present disclosure is shown.

[0006] Figure 2 A schematic block diagram of a computing device that can employ a method used in an AI accelerator chip according to an embodiment of the present disclosure is shown. Detailed Implementation

[0007] Features and exemplary embodiments of various aspects of this disclosure will now be described in detail. Numerous specific details are set forth in the following detailed description in order to provide a thorough understanding of this disclosure. However, it will be apparent to those skilled in the art that this disclosure may be practiced without requiring some of these specific details. The following description of embodiments is merely intended to provide a better understanding of this disclosure by illustrating examples. This disclosure is by no means limited to any specific configuration presented below, but covers any modifications, substitutions, and improvements to elements, components, and algorithms without departing from the spirit of this disclosure. Well-known structures and techniques are not shown in the accompanying drawings and the following description to avoid unnecessarily obscuring this disclosure.

[0008] Furthermore, the various operations will be described as multiple discrete operations in a manner most conducive to understanding the illustrative embodiments; however, the order of description should not be construed as implying that these operations must depend on the order. In particular, these operations do not need to be performed in the order presented.

[0009] The phrases “in an embodiment,” “in one embodiment,” and “in some embodiments” are used repeatedly throughout this document. These phrases do not typically refer to the same embodiment; however, they may refer to the same embodiment. Unless the context otherwise requires, the terms “comprising,” “having,” and “including” are synonyms. The phrases “A or B” and “A / B” mean “(A), (B), or (A and B).”

[0010] Currently, the field of high-performance computing widely adopts the Triton and other tile programming models to optimize operator performance. However, at the hardware implementation level, the industry lacks a dedicated acceleration solution that can deeply adapt to this programming model and has asynchronous scheduling capabilities between hardware engines.

[0011] Existing general-purpose AI accelerator chips suffer from one or more limitations when executing AI model operators: 1) Rigid synchronization mechanisms, often relying on expensive synchronization barriers for instruction alignment. This leads to waiting gaps between access, computation, and the memory engine, preventing fine-grained pipeline parallelism and causing pipeline bubbles that severely reduce computational efficiency. 2) Redundant control logic, typically using software polling or frequent processor instruction transfer to manage data flow. This not only consumes valuable instruction bandwidth but also introduces significant software interrupts and synchronization overhead. In high-throughput scenarios, this stop-and-go, discontinuous execution mode greatly limits the effective utilization of on-chip memory and computing units.

[0012] In view of the above-mentioned limitations of existing general-purpose AI accelerator chips in executing AI model operators, this disclosure provides a method for use in AI accelerator chips that can realize asynchronous scheduling between hardware engines in AI accelerator chips based on programming model semantics, thereby improving the hardware utilization efficiency in AI accelerator chips.

[0013] Figure 1 A schematic flowchart of a method used in an AI accelerator chip according to an embodiment of this disclosure is shown. Figure 1 As shown, the method 100 used in an AI accelerator chip includes: S102, statically dividing multiple physical buffers in the shared memory of the AI ​​accelerator chip according to a preset pipeline number, and locking the base address of each of the multiple physical buffers; S104, parsing the computation and memory allocation logic of the operator blocks of the AI ​​model to be run on the AI ​​accelerator chip, and inserting synchronization tokens on one or more intermediate representation nodes of the AI ​​model based on the parsing result, wherein the synchronization token inserted on any intermediate representation node of the AI ​​model contains information about the synchronization dependency of the hardware operation represented by the intermediate representation node in the physical pipeline executed by the hardware engine in the AI ​​accelerator chip; and S106, generating a hardware instruction stream based on the intermediate representation of the AI ​​model, and interleaving and rearranging the instruction blocks in the hardware instruction stream based on the synchronization tokens inserted on one or more intermediate representation nodes of the AI ​​model, wherein the hardware engine in the AI ​​accelerator chip works in cooperation with the multiple physical buffers to realize the asynchronous execution of the physical pipeline executed by the hardware engine in the AI ​​accelerator chip based on the hardware instruction stream with its instruction blocks interleaved and rearranged.

[0014] Here, the physical pipeline executed by the hardware engines in an AI accelerator chip refers to the pipelined sequence of operations formed when multiple hardware engines in the AI ​​accelerator chip collaboratively execute tasks. These hardware engines include, but are not limited to, Direct Memory Access Load (DMA0), Vector Processing Unit (VPU), Direct Memory Access Store (DMA1), and Sparse Processing Unit (SPU). In other words, the physical pipeline executed by the hardware engines in an AI accelerator chip is an engine-level pipeline, with a scheduling granularity of the entire operator block and a parallel unit of the hardware engines on the AI ​​accelerator chip. The synchronization token inserted into an intermediate representation node of the AI ​​model indicates which subsequent hardware engine events (e.g., waking up a waiting VPU computing engine) need to be triggered after the hardware operation represented by that intermediate representation node (e.g., loading the DMA loading engine) is completed. It also indicates which preceding hardware engine events (e.g., waiting for the DMA loading engine to complete) need to be waited for before the hardware operation represented by that intermediate representation node (e.g., a VPU computing operation) can begin.

[0015] In some embodiments, the method 100 used in an AI accelerator chip further includes: generating hardware notification instructions based on synchronization tokens inserted on one or more intermediate representation nodes of the AI ​​model and arranging the hardware notification instructions into a hardware instruction stream whose instruction blocks are interleaved and rearranged.

[0016] In some embodiments, interleaving and rearranging instruction blocks in the hardware instruction stream includes inserting the prefetch instruction for the (i+1)th operator block of the AI ​​model into the execution cycle of the computation instruction for the ith operator block of the AI ​​model, where i is an integer greater than or equal to 1.

[0017] In some embodiments, the shared memory of an AI accelerator chip refers to its Global Buffer (GLB). Multiple physical buffers partitioned from the GLB are used to eliminate waiting caused by data dependencies, achieving a high degree of overlap between computation and data transfer. Specifically, statically partitioning multiple physical buffers from the GLB and locking the base address of each physical buffer serves the following purposes:

[0018] 1) Multiple physical buffers serve as intermediate storage, decoupling the different hardware engines within the AI ​​accelerator chip in time. For example, while the computing engine is processing data in one physical buffer, the data loading engine can simultaneously fill data into another physical buffer without waiting for the other.

[0019] 2) By statically dividing multiple physical buffers and locking their base addresses, it can be ensured that each physical buffer has a fixed physical location, avoiding address conflicts caused by dynamically allocating physical buffers, so that the hardware engine in the AI ​​accelerator chip can obtain non-conflicting operation objects.

[0020] 3) Multiple physical buffers constitute the "platforms" of the physical pipeline executed by the hardware engine in the AI ​​accelerator chip. Data flows between different physical buffers like a train and is processed by different hardware engines, thereby realizing the parallel execution of the physical pipeline executed by the hardware engine in the AI ​​accelerator chip.

[0021] In some embodiments, the preset pipeline has at least two stages, and the number of physical buffers allocated from the shared memory of the AI ​​accelerator chip is no more than (i.e., equal to or less than) the preset pipeline stages. Physical buffers are used to temporarily store and transfer data between different pipeline stages to support parallel and overlapping execution between multiple hardware engines. For example, in a dual-buffered implementation, two physical buffers can be allocated from shared memory. Multiple hardware engines alternately perform data loading, data computation, and data storage operations around these two physical buffers. Specifically, at one moment, the first physical buffer can be used to handle the current batch of data loading operations, while the second physical buffer can be used to handle the previous batch of data computation operations; in subsequent moments, the uses of the first and second physical buffers are switched. For scenarios requiring output results, the data that has completed computation can be further stored by the data storage engine. By alternately reusing the two physical buffers, different processing stages can overlap in time, thereby improving the execution efficiency of the AI ​​accelerator chip.

[0022] In some embodiments, the hardware engines within the AI ​​accelerator chip communicate with each other via a hardware-software co-operational signal handshake protocol. For example, when the DMA0 loading engine completes data loading, it directly notifies the VPU computing engine that depends on it via a hardware signal. Upon receiving the hardware signal from the DMA0 loading engine, the VPU computing engine immediately begins computation without software polling or waiting for global barriers. Similarly, when the VPU computing engine completes computation, it wakes up the waiting DMA1 storage engine via a hardware signal to write the result back. Here, the entire processing flow is driven by the flow of data between the various hardware engines (i.e., implementing data-driven asynchronous execution of a physical pipeline executed by the hardware engines within the AI ​​accelerator chip).

[0023] In some embodiments, a dedicated asynchronous scheduling optimization process (Pass) is introduced in the compiler backend, including:

[0024] A. Based on the preset pipeline stages, statically divide the shared memory of the AI ​​accelerator chip into multiple physical buffers and lock the base address of each physical buffer.

[0025] B. By analyzing the computation and memory allocation logic of the operator blocks in the AI ​​model, the following analysis results can be obtained: 1) Which operations will occupy which hardware engines in the AI ​​accelerator chip (e.g., DMA loading engine, VPU computing engine, etc.); 2) Whether each hardware engine in the AI ​​accelerator chip is a data producer or a data consumer at a specific stage; 3) When the physical buffer in the AI ​​accelerator chip is occupied and when it is released.

[0026] C. Based on the information contained in the parsing results, analyze the intermediate representation of the AI ​​model. When traversing to a specific intermediate representation node (e.g., matrix multiplication or data loading node), combine the information parsed in process A (e.g., this intermediate representation node corresponds to the hardware operation of the computing engine, and it depends on the data in physical buffer 0) to insert a synchronization token on this intermediate representation node. This synchronization token actually tells the hardware engine "when you execute the hardware operation corresponding to this intermediate representation node, you need to follow specific synchronization rules (e.g., wait for a hardware engine event, or trigger a hardware engine event after completion)".

[0027] D. Generate a hardware instruction stream based on the intermediate representation of the AI ​​model, and interleave and rearrange the instruction blocks in the hardware instruction stream based on the synchronization tokens inserted on one or more intermediate representation nodes of the AI ​​model. Then, based on the synchronization tokens inserted on one or more intermediate representation nodes of the AI ​​model, generate hardware notification instructions and arrange the hardware notification instructions into the hardware instruction stream whose instruction blocks have been interleaved and rearranged.

[0028] In the above asynchronous scheduling optimization process, by statically dividing the physical buffer and interleaving and rearranging the instruction blocks in the hardware instruction stream, the startup time of the data producer and consumer is precisely staggered. Subsequently, the high-speed signal handshake between hardware engines can ensure that the staggered pipeline can be seamlessly connected, thereby eliminating the waiting time between hardware engines to the greatest extent and greatly reducing pipeline "cavitation".

[0029] Figure 2 A schematic block diagram of a computing device that can employ a method used in an AI accelerator chip according to embodiments of the present disclosure is shown. Figure 2As shown, computing device 200 may include bus 202 or other communication mechanisms for transmitting information, and one or more hardware processors 204 coupled to bus 202 for processing information. The one or more hardware processors 204 may include, for example, one or more general-purpose microprocessors and / or one or more AI accelerator chips.

[0030] like Figure 2 As shown, in some embodiments, computing device 200 may further include main memory 206 coupled to bus 202. Main memory 206 is used to store information and instructions executed by one or more processors 204, such as random access memory (RAM), cache, and / or other dynamic storage devices. Main memory 206 may also be used to store temporary variables or other intermediate information during the execution of instructions executed by one or more processors 204. When these instructions are stored in storage media accessible to one or more processors 204, they can cause computing device 200 to become a dedicated machine customized to perform the operations specified in the instructions. Storage device 208 may include non-volatile and / or volatile storage media. Non-volatile storage media may include, for example, optical discs or magnetic disks. Volatile storage media may include dynamic memory. Common forms of storage media may include, for example, floppy disks, hard disks, solid-state drives, magnetic tape, or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, DRAM, PROM, and EPROM, FLASH-EPROM, NVRAM, any other memory chip or cartridge, or their networking versions.

[0031] like Figure 2 As shown, in some embodiments, computing device 200 may further include one or more communication interfaces or network interfaces 210 coupled to bus 202. Network interface 210 may provide bidirectional data communication coupling to one or more network links connected to one or more networks. As another example, network interface 210 may be a local area network (LAN) card to provide data communication connectivity to a LAN-compatible (or WAN component communicating with a WAN) network. Wireless links may also be implemented.

[0032] The execution of certain operations can be distributed across processors rather than residing within a single machine, but rather deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0033] Each of the processes, methods, and algorithms described in the preceding sections may be embodied in code modules executed by one or more computer systems or computer processors including computer hardware, and may be fully or partially automated by these code modules. The processes and algorithms may be implemented, partially or fully, in dedicated circuit systems.

[0034] When the functions disclosed herein are implemented as software functional units and sold or used as stand-alone products, they may be stored in a processor-executable, non-volatile, computer-readable storage medium. Specific technical solutions (all or part) disclosed herein, or aspects contributing to the prior art, may be embodied in the form of a software product. The software product may be stored in a storage medium and includes several instructions that cause a computing device (which may be a personal computer, server, network device, etc.) to perform all or some steps of the methods of the embodiments of this application. The storage medium may include a flash drive, portable hard disk drive, ROM, RAM, magnetic disk, optical disk, other media operable to store program code, or any combination thereof.

[0035] Specific embodiments further provide a system including a processor and a non-transitory computer-readable storage medium storing instructions executable by the processor to cause the system to perform operations corresponding to steps in any method of the embodiments disclosed above. Specific embodiments further provide a non-transitory computer-readable storage medium storing instructions executable by one or more processors to cause the one or more processors to perform operations corresponding to steps in any method of the embodiments disclosed above.

[0036] The embodiments disclosed herein can be implemented via a cloud platform, server, or server cluster (collectively referred to below as the "Service System") that interacts with a client. The client can be a terminal device or a client registered by a user at the platform, wherein the terminal device can be a mobile terminal, a personal computer (PC), or any device that can have the platform application installed.

[0037] The various features and processes described above can be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Additionally, certain method or process blocks may be omitted in some embodiments. The methods and processes described herein are not limited to any particular order, and their associated blocks or states may be executed in other suitable orders. For example, described blocks or states may be executed in an order other than that specifically disclosed, or multiple blocks or states may be combined into a single block or state. Example blocks or states may be executed sequentially, in parallel, or in some other manner. Blocks or states may be added to or removed from the disclosed example embodiments. The exemplary systems and components described herein may be configured differently than described. For example, components may be added to, removed from, or rearranged compared to the disclosed example embodiments.

[0038] The various operations of the exemplary methods described herein can be performed at least in part by an algorithm. The algorithm may be included in program code or instructions stored in memory (e.g., the aforementioned non-transitory computer-readable storage medium). This algorithm may include a machine learning algorithm. In some embodiments, the machine learning algorithm may not explicitly refer to the computer as performing the function but may learn from training data to generate a predictive model of the function.

[0039] The various operations of the exemplary methods described herein can be performed, at least in part, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, these processors can constitute an engine of processor implementation that operates to perform one or more of the operations or functions described herein.

[0040] Similarly, the methods described herein may be implemented at least in part by a processor, wherein one or more specific processors are instances of hardware. For example, at least some operations of the methods may be performed by one or more processors or an engine implemented by a processor. Furthermore, one or more processors may also be operable to support the execution of relevant operations in a “cloud computing” environment or as the execution of relevant operations in a “Software as a Service” (SaaS) context. For example, at least some operations may be performed by a group of computers (as an example of a machine containing processors), wherein these operations are accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application programming interfaces (APIs)).

[0041] The execution of certain operations can be distributed across processors rather than residing within a single machine, and can be deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0042] Throughout this specification, multiple instances may be implemented as components, operations, or structures described as a single instance. Although individual operations of one or more methods are illustrated and described as separate operations, one or more of these individual operations may be performed simultaneously, and not necessarily in the order illustrated. Structures and functions presented as separate components in the example configuration may be implemented as composite structures or components. Similarly, structures and functions presented as single components may be implemented as single components. These and other variations, modifications, additions, and improvements fall within the scope of this document.

[0043] As used herein, "or" is inclusive rather than exclusive unless explicitly indicated by the context. Therefore, in this document, "A, B, or C" means "A, B, A and B, A and C, B and C, or A, B, and C" unless explicitly indicated by the context. Furthermore, "and" is combined and separate unless explicitly indicated by the context. Therefore, in this document, "A and B" means "A and B, combined or separate" unless explicitly indicated by the context. Additionally, multiple instances of resources, operations, or structures described herein may be provided as a single instance. Furthermore, the boundaries between various resources, operations, engines, and data storage devices are somewhat arbitrary and specific operations are illustrated within the context of a particular illustrative configuration. Other functional assignments are foreseeable and fall within the scope of various embodiments of this disclosure. Generally, structures and functions presented as individual resources in example configurations may be implemented as combined structures or resources. Similarly, structures and functions presented as single resources may be implemented as single resources. These and other changes, modifications, additions, and improvements fall within the scope of the embodiments of this disclosure as expressed in the appended claims. Therefore, this specification and drawings should be considered illustrative rather than restrictive.

[0044] The terms “comprising” or “including” are used to indicate the presence of a subsequently claimed feature, but do not preclude the addition of other features. Unless otherwise specifically stated or otherwise understood in the context in which they are used, conditional language such as “may,” “can,” “may,” and “can” is generally intended to convey that certain embodiments include certain features, components, and / or steps that are not included in other embodiments. Therefore, this conditional language is generally not intended to imply that one or more embodiments require features, components, and / or steps in any way, or that one or more embodiments must include logic for determining whether such features, components, and / or steps are included in or performed in any particular embodiment, with or without user input or prompts.

[0045] Although the general outline of the subject matter has been described with reference to specific exemplary embodiments, various modifications and changes may be made to these embodiments without departing from the broad scope of embodiments of this disclosure. Where more than one embodiment is disclosed, these embodiments of the subject matter may be referred to individually or collectively herein as the term "invention," this is for convenience only and is not intended to automatically limit the scope of this application to any single disclosure or concept.

[0046] The embodiments illustrated herein are described in detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be used and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of this disclosure. Therefore, “implementation” is not intended to be limiting, and the scope of the various embodiments is defined only by the appended claims and their full scope.

Claims

1. A method for use in an AI accelerator chip, characterized in that, include: According to the preset pipeline stage, multiple physical buffers are statically partitioned in the shared memory of the AI ​​accelerator chip, and the base address of each of the multiple physical buffers is locked. The computation and memory allocation logic of the operator blocks of the AI ​​model to be run on the AI ​​accelerator chip is parsed, and based on the parsing results, synchronization tokens are inserted into one or more intermediate representation nodes of the AI ​​model. The synchronization token inserted into any intermediate representation node of the AI ​​model contains information about the synchronization dependencies of the hardware operations represented by that intermediate representation node within the physical pipeline executed by the hardware engine in the AI ​​accelerator chip; and A hardware instruction stream is generated based on the intermediate representation of the AI ​​model, and the instruction blocks in the hardware instruction stream are interleaved and rearranged based on synchronization tokens inserted on one or more intermediate representation nodes of the AI ​​model. The hardware engine in the AI ​​accelerator chip works in conjunction with the multiple physical buffers to achieve asynchronous execution of the physical pipeline executed by the hardware engine in the AI ​​accelerator chip based on the hardware instruction stream with its instruction blocks interleaved and rearranged. The interleaving and rearrangement of the instruction blocks in the hardware instruction stream includes inserting the prefetch instruction for the (i+1)th operator block of the AI ​​model into the execution cycle of the computation instruction for the ith operator block of the AI ​​model, where i is an integer greater than or equal to 1.

2. The method according to claim 1, characterized in that, The hardware engines in the AI ​​accelerator chip communicate with each other via a hardware-software collaborative signal handshake protocol.

3. The method according to claim 1, characterized in that, Also includes: Based on synchronization tokens inserted on one or more intermediate representation nodes of the AI ​​model, hardware notification instructions are generated and arranged into a hardware instruction stream whose instruction blocks are interleaved and rearranged.

4. The method according to claim 1, characterized in that, The preset number of pipeline stages is at least two.

5. The method according to claim 1, characterized in that, The number of the multiple physical buffers is equal to or less than the preset pipeline stage number.

6. A computer program product comprising computer-executable instructions, characterized in that, When executed by a processor, the computer-executable instructions cause the processor to perform the method of any one of claims 1 to 5.

7. A non-transitory computer-readable storage medium storing computer-executable instructions thereon, characterized in that, When executed by a processor, the computer-executable instructions cause the processor to perform the method of any one of claims 1 to 5.

Images