Computing scheduling method and device for AI chip, electronic equipment and storage medium

Abstract

A computing scheduling method and apparatus for an AI chip, an electronic device, and a storage medium are provided. The method includes: a task distribution unit sending information of instructions that need to be loaded in advance to a prefetch unit; in response to the prefetch unit receiving the information of the instructions that need to be loaded in advance, the prefetch unit sending, based on the information of the instructions that need to be loaded in advance, an instruction prefetch request corresponding to the instructions that need to be loaded in advance to a memory; in response to the memory receiving the instruction prefetch request, the memory sending the instructions that need to be loaded in advance to the prefetch unit; and in response to the prefetch unit receiving the instructions that need to be loaded in advance, the prefetch unit sending the instructions that need to be loaded in advance to a plurality of computing cores to cache the instructions that need to be loaded in advance to a cache corresponding to each computing core.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a computation scheduling method, apparatus, electronic device, and computer-readable storage medium for AI chips. Background Technology

[0002] AI chips utilize multiple hardware computing cores. Each individual hardware computing core reads instructions from external storage and executes those instructions to complete the computational function of a single operator.

[0003] Each complete AI computation typically involves multiple operators, requiring numerous hardware computing cores to perform calculations in a specific order. This involves both parallel execution and adherence to certain sequential rules. Therefore, a dedicated hardware scheduling unit is needed to monitor the status of each hardware computing core in real time and schedule the next operator computation task to an idle, available computing core in the shortest possible time.

[0004] The methods described in this section are not necessarily methods that had been previously conceived or adopted. Unless otherwise specified, no method described in this section should be assumed to be prior art simply because it is included in this section. Similarly, unless otherwise specified, the issues mentioned in this section should not be considered to be accepted in any prior art. Summary of the Invention

[0005] This disclosure provides computation scheduling methods, apparatus, electronic devices, and storage media for AI chips.

[0006] According to one aspect of this disclosure, a computation scheduling method for an AI chip is provided, wherein the AI ​​chip includes a task distribution unit, multiple computing cores, a prefetch unit, and a memory, and wherein each of the multiple computing cores includes a corresponding cache, and the memory stores instructions. The method includes: the task distribution unit sending information about instructions that need to be preloaded to the prefetch unit; in response to the prefetch unit receiving information about instructions that need to be preloaded, the prefetch unit sending an instruction prefetch request corresponding to the instructions that need to be preloaded to the memory based on the information about the instructions that need to be preloaded; in response to the memory receiving the instruction prefetch request, the memory sending the instructions that need to be preloaded to the prefetch unit; and in response to the prefetch unit receiving the instructions that need to be preloaded, the prefetch unit sending the instructions that need to be preloaded to the multiple computing cores to cache the instructions that need to be preloaded into the cache corresponding to each computing core.

[0007] According to another aspect of this disclosure, a computation scheduling apparatus for an AI chip is provided, wherein the AI ​​chip includes a task distribution unit, a plurality of computation cores, a prefetch unit, and a memory, and wherein each of the plurality of computation cores includes a corresponding cache, and the memory stores instructions. The apparatus is configured to perform a method comprising the following steps: the task distribution unit sends information about instructions that need to be preloaded to the prefetch unit; in response to the prefetch unit receiving information about instructions that need to be preloaded, the prefetch unit sends an instruction prefetch request corresponding to the instructions that need to be preloaded to the memory based on the information about the instructions that need to be preloaded; in response to the memory receiving the instruction prefetch request, the memory sends the instructions that need to be preloaded to the prefetch unit; and in response to the prefetch unit receiving the instructions that need to be preloaded, the prefetch unit sends the instructions that need to be preloaded to the plurality of computation cores to cache the instructions that need to be preloaded into the cache corresponding to each computation core.

[0008] According to another aspect of this disclosure, an electronic circuit is provided, comprising: a circuit configured to perform the steps of the above-described method.

[0009] According to another aspect of this disclosure, an electronic device is provided. The electronic device includes: a processor; and a memory storing a program including instructions that, when executed by the processor, cause the processor to perform the methods described above.

[0010] According to another aspect of this disclosure, a non-transitory computer-readable storage medium storing a program is provided. The program includes instructions that, when executed by a processor of an electronic device, cause the electronic device to perform the methods described above.

[0011] According to another aspect of this disclosure, a computer program product is provided. This computer program product includes a computer program that, when executed by a processor, implements the above-described method.

[0012] According to another aspect of this disclosure, an AI chip is provided. The AI ​​chip includes the aforementioned electronic circuitry.

[0013] According to embodiments of this disclosure, instructions can be dynamically preloaded during multi-operator computation by setting up a separate prefetch unit, which can reduce the instruction loading time of each hardware computing core startup and improve the equivalent computing efficiency of the computing core. At the same time, by broadcasting instructions to multiple computing cores in parallel, external memory bandwidth is saved and access latency is reduced.

[0014] These and other aspects of this disclosure will be apparent from the embodiments described below, and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0015] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0016] Figure 1 A flowchart illustrating an exemplary process of a computation scheduling method for an AI chip according to an embodiment of the present disclosure is shown; Figure 2 An exemplary block diagram of a computation scheduling apparatus for an AI chip according to embodiments of the present disclosure is shown. Figure 3 The diagram shows a specific flowchart of a computation scheduling method for AI chips in the prior art; Figure 4 The diagram shows a detailed flowchart of an exemplary process for a computation scheduling method for an AI chip according to an embodiment of the present disclosure; Figure 5 Another specific flowchart of an exemplary process for a computation scheduling method for an AI chip according to an embodiment of the present disclosure is shown; Figure 6 The diagram illustrates the execution timeline of an example in the prior art without flexible prefetching and the execution timeline after flexible prefetching using the computation scheduling method for AI chips according to embodiments of this disclosure; and Figure 7 This is a block diagram illustrating an example of an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0017] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0018] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0019] In related technologies, the conventional approach is to use a dedicated task distribution unit (also known as a task scheduling unit) to monitor the busy / idle status of each computing core. When an idle computing core is detected, the instruction address of the computing task is sent to that core, which then actively loads the code and executes it. However, as AI chips become larger, the latency of external memory access also increases. If a new instruction read information is received only after the hardware computing core is completely idle, there is a long loading time when loading instructions from external memory, during which the hardware computing core remains in an idle state. For some operators with relatively short computation times, the loading time accounts for a large proportion of the overall computation time, effectively causing a decrease in computational efficiency. To address the aforementioned problems in related technologies, this disclosure provides a computation scheduling method for AI chips. By setting up a separate prefetch unit, instructions are dynamically preloaded during multi-operator computation, reducing the instruction loading time of each hardware computing core startup and improving the equivalent computational efficiency of the computing core. Furthermore, by broadcasting instructions to multiple computing cores for parallel loading, external memory bandwidth is saved, and access latency is reduced. The embodiments of this disclosure are described in detail below with reference to the accompanying drawings.

[0020] Figure 1 A flowchart illustrating an exemplary process of a computation scheduling method 100 for an AI chip according to an embodiment of the present disclosure is shown, wherein the AI ​​chip may include a task distribution unit, a plurality of computation cores, a prefetch unit and a memory, and wherein each of the plurality of computation cores may include a corresponding cache, and the memory stores instructions.

[0021] In step S102, the task distribution unit can send information about instructions that need to be preloaded to the prefetching unit.

[0022] In step S104, in response to the prefetch unit receiving information about instructions that need to be preloaded, the prefetch unit can send an instruction prefetch request corresponding to the instructions that need to be preloaded to the memory based on the information about the instructions that need to be preloaded.

[0023] In step S106, in response to the memory receiving an instruction prefetch request, the memory may send the instruction that needs to be preloaded to the prefetch unit.

[0024] In step S108, in response to the prefetch unit receiving an instruction that needs to be preloaded, the prefetch unit can send the instruction that needs to be preloaded to multiple computing cores to cache the instruction that needs to be preloaded into the cache corresponding to each computing core.

[0025] The computation scheduling method for AI chips provided by the embodiments of this disclosure can dynamically preload instructions during multi-operator computation by setting up a separate prefetch unit, which can reduce the instruction loading time of each hardware computing core startup and improve the equivalent computing efficiency of the computing core. At the same time, by broadcasting instructions to multiple computing cores in parallel, external memory bandwidth is saved and access latency is reduced.

[0026] The steps of method 100 are described in detail below.

[0027] In step S102, the task distribution unit can send information about instructions that need to be preloaded to the prefetching unit.

[0028] In some embodiments, the task distribution unit has multiple operator execution queues, and each of the multiple operator execution queues includes one or more operators.

[0029] The task distribution unit is the control center of the AI ​​chip. Its function is to receive commands from the CPU and then efficiently and massively distribute computing tasks to tens of thousands of computing cores inside the GPU for execution.

[0030] refer to Figure 5 The example task distribution unit 501 internally includes multiple operator execution queues, such as operator queue 510, operator queue 520, and operator queue 530. Operator queue 510 includes operators 511, 512, 513, and 514; operator queue 520 includes operators 521, 522, 523, and 524; and operator queue 530 includes operators 531, 532, 533, and 534. It should be noted that... Figure 5 The number of operator queues and operators shown are for illustrative purposes only and are not intended to be limiting.

[0031] In some embodiments, the information about instructions that need to be preloaded includes the instruction address in memory and the prefetch length of the instructions that need to be preloaded.

[0032] Therefore, instructions that need to be preloaded can be read from memory using the instruction address and prefetch length information of the instructions that need to be preloaded.

[0033] In some examples, the instructions that need to be preloaded can be either complete instructions for implementing a certain operator's computation, or fragments of complete instructions for implementing a certain operator's computation.

[0034] In step S104, in response to the prefetch unit receiving information about instructions that need to be preloaded, the prefetch unit can send an instruction prefetch request corresponding to the instructions that need to be preloaded to the memory based on the information about the instructions that need to be preloaded.

[0035] In some examples, after receiving the instruction address and prefetch length in memory of the instruction that needs to be preloaded, the prefetch unit sends an instruction prefetch request to memory corresponding to the instruction that needs to be preloaded.

[0036] In step S106, in response to the memory receiving an instruction prefetch request, the memory may send the instruction that needs to be preloaded to the prefetch unit.

[0037] In some examples, the memory receives an instruction prefetch request from the prefetch unit. The memory can then read the instruction that needs to be prefetched from the instruction address in the memory and the prefetch length, and send it to the prefetch unit.

[0038] In step S108, in response to the prefetch unit receiving an instruction that needs to be preloaded, the prefetch unit can send the instruction that needs to be preloaded to multiple computing cores to cache the instruction that needs to be preloaded into the cache corresponding to each computing core.

[0039] In some examples, after the prefetch unit receives an instruction that needs to be preloaded, it broadcasts the preloaded instruction to multiple compute cores to cache the instruction that needs to be preloaded in the cache corresponding to each compute core, so that multiple compute cores can load the instruction that needs to be preloaded in parallel in subsequent compute tasks.

[0040] In some embodiments, method 100 may further include: a task distribution unit determining whether the next operator to be executed in an operator execution queue needs to be preloaded with instructions based on operator information at multiple queue heads of one of the multiple operator execution queues.

[0041] In some embodiments, method 100 may further include: a task distribution unit sending operator information to multiple computing cores; multiple computing cores determining whether the operator corresponding to the operator information needs to be preloaded based on the operator information; and in response to determining that the operator corresponding to the operator information needs to be preloaded, each of the multiple computing cores reads the preloaded instructions that have been cached in its corresponding cache from its corresponding cache.

[0042] In some embodiments, the operator information includes a prefetch flag, operator priority, the address of the instruction in memory that needs to be preloaded, and the prefetch length, wherein the prefetch flag is used to determine whether the corresponding operator needs to preload the instruction.

[0043] In some examples, the task dispatch unit can determine the next operator to be executed in multiple operator queues based on the operator priority in the operator information, and then determine whether the operator needs to preload instructions based on the prefetch flag in the operator information of the next operator to be executed. Similarly, multiple computation cores can also determine whether an operator needs to preload instructions based on the prefetch flag in the operator information sent by the task dispatch unit.

[0044] In some embodiments, operator information can be flexibly configured via software. Operator information may include information about instructions that need to be pre-loaded, and this information can be flexibly configured via software to achieve flexible prefetching strategies.

[0045] In some embodiments, the memory can be external storage. Taking a GPU as an example, external storage, also known as video memory, is a high-performance memory tightly integrated with the GPU computing core, specifically designed to store the massive amounts of data that the GPU needs to process quickly. The physical location of external storage is typically on the graphics card board, directly connected to the GPU chip via a high-speed bus.

[0046] In some embodiments, the AI ​​chip can be any one of a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a neural network processor (NPU).

[0047] AI computing has two core computational characteristics: 1) massively parallel processing: the computation of neural networks can be decomposed into a massive number of matrix multiplications and additions, which are independent of each other and can be performed simultaneously; 2) high-throughput data transfer: it requires frequent transfer of large amounts of weight and input data between memory and processing units. General-purpose CPUs are inefficient in handling these AI tasks because their core design is low latency and strong versatility. They are good at handling complex logic such as branch prediction and out-of-order execution, but their parallel computing units (number of computing cores) are limited, and memory bandwidth often becomes a bottleneck.

[0048] Therefore, large-scale parallel computing can be achieved using AI chips such as graphics processing units (GPUs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or neural network processors (NPUs).

[0049] Therefore, by setting up a separate prefetch unit and flexibly configuring the operator information of each operator in advance via software, instructions or instruction fragments can be dynamically preloaded during multi-operator computation. This reduces the instruction loading time of the hardware computing core each time it starts, improving the equivalent computational efficiency of the computing core. Simultaneously, broadcasting the instructions that need to be preloaded allows multiple computing cores to load them in parallel, saving external memory bandwidth, reducing access latency, and avoiding task scheduling limitations caused by unequal loading between different computing cores. This achieves absolute transparency in task scheduling and does not introduce additional software burden.

[0050] According to embodiments of this disclosure, a computing scheduling device 200 for an AI chip is also provided. Figure 2 The diagram illustrates an exemplary block diagram of a computation scheduling apparatus 200 for an AI chip according to an embodiment of the present disclosure. The computation scheduling apparatus 200 for an AI chip includes a task distribution unit 210, a plurality of computation cores 230, a prefetch unit 220, and a memory 240. Each of the plurality of computation cores 230 includes a corresponding cache 231 (the diagram simplifies the plurality of computation cores to a single block diagram for illustrative purposes only; multiple caches may exist). The memory 240 stores instructions. The apparatus 200 performs a method including the following steps: the task distribution unit sends information about instructions that need to be preloaded to the prefetch unit; in response to the prefetch unit receiving information about instructions that need to be preloaded, the prefetch unit sends an instruction prefetch request corresponding to the instructions that need to be preloaded to the memory based on the information about the instructions that need to be preloaded; in response to the memory receiving the instruction prefetch request, the memory sends the instructions that need to be preloaded to the prefetch unit; and in response to the prefetch unit receiving the instructions that need to be preloaded, the prefetch unit sends the instructions that need to be preloaded to the plurality of computation cores to cache the instructions that need to be preloaded into the cache corresponding to each computation core.

[0051] Here, the operation of each unit of the device is similar to the operation of steps S102 to S108 described above, and will not be repeated here.

[0052] According to another aspect of this disclosure, an electronic circuit is also provided, including a circuit configured to perform the steps of the above-described method.

[0053] According to another aspect of this disclosure, an electronic device is also provided, comprising: a processor; and a memory storing a program, the program including instructions that, when executed by the processor, cause the processor to perform the methods described above.

[0054] According to another aspect of this disclosure, a non-transitory computer-readable storage medium storing a program is also provided, the program including instructions that, when executed by a processor of an electronic device, cause the electronic device to perform the method described above.

[0055] According to another aspect of this disclosure, a computer program product is also provided, including a computer program that, when executed by a processor, implements the above-described method.

[0056] According to another aspect of this disclosure, an AI chip is also provided. This AI chip includes the aforementioned electronic circuitry.

[0057] Figure 3The diagram shows a specific flowchart of a computation scheduling method for AI chips in the prior art.

[0058] refer to Figure 3 , Figure 3 In the AI ​​chip, the typical data path topology and message path are as follows: The task distribution unit 310 notifies the idle computing core among the multiple computing cores 320 of the new operator instruction address. The computing core 320 sends the instruction read address to the external storage 330, and then the external storage 330 returns the read instruction to the computing core 320. Figure 4 The diagram shows a detailed flowchart of an exemplary process 400 of a computation scheduling method 100 for an AI chip according to an embodiment of the present disclosure.

[0059] The specific flow of an exemplary process 400 of the computation scheduling method 100 for an AI chip according to embodiments of the present disclosure may include the following steps: Task distribution unit 410 notifies prefetching unit 440 to prefetch instructions; The prefetch unit 440 sends the read address of the instruction to the external memory 430; External memory 430 returns the read instruction to prefetch unit 440; The prefetch unit 440 broadcasts the prefetched instructions to all computing cores 420 to be cached in the corresponding cache 421 of each computing core 420. This operation does not require waiting for the computing core 420 to enter an idle state. It mainly loads the instructions that need to be loaded in advance and masks the start loading time of the instruction in the computing time of the previous operator. The task distribution unit 410 notifies multiple computing cores 420 to perform computing tasks.

[0060] Therefore, all computing cores 420 can minimize the startup time of the next computing task by directly reading the instructions for the next computing task from their corresponding cache 421. Furthermore, from... Figure 4 It can also be seen that by adding a separate prefetch unit, the number of times multiple computing cores access external memory can be reduced. All computing cores can be preloaded through a single access and broadcast by the prefetch unit, thereby saving external memory bandwidth, reducing access latency, and avoiding task scheduling limitations caused by unequal loading of different computing cores. It is absolutely transparent to task scheduling and does not bring any additional software burden.

[0061] Figure 5 Another specific flowchart of an exemplary process 500 of a computation scheduling method 100 for an AI chip according to an embodiment of the present disclosure is shown.

[0062] Continue to refer to Figure 5The task distribution unit 501 performs a multi-queue judgment 540 based on the operator information of multiple queue heads to determine whether the next operator to be executed in the operator execution queue needs to preload instructions. In some examples, the operator information may include a prefetch flag, operator priority, the instruction address of the instruction that needs to be preloaded in the memory, and the prefetch length, wherein the prefetch flag is used to determine whether the corresponding operator needs to preload instructions.

[0063] Specifically, the task distribution unit 501 can determine the next operator to be executed in the multiple operator execution queues by the operator priority in the operator information, and then determine whether the operator needs to preload instructions according to the prefetch flag in the operator information.

[0064] Figure 5 The rest of the parts and Figure 4 This is a corresponding relationship, which will not be elaborated upon here. It should be understood that... Figure 4 The specific process and Figure 5 The specific examples shown are for illustrative purposes only and not for limiting purposes. For example, external storage is only used as an example of memory, and other memory besides external storage can also be used as memory for storing instructions.

[0065] Figure 6 The diagram illustrates an execution timeline without flexible prefetching, as exemplified in the prior art, and an execution timeline with flexible prefetching using a computation scheduling method for AI chips according to embodiments of this disclosure.

[0066] It can be seen that without flexible prefetching, the computing core needs to load the next order operator after completing the previous order operator calculation, which takes a long time. However, with flexible prefetching using the computing scheduling method for AI chips according to the embodiments of this disclosure, the previous order operator calculation time and the next order operator loading time can overlap, so that the computing core does not need to wait after completing the previous order operator calculation, the time to start the next order operator calculation is reduced, and the equivalent computing efficiency of the computing core is improved.

[0067] See Figure 7 The electronic device 700 will now be described as an example of a hardware device (electronic device) that can be applied to various aspects of this disclosure. The electronic device 700 can be any machine configured to perform processing and / or computation, and can be, but is not limited to, a workstation, server, desktop computer, laptop computer, tablet computer, personal digital assistant, robot, smartphone, in-vehicle computer, or any combination thereof. The computation scheduling method 100 described above for AI chips can be implemented wholly or at least partially by the electronic device 700 or similar devices or systems.

[0068] Electronic device 700 may include elements that are connected to or communicate with bus 702 (possibly via one or more interfaces). For example, electronic device 700 may include bus 702, one or more processors 704, one or more input devices 706, and one or more output devices 708. The one or more processors 704 may be any type of processor and may include, but are not limited to, one or more general-purpose processors and / or one or more dedicated processors (e.g., special-purpose chips). Input devices 706 may be any type of device capable of inputting information to electronic device 700 and may include, but are not limited to, a mouse, keyboard, touchscreen, microphone, and / or remote control. Output devices 708 may be any type of device capable of presenting information and may include, but are not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Electronic device 700 may also include a non-transitory storage device 710. The non-transitory storage device can be any storage device that is non-transitory and capable of storing data, including but not limited to disk drives, optical storage devices, solid-state storage, floppy disks, flexible disks, hard disks, magnetic tapes or any other magnetic media, optical discs or any other optical media, ROM (read-only memory), RAM (random access memory), cache memory and / or any other memory chip or cartridge, and / or any other medium from which a computer can read data, instructions, and / or code. The non-transitory storage device 710 can be detached from an interface. The non-transitory storage device 710 may have data / programs (including instructions) / code for implementing the methods and steps described above. Electronic device 700 may also include a communication device 712. The communication device 712 can be any type of device or system that enables communication with external devices and / or with a network, and may include, but is not limited to, modems, network interface cards, infrared communication devices, wireless communication devices and / or chipsets, such as Bluetooth™ devices, 802.11 devices, Wi-Fi devices, Wi-Max devices, cellular communication devices, and / or the like.

[0069] The electronic device 700 may also include working memory 714, which may be any type of working memory that can store programs (including instructions) and / or data useful for the operation of the processor 704, and may include, but is not limited to, random access memory and / or read-only memory devices.

[0070] Software elements (programs) may reside in working memory 714, including but not limited to operating system 716, one or more application programs 718, drivers, and / or other data and code. Instructions for performing the above methods and steps may be included in one or more application programs 718, and the above-described computation scheduling method 100 for an AI chip can be implemented by the processor 704 reading and executing the instructions of one or more application programs 718. More specifically, in the above-described computation scheduling method 700 for an AI chip, steps S102-S108 can be implemented, for example, by the processor 704 executing the application program 718 having instructions for steps S102-S108. Furthermore, other steps in the above-described computation scheduling method 700 for an AI chip can be implemented, for example, by the processor 704 executing the application program 718 having instructions for executing the corresponding steps. The executable code or source code of the instructions of the software elements (programs) may be stored in a non-transitory computer-readable storage medium (e.g., the above-described storage device 710) and may be stored in working memory 714 during execution (possibly compiled and / or installed). The executable code or source code of the instructions for software elements (programs) can also be downloaded from a remote location.

[0071] It should also be understood that various modifications can be made depending on specific requirements. For example, custom hardware can also be used, and / or specific elements can be implemented using hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. For example, some or all of the disclosed methods and apparatus can be implemented by programming hardware (e.g., programmable logic circuits including field-programmable gate arrays (FPGAs) and / or programmable logic arrays (PLAs)) using logic and algorithms according to this disclosure in assembly language or hardware programming languages ​​(such as Verilog, VHDL, C++).

[0072] It should also be understood that the aforementioned methods can be implemented using a server-client model. For example, the client can receive user input data and send it to the server. Alternatively, the client can receive user input data, perform a portion of the processing described in the aforementioned methods, and send the resulting data to the server. The server can receive data from the client, execute the aforementioned methods or a portion thereof, and return the execution result to the client. The client can receive the execution result from the server and, for example, present it to the user via an output device.

[0073] It should also be understood that the components of electronic device 700 can be distributed across a network. For example, some processing can be performed by one processor, while other processing can be performed by another processor located far away from that processor. Other components of computing system 700 can also be distributed similarly. Thus, electronic device 700 can be interpreted as a distributed computing system that performs processing in multiple locations.

[0074] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. A computation scheduling method for AI chips, wherein, The AI ​​chip includes a task distribution unit, multiple computing cores, a prefetch unit, and a memory. The task distribution unit internally includes multiple operator execution queues, and each operator execution queue includes one or more operators. Each computing core includes a corresponding cache. The memory stores instructions. The method includes: The task distribution unit determines whether the next operator to be executed in the operator execution queue needs to be preloaded with instructions based on the operator information at the head of multiple queues in the multiple operator execution queues; The task distribution unit sends information about instructions that need to be preloaded to the prefetching unit; In response to the prefetch unit receiving information about the instruction that needs to be preloaded, the prefetch unit sends an instruction prefetch request corresponding to the instruction that needs to be preloaded to the memory based on the information about the instruction that needs to be preloaded. In response to the memory receiving the instruction prefetch request, the memory sends an instruction that needs to be preloaded to the prefetch unit; In response to the prefetch unit receiving the instruction that needs to be preloaded, the prefetch unit sends the instruction that needs to be preloaded to the plurality of computing cores to cache the instruction that needs to be preloaded into the cache corresponding to each computing core; The task distribution unit sends the operator information to the multiple computing cores; The multiple computational cores determine whether the operator corresponding to the operator information needs to have its instructions pre-loaded based on the operator information; and In response to determining that the operator information corresponds to an instruction that needs to be preloaded, each of the plurality of computational cores reads the instruction that needs to be preloaded from its corresponding cache.

2. The method as described in claim 1, wherein, The operator information includes a prefetch flag, operator priority, the instruction address in the memory of the instruction that needs to be preloaded, and the prefetch length. The prefetch flag is used to determine whether the corresponding operator needs to preload the instruction.

3. The method as described in claim 1, wherein, The information about the instructions that need to be preloaded includes the instruction address and prefetch length of the instructions in the memory.

4. The method of claim 1, wherein, The operator information can be flexibly configured through software.

5. The method of claim 1, wherein, The AI ​​chip can be any one of a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a neural network processor (NPU).

6. The method of claim 1, wherein, The memory is external storage.

7. A computation scheduling device for AI chips, wherein, The AI ​​chip includes a task distribution unit, multiple computing cores, a prefetch unit, and a memory. The task distribution unit internally includes multiple operator execution queues, and each operator execution queue includes one or more operators. Each computing core includes a corresponding cache. The memory stores instructions. The device is used to execute a method including the following steps: The task distribution unit determines whether the next operator to be executed in the operator execution queue needs to be preloaded with instructions based on the operator information at the head of multiple queues in the multiple operator execution queues; The task distribution unit sends information about instructions that need to be preloaded to the prefetching unit; In response to the prefetch unit receiving information about the instruction that needs to be preloaded, the prefetch unit sends an instruction prefetch request corresponding to the instruction that needs to be preloaded to the memory based on the information about the instruction that needs to be preloaded. In response to the memory receiving the instruction prefetch request, the memory sends an instruction that needs to be preloaded to the prefetch unit; In response to the prefetch unit receiving the instruction that needs to be preloaded, the prefetch unit sends the instruction that needs to be preloaded to the plurality of computing cores to cache the instruction that needs to be preloaded into the cache corresponding to each computing core; The task distribution unit sends the operator information to the multiple computing cores; The multiple computational cores determine whether the operator corresponding to the operator information needs to have its instructions pre-loaded based on the operator information; and In response to determining that the operator information corresponds to an instruction that needs to be preloaded, each of the plurality of computational cores reads the instruction that needs to be preloaded from its corresponding cache.

8. An electronic circuit, comprising: A circuit configured to perform the steps of the method according to any one of claims 1 to 6.

9. An electronic device, comprising: processor; as well as A memory storing a program, the program comprising instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 6.

10. A non-transitory computer-readable storage medium storing a program, the program comprising instructions that, when executed by a processor of an electronic device, cause the electronic device to perform the method according to any one of claims 1 to 6.

11. A computer program product comprising a computer program, wherein, The computer program, when executed by a processor, implements the method according to any one of claims 1 to 6.

12. An AI chip, the AI ​​chip comprising the electronic circuitry according to claim 8.

Images