Method for executing attention mechanism calculation task and related product

Abstract

The invention discloses a method for executing an attention mechanism calculation task by using an artificial intelligence processor and a related product, and relates to the technical field of artificial intelligence. The artificial intelligence processor may be included in a combined processing device, which may also include an interface device and other processing devices. And the artificial intelligence processor interacts with other processing devices to jointly complete calculation operation specified by a user. The combined processing device can further comprise a storage device, and the storage device is connected with the artificial intelligence processing device and the other processing devices and used for storing data of the artificial intelligence processing device and the other processing devices. According to the embodiment of the invention, a self-adaptive scheduling scheme is provided for an attention mechanism calculation task, the parallel calculation capability of an artificial intelligence processor can be fully exerted, and the situation of idle calculation power is avoided.

Description

Technical Field

[0001] The present disclosure generally relates to the field of artificial intelligence technology. More specifically, the present disclosure relates to a method for using an artificial intelligence processor to perform attention mechanism computing tasks, an artificial intelligence processor, a chip, and a board. Background Art

[0002] Currently, deep learning algorithms based on the attention mechanism and its variants have shown strong performance in tasks such as natural language processing, image processing, and assisted programming, and their applications have been widely popularized at an unprecedented rate. However, training and using this type of algorithm for reasoning involves a huge amount of computation, and how to improve the computational speed of the attention mechanism and its variants has become an important issue that limits the further development of this type of algorithm.

[0003] Artificial intelligence processor chips are usually designed to efficiently execute deep learning and machine learning algorithms. In order to process large amounts of data and complex algorithms, artificial intelligence processor chips usually need to have a high degree of parallel computing capabilities and be able to process multiple data points at the same time. Single Instruction Multiple Data (SIMD) is a parallel computing architecture that uses multiple computing units to execute the same instruction at the same time. A single instruction acts on multiple data, thereby achieving parallel processing of multiple data and increasing the running speed of the program. In high-throughput scenarios, in order to maximize the performance of SIMD, the data to be calculated needs to be distributed as evenly as possible to each computing unit. This can avoid the situation where some computing units are idle waiting for data while other computing units are still processing, thereby improving the overall processing efficiency.

[0004] However, for some complex computing tasks, such as the attention mechanism computing task mentioned above, task splitting is cumbersome and complicated. Although attempts are made to evenly distribute computing tasks across multiple computing units according to the size of the computing tasks, in practical applications, computing bottlenecks often occur in multiple computing units.

[0005] In view of this, there is an urgent need to provide a solution for using an artificial intelligence processor to perform attention mechanism calculation tasks, so as to realize the calculation of the attention mechanism at a faster running speed and avoid wasting computing power. Summary of the invention

[0006] In order to at least solve one or more technical problems described in the above background technology section, the present disclosure proposes the following technical solutions and multiple embodiments thereof.

[0007] In a first aspect, the present disclosure proposes a method for using an artificial intelligence processor to perform an attention mechanism computing task, comprising: dividing the attention mechanism computing task into multiple unit computing tasks; until the attention mechanism computing task is completed, each operation unit cyclically executes: requesting a unit computing task; and completing the requested unit computing task; wherein the artificial intelligence processor includes at least two of the operation units.

[0008] In a second aspect, the present disclosure discloses an artificial intelligence processor, comprising: a control unit, configured to divide an attention mechanism computing task into multiple unit computing tasks; an operation unit, configured to cyclically perform the following operations until the attention mechanism computing task is completed: requesting a unit computing task; and completing the requested unit computing task; wherein the artificial intelligence processor includes at least two of the operation units.

[0009] In a third aspect, the present disclosure discloses a chip, whose configuration includes an artificial intelligence processor according to the second aspect.

[0010] In a fourth aspect, the present disclosure discloses a board comprising the chip according to the third aspect.

[0011] By utilizing the method, artificial intelligence processor, chip and board proposed in this disclosure to use an artificial intelligence processor to perform attention mechanism computing tasks, the computing unit that completes the current round of tasks can automatically request to obtain the tasks to be processed in the next round, which is adaptive and can solve the problem of unbalanced task distribution, avoid some computing units being idle, and give full play to the parallel computing power of the computing units. Moreover, this adaptive scheduling scheme of the embodiment of the disclosure also avoids designing a complex load balancing algorithm, which causes development and maintenance difficulties. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] By reading the detailed description below with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0013] Figure 1 A schematic diagram showing the structure of a board according to an embodiment of the present disclosure;

[0014] Figure 2 A schematic diagram showing the structure of a combined processing device in a chip according to an embodiment of the present disclosure;

[0015] Figure 3 An exemplary internal structure diagram of an artificial intelligence processor in some embodiments of the present disclosure is shown;

[0016] Figure 4 An exemplary structural diagram of a processor core in some embodiments of the present disclosure is shown;

[0017] Figure 5 An exemplary schematic diagram showing a processor core writing data to a processor core of another cluster in some embodiments of the present disclosure;

[0018] Figure 6 An exemplary schematic diagram of a two-layer three-stage pipeline in some embodiments of the present disclosure is shown;

[0019] Figure 7 An exemplary block diagram of a computing module of some embodiments of the present disclosure is shown;

[0020] Figure 8 An exemplary flow chart of a method for using an artificial intelligence processor to perform attention mechanism computing tasks in some embodiments of the present disclosure is shown;

[0021] Fig. 9 An exemplary schematic diagram of the self-attention mechanism calculation process in some embodiments of the present disclosure is shown;

[0022] Fig.10 An exemplary schematic diagram showing the calculation process of the multi-head self-attention mechanism in some embodiments of the present disclosure;

[0023] Fig.11 A schematic diagram showing the impact of mask operations on the execution time of the operation unit. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0025] It should be understood that the terms "include" and "comprising" used in the specification and claims of the present disclosure indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0026] It should also be understood that the terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. As used in this disclosure and claims, the singular forms of "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" used in this disclosure and claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations.

[0027] As used in this specification and claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0028] The specific implementation of the present disclosure is described in detail below with reference to the accompanying drawings.

[0029] The working principle of the attention mechanism is mainly to let the model pay attention to the information at different positions in the input sequence and capture long-distance dependencies. As a variant of the attention mechanism, the self-attention mechanism can handle internal dependencies in the sequence without relying on external information. At present, deep learning algorithms based on the attention mechanism and its variants have shown strong performance in tasks such as natural language processing, image processing, and auxiliary programming, and their applications have been widely popularized at an unprecedented rate. However, training and using this type of algorithm for reasoning involves a huge amount of computing. How to improve the computing speed of the attention mechanism and its variants has become an important issue that restricts the further development of this type of algorithm.

[0030] Parallel technology is an important method for solving large-scale computing tasks. SIMD, as a parallel computing architecture, uses multiple computing units to execute the same instruction at the same time, thereby realizing parallel processing of multiple data and improving the running speed of the program. The SIMD architecture realizes instruction-level parallelism, that is, a single instruction acts on multiple data. This requires that when using the SIMD architecture to execute complex algorithms, process large amounts of data, and other high-throughput scenarios, the computing data should be distributed as evenly as possible to each computing unit.

[0031] For the self-attention mechanism and its variants, the prior art usually handles this type of computing task under the SIMD architecture by evenly splitting it into various computing units in the SIMD architecture according to the scale of the computing task. For example, for the computing task of the multi-head attention mechanism, its scale is mainly determined by the batch size, the number of heads, and the sequence length. The batch size can refer to the number of samples processed in a batch cycle of model training, or the number of samples processed simultaneously in a batch of the model in the inference phase. The number of heads refers to the number of self-attention heads (AttentionHead) obtained by splitting the Q (query), K (key), and V (value) matrices on the word embedding dimension or the model dimension. The sequence length refers to the number of elements in the input sequence. For example, when processing text data, the sequence length can be the number of words in a sentence. When splitting the multi-head attention mechanism computing task, the common method in the prior art is to evenly distribute the total number SUM = Batch Size*Head Num*Sequence Length to each computing unit.

[0032] The inventors of the present disclosure have found that there are the following problems in the prior art: Due to the presence of a mask operation in the multi-head attention calculation, this mask operation will cause part of the data in the dot product result of Q and K to become invalid values, which has no effect on subsequent calculations. Therefore, the existing splitting method that evenly distributes the computing tasks to each computing unit according to the scale of the attention mechanism computing task will actually lead to uneven computing loads between computing units, resulting in some idle computing power. Artificial intelligence processors are usually designed to handle highly parallel computing tasks. In order to simplify the design of the processor, it is generally not necessary to implement complex task scheduling logic. Therefore, in order to solve the above-mentioned problem of uneven load, according to the conventional design ideas, the impact of the mask operation will be analyzed in detail before the task is assigned to the computing unit, and the task will be split in consideration of this impact to distribute it as evenly as possible. However, this solution requires the design of a very complex load balancing algorithm, which is very difficult to develop. Moreover, if the mask operation changes with different attention mechanism computing tasks, it is necessary to analyze and design task splitting schemes for different attention mechanism computing tasks respectively, which increases the workload. Furthermore, if operations such as mask operations that affect computational load vary in a single attention mechanism computation task, it is almost impossible to accurately analyze their impact and perform effective task splitting.

[0033] In view of this, the inventors break away from the conventional design ideas and do not try to build a complex task balancing algorithm to implement the task splitting scheme, which is then distributed to the computing units, and the computing units passively accept and execute the assigned computing tasks. Instead, the computing units can actively request the next computing task based on their own completion status, that is, to achieve adaptive scheduling. Therefore, the present disclosure proposes an artificial intelligence processor and a method for using an artificial intelligence processor to perform attention mechanism computing tasks, so as to make full use of the parallel computing power of multiple computing units, thereby completing the attention mechanism computing tasks at a faster running speed.

[0034] It should be noted that in the disclosed embodiments, the attention mechanism computing task can exist in applications in various fields, and the data involved can be of various types. For example, some major application fields of the attention mechanism include: machine translation, sentiment analysis, text summarization, text classification, named entity recognition, etc. in natural language processing (NLP), and the corresponding data types can include text data, vocabulary embedding, sentence structure, etc.; facial recognition, image classification, target detection, image segmentation, etc. in computer vision (CV), and the corresponding data types can include pixel data, image features, etc.; intelligent assistants, automatic subtitle generation, speech to text conversion, etc. in speech recognition (ASR), and the corresponding data types can include audio signals, spectrograms, etc.; generative AI applications, etc. Generative AI refers to artificial intelligence technology that uses complex algorithms, models and rules to learn from large-scale data sets to create new original content, such as but not limited to the ability to create multiple types of content such as text, pictures, sounds, videos and codes. Accordingly, the input data processed by the attention mechanism computing task can be image data, audio data, video data, voice data, text data, document data, etc. The output of the attention mechanism computation task can include a likelihood score that an image belongs to a particular object category, a likelihood score that a document is about a particular topic, a likelihood score that a text segment in a target language is a correct translation of a text segment in a source language, or a likelihood score that a text segment is a correct transcription of a spoken utterance, etc.

[0035] Figure 1 FIG. 1 is a schematic diagram showing the structure of a board 10 according to an embodiment of the present disclosure. Figure 1As shown, the board 10 includes a chip 101, which is a system-on-chip, or a system on chip (SoC), which integrates one or more combined processing devices. The combined processing device is an artificial intelligence computing unit that supports various deep learning and machine learning algorithms to meet the intelligent processing needs in complex scenarios in the fields of computer vision, speech, natural language processing, data mining, etc. In particular, deep learning technology is widely used in the field of cloud intelligence. A notable feature of cloud-based intelligent applications is the large amount of input data, which places high demands on the storage capacity and computing power of the platform. The board 10 of this embodiment is suitable for cloud-based intelligent applications and has huge off-chip storage, on-chip storage, and a large amount of computing power.

[0036] The chip 101 is connected to an external device 103 via an external interface device 102. The external device 103 is, for example, a server, a computer, a camera, a display, a mouse, a keyboard, a network card, or a wifi interface. The data to be processed can be transmitted from the external device 103 to the chip 101 via the external interface device 102. The calculation result of the chip 101 can be transmitted back to the external device 103 via the external interface device 102. According to different application scenarios, the external interface device 102 can have different interface forms, such as a PCIe interface.

[0037] The board 10 also includes a storage device 104 for storing data, which includes one or more storage units 105. The storage device 104 is connected to the control device 106 and the chip 101 through a bus and transmits data. The control device 106 in the board 10 is configured to control the state of the chip 101. To this end, in an application scenario, the control device 106 may include a microcontroller (MCU).

[0038] Figure 2 1 is a block diagram showing the combined processing device in the chip 101 of this embodiment. Figure 2 As shown in , the combined processing device 20 includes an artificial intelligence processor 201, an interface device 202, a processing device 203 and a storage device 204.

[0039] The artificial intelligence processor 201 is configured to execute user-specified operations, and is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0040] The interface device 202 is used to transmit data and control instructions between the artificial intelligence processor 201 and the processing device 203. For example, the artificial intelligence processor 201 can obtain input data from the processing device 203 via the interface device 202 and write it into the storage device on the artificial intelligence processor 201 chip. Further, the artificial intelligence processor 201 can obtain control instructions from the processing device 203 via the interface device 202 and write them into the control cache on the artificial intelligence processor 201 chip. Alternatively or optionally, the interface device 202 can also read data in the storage device of the artificial intelligence processor 201 and transmit it to the processing device 203.

[0041] The processing device 203, as a general processing device, performs basic controls including but not limited to data handling, starting and / or stopping the artificial intelligence processor 201. Depending on the implementation, the processing device 203 can be a central processing unit (CPU), a graphics processing unit (GPU), or one or more types of processors in other general and / or special processors, including but not limited to digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, only with respect to the artificial intelligence processor 201 disclosed in the present invention, it can be regarded as having a single-core structure or a homogeneous multi-core structure. However, when the artificial intelligence processor 201 and the processing device 203 are integrated and considered together, the two are regarded as forming a heterogeneous multi-core structure.

[0042] The storage device 204 may be DRAM, which is an off-chip memory used to store data to be processed. It is DDR and is typically 16G or larger in size. It is used to store data of the artificial intelligence processor 201 and / or the processing device 203 .

[0043] Figure 3The figure shows a schematic diagram of the internal structure of the artificial intelligence processor 201. The artificial intelligence processor 201 is used to process input data such as computer vision, speech, natural language, and data mining. The artificial intelligence processor 201 in the figure adopts a multi-core hierarchical structure design. The artificial intelligence processor 201 is a system on a chip, which includes multiple clusters, and each cluster includes multiple processor cores. In other words, the artificial intelligence processor 201 is composed of a hierarchy of system on a chip-cluster-processor core.

[0044] At the system-on-chip level, Figure 3 As shown, the artificial intelligence processor 201 includes an external storage controller 301, a peripheral communication module 302, an on-chip interconnect module 303, a synchronization module 304 and multiple clusters 305.

[0045] There may be multiple external storage controllers 301, two of which are shown in the figure as an example, and are used to respond to access requests issued by the processor core and access external storage devices, such as Figure 2 DRAM 204 in the chip, thereby reading data from outside the chip or writing data to outside the chip. The peripheral communication module 302 is used to receive the control signal from the processing device 203 through the interface device 202, and start the artificial intelligence processor 201 to perform tasks. The on-chip interconnect module 303 connects the external storage controller 301, the peripheral communication module 302 and multiple clusters 305 to transmit data and control signals between each module. The synchronization module 304 is a global synchronization barrier controller (GBC) used to coordinate the work progress of each cluster and ensure information synchronization. Multiple clusters 305 are the computing cores of the artificial intelligence processor 201. Four are shown as examples in the figure. With the development of hardware, the artificial intelligence processor 201 disclosed in the present invention can also include 8, 16, 64, or even more clusters 305. Clusters 305 are used to efficiently execute deep learning algorithms.

[0046] At the cluster level, Figure 3 As shown, each cluster 305 includes multiple processor cores (IPU cores) 306 and a memory core (MEM core) 307 .

[0047] The figure shows four processor cores 306 as an example, but the present disclosure does not limit the number of processor cores 306 . Figure 4 The internal structure diagram of the processor core is shown. Each processor core 306 includes three modules: a control module 41 , a calculation module 42 and a storage module 43 .

[0048] The control module 41 is used to coordinate and control the operation of the computing module 42 and the storage module 43 to complete the deep learning task, and includes an instruction fetch unit (IFU) 411 and an instruction decode unit (IDU) 412. The instruction fetch unit 411 is used to obtain instructions from the processing device 203, and the instruction decode unit 412 decodes the obtained instructions and sends the decoding results to the computing module 42 and the storage module 43 as control information.

[0049] The operation module 42 includes a vector operation unit 421 and a matrix operation unit 422. The vector operation unit 421 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformation; the matrix operation unit 422 is responsible for the core calculation of the deep learning algorithm, such as matrix multiplication and convolution.

[0050] The storage module 43 is used to store or move related data, including a neuron RAM (NRAM) 431, a weight RAM (WRAM) 432, an input / output direct memory access module (IODMA) 433, and a move direct memory access module (MVDMA) 434. NRAM 431 is used to store feature maps and intermediate results calculated by the processor core 306; WRAM 432 is used to store the weights of the deep learning network; IODMA 433 controls the memory access between NRAM 431 / WRAM 432 and DRAM 204 through the broadcast bus 309; MVDMA 434 is used to control the memory access between NRAM 431 / WRAM 432 and SRAM 308.

[0051] Back to Figure 3 The storage core 307 is mainly used for storage and communication, that is, to store shared data or intermediate results between the processor cores 306, and to perform communication between the cluster 305 and the DRAM 204, communication between the clusters 305, and communication between the processor cores 306. In other embodiments, the storage core 307 has the capability of scalar operation and is used to perform scalar operation.

[0052] The storage core 307 includes a shared memory unit (SRAM) 308, a broadcast bus 309, a cluster direct memory access module (cluster direct memory access, CDMA) 310 and a global direct memory access module (global direct memory access, GDMA) 311. The SRAM 308 plays the role of a high-performance data transfer station. The data reused between different processor cores 306 in the same cluster 305 does not need to be obtained from the DRAM 204 by each processor core 306, but is transferred between the processor cores 306 via the SRAM 308. The storage core 307 only needs to quickly distribute the reused data from the SRAM 308 to multiple processor cores 306, so as to improve the efficiency of inter-core communication and greatly reduce on-chip and off-chip input / output access.

[0053] The broadcast bus 309, CDMA 310 and GDMA 311 are respectively used to perform communication between processor cores 306, communication between clusters 305 and data transmission between clusters 305 and DRAM 204. They will be described below.

[0054] The broadcast bus 309 is used to complete high-speed communication between the processor cores 306 in the cluster 305. The broadcast bus 309 of this embodiment supports inter-core communication modes including unicast, multicast and broadcast. Unicast refers to point-to-point (i.e., single processor core to single processor core) data transmission, multicast is a communication mode of transmitting a copy of data from SRAM 308 to specific processor cores 306, and broadcast is a communication mode of transmitting a copy of data from SRAM 308 to all processor cores 306, which is a special case of multicast.

[0055] CDMA 310 is used to control memory access to SRAM 308 between different clusters 305 within the same artificial intelligence processor 201 . Figure 5 A schematic diagram is shown when a processor core wants to write data to a processor core of another cluster to illustrate the working principle of CDMA 310. In this application scenario, the same artificial intelligence processor includes multiple clusters. For the convenience of explanation, only cluster 0 and cluster 1 are shown in the figure. Cluster 0 and cluster 1 each include multiple processor cores. For the convenience of explanation, cluster 0 in the figure only shows processor core 0, and cluster 1 only shows processor core 1. Processor core 0 wants to write data to processor core 1.

[0056] First, processor core 0 sends a unicast write request to write data into the local SRAM 0. CDMA 0 acts as the master and CDMA 1 acts as the slave. The master pushes the write request to the slave, that is, the master sends the write address AW and write data W to transfer the data to SRAM 1 of cluster 1. Then the slave sends a write response B in response. Finally, processor core 1 of cluster 1 sends a unicast read request to read the data from SRAM 1.

[0057] Back to Figure 3 , GDMA 311 cooperates with external memory controller 301 to control the access of SRAM 308 of cluster 305 to DRAM 204, or reads data from DRAM 204 to SRAM 308. As can be seen from the above, the communication between DRAM 204 and NRAM 431 or WRAM 432 can be realized through two channels. The first channel is to directly connect DRAM 204 and NRAM 431 or WRAM 432 through IODMA 433; the second channel is to first transfer data between DRAM 204 and SRAM 308 through GDMA 311, and then transfer data between SRAM 308 and NRAM 431 or WRAM 432 through MVDMA 434. Although it seems that the second channel requires more components to participate and the data flow is longer, in fact, in some embodiments, the bandwidth of the second channel is much greater than that of the first channel, so the communication between DRAM 204 and NRAM 431 or WRAM 432 may be more efficient through the second channel. The embodiments of the present disclosure can select a data transmission channel according to the hardware conditions thereof.

[0058] In other embodiments, the functions of GDMA 311 and IODMA 433 can be integrated into the same component. For the convenience of description, this disclosure regards GDMA 311 and IODMA 433 as different components. For those skilled in the art, as long as the functions they implement and the technical effects they achieve are similar to those of this disclosure, they fall within the protection scope of this disclosure. Furthermore, the functions of GDMA 311, IODMA 433, CDMA 310, and MVDMA 434 can also be implemented by the same component. Similarly, as long as the functions they implement and the technical effects they achieve are similar to those of this disclosure, they fall within the protection scope of this disclosure.

[0059] An important reason why the computing device 201 has strong computing power is the three-level computing hierarchy of system on chip - cluster - processor core, combined with a three-layer memory design such as DRAM-SRAM-NRAM / WRAM, which enables data to be cached and calculated at the appropriate level to form a sufficient pipeline.

[0060] When the computing device 201 performs computing, the process can be divided into the following three phases: loading phase (load): loading data; computing phase (compute): moving data, computing, and moving intermediate results; and storing phase (store): storing results.

[0061] In more detail, in some embodiments, a two-layer three-stage pipeline may be used, such as Figure 6 As shown, the first-level loading stage 601, calculation stage 602, and storage stage 603 occur in the cluster level. The first-level loading stage 601 is that the GDMA 330 loads data from the DRAM 204 to the SRAM 308, the first-level calculation stage 602 is that the cluster 305 calculates the loaded on-chip unit map and generates calculation results, and the first-level storage stage 603 is that the GDMA 330 stores the calculation results from the SRAM 308 back to the DRAM 204.

[0062] Since the cluster 305 includes multiple processor cores 306, the first layer calculation stage 602 actually divides the on-chip unit map into corresponding sub-maps through the storage core 307, and broadcasts them to at least one processor core 306 for calculation. Therefore, the second layer three-level waterline occurs in the processor core 306. In more detail, the second layer loading stage 604 is to load the sub-map from the SRAM 308 to the NRAM 431 by the MVDMA 434, the second layer calculation stage 605 is to move the sub-map and the sub-weight to the operation module 42 for calculation, and then move the intermediate result back to the NRAM 431, and the second layer storage stage 606 is to store the intermediate result from the NRAM 431 back to the SRAM 308 by the MVDMA 434.

[0063] The first-level pipeline means that the first-level loading stage 601, the first-level calculation stage 602 and the first-level storage stage 603 can be performed in parallel. Now, taking the case where the same cluster 305 wants to process the j-th on-chip unit diagram, the j+1-th on-chip unit diagram and the j+2-th on-chip unit diagram as an example, first, the j-th on-chip unit diagram is loaded into the SRAM 308 in the first-level loading stage 601. Then, the j-th on-chip unit diagram is calculated in the first-level calculation stage 602, and the first calculation result is transferred back to the SRAM 308. While the j-th on-chip unit diagram is being calculated, the j+1-th on-chip unit diagram is loaded into the SRAM 308 in the first-level loading stage 607. When the first calculation result is stored back to DRAM 204 in the first level storage-back stage 603, the j+1th on-chip unit map is calculated in the first level calculation stage 608, and the second calculation result is transferred back to SRAM 308, and the j+2th on-chip unit map is loaded into SRAM 308 in the first level loading stage 610. The first level pipeline is rolled in this way.

[0064] In order to cooperate with the operation of the aforementioned pipeline, the SRAM 308 of this embodiment includes two storage spaces: a ping storage unit and a pong storage unit. The data flow is divided into three types according to the ping-pong attribute of the SRAM 308: input / output ping-pong (IOparity), input ping-pong (input parity), and no ping-pong (no parity). Input / output ping-pong can support parallel loading, calculation, and storage. To realize input / output ping-pong, the ping storage unit and the pong storage unit need to be completely equal, and are used for loading and storage respectively. Input ping-pong only supports storage and calculation in parallel, which will increase the time of transportation on the SRAM 308. Compared with input / output ping-pong, the ping storage unit and the pong storage unit do not need to be completely equal, but an additional cache (cache) of the same size as the storage space needs to be allocated. No ping-pong means that loading / storage and calculation are serial, and space does not need to be additionally allocated.

[0065] In order to implement the aforementioned first-level pipeline, the SRAM 308 of some embodiments has ping memory cells and pong memory cells of the same size to achieve the input / output ping-pong effect. Figure 6 For illustration, the storage area involved in the first-level loading stage 601, the first-level calculation stage 602 and the first-level storage-back stage 603 of the j-th on-chip unit diagram is limited to the ping storage unit, while the storage area involved in the first-level loading stage 607, the first-level calculation stage 608 and the first-level storage-back stage 609 of the j+1-th on-chip unit diagram is limited to the pong storage unit, and the storage area involved in the first-level loading stage 610, the first-level calculation stage 611 and the first-level storage-back stage 612 of the j+2-th on-chip unit diagram is again limited to the ping storage unit, and in this way, the ping storage unit and the pong storage unit are used alternately for storage.

[0066] The second-level pipeline means that the second-level loading stage 604, the second-level calculation stage 605, and the second-level storage stage 606 can be processed in parallel. Take the case where the same processor core 306 wants to process the i-th sub-graph, the i+1-th sub-graph, and the i+2-th sub-graph in the j-th on-chip unit graph as an example. First, the i-th sub-graph is broadcast to the NRAM 431 in the second-level loading stage 604. Then, the i-th sub-graph is calculated in the second-level calculation stage 605 to generate the i-th intermediate result, and the i-th intermediate result is transferred back to the NRAM 431. At the same time, the i+1-th sub-graph is broadcast to the NRAM 431 in the second-level loading stage 613. The i-th intermediate result is stored back to the SRAM 308 in the second-level storage-back stage 606, and at the same time, the i+1-th sub-graph is calculated in the second-level calculation stage 614 to generate the i+1-th intermediate result, and the i+1-th intermediate result is moved back to the NRAM 431, and the i+2-th sub-graph is loaded into the NRAM 431 in the second-level loading stage 615.

[0067] Considering that the tasks of each cluster 305 are different and the completion times are naturally inconsistent, the synchronization module 304 of some embodiments may use synchronization barrier instructions to synchronize the task completion times to avoid timing errors.

[0068] In order to improve the processing efficiency and throughput of the calculation, the operation module 42 in the processor core 306 can be designed to support multi-stage pipeline operations.

[0069] Figure 7 FIG. 1 shows a block diagram of a computing module according to some embodiments of the present disclosure. The computing module here can be a computing module in a single processor core, or a computing module considered jointly by multiple processor cores. Figure 7 As shown, the operation module 42 may include multiple operation units, each of which may be configured as a multi-stage operation pipeline, and multiple operation units constitute multiple multi-stage operation pipelines.

[0070] Figure 7The first group of pipeline operation circuits 701, the second group of pipeline operation circuits 702 and the third group of pipeline operation circuits 703 are shown, wherein each group of pipeline operation circuits can constitute a multi-stage operation pipeline in the context of the present disclosure. Taking the first group of pipeline operation circuits 701 constituting the first multi-stage operation pipeline as an example, it can perform pipeline operations including 1-1 level pipeline operation, 1-2 level pipeline operation, 1-3 level pipeline operation... 1-N level pipeline operation, a total of N levels of pipeline operations. Similarly, the second and third groups of pipeline operation circuits also have a structure that supports N levels of pipeline operations. Through such an exemplary architecture, those skilled in the art can understand that the multiple groups of pipeline operation circuits disclosed in the present disclosure can constitute multiple multi-stage operation pipelines, and these multiple multi-stage operation pipelines can execute their respective multiple operation instructions in parallel. These operation instructions can be obtained by parsing the calculation instructions.

[0071] In order to perform the above-mentioned pipeline operations at each level, an operation circuit including one or more operators can be arranged at each level to execute the corresponding operation instructions to implement the operation at that level. In some embodiments, in response to receiving multiple operation instructions, one or more groups of pipeline operation circuits disclosed herein can be configured to perform multi-data operations, such as executing single instruction multiple data SIMD instructions. For example, the aforementioned multiple operation instructions can be obtained by the control module in the processor core 306 parsing the received calculation instructions, and the operation code of the calculation instruction can represent multiple operations performed by the multi-stage operation pipeline.

[0072] In order to realize multi-stage pipeline operation, each stage of the operation circuit that performs each stage of pipeline operation may include but is not limited to one or more of the following operators or circuits: random number processing circuit, addition and subtraction circuit, subtraction circuit, table lookup circuit, parameter configuration circuit, multiplier, pooler, comparator, absolute value circuit, logic operator, position index circuit or filter.

[0073] The above describes the hardware implementation of the embodiments of the present disclosure. Based on the above hardware environment, the present disclosure proposes a solution for using an artificial intelligence processor to perform attention mechanism calculation tasks. The solution can make full use of the parallel computing power of multiple computing units in the artificial intelligence processor to avoid the situation where some computing units are idle, thereby completing the attention mechanism calculation task at a faster running speed.

[0074] Figure 8 An exemplary flow chart of a method for using an artificial intelligence processor to perform attention mechanism computing tasks in some embodiments of the present disclosure is shown. Figure 8As shown, in step 801, the attention mechanism computing task is divided into a plurality of unit computing tasks. Then, in step 802, until the attention mechanism computing task is completed, each computing unit in the artificial intelligence processor cyclically executes: requesting a unit computing task and completing the requested unit computing task. The artificial intelligence processor includes at least two of the above computing units. Step 801 can be, for example, Figure 4 The control module 41 in the processor core 306 of the artificial intelligence processor shown in the figure is executed, which can also be called a control unit. Step 802 is executed by each computing unit, corresponding to Figure 4 The computing module 42 in the processor core 306 is shown. As mentioned above, the computing module 42 may include multiple computing units, each of which may be configured as a multi-stage computing pipeline, and multiple computing units constitute multiple multi-stage computing pipelines, which may execute their own unit computing tasks in parallel.

[0075] It is understandable that according to the method proposed in this disclosure using an artificial intelligence processor to perform attention mechanism computing tasks, the computing unit that completes the current round of tasks can automatically request to obtain the tasks that need to be processed in the next round. This task scheduling method is adaptive, can solve the problem of unbalanced task distribution, avoids the idleness of certain computing units, and gives full play to the parallel computing power of the computing units. Moreover, this adaptive scheduling scheme of the embodiment of the present disclosure also avoids the design of complex load balancing algorithms, resulting in development difficulty and maintenance difficulty. It is understandable that the attention mechanism can be an attention mechanism and its variants, including but not limited to self-attention mechanism, multi-head self-attention mechanism (Multi-Head Self-Attention), and self-attention mechanism with various mask operations, multi-head self-attention mechanism, etc.

[0076] Fig. 9 An exemplary schematic diagram of the self-attention mechanism calculation process in some embodiments of the present disclosure is shown. Fig. 9 As shown, for the input sequence data S = [S1; S2; S3; S4], first use three matrices W_Q, W_K, and W_V to perform linear transformation on it to obtain the query (Query, Q) matrix, key (Key, K) matrix, and value (Value, V) matrix; secondly, perform inner product operation on Q and K to obtain the original score (Logits) matrix; then perform Softmax operation on Logits row by row to obtain the attention score (Score) matrix; finally, multiply Score by V to obtain the output (Output, O) of the self-attention mechanism.

[0077] like Fig. 9As shown in the steps indicated by the dotted arrows in the figure, the mask operation is introduced into the self-attention mechanism to obtain the Masked Self-Attention mechanism. The mask is usually a matrix with the same shape as the input sequence, in which the elements are 0 or 1. When calculating the attention weight, the mask is combined with the original score matrix element by element to control which positions of information can be paid attention to and which positions of information need to be blocked. For example: In the Transformer-based natural language processing model, the length of the input sequence is often inconsistent. In order to batch these sequences, the shorter sequences are usually padded (padding) to make their length consistent with the longest sequence. However, these padding values ​​have no meaning in actual semantics. If they are involved in the calculation of the attention score, noise will be introduced and the performance of the model will be affected. Therefore, the padding mask (PaddingMask) operation can be introduced in the Transformer to mask the positions corresponding to the padding values. For another example: In order to ensure that the model can only see the sequence information before the current position when predicting the next word, but cannot see the future sequence information, a looking-ahead mask operation can be introduced in the Transformer to mask the position after the current position.

[0078] As an extension of the self-attention mechanism, the multi-head self-attention mechanism repeatedly calculates the attention mechanism multiple times. Each calculation of the attention mechanism can be regarded as an attention head. Through multiple self-attention heads, the complex relationships in the input sequence can be captured from different subspaces or angles, thereby enhancing the expressiveness of the model.

[0079] Fig.10 An exemplary schematic diagram of the calculation process of the multi-head self-attention mechanism in some embodiments of the present disclosure is shown. Fig.10As shown, firstly, the input S = [S1; S2; S3; S4] is linearly transformed to obtain Q, K, and V matrices respectively, and then a split operation is performed on the word embedding dimension or model dimension (ModelDimension, which is used to represent the feature dimension of sequence elements in the model calculation process) to obtain multiple attention heads. Specifically, Q is split to obtain Q0 and Q1 matrices, K is split to obtain K0 and K1 matrices, and V is split to obtain V0 and V1 matrices. Then, similar to the self-attention mechanism, based on Q0, K0, and V0, O_0 is calculated by inner product, Softmax, and matrix multiplication, and based on Q1, K1, and V1, O_1 is calculated by inner product, Softmax, and matrix multiplication; finally, O_0 and O_1 are concatenated to obtain the output O of the multi-head self-attention mechanism. Accordingly, a mask operation is introduced in the attention score calculation operation of the multi-head attention mechanism to obtain the Masked Multi-Head Self-Attention mechanism.

[0080] For the convenience of description, Fig. 9 and Fig.10 The calculation process of the self-attention mechanism and the multi-head self-attention mechanism is explained by taking a single sample as an example. It is understandable that in actual application scenarios, such as during training, the model can process samples in batches, calculate and process Batch Size samples in a batch cycle, and then update the model parameters. For another example, during the inference process, the model can process samples in batches, complete the calculation and processing of Batch Size samples in one batch, and improve the inference efficiency.

[0081] Back to Figure 8 ,It can be understood that according to the scale of the attention mechanism ,computation task, the attention mechanism computation task can be divided into ,multiple unit computation tasks.

[0082] In some embodiments, the step of dividing the attention mechanism computing task into multiple unit computing tasks may include: determining the maximum amount of data that a single computing unit can calculate at one time; and dividing the attention mechanism computing task into multiple unit computing tasks in the sequence length dimension according to the maximum amount of data. Accordingly, in some embodiments, the control unit in the artificial intelligence processor may be further configured to: determine the maximum amount of data that a single computing unit can calculate at one time; and divide the attention mechanism computing task into multiple unit computing tasks in the sequence length dimension according to the maximum amount of data.

[0083] It can be understood that in order to give full play to the computing power of the computing unit, it is preferred that the divided unit computing tasks can fully utilize the computing bandwidth of the computing unit, thereby reducing the number of operations and speeding up the processing speed. The computing bandwidth can be known in advance according to the specific hardware configuration, such as 512B. In addition, since the sequence length dimension is usually larger than the batch size and head number dimensions, it is preferred to split the attention mechanism computing task on the sequence length dimension. It can be understood that, depending on the specific situation, it can also be split on one or more other dimensions.

[0084] In some embodiments, the sequence length of the attention mechanism calculation task is seq_q, the partition size is tile_q, the batch is B, the number of heads of the attention mechanism is h, and the maximum amount of data that can be calculated by a single operation unit at a time is Nmax, where tile_q×B×h≤Nmax. Preferably, tile_q×B×h=Nmax.

[0085] In order to manage the scheduling of attention mechanism computing tasks and confirm the completion of tasks, in some embodiments, the above method may also include: determining the total number of attention mechanism computing tasks; and storing the total number in a shared cache. Accordingly, for the artificial intelligence processor disclosed in the present disclosure, it also includes a shared cache for storing the total number of attention mechanism computing tasks. The shared cache here refers to the cache space shared by the computing units in the artificial intelligence processor 201. In some embodiments, the shared cache can be located in a high-speed storage area inside the artificial intelligence processor 201, such as a shared memory shared between multiple clusters. In other embodiments, the shared cache can also be located outside the artificial intelligence processor 201, for example Figure 2 On the storage device 204.

[0086] The total number of attention mechanism computing tasks can be described or characterized in a variety of ways. In some embodiments, it can be directly characterized by the number of data in the attention mechanism computing task. For example, for a multi-head attention mechanism computing task, its scale can be described by three dimensions: Batch Size, Head Num, and Sequence Length. The total number of computing tasks can be expressed as BatchSize*HeadNum*SequenceLength. In other embodiments, the number of unit computing tasks divided can be used to characterize it. For example, when the sequence length seq_q is split according to the split size tile_q of the sequence length dimension, ceil(seq_q / tile_q) unit computing tasks can be obtained. The total number of computing tasks can be expressed as ceil(seq_q / tile_q), where ceil means rounding up.

[0087] By counting the total number of attention mechanism calculation tasks, and maintaining the unprocessed (remaining) unit calculation tasks by updating the total number when the operation unit requests a unit calculation task, it can be ensured that all unit calculation tasks are executed.

[0088] In some embodiments, the step of requesting a unit computing task in step 802 may include: the computing unit initiates the requesting unit computing task through an atomic operation. Accordingly, for the artificial intelligence processor disclosed in the present disclosure, the computing unit is further configured to: initiate the requesting unit computing task through an atomic operation.

[0089] It is understandable that the process of a computing unit requesting a unit computing task involves concurrent access issues. When multiple computing units request a unit computing task at the same time, atomic operations can ensure data consistency and correctness. Atomic operations can accurately avoid mutual interference between computing units and ensure that tasks are correctly obtained.

[0090] In some embodiments, the computing unit may use a three-stage data pipeline including at least load-calculate-store back to execute the unit computing task. Figure 6 Describing the two-layer three-stage pipeline, depending on the specific storage architecture, a three-stage data pipeline including at least loading-computing-restoring can be configured. In this case, the timing for the computing unit to request the next unit computing task does not need to wait until the current unit computing task is completely completed (for example, restoring to the memory), but can be advanced to the loading stage of the current unit computing task and immediately initiate the task request. This ensures that the data pipeline is seamlessly connected and continues to flow, further shortening the overall processing time.

[0091] In some embodiments, the step of initiating the request unit computing task by the operation unit through an atomic operation includes: accessing the shared cache to determine whether there is an unexecuted unit computing task; in response to the existence of an unexecuted unit computing task, requesting a unit computing task; and updating the total number according to the requested unit computing task. Accordingly, for the artificial intelligence processor disclosed in the present disclosure, the operation unit is further configured to implement the above steps.

[0092] It is understandable that in the initial stage of executing the attention mechanism computing task, the total number of unit computing tasks is recorded in the shared cache; after the operation unit successfully requests the unit computing task, the number of the requested unit computing tasks can be subtracted from the total number, for example, by 1, so as to update the total number. In some embodiments, in response to the total number of the attention mechanism computing tasks being greater than 0, it can be determined that there are unexecuted unit computing tasks. When the total number is less than or equal to 0, it indicates that the attention mechanism computing task has been completed.

[0093] It can be considered that when using multiple computing units to perform attention mechanism computing tasks, due to the different number of invalid elements between unit computing tasks caused by masking operations, some computing units quickly completed the computing tasks, so that they have been in an idle state in the later stage, while other computing units have been performing computing tasks at full load, resulting in a load imbalance problem. The adaptive scheduling scheme proposed by the disclosed embodiment for the load imbalance problem first divides the attention mechanism computing task into unit computing tasks, and obtains the total number of unit computing tasks SUM. Subsequently, a counter is set in the shared cache of the artificial intelligence processor to record the number of remaining unprocessed unit computing tasks, and the value of the counter is initialized to SUM. Then, each computing unit applies for a unit computing task through an atomic operation, and after the execution of the applied unit computing task is completed, it applies for other unit computing tasks again through an atomic operation, so that each computing unit is always in a computing state, thereby improving the utilization rate of the computing unit. Among them, when each operation unit applies for a unit calculation task through an atomic operation, it first determines whether the counter is greater than 0. If the counter is greater than 0, it means that a unit calculation task can be applied for, and then a unit calculation task is applied for, and the counter is reduced by 1; otherwise, it means that all unit calculation tasks have been executed and the attention mechanism calculation task has ended.

[0094] As mentioned above, the computing units in the artificial intelligence processor in the disclosed embodiment can be configured into a multi-stage computing pipeline; multiple computing units constitute multiple multi-stage computing pipelines. Multiple multi-stage pipelines execute their respective unit computing tasks in parallel.

[0095] Specifically, according to the above Figure 7 The exemplary architecture of the multi-stage operation pipeline is described. Those skilled in the art will appreciate that multiple multi-stage operation pipelines can process large-scale operands in parallel for the same instruction.

[0096] In some embodiments, the step of completing the requested unit computing task includes multi-level operations. The multi-level operations include: calculating the original attention score for the query data block and the key data block of the requested unit computing task; performing a masking operation on the original attention score to obtain a masked original attention score; performing a Softmax operation on the masked original attention score to obtain a masked attention score; and calculating the output of the requested unit computing task based on the masked attention score and the value data block of the requested unit computing task.

[0097] Accordingly, for the artificial intelligence processor disclosed in the embodiment of the present disclosure, each of the multi-stage operation pipelines therein can be configured to complete the requested unit calculation task according to the above steps. For example, the above operation can utilize multiple first multipliers (first stage) included in the multi-stage pipeline operation circuit structure, at least one addition tree (second stage) to perform the operation of calculating the original attention score (that is, the dot product operation of Q and K), utilize multiple second multipliers (third stage) to perform the mask operation (that is, the bitwise multiplication operation of the original attention score and the mask template), utilize nonlinear operators (fourth stage) to perform Softmax operations, and utilize multiple third multipliers (fifth stage) to obtain the output of the unit calculation task (that is, the multiplication operation of the mask attention score and V). It can also be understood that some pipeline stages can be further subdivided. For example, the pipeline stage performing the Softmax operation can further include several sub-stages, such as comparators (to obtain the maximum value), exponential operations, and the like.

[0098] As mentioned above, at least due to the above mask operation, part of the data in the dot product result of Q and K will become invalid values, which has no effect on the subsequent calculation. In this case, the unused one or more stages of pipeline operation circuits can be bypassed, that is, one or more stages of the multi-stage pipeline operation circuit can be selectively used according to the needs of the operation operation, without requiring the operation operation to pass through all the multi-stage pipeline operations. For example, for the invalid data after the above mask operation, the subsequent operation pipeline stage can be bypassed, thereby eliminating the invalid operation.

[0099] Since invalid operations can be omitted, in some embodiments, the above-mentioned multi-level operation steps also include: in response to the different numbers of invalid elements in the masked attention scores, the time required for the multiple multi-level operation pipelines corresponding to the above-mentioned multiple operation units to complete their respective unit calculation tasks is inconsistent.

[0100] Fig.11 A schematic diagram of the effect of masking operation on the execution time of the operation unit is shown to better understand the technical effect of the disclosed embodiment. As shown in the figure, the blocks in the figure represent the data to be processed, where it is evenly split into 4 parts in the sequence length dimension, and processed in parallel by 2 operation units, and the attention mechanism operation is performed on the corresponding data. Due to the existence of the masking operation, part of the data in the dot product results of Q and K becomes invalid values, and the upper triangle part of the figure is shaded to represent invalid values. The number of invalid values ​​in each split data is different, so the amount of calculation of each operation unit will be different, and the time required to complete the calculation of the data block will be different.

[0101] If data block 1 is allocated to computing unit 1 and data block 2 is allocated to computing unit 2 according to the allocation scheme of the prior art, computing unit 1 will usually finish executing faster than computing unit 2. However, computing unit 1 can only be allocated the operation of the next data block after computing unit 2 finishes the operation. Therefore, although the computing tasks on each computing unit are evenly distributed, it will actually lead to uneven computing loads among computing units, resulting in some computing power being idle.

[0102] In contrast, according to the adaptive scheduling scheme of the disclosed embodiment, data block 1 is assigned to operation unit 1, and data block 2 is assigned to operation unit 2. Similarly, operation unit 1 will be executed faster than operation unit 2. However, at this time, operation unit 1 can immediately request the next data block to perform operations, such as data block 3, without waiting for operation unit 2 to complete the operation. In this way, after each operation unit finishes the execution of the current unit computing task, it can autonomously request the remaining unit computing tasks. Before the overall attention mechanism computing task is completed, all operation units are in a computing state. In other words, the multiple multi-level computing pipelines are always in a "flowing" state and will not be idle, thereby improving the execution speed of the attention mechanism computing task.

[0103] Although the adaptive scheduling scheme of the disclosed embodiment is described above in conjunction with the attention mechanism computing task, those skilled in the art will appreciate that the adaptive scheduling scheme can also be applied to other scenarios, such as when load balancing is required for complex algorithms implemented on artificial intelligence processors, thereby avoiding the design of complex load balancing algorithms.

[0104] Based on the above description, those skilled in the art can understand that the artificial intelligence processor of the embodiment of the present disclosure can be implemented in a chip, so the present disclosure also discloses a chip, which includes the artificial intelligence processor as described in any of the above embodiments. In addition, the present disclosure also discloses a board, which includes the above chip.

[0105] According to different application scenarios, the electronic equipment or device disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablet computers, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, driving recorders, navigators, sensors, cameras, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, automatic driving terminals, transportation, household appliances, and / or medical equipment. The transportation includes airplanes, ships and / or vehicles; the household appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, electric lights, gas stoves, and range hoods; the medical equipment includes magnetic resonance imaging, ultrasound machines and / or electrocardiographs. The electronic equipment or device disclosed herein may also be applied to the Internet, IoT, data centers, energy, transportation, public administration, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and medical fields. Further, the electronic equipment or device disclosed herein may also be used in cloud, edge, and terminal applications related to artificial intelligence, big data, and / or cloud computing. In one or more embodiments, electronic devices or devices with high computing power according to the disclosed solution can be applied to cloud devices (such as cloud servers), while electronic devices or devices with low power consumption can be applied to terminal devices and / or edge devices (such as smart phones or cameras). In one or more embodiments, the hardware information of the cloud device and the hardware information of the terminal device and / or edge device are compatible with each other, so that according to the hardware information of the terminal device and / or edge device, appropriate hardware resources can be matched from the hardware resources of the cloud device to simulate the hardware resources of the terminal device and / or edge device, so as to complete the unified management, scheduling and collaborative work of end-to-end or cloud-edge-to-end.

[0106] It should be noted that, for the purpose of simplicity, the present disclosure describes some methods and embodiments thereof as a series of actions and combinations thereof, but those skilled in the art will appreciate that the scheme of the present disclosure is not limited by the order of the actions described. Therefore, based on the disclosure or teaching of the present disclosure, those skilled in the art will appreciate that some of the steps therein may be performed in other orders or simultaneously. Further, those skilled in the art will appreciate that the embodiments described in the present disclosure may be regarded as optional embodiments, i.e., the actions or modules involved therein may not necessarily be necessary for the implementation of one or some of the schemes of the present disclosure. In addition, depending on the different schemes, the present disclosure also has different focuses on the description of some embodiments. In view of this, those skilled in the art may appreciate the parts that are not described in detail in a certain embodiment of the present disclosure, and may also refer to the relevant descriptions of other embodiments.

[0107] In terms of specific implementation, based on the disclosure and teaching of this disclosure, those skilled in the art can understand that several embodiments disclosed in this disclosure can also be implemented by other methods not disclosed herein. For example, with respect to the various units in the electronic device or device embodiments described above, this article divides them on the basis of considering logical functions, and there may be other division methods in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. In terms of the connection relationship between different units or components, the connection discussed in the above text in conjunction with the accompanying drawings can be a direct or indirect coupling between units or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection using an interface, wherein the communication interface can support electrical, optical, acoustic, magnetic or other forms of signal transmission.

[0108] In the present disclosure, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed on multiple network units. In addition, according to actual needs, some or all of the units may be selected to achieve the purpose of the scheme described in the embodiments of the present disclosure. In addition, in some scenarios, multiple units in the embodiments of the present disclosure may be integrated into one unit or each unit may exist physically separately.

[0109] In some implementation scenarios, the above-mentioned integrated unit can be implemented in the form of a software program module. If implemented in the form of a software program module and sold or used as an independent product, the integrated unit can be stored in a computer-readable memory. Based on this, when the scheme of the present disclosure is embodied in the form of a software product (such as a computer-readable storage medium), the software product can be stored in a memory, which may include several instructions to enable a computer device (such as a personal computer, a server or a network device, etc.) to perform some or all of the steps of the method described in the embodiment of the present disclosure. The aforementioned memory may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0110] In some other implementation scenarios, the above-mentioned integrated unit can also be implemented in the form of hardware, that is, a specific hardware circuit, which may include digital circuits and / or analog circuits, etc. The physical implementation of the hardware structure of the circuit may include but is not limited to physical devices, and the physical devices may include but are not limited to devices such as transistors or memristors. In view of this, the various devices described herein (such as artificial intelligence processors or other processing devices) can be implemented by appropriate hardware processors, such as CPUs, GPUs, FPGAs, DSPs, and ASICs, etc. Further, the aforementioned storage unit or storage device may be any appropriate storage medium (including magnetic storage media or magneto-optical storage media, etc.), which may be, for example, a variable resistive memory (Resistive Random Access Memory, RRAM), a dynamic random access memory (Dynamic Random Access Memory, DRAM), a static random access memory (Static Random Access Memory, SRAM), an enhanced dynamic random access memory (Enhanced Dynamic Random Access Memory, EDRAM), a high bandwidth memory (High Bandwidth Memory, HBM), a hybrid memory cube (Hybrid Memory Cube, HMC), ROM and RAM, etc.

[0111] Although multiple embodiments of the present disclosure have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Those skilled in the art may think of many changes, modifications, and alternatives without departing from the thought and spirit of the present disclosure. It should be understood that in the process of practicing the present disclosure, various alternatives to the embodiments of the present disclosure described herein may be adopted. The attached claims are intended to define the scope of protection of the present disclosure, and therefore cover equivalents or alternatives within the scope of these claims.

Claims

1. A method for performing an attention mechanism computing task using an artificial intelligence processor, comprising: Divide the attention mechanism calculation task into multiple unit calculation tasks; Until the attention mechanism calculation task is completed, each operation unit executes cyclically: Request unit calculation task; as well as Complete the requested unit calculation task; Wherein, the artificial intelligence processor includes at least two of the computing units.

2. The method of claim 1, wherein: The steps of dividing the attention mechanism calculation task into multiple unit calculation tasks include: Determine the maximum amount of data that a single computing unit can calculate at one time; According to the maximum data amount, the attention mechanism calculation task is divided into multiple unit calculation tasks in the sequence length dimension.

3. The method of claim 2, wherein: The computing units in the artificial intelligence processor are configured as a multi-stage computing pipeline; Multiple computing units form multiple multi-stage computing pipelines; The plurality of multi-stage pipelines execute respective unit computing tasks in parallel.

4. The method of claim 3, wherein: The steps of completing the requested unit calculation task include multi-level operations; wherein the multi-level operations include: Calculate the raw attention score for the query data block and key data block of the requested unit computation task; Performing a mask operation on the original attention score to obtain a masked original attention score; Performing a Softmax operation on the masked original attention score to obtain a masked attention score; Calculate the output of the requested unit computation task according to the masked attention score and the value data block of the requested unit computation task.

5. The method of claim 4, wherein: The multi-level operation steps also include: In response to the different numbers of invalid elements in the masked attention scores, the times required for the multiple levels of multiple operation pipelines to complete their respective unit computing tasks are inconsistent.

6. The method according to any one of claims 2 to 5, wherein: The sequence length of the attention mechanism calculation task is seq_q, the partition size is tile_q, the batch is B, the number of heads of the attention mechanism is h, and the maximum amount of data that a single operation unit can calculate at a time is Nmax, where tile_q×B×h≤Nmax.

7. The method according to any one of claims 1 to 6, wherein: The method further comprises: Determining the total number of attention mechanism computation tasks; and The total quantity is stored in a shared cache.

8. The method of claim 7, wherein: The step of requesting a unit computing task comprises: The computing unit initiates the request unit computing task through an atomic operation.

9. The method according to claim 8, wherein: The operation unit uses a three-stage data pipeline including at least load-calculate-store back to execute the unit computing task, and the step of the operation unit initiating the request unit computing task through atomic operation includes: The request is initiated in response to completion of a loading phase of the current unit computing task.

10. According to any one of the methods of claims 8-9, the step of the computing unit initiating the request unit computing task through an atomic operation comprises: Accessing the shared cache to determine whether there is an unexecuted unit computing task; In response to the presence of an unexecuted unit computing task, requesting the unit computing task; as well as The total quantity is updated according to the requested unit calculation task.

11. The method of claim 10, wherein: The steps of determining whether there are unexecuted unit calculation tasks include: In response to the total number of the attention mechanism computing tasks being greater than 0, it is determined that there are unexecuted unit computing tasks.

12. An artificial intelligence processor, comprising: A control unit configured to divide the attention mechanism computation task into a plurality of unit computation tasks; A computing unit configured to cyclically perform the following operations until the attention mechanism calculation task is completed: Request unit calculation task; as well as Complete the requested unit calculation task; Wherein, the artificial intelligence processor includes at least two of the computing units.

13. The artificial intelligence processor of claim 12, wherein: The control unit is further configured to: Determine the maximum amount of data that a single computing unit can calculate at one time; According to the maximum data amount, the attention mechanism calculation task is divided into multiple unit calculation tasks in the sequence length dimension.

14. The artificial intelligence processor of claim 13, wherein: The computing unit is configured as a multi-stage computing pipeline; Multiple computing units form multiple multi-stage computing pipelines; The plurality of multi-stage operation pipelines execute respective unit computing tasks in parallel.

15. The artificial intelligence processor of claim 14, wherein: Each multi-stage operation pipeline of the artificial intelligence processor is configured to complete the requested unit computing task as follows: Calculate the raw attention score for the query data block and key data block of the requested unit computation task; Performing a mask operation on the original attention score to obtain a masked original attention score; Performing a Softmax operation on the masked original attention score to obtain a masked attention score; Calculate the output of the requested unit computation task according to the masked attention score and the value data block of the requested unit computation task.

16. The artificial intelligence processor of claim 15, wherein: In response to the different numbers of invalid elements in the masked attention scores, the artificial intelligence processor requires inconsistent times for the multiple multi-stage operation pipelines to complete their respective unit computing tasks.

17. The artificial intelligence processor according to any one of claims 13 to 16, wherein: The sequence length of the attention mechanism calculation task is seq_q, the partition size is tile_q, the batch is B, the number of heads of the attention mechanism is h, and the maximum amount of data that a single computing unit can calculate at a time is Nmax, tile_q×B×h≤Nmax.

18. The artificial intelligence processor according to any one of claims 12 to 17, wherein: The artificial intelligence processor also includes: A shared cache for storing the total number of attention mechanism calculation tasks.

19. The artificial intelligence processor of claim 18, wherein: The computing unit is further configured to: The request unit computing task is initiated through an atomic operation.

20. The artificial intelligence processor of claim 19, wherein the computing unit uses a three-stage data pipeline including at least load-calculate-store back to execute the unit computing task, and the computing unit is further configured to: The request is initiated in response to completion of a loading phase of the current unit computing task.

21. The artificial intelligence processor according to any one of claims 18 to 19, wherein: The computing unit is further configured to: Accessing the shared cache to determine whether there is an unexecuted unit computing task; In response to the presence of an unexecuted unit computing task, requesting the unit computing task; as well as The total quantity is updated according to the requested unit calculation task.

22. A chip, comprising an artificial intelligence processor as described in any one of claims 12 to 21.

23. A board comprising the chip according to claim 22.

Images