Computing devices, methods, and related products that perform neural network models

Abstract

This disclosure discloses a computing device for executing neural network models, a method for executing neural network models using the computing device, and related products. The computing device may be included in a combined processing unit, which may further include an interface device and other processing units. The computing device interacts with other processing units to jointly complete user-specified computational operations. The combined processing unit may also include a storage device connected to both the computing device and the other processing units to store data from them. The disclosed solution utilizes a multi-core architecture to effectively integrate the operations of the neural network output layer, reducing the number of interactions with external storage circuits and the amount of input / output data, thereby improving machine processing efficiency.

Description

Technical Field

[0001] This disclosure generally relates to the field of neural network models. More specifically, this disclosure relates to computing devices for executing neural network models, methods for executing neural network models using computing devices, chips, and circuit boards. Background Technology

[0002] With the rapid development of artificial intelligence, neural network models are widely used in language modeling, such as machine translation and speech recognition. These models calculate the probability of words while considering partially generated sequences.

[0003] Figure 1 A simplified diagram of the output layer of a neural network language model is shown. As illustrated, the probabilities of these words are typically calculated by projecting them onto the dictionary space using the projection layer 110 (also known as a fully connected layer, generally a large matrix multiplication). Then, the K words with the highest probabilities are selected through the Softmax and TopK functions of the Softmax layer 120 and TopK layer 130. It's understandable that in some neural network models, the processing order of the Softmax and TopK layers can be reversed. Here, we take the more general scenario of first using the Softmax layer and then the TopK layer as an example. Currently, the dictionaries of the projection layers are quite large, and calculating the projection layer, Softmax, and TopK layers separately would incur significant memory access overhead.

[0004] With the development of hardware technology, modern intelligent processors mostly adopt a multi-core parallel framework. How to efficiently solve the TopK problem of large-scale data in a multi-core parallel framework, effectively reduce the memory access overhead of external storage circuits, and maximize the parallelism of data are urgent problems that neural network language models need to solve. Summary of the Invention

[0005] In order to at least partially solve one or more of the technical problems mentioned in the background art, this disclosure provides a method, apparatus and computer-readable storage medium for deep fusion calculation of the output layer of a neural network model.

[0006] In a first aspect, this disclosure discloses a computing device for executing a neural network model, the computing device including a plurality of processing cores and an on-chip network supporting communication between the cores, the output layer of the neural network model including a projection layer, a softmax layer and a TopK layer, wherein the computing device is configured to: load input data of the projection layer from an off-chip memory circuit; perform a fusion operation of the projection layer, softmax layer and TopK layer on the plurality of processing cores for the input data; perform reduction processing on the fusion operation result of the plurality of processing cores using the on-chip network to obtain a final result; and output the final result to the off-chip memory circuit.

[0007] In a second aspect, this disclosure provides a chip that includes the computing device of the first aspect.

[0008] In a third aspect, this disclosure provides a board including the chip described in the second aspect above.

[0009] In a fourth aspect, this disclosure provides a method for performing computations on a neural network model using the computing device described in the first aspect.

[0010] By using the deep fusion calculation method, apparatus and computer-readable storage medium for the output layer of neural network models provided above, the embodiments disclosed herein can efficiently fuse the projection layer, Softmax layer and TopK layer, effectively reduce the memory access overhead of external storage circuits, and greatly improve the computational efficiency of the output layer, especially the TopK layer. Attached Figure Description

[0011] The above and other objects, features, and advantages of exemplary embodiments of this disclosure will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of this disclosure are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein:

[0012] Figure 1 This diagram shows a simplified illustration of the output layer of a neural network language model.

[0013] Figure 2a This diagram shows the structure of the board card according to an embodiment of this disclosure;

[0014] Figure 2b This diagram illustrates the structure of the combined processing apparatus according to an embodiment of the present disclosure.

[0015] Figure 3 This diagram illustrates the internal structure of a multi-core computing device according to an embodiment of the present disclosure.

[0016] Figure 4 This example illustrates how CDMA works.

[0017] Figure 5 This diagram illustrates a computational partitioning scheme according to some embodiments of this disclosure;

[0018] Figure 6 A schematic flowchart of a deep fusion method for the output layer of a neural network model according to some embodiments of this disclosure is shown;

[0019] Figure 7 A schematic flowchart illustrating a fused computing method executed on each processing core according to some embodiments of this disclosure; and

[0020] Figure 8A schematic flowchart illustrating the specification processing according to some embodiments of this disclosure is shown. Detailed Implementation

[0021] The technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0022] It should be understood that the terms "first," "second," "third," and "fourth," etc., that may appear in the claims, specification, and drawings of this disclosure are used to distinguish different objects, rather than to describe a specific order. The terms "comprising" and "including" as used in the specification and claims of this disclosure indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0023] It should also be understood that the terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0024] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection."

[0025] The specific embodiments disclosed herein will now be described in detail with reference to the accompanying drawings.

[0026] Exemplary hardware environment

[0027] Figure 2a A schematic diagram of the structure of a board 20 according to an embodiment of this disclosure is shown. Figure 2aAs shown, board 20 includes chip 201, which is a system-on-chip (SoC) integrating one or more combined processing units. These combined processing units are artificial intelligence computing units used to support various deep learning and machine learning algorithms, meeting the intelligent processing needs of complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in cloud intelligence. A significant characteristic of cloud intelligence applications is the large volume of input data, placing high demands on the platform's storage and computing capabilities. Board 20 in this embodiment is suitable for cloud intelligence applications, possessing massive off-chip storage, on-chip storage, and powerful computing capabilities.

[0028] Chip 201 is connected to external device 203 via external interface device 202. External device 203 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 203 to chip 201 via external interface device 202. The calculation results from chip 201 can be transmitted back to external device 203 via external interface device 202. Depending on the application scenario, external interface device 202 may have different interface types, such as a PCIe interface.

[0029] The board 20 also includes a storage device 204 for storing data, which includes one or more memory cells 205. The storage device 204 is connected to and transmits data with the controller 206 and the chip 201 via a bus. The controller 206 in the board 20 is configured to regulate the state of the chip 201. Therefore, in one application scenario, the controller 206 may include a microcontroller (MCU).

[0030] Figure 2b This is a structural diagram illustrating the combined processing device in chip 201 of this embodiment. (As shown...) Figure 2b As shown, the combined processing device 210 includes a computing device 211, an interface device 212, a processing device 213, and a storage device 214.

[0031] The computing device 211 is configured to perform user-specified operations. It is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 213 through the interface device 212 to jointly complete the user-specified operations.

[0032] Interface device 212 is used to transmit data and control commands between computing device 211 and processing device 213. For example, computing device 211 can obtain input data from processing device 213 via interface device 212 and write it to on-chip storage device of computing device 211. Further, computing device 211 can obtain control commands from processing device 213 via interface device 212 and write them to on-chip control cache of computing device 211. Alternatively or optionally, interface device 212 can also read data from storage device of computing device 211 and transmit it to processing device 213.

[0033] Processing device 213, as a general-purpose processing device, performs basic control including but not limited to data transfer, and starting and / or stopping computing device 211. Depending on the implementation, processing device 213 may be one or more types of processors, including but not limited to digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, for the purposes of this disclosure only, computing device 211 can be considered to have a single-core structure or a homogeneous multi-core structure. However, when computing device 211 and processing device 213 are considered together, they are considered to form a heterogeneous multi-core structure.

[0034] Storage device 214 is used to store data to be processed. It may be DRAM or DDR memory, typically 16G or larger in size, and is used to store data of computing device 211 and / or processing device 213.

[0035] Figure 3 A schematic diagram of the internal structure of the computing device 211 as a multi-core is shown. The multi-core computing device 300 adopts a hierarchical structure design. As a system-on-a-chip, the multi-core computing device 300 includes at least one computing cluster, and each computing cluster includes multiple processor cores. In other words, the multi-core computing device 300 is constructed in a hierarchical structure of system-on-a-chip, computing cluster, and processor core.

[0036] From the perspective of system-on-a-chip hierarchy, such as Figure 3As shown, the multi-core computing device 300 includes an external storage controller 31, a peripheral communication module 32, an on-chip interconnect module 33 (or on-chip network), a synchronization module 34, and multiple computing clusters 35.

[0037] There can be multiple external storage controllers 31; two are shown exemplarily in the figure. These controllers are used to respond to access requests issued by the processing core to access external storage devices, such as... Figure 2b The DRAM 214 in the chip allows data to be read from or written to external devices. The peripheral communication module 32 receives control signals from the processing device 213 via the interface device 212, initiating the computing device 211 to execute tasks. The on-chip interconnect module 33 connects the external memory controller 31, the peripheral communication module 32, and multiple computing clusters 35 to transmit data and control signals between modules. The synchronization module 34 is a global barrier controller (GBC) used to coordinate the working progress of each computing cluster, ensuring information synchronization. The multiple computing clusters 35 are the computing cores of the multi-core computing device 300. Four are shown exemplary in the figure; however, with hardware development, the multi-core computing device 300 disclosed herein may also include 8, 16, 64, or even more computing clusters 35. The computing clusters 35 are used to efficiently execute deep learning algorithms.

[0038] From the perspective of the hierarchy of computational clusters, such as Figure 3 As shown in the upper right corner, each computing cluster 35 includes a processing unit 302 and a memory core 304. The processing unit 302 performs various computational tasks. In some implementations, the processing unit may be a multi-core architecture, for example, including multiple processing cores (IPUcores) 311-1 to 311-n, to perform tasks such as large-scale vector computation. This disclosure does not limit the number of processing cores 311.

[0039] The internal architecture of the processing core 311 is as follows Figure 3As shown below. Each processing core 311 may have multiple computing modules 324-1 to 324-m for performing computing tasks, and a local storage module 323 required for performing computing tasks. It should be noted that the local storage module 323 may include various communication modules for exchanging data with external storage units. For example, the local storage module 323 may include a communication module 321 for communicating with the shared storage module 315 in the storage core 304. The communication module 321 may be, for example, a move direct memory access (MVDMA) module. The local storage module 323 may also include a communication module 322 for exchanging data with off-chip memory, such as DRAM 308. The communication module 322 may be, for example, an input / output direct memory access (IODMA) module. The IODMA 322 controls the NRAM / WRAM in the local storage module 323. Figure 3 (Not shown) Access to DRAM 308; MVDMA 321 is used to control access to NRAM / WRAM in local storage module 323 and shared storage module 315.

[0040] continue Figure 3 In the upper right view, storage core 304 is mainly used for storage and communication, namely storing shared data or intermediate results between processing cores 311, and performing communication between computing clusters 35 and DRAM 308, communication between computing clusters 35, and communication between processing cores 311. In other embodiments, storage core 304 has scalar operation capabilities to perform scalar operations to realize computational tasks in data communication.

[0041] The storage core 304 includes a large shared memory module (SRAM) 315, a broadcast bus 314, a cluster direct memory access (CDMA) module 318, a global direct memory access (GDMA) module 316, and a communication-time computing module 317. The SRAM 315 acts as a high-performance data relay station. Data multiplexed between different processing cores 311 within the same computing cluster 35 does not need to be obtained from the DRAM 308 by each processing core 311. Instead, it is relayed between processing cores 311 via the SRAM 315. The storage core 304 only needs to quickly distribute the multiplexed data from the SRAM 315 to multiple processing cores 311 to improve inter-core communication efficiency and greatly reduce on-chip and off-chip input / output access.

[0042] Broadcast bus 314, CDMA 318, and GDMA 316 are used to perform communication between processing cores 311, communication between computing clusters 35, and data transfer between computing clusters 35 and DRAM 308, respectively. These will be explained separately below.

[0043] The broadcast bus 314 is used to complete high-speed communication between the processing cores 311 within the computing cluster 35. In this embodiment, the broadcast bus 314 supports inter-core communication methods including unicast, multicast, and broadcast. Unicast refers to point-to-point (e.g., data transmission from one processing core to another) data transmission. Multicast is a communication method that transmits a copy of data from SRAM 315 to several specific processing cores 311. Broadcast is a communication method that transmits a copy of data from SRAM 315 to all processing cores 311, and is a special case of multicast.

[0044] CDMA 318 is used to control access to SRAM 315 between different computing clusters 35 within the same computing device 300.

[0045] GDMA 316 works in conjunction with external memory controller 31 to control memory access from SRAM 315 to DRAM 308 in computing cluster 35, or to read data from DRAM 308 into SRAM 315. As described above, communication between DRAM 308 and NRAM / WRAM in local memory module 323 can be achieved through two channels. The first channel is a direct connection between DRAM 308 and local memory module 323 via IODMA 322; the second channel involves first transmitting data between DRAM 308 and SRAM 315 via GDMA 316, and then transmitting data between SRAM 315 and local memory module 323 via MVDMA 321. Although the second channel appears to require more components and has a longer data flow, in some embodiments, the bandwidth of the second channel is significantly greater than that of the first channel. Therefore, communication between DRAM 308 and local memory module 323 may be more efficient via the second channel. The embodiments disclosed herein can select the data transmission channel based on their hardware capabilities.

[0046] In some embodiments, the storage core 304 can serve as a cache layer within the computing cluster 35, thereby expanding communication bandwidth. Furthermore, the storage core 304 can also facilitate communication with other computing clusters 35. For example, the storage core 304 can implement communication functions such as broadcasting, scattering, gathering, reducing, and all-reduce between computing clusters 35. Broadcasting refers to distributing the same data to all computing clusters; scattering refers to distributing different data to different computing clusters; gathering refers to aggregating data from multiple computing clusters; reducing refers to performing operations on data from multiple computing clusters according to a specified mapping function to obtain the final result and sending it to a specific computing cluster; and the difference between all-reduce and scattering is that the latter only sends the final result to one computing cluster, while all-reduce sends it to all computing clusters.

[0047] The communication-time computing module 317 can be used to complete computational tasks in communication processes, such as those involving the aforementioned protocols and full protocols, without relying on the processing unit 302, thereby improving communication efficiency and achieving the effect of "in-memory computing". Depending on different hardware implementations, the communication-time computing module 317 and the shared storage module 315 can be integrated in the same or different components. This disclosure embodiment is not limited in this respect, and as long as the functions implemented and the technical effects achieved are similar to those disclosed herein, they are all within the protection scope of this disclosure.

[0048] Figure 4 This diagram illustrates the operation of CDMA 318 when one processing core attempts to write data to another processing core in a computing cluster. In this application scenario, the same computing device includes multiple computing clusters. For simplicity, only computing cluster 0 and computing cluster 1 are shown in the diagram. Both clusters include multiple processing cores; similarly, for ease of explanation, only processing core 0 is shown for cluster 0, and only processing core 1 is shown for cluster 1. Processing core 0 intends to write data to processing core 1.

[0049] First, processing core 0 sends a unicast write request to write data into its local SRAM 0. CDMA 0 acts as the master and CDMA 1 acts as the slave. The master pushes the write request to the slave, that is, the master sends the write address AW and the write data W to transfer the data to SRAM 1 of computing cluster 1. Then, the slave sends a write response B as a reply. Finally, processing core 1 of computing cluster 1 sends a unicast read request to read the data from SRAM 1.

[0050] Exemplary neural network model computation scheme

[0051] Based on the aforementioned hardware environment, this disclosed embodiment provides a deep fusion computing scheme for the output layer of a neural network model. The fusion starts from the projection layer (fully connected layer) and then fuses the Softmax layer and TopK layer. This reduces the number of interactions and data volume with external storage circuits (such as DDR) during computation, maximizes the advantages of multi-core parallel computing, and improves computational efficiency.

[0052] As mentioned earlier, the projection layer, also known as the fully connected layer, maps the high-dimensional feature map extracted from the feature extraction into a one-dimensional feature vector. This feature vector contains all the feature information and can be converted into the probability of being classified into each category. In neural network language models, this is converted into the probability of words. The calculation of the projection layer is generally a large-scale matrix multiplication operation.

[0053] The Softmax layer normalizes the output, outputting the classification probability. The calculation process is as follows:

[0054]

[0055] Among them, z j It is the j-th input to the softmax layer. As can be seen from the softmax calculation process, the input and output data sizes are the same; through normalization, larger values ​​are highlighted while smaller values ​​are suppressed, thus significantly suppressing secondary features and determining the classification probability.

[0056] The TopK layer, as the name suggests, selects the K probabilities with the highest probabilities from the output. Various algorithms can be used to calculate the TopK layer, including but not limited to heap sort, quick sort, and bubble sort.

[0057] Traditional methods of computing each layer separately result in a massive amount of I / O data. For example, with a dictionary of 90,000 characters, if each operator (operation of each layer) is called only once, the result of the mapping layer needs to be imported from external storage circuitry (e.g., DDR) twice. When the base library is very large (e.g., the dictionary is long in Natural Language Processing (NLP) networks, and there are many face IDs in Computer Vision (CV) networks), the I / O performance of the intelligent computing unit deteriorates drastically under large step sizes. Therefore, there is an urgent need for optimization schemes to avoid large step sizes of DDR access.

[0058] The computational partitioning scheme of the neural network model in this embodiment is described below.

[0059] Figure 5A schematic diagram illustrating computational partitioning according to some embodiments of this disclosure is shown. As illustrated, taking an NLP network as an example, assuming the input word sequence is x, the hidden representation length of the sequence (equal to the word vector length) is B, and the dictionary length is L, the dictionary length can be divided into N equal parts based on the number of processing kernels N. Each processing kernel needs to calculate a matrix multiplication of size L / N. Further assuming that each processing kernel processes a data size of l at a time...

[0060] After computation is partitioned on a multi-core computing device, deep fusion computation of the allocated output layer can be performed on it.

[0061] Figure 6 A schematic flowchart of a deep fusion method for the output layer of a neural network model according to some embodiments of this disclosure is shown.

[0062] As shown in the figure, in step 610, the computing device loads the input data for the projection layer from off-chip memory circuitry. For the NLP network model, the loaded input data may include the word sequence x to be processed and a dictionary. As described in the preceding hardware architecture description, there can be two loading paths. In one example, the computing device can load data from off-chip memory circuitry (e.g., Figure 3 The DRAM 308 is then transferred to the local storage module 323 of the processing core via the shared storage module 315. In another example, the computing device may also load data directly from the DRAM 308 to the local storage module 323. The embodiments disclosed herein are not limited in this respect.

[0063] Next, in step 620, a fusion operation of the projection layer, softmax layer, and TopK layer is performed on each processing core for the corresponding input data. The fusion process will be described in detail later.

[0064] Next, in step 630, the fusion operation results of multiple processing cores are reduced to obtain the final result. Since each processing core only processes a portion of the dictionary, it is necessary to summarize the operation results of multiple processing cores. In some embodiments, on-chip communication reduction functionality can be used to implement the summarization of operation results from multiple processing cores, as described in detail below.

[0065] Finally, in step 640, the final result is output to the external memory circuit.

[0066] As can be seen from the above process, since the calculations of the projection layer, softmax layer, and TopK layer are fused, only one memory access is required for the off-chip memory circuit. Compared to calculating each layer separately, where each layer's calculation involves memory access to the off-chip memory circuit, the embodiments disclosed in this paper can reduce the number of memory accesses. This can save significant memory access overhead for language models with large dictionaries or CV models with large databases.

[0067] The following describes in detail the fusion and reduction processing of some embodiments of this disclosure. It is understood that although a detailed description is provided using an NLP network model as an example, these descriptions can be similarly applied to other network models, such as CV networks.

[0068] Figure 7 A schematic flowchart illustrating a fusion operation method performed on each processing core according to some embodiments of this disclosure is shown. Each processing core can progressively load the dictionary allocated to it for processing in steps of a size l of the amount of data it can process at one time, and cyclically perform the fusion operation.

[0069] As shown in the figure, in step 710, each processing core first loads the input sequence x into its local storage circuit (e.g., Figure 3 The local storage module 323 in the processing core loads a dictionary of a certain length, for example, Δl, and performs matrix multiplication operations. The data loaded by the processing core is, for example, from external storage circuitry (e.g.,...). Figure 3 It comes from DRAM 308, which can be transferred to local storage module 323 via shared storage module 315, or directly from DRAM 308 to local storage module 323.

[0070] Matrix multiplication results are cached in a shared memory core (e.g., Figure 3 The shared storage module 315 in the middle is used to repeat this process until the dictionary of length l is calculated. The matrix multiplication result obtained by executing this step for the i-th time can be denoted as the sequence Z = [z i*l+1 ,…,z i*l+l ].

[0071] Next, in step 720, the matrix multiplication result, i.e., sequence Z, cached in the shared memory core, is loaded into the local storage circuitry of the processing core. Then, the K largest values ​​from sequence Z can be selected. Since these K largest values ​​are selected based on the results of operations on the dictionary portion of length l processed in the current loop, which is only a part of the entire dictionary, they are also called K local maxima. Various sorting methods can be used to calculate TopK.

[0072] In some embodiments, heap sort can be used to calculate TopK.

[0073] Heapsort is a sorting algorithm that utilizes a heap data structure. A heap is an approximately complete binary tree structure that satisfies the heap property: the key or index of a child node is always less than (or greater than) its parent node. A max-heap is where the value of each node is greater than or equal to the values ​​of its left and right child nodes, and is used for ascending order in heapsort; or a min-heap is where the value of each node is less than or equal to the values ​​of its left and right child nodes, and is used for descending order in heapsort.

[0074] The basic idea of ​​heap sort is to build an unordered sequence into a heap, choosing a max-heap or min-heap based on the ascending or descending order requirement; swap the top element of the heap with the last element, and "sink" the top (largest or smallest) element to the end of the array; readjust the structure to satisfy the heap definition, and then continue to swap the top element of the heap with the current last element, repeating the adjustment and swapping steps until the entire sequence is ordered.

[0075] In the TopK calculation of the disclosed embodiment, a min-heap can be used to achieve descending order, thereby obtaining the largest K values.

[0076] Specifically, a min-heap P can be built for sequence Z, and the K largest values ​​corresponding to the sequence can be selected using the heap sort method. The indices of these K values ​​can then be recorded in array P. index middle.

[0077] In other embodiments, TopK calculations can be implemented using methods such as bubble sort or quick sort. This disclosure does not limit the scope of the embodiments in this regard.

[0078] Next, in step 730, based on the calculation results of the projection layer in step 710 and the maximum value among the K local maxima determined in step 720, the sum of local probabilities in the softmax operation is updated.

[0079] In the original Softmax layer, the Softmax operator is defined using the above formula (1). However, in actual hardware, the range of data it can represent is limited, and the exponential operation of the above formula (1) is very likely to overflow. Therefore, to prevent overflow, the above formula (1) can be replaced by formula (2):

[0080]

[0081] Therefore, in some embodiments, the maximum value in the min-heap P can be recorded as max1. Then, the maximum value max1 in the heap is uniformly subtracted from the sequence Z (the result of the projection layer). After that, the exponential function value exp(Z-max1) of the sequence and their sum are obtained. In a specific instance, the vector activation instruction is used to obtain the result. The value of is then used to calculate the sum of the sequence using the Reducesum command, denoted as:

[0082]

[0083] Then, record the K maxima at the top of the heap (the maxima are the original values) and the exponential function sum S′ of the sequence. This allows us to obtain the K maxima and their exponential function sum for each processing core processing the amount of data l each time.

[0084] The above steps 710, 720 and 730 are executed repeatedly, so that each processing core processes the next round of data volume l.

[0085] During the loop, in step 720, when determining the K local maxima, the K local maxima determined in the previous loop need to be incorporated into the calculation result of the projection layer in the current loop, and the updated K local maxima are determined from it. That is, for this batch of data l, the sequence Z obtained together with the K local maxima selected in the previous loop are then sorted using various sorting methods such as heap sort to select K local maxima, which is to update the K local maxima.

[0086] During the loop, in step 730, the new maximum value in the heap is recorded as max1. The sequence Z obtained in step 710 of this loop is uniformly subtracted from the current maximum value max1, and the vector activation instruction is used to obtain the result. Then, the Reducesum instruction is used to calculate the exponent S′ of the sequence (see formula (3)), and it is accumulated with the previously calculated local probability Sp to obtain the updated local probability value Sp, where the update formula is:

[0087]

[0088]

[0089] Where max0 is the maximum value among the K local maxima in the previous loop, max1 is the maximum value among the K local maxima in the current loop, and z i Let i = 1, ..., l be the result of the projection layer in the current loop. It can be understood that during the first calculation, the value of S′ needs to be assigned to Sp, where Sp is the sum of local probabilities of the sequence.

[0090] This process continues until each processing core has finished calculating the dictionary of L / N assigned to it.

[0091] The logic of the above calculation method can be represented by the following pseudocode, as shown in Table 1.

[0092]

[0093] Table 1

[0094] Alternatively, in some embodiments, other methods can be used to perform TopK and Softmax calculations. Table 2 provides the pseudocode for another calculation method.

[0095]

[0096] Table 2

[0097] In Table 2, _sum, _exp, and _max are all vector calculation instructions. Comparing Table 1 and Table 2, it can be seen that the calculation methods differ starting from line 10, that is, the TopK calculation methods are different.

[0098] Specifically, in Table 2, the __max instruction is used to sort the data one by one. After sorting one data, its original value is written as the minimum value. This process is repeated K times to obtain the K maximum values.

[0099] The above describes the dictionary-related calculations for each processing core, which are assigned to its L / N. As described above, during the calculations at the projection layer, the results are cached in the shared memory module, eliminating the need to store them back to off-chip memory, thus reducing I / O memory access pressure. Furthermore, by first selecting the K maximum values ​​from the matrix multiplication results before performing Softmax-related operations, the computational load can be effectively reduced.

[0100] After each processing core has completed the fusion operation, the calculation results of each processing core need to be aggregated and processed.

[0101] Figure 8 A schematic flowchart illustrating the specification processing according to some embodiments of this disclosure is shown.

[0102] As shown in the figure, in step 810, the results of the fusion operations of each processing core can be aggregated into the first shared memory core via on-chip communication. Specifically, the array P (i.e., the K maximum values) obtained from the N processing cores, P... index (That is, the index of the K maximum values), data Sp (the denominator value in the Softmax calculation, that is, the sum of the local probabilities of each group), and m0 (the maximum value) are all reduced to the shared memory core of each computing cluster through on-chip communication.

[0103] Next, in step 820, K global maxima are determined from the K local maxima of the aggregated processing cores. Specifically, the arrays P and P0 obtained from the N cores can be used as follows: index Cascaded into a new array P new and P n-index These stores the K local maxima and their corresponding indices obtained from each kernel. P is then obtained using heap sort or other sorting methods.new Find the first K maximum values ​​and assign each maximum value to P. n-index The indices of the corresponding positions stored inside are retrieved to form the INDEX array. At this point, the K maximum values ​​are the global maximum values.

[0104] Finally, in step 830, the sum of global probabilities in the softmax operation is calculated for the K global maximum values ​​to obtain the final result.

[0105] In some embodiments, the above softmax operation includes calculating the global probability S to obtain the final result as follows:

[0106]

[0107] s[k]=e P[k]-temp / S,k=1,…,K(6)

[0108] Where P[k], k=1,…,K are the K global maximum values, P[1] and temp are the maximum values ​​among the K global maximum values, N is the number of processing cores, and maxValue is... j Sp is the maximum of the local maxima on the j-th processing kernel. j Let s[k] be the final local probability sum calculated on the j-th processing kernel, and s[k] be the probability of the k-th global maximum value.

[0109] Therefore, we can return the K largest values ​​(i.e., the K global maximum values) of the entire word sequence and their corresponding indices, as well as their corresponding probability values ​​s[k].

[0110] The logic of the above calculation method can be represented by the following pseudocode: Table 3.

[0111]

[0112] Table 3

[0113] As can be seen from the above specification calculation process, the specification calculation in this disclosed embodiment makes full use of the CDMA communication capability between computing clusters, reduces the communication time with off-chip memory units, and thus greatly improves the computing efficiency.

[0114] The above description, in conjunction with the hardware structure and flowchart, outlines the deep fusion scheme for the output layer of the neural network model in this disclosure. Similarly, some embodiments of this disclosure also provide chips and boards that include the computing device described above for performing deep fusion calculations of the output layer of the neural network model. These may include the corresponding features described above, which will not be repeated here.

[0115] Depending on the application scenario, the electronic devices or apparatus disclosed herein may include servers, cloud servers, server computing clusters, data processing devices, robots, computers, printers, scanners, tablets, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, dashcams, navigators, sensors, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, autonomous driving terminals, vehicles, home appliances, and / or medical devices. The vehicles include airplanes, ships, and / or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs. The electronic devices or apparatus disclosed herein can also be applied in fields such as the Internet, IoT, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and healthcare. Furthermore, the electronic devices or apparatus disclosed herein can also be used in cloud, edge, and terminal applications related to artificial intelligence, big data, and / or cloud computing. In one or more embodiments, the high-computing-power electronic devices or apparatuses according to the present disclosure can be applied to cloud devices (e.g., cloud servers), while the low-power electronic devices or apparatuses can be applied to terminal devices and / or edge devices (e.g., smartphones or cameras). In one or more embodiments, the hardware information of the cloud devices and the hardware information of the terminal devices and / or edge devices are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud devices to simulate the hardware resources of the terminal devices and / or edge devices based on the hardware information of the terminal devices and / or edge devices, so as to complete the unified management, scheduling and collaborative work of end-to-cloud or cloud-edge-end integration.

[0116] It should be noted that, for the sake of brevity, this disclosure describes some methods and their embodiments as a series of actions and combinations thereof. However, those skilled in the art will understand that the solutions disclosed herein are not limited by the order of the described actions. Therefore, based on the disclosure or teachings of this document, those skilled in the art will understand that some steps can be performed in a different order or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in this disclosure can be considered optional embodiments, that is, the actions or modules involved are not necessarily essential for the implementation of one or more solutions disclosed herein. In addition, depending on the solution, the description of some embodiments in this disclosure may have different emphases. In view of this, those skilled in the art will understand that parts not described in detail in a certain embodiment of this disclosure can also be referred to the relevant descriptions of other embodiments.

[0117] In terms of specific implementation, based on the disclosure and teachings of this document, those skilled in the art will understand that several embodiments disclosed herein can also be implemented in other ways not disclosed herein. For example, regarding the various units in the electronic device or apparatus embodiments described above, this document has divided them based on logical functions, but in actual implementation, there may be other ways of division. As another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. Regarding the connection relationships between different units or components, the connections discussed above in conjunction with the accompanying drawings can be direct or indirect couplings between units or components. In some scenarios, the aforementioned direct or indirect couplings involve communication connections utilizing interfaces, where the communication interface can support electrical, optical, acoustic, magnetic, or other forms of signal transmission.

[0118] In this disclosure, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network units. Furthermore, depending on actual needs, some or all of the units can be selected to achieve the purpose of the solution described in the embodiments of this disclosure. Additionally, in some scenarios, multiple units in the embodiments of this disclosure may be integrated into one unit or each unit may exist physically independently.

[0119] In other implementation scenarios, the integrated units described above can also be implemented in hardware, i.e., as specific hardware circuits, which may include digital circuits and / or analog circuits. The physical implementation of the circuit's hardware structure may include, but is not limited to, physical devices, which may include, but are not limited to, transistors or memristors. Therefore, the various devices described herein (e.g., computing devices or other processing devices) can be implemented using appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage units or storage devices can be any suitable storage medium (including magnetic storage media or magneto-optical storage media), such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), ROM, and RAM.

[0120] The embodiments of this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this disclosure. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

Claims

1. A computing device for executing a neural network model, the computing device comprising a plurality of processing cores and an on-chip network supporting inter-core communication, wherein the output layer of the neural network model comprises a projection layer, a softmax layer, and a TopK layer, wherein the computing device is used for: The input data of the projection layer is loaded from the off-chip memory circuitry; The fusion operation of the projection layer, softmax layer and TopK layer is performed on the input data on the multiple processing cores; The on-chip network is used to reduce the fusion operation results of the multiple processing cores to obtain the final result; as well as The final result is output to the off-chip memory circuit.

2. The computing device of claim 1, wherein each processing core is configured to perform the fusion operation as follows: The projection layer operation is performed on the input word sequence x and the dictionary assigned to the current processing kernel; Determine K local maxima from the computation results of the projection layer; and Based on the calculation results of the projection layer and the maximum value among the K local maxima, update the local probabilities in the softmax operation.

3. The computing device of claim 2, wherein each processing core is configured to progressively load the dictionary allocated to it for processing, with a step size of l of the amount of data it can process at one time, and cyclically execute the fusion operation.

4. The computing device of claim 3, further comprising a shared memory core shared by a plurality of processing cores, wherein each processing core is further configured to: Each time, a matrix multiplication operation is performed between the input word sequence x and a portion of l, resulting in a matrix multiplication result; and The matrix multiplication result is cached on the shared storage core until the amount of data in l has been processed.

5. The computing device of claim 4, wherein the processing core is further configured to: Load the cached matrix multiplication result from the shared memory core; and The K local maxima are determined from the matrix multiplication result.

6. The computing apparatus of claim 5, wherein the processing core is configured to determine the K local maxima using any of the following sorting methods: Heap sort, bubble sort, and quick sort.

7. The computing device according to any one of claims 5-6, wherein the processing core is configured to: The K local maxima determined in the previous loop are incorporated into the calculation results of the projection layer in the current loop, and the updated K local maxima are determined from them.

8. The computing device according to any one of claims 2-6, further comprising a shared memory core, and the computing device configured to perform reduction processing as follows: The results of the fusion operation of each processing core are aggregated into the shared memory core through the on-chip network; Determine K global maxima from the K local maxima of the aggregated processing cores; and The final result is obtained by calculating the sum of global probabilities in the softmax operation for the K global maximum values.

9. The computing device according to any one of claims 2-6, wherein the number of processing cores is N, the size of the dictionary is L, and each processing core is assigned to process the dictionary of size L / N.

10. The computing device according to any one of claims 2-6, wherein the input word sequence x comprises one or more word sequences, and the computing device is configured to allocate computational processing of the one or more word sequences according to the number of processing cores.

11. A chip comprising a computing device according to any one of claims 1-10.

12. A circuit board comprising the chip according to claim 11.

13. A method for performing computation of a neural network model using the computing device according to any one of claims 1-10.

Images