Devices, boards, methods, and readable storage media for integrating neural networks

Abstract

This disclosure relates to apparatus, boards, methods, and readable storage media for incorporating neural networks. The computing device disclosed herein is included in an integrated circuit device, which includes a general-purpose interconnect interface and other processing devices. The computing device interacts with other processing devices to jointly perform user-specified computational operations. The integrated circuit device may also include a storage device connected to the computing device and other processing devices respectively, for data storage of the computing device and other processing devices.

Description

Technical Field

[0001] This disclosure generally relates to the field of neural networks. More specifically, this disclosure relates to apparatus, boards, methods, and readable storage media for incorporating neural networks. Background Technology

[0002] A neural network is a system of multiple neurons connected according to certain rules. It is generally composed of the following four types of layer structures: input layer, convolution layer, pooling layer, and fully connected layer.

[0003] The input layer extracts a portion of the input data and transforms it into a feature matrix, containing the features corresponding to that portion. The convolutional layer is configured to receive the feature matrix from the input layer and extract features from the input data through convolution operations. Multiple convolutional layers can be built in practice. The pooling layer is configured to replace a region of data with a single value, typically the maximum or average value among all values ​​in that region. Pooling reduces model size and improves computational speed without losing too much information. The fully connected layer acts as a classifier in the entire convolutional neural network, essentially transforming the feature space, extracting and integrating all the useful information from the previous layers, and comparing information based on different classifications to determine whether the input data is similar to the comparison target.

[0004] With the development of technology, the number of layers in neural networks is increasing. Taking the classic VGG architecture as an example, VGG-A has 11 weight layers, VGG-B has 13 weight layers, VGG-C has 16 weight layers, VGG-D has 16 weight layers, and VGG-E has 19 weight layers. Here, "convolutional layer" and "fully connected layer" generally refer to weight layers. Some neural networks even have hundreds of layers. Furthermore, with the increase in the number of layers, the number of parameters in the neural network also increases exponentially; for example, AlexNet has 60 million parameters involved in the computation.

[0005] Multi-layered and multi-parameter computations require a large number of on-chip and off-chip input / output accesses, which consume significant resources and delay computation time. Therefore, a mechanism to reduce input / output accesses is urgently needed in the field of artificial intelligence. Summary of the Invention

[0006] In order to at least partially solve the technical problems mentioned in the background art, the present disclosure provides an apparatus, board, method and readable storage medium for incorporating neural networks.

[0007] In one aspect, this disclosure discloses an integrated circuit apparatus for dynamically fusing neural networks according to a fusion strategy, including a processing unit and a computing unit. The processing unit is configured to: select a starting layer of a template fusion unit according to a starting rule of the fusion strategy; and perform fusion based on the starting layer, checking rules within the fusion strategy to establish the template fusion unit. The computing unit is configured to perform neural network computation based on the template fusion unit.

[0008] In another aspect, this disclosure discloses a board including an integrated circuit device according to the foregoing.

[0009] In another aspect, this disclosure discloses a method for dynamically fusing neural networks according to a fusion strategy, comprising: selecting a starting layer of a template fusion unit according to the starting rules of the fusion strategy; performing fusion based on the starting layer, checking the rules of the fusion strategy to establish the template fusion unit; and performing neural network computation based on the established template fusion unit.

[0010] In another aspect, this disclosure discloses a computer-readable storage medium storing computer program code that dynamically fuses neural networks according to a fusion strategy, wherein the aforementioned method is executed when the computer program code is run by a processing device.

[0011] This disclosure dynamically determines the template fusion unit by setting a fusion strategy, fusions multiple layers in the neural network to form a new custom layer, and loads the data required to compute the template fusion unit at once to reduce input / output overhead. Attached Figure Description

[0012] The above and other objects, features, and advantages of exemplary embodiments of this disclosure will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of this disclosure are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding portions wherein:

[0013] Figure 1 This is a structural diagram of the board card shown in the embodiment of this disclosure;

[0014] Figure 2 This is a structural diagram illustrating an integrated circuit device according to an embodiment of this disclosure;

[0015] Figure 3 This is a schematic diagram illustrating the internal structure of a computing device according to an embodiment of the present disclosure;

[0016] Figure 4 This is a schematic diagram illustrating the internal structure of the processor core in an embodiment of this disclosure;

[0017] Figure 5This is a diagram illustrating when one processor core wants to write data to another processor core in a different cluster;

[0018] Figure 6 This is a schematic diagram illustrating the AlexNet model;

[0019] Figure 7 This is a schematic diagram illustrating an exemplary neural network model;

[0020] Figure 8 This is a schematic diagram illustrating the fusion of two convolutional layers in an embodiment of this disclosure;

[0021] Figure 9 This is a schematic diagram showing the formats of NCHW and NHWC;

[0022] Figure 10 This is a flowchart illustrating the template fusion unit performing neural network computation in an embodiment of this disclosure;

[0023] Figure 11 This is a flowchart illustrating the dynamic fusion of neural networks according to a fusion strategy in an embodiment of this disclosure; and

[0024] Figure 12 This is a flowchart illustrating the template fusion unit performing neural network calculations in an embodiment of this disclosure. Detailed Implementation

[0025] The technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, not all of them. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0026] It should be understood that the terms "first," "second," "third," and "fourth," etc., in the claims, specification, and drawings of this disclosure are used to distinguish different objects, not to describe a specific order. The terms "comprising" and "including" as used in the specification and claims of this disclosure indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0027] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0028] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection."

[0029] The specific embodiments disclosed herein will now be described in detail with reference to the accompanying drawings.

[0030] Neural networks consist of input layers, convolutional layers, activation functions, pooling layers, and fully connected layers. They can have anywhere from a few to hundreds of layers. Each layer executes an operator; for example, a convolutional layer executes a convolution operator. The number of operators executed depends on the number of layers. In this disclosure, when a specific layer is mentioned, it refers to the operator corresponding to that layer.

[0031] In neural network computation, the input information and the output results of each layer of the model are different in each inference computation. They are considered variable data, which are generally represented by feature maps (matrices). In this disclosure, the input information of the entire neural network model and the input maps of each layer of the model are collectively referred to as feature maps. Once the feature maps are loaded onto the on-chip memory component, they are referred to as on-chip cell maps in this disclosure. The parameters of the trained network model are usually not frequently changed after training is stable, or they can be compiled and generated after the network topology and hardware parameters are determined and do not change during the computation process. Therefore, they can be regarded as constant data. Constant data includes, but is not limited to, weights, biases, device hardware instructions, the mean and variance of batch normalization, etc. In this disclosure, weights are used to represent all constant data. When "data" is mentioned in this disclosure, it generally refers to the graph structure that allows the operation of corresponding operators in the neural network model to be fused together according to the fusion strategy. The variable data and constant data involved in this graph structure are the feature maps plus the corresponding weights.

[0032] Figure 1 A schematic diagram of the structure of a board 10 according to an embodiment of this disclosure is shown. Figure 1As shown, board 10 includes chip 101, which is a system-on-chip (SoC) integrating one or more combined processing units. These combined processing units are artificial intelligence computing units used to support various deep learning and machine learning algorithms, meeting the intelligent processing needs of complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in cloud intelligence. A significant characteristic of cloud intelligence applications is the large volume of input data, placing high demands on the platform's storage and computing capabilities. Board 10 in this embodiment is suitable for cloud intelligence applications, possessing massive off-chip storage, on-chip storage, and substantial computing power.

[0033] Chip 101 is connected to external device 103 via external interface device 102. External device 103 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 103 to chip 101 via external interface device 102. The calculation results from chip 101 can be transmitted back to external device 103 via external interface device 102. Depending on the application scenario, external interface device 102 may have different interface forms, such as a PCIe interface.

[0034] The board 10 also includes a storage device 104 for storing data, which includes one or more memory cells 105. The storage device 104 is connected to and transmits data with the controller 106 and the chip 101 via a bus. The controller 106 in the board 10 is configured to regulate the state of the chip 101. Therefore, in one application scenario, the controller 106 may include a microcontroller (MCU).

[0035] Figure 2 This is a structural diagram illustrating the combined processing device in chip 101 of this embodiment. (As shown) Figure 2 As shown, the combined processing device 20 includes a computing device 201, an interface device 202, a processing device 203, and a DRAM 204.

[0036] The computing device 201 is configured to perform user-specified operations. It is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0037] Interface device 202 is used to transmit data and control commands between computing device 201 and processing device 203. For example, computing device 201 can obtain input data from processing device 203 via interface device 202 and write it to on-chip storage device of computing device 201. Further, computing device 201 can obtain control commands from processing device 203 via interface device 202 and write them to on-chip control cache of computing device 201. Alternatively or optionally, interface device 202 can also read data from storage device of computing device 201 and transmit it to processing device 203.

[0038] Processing device 203, as a general-purpose processing device, performs basic control including but not limited to data transfer, and starting and / or stopping computing device 201. Depending on the implementation, processing device 203 may be one or more types of processors, including but not limited to digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, computing device 201 disclosed herein can be considered as having a single-core structure or a homogeneous multi-core structure. However, when computing device 201 and processing device 203 are considered together, they are considered to form a heterogeneous multi-core structure.

[0039] DRAM 204 is used to store data to be processed. It is DDR memory, typically 16G or larger in size, and is used to store data in computing device 201 and / or processing device 203.

[0040] Figure 3 A schematic diagram of the internal structure of computing device 201 is shown. Computing device 201 is used to process input data such as computer vision, speech, natural language, and data mining. The computing device 201 in the figure adopts a multi-core hierarchical structure design. As a system-on-a-chip, computing device 201 includes multiple clusters, and each cluster includes multiple processor cores. In other words, computing device 201 is constructed in a hierarchical structure of system-on-a-chip, cluster, and processor core.

[0041] From the perspective of system-on-a-chip hierarchy, such as Figure 3As shown, the computing device 201 includes an external storage controller 301, a peripheral communication module 302, an on-chip interconnect module 303, a synchronization module 304, and multiple clusters 305.

[0042] There can be multiple external storage controllers 301; two are shown as an example in the figure. These controllers are used to respond to access requests from the processor core to access external storage devices, such as… Figure 2 The DRAM 204 in the chip allows data to be read from or written to external devices. The peripheral communication module 302 receives control signals from the processing device 203 via the interface device 202, initiating the computing device 201 to execute tasks. The on-chip interconnect module 303 connects the external storage controller 301, the peripheral communication module 302, and multiple clusters 305 to transmit data and control signals between modules. The synchronization module 304 is a global barrier controller (GBC) used to coordinate the working progress of each cluster and ensure information synchronization. The multiple clusters 305 are the computing core of the computing device 201. Four are shown exemplary in the figure; however, with hardware development, the computing device 201 disclosed herein may also include 8, 16, 64, or even more clusters 305. The clusters 305 are used to efficiently execute deep learning algorithms.

[0043] From the perspective of cluster hierarchy, such as Figure 3 As shown, each cluster 305 includes multiple processor cores (IPU cores) 306 and one memory core (MEM core) 307.

[0044] Four processor cores 306 are shown in the figure as an example; this disclosure does not limit the number of processor cores 306. Its internal architecture is as follows: Figure 4 As shown. Each processor core 306 includes three main modules: a control module 41, an arithmetic module 42, and a storage module 43.

[0045] The control module 41 coordinates and controls the operation of the computation module 42 and the storage module 43 to complete the deep learning task. It includes an instruction fetch unit (IFU) 411 and an instruction decode unit (IDU) 412. The instruction fetch unit 411 fetches instructions from the processing device 203, and the instruction decode unit 412 decodes the fetched instructions and sends the decoding result as control information to the computation module 42 and the storage module 43.

[0046] The computation module 42 includes a vector operation unit 421 and a matrix operation unit 422. The vector operation unit 421 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformations; the matrix operation unit 422 is responsible for the core computations of deep learning algorithms, namely matrix multiplication and convolution.

[0047] Storage module 43 is used to store or move related data, including neuron RAM (NRAM) 431, weight RAM (WRAM) 432, input / output direct memory access (IODMA) 433, and move direct memory access (MVDMA) 434. NRAM 431 is used to store feature maps for computation by processor core 306 and intermediate results after computation; WRAM 432 is used to store the weights of the deep learning network; IODMA 433 controls the memory access of NRAM 431 / WRAM 432 and DRAM 204 through broadcast bus 309; MVDMA 434 controls the memory access of NRAM 431 / WRAM 432 and SRAM 308.

[0048] Back Figure 3 The storage core 307 is primarily used for storage and communication, namely storing shared data or intermediate results among processor cores 306, and performing communication between cluster 305 and DRAM 204, communication between clusters 305, and communication between processor cores 306. In other embodiments, the storage core 307 has scalar operation capabilities and is used to perform scalar operations.

[0049] Storage core 307 includes a shared memory unit (SRAM) 308, a broadcast bus 309, a cluster direct memory access (CDMA) module 310, and a global direct memory access (GDMA) module 311. SRAM 308 acts as a high-performance data relay station. Data multiplexed between different processor cores 306 within the same cluster 305 does not need to be obtained from DRAM 204 by each processor core 306 individually. Instead, it is relayed between processor cores 306 via SRAM 308. Storage core 307 only needs to quickly distribute the multiplexed data from SRAM 308 to multiple processor cores 306, thereby improving inter-core communication efficiency and significantly reducing on-chip and off-chip I / O access.

[0050] Broadcast bus 309, CDMA 310, and GDMA 311 are used to perform communication between processor cores 306, communication between clusters 305, and data transfer between cluster 305 and DRAM 204, respectively. These will be explained below.

[0051] The broadcast bus 309 is used to complete high-speed communication between the processor cores 306 within the cluster 305. In this embodiment, the broadcast bus 309 supports inter-core communication methods including unicast, multicast, and broadcast. Unicast refers to point-to-point (i.e., data transmission from one processor core to another) data transmission. Multicast is a communication method that transmits a piece of data from SRAM 308 to several specific processor cores 306. Broadcast is a communication method that transmits a piece of data from SRAM 308 to all processor cores 306, and is a special case of multicast.

[0052] CDMA 310 is used to control SRAM 308 access between different clusters 305 within the same computing device 201. Figure 5 This diagram illustrates the operation of CDMA310 when one processor core attempts to write data to another processor core in a different cluster. In this application scenario, the same computing device comprises multiple clusters. For simplicity, only clusters 0 and 1 are shown in the diagram. Both clusters 0 and 1 contain multiple processor cores; similarly, for ease of explanation, only processor core 0 is shown in cluster 0, and only processor core 1 is shown in cluster 1. Processor core 0 intends to write data to processor core 1.

[0053] First, processor core 0 sends a unicast write request to write data into its local SRAM 0. CDMA 0 acts as the master and CDMA 1 acts as the slave. The master pushes the write request to the slave, that is, the master sends the write address AW and the write data W to transmit the data to SRAM 1 of cluster 1. Then, the slave sends a write response B as a response. Finally, processor core 1 of cluster 1 sends a unicast read request to read the data from SRAM 1.

[0054] Back Figure 3The GDMA 311, in conjunction with the external memory controller 301, controls memory access from the SRAM 308 to the DRAM 204 of the cluster 305, or reads data from the DRAM 204 into the SRAM 308. As previously described, communication between the DRAM 204 and the NRAM 431 or WRAM 432 can be achieved through two channels. The first channel is a direct connection between the DRAM 204 and the NRAM 431 or WRAM 432 via the IODAM 433; the second channel involves first transmitting data between the DRAM 204 and SRAM 308 via the GDMA 311, and then transmitting data between the SRAM 308 and the NRAM 431 or WRAM 432 via the MVDMA 434. Although the second channel appears to require more components and has a longer data flow, in some embodiments, the bandwidth of the second channel is actually much greater than that of the first channel. Therefore, communication between the DRAM 204 and the NRAM 431 or WRAM 432 may be more efficient via the second channel. The embodiments disclosed herein allow for the selection of data transmission channels based on the hardware capabilities available.

[0055] In other embodiments, the functions of GDMA 311 and IODMA 433 can be integrated into the same component. For ease of description, this disclosure treats GDMA 311 and IODMA 433 as different components. For those skilled in the art, any component whose implemented functions and achieved technical effects are similar to those disclosed herein falls within the scope of protection of this disclosure. Furthermore, the functions of GDMA 311, IODMA 433, CDMA 310, and MVDMA 434 can also be implemented by the same component. Similarly, any component whose implemented functions and achieved technical effects are similar to those disclosed herein falls within the scope of protection of this disclosure.

[0056] The neural network structures related to this disclosure are divided into two categories: long-chain structures and block structures. Long-chain structures refer to neural network models composed of layers connected in a single chain, with each layer having only one input and one output, and the overall structure is single-branched, such as the VGG16 model or... Figure 6 The AlexNet model shown is an example. A block structure refers to a subnetwork in a neural network that has only one input and one output, but contains multiple branches. That is, some layers of the subnetwork have multiple inputs or outputs, such as the resblock structure of resnet50 and the block structure of inception_v3. Figure 7A schematic diagram of an exemplary neural network model is shown, comprising subnetwork 701 and subnetwork 702. Subnetwork 701 has only one input and one output, and includes layers one through six. Layer one has two outputs, and layer six has two inputs. Therefore, subnetwork 701 includes two branches: one branch is layer one → layer two → layer three → layer six, and the other branch is layer one → layer four → layer five → layer six. Subnetwork 701 constitutes a block structure. Similarly, subnetwork 702 also constitutes a block structure.

[0057] Performing computations at each layer of deep learning requires numerous off-chip and on-chip accesses, particularly reading input data from DRAM 204 into computing device 201 and storing the computation results from computing device 201 back into DRAM 204. These frequent accesses consume significant hardware resources. To address this issue, this disclosure significantly reduces off-chip and on-chip data transfers by fusing adjacent layers of the neural network.

[0058] Figure 8 This diagram illustrates the fusion of two convolutional layers. The input to the first convolutional layer 810 is a 7×7 feature map 801. This layer convolves feature map 801 with a 3×3 kernel (not shown) to obtain feature map 802 of the first convolutional layer 810. The value of the 5×5 feature submap 804 affects the 3×3 feature submap 805. Assuming a stride of 1, after calculating the 5×5 feature submap 804, the first convolutional layer 810 will then calculate the 5×5 feature submap 806, and the value of the 5×5 feature submap 806 will affect the 3×3 feature submap 807.

[0059] During the computation of the second convolutional layer 811, feature map 802 becomes the input to the second convolutional layer 811, and is also convolved with a 3×3 kernel to obtain feature map 803 of the second convolutional layer 811. The value of the 3×3 feature sub-map 805 affects the 1×1 feature sub-map 808 within feature map 803. After calculating the 3×3 feature sub-map 805, the second convolutional layer 811 then calculates the 3×3 feature sub-map 807, and the value of the 3×3 feature sub-map 807 affects the 1×1 feature sub-map 809 within feature map 803.

[0060] If not merged, during the first convolutional layer 810, the computing device 201 reads a 5×5 feature sub-map 804 from DRAM 204, calculates it, and then stores a 3×3 feature sub-map 805 back into DRAM 204. Next, it reads a 5×5 feature sub-map 806 from DRAM 204, calculates it, and then stores a 3×3 feature sub-map 807 into DRAM 204. During the second convolutional layer 811, it similarly reads a 3×3 feature sub-map 805 from DRAM 204, calculates it, and then stores a 1×1 feature sub-map 808 into DRAM 204. Then, it reads a 3×3 feature sub-map 807 from DRAM 204, calculates it, and then stores a 1×1 feature sub-map 809 into DRAM 204. As can be seen from the above description, feature map 802 is repeatedly read and stored off-chip as intermediate data, which consumes considerable system resources.

[0061] If the first convolutional layer 810 and the second convolutional layer 811 are fused, that is, the feature map 802 is stored in NRAM 431 (the weights of the first convolutional layer 810 and the second convolutional layer 811 can also be stored in WRAM 432), the number of accesses between the computing device 201 and DRAM 204 can be reduced, thereby improving the overall execution efficiency of the neural network. Since the feature maps involved in the fusion (such as feature map 801, feature map 802, and feature map 803) look like an inverted pyramid in the context logic of the neural network model, this is called pyramid fusion.

[0062] Pyramid fusion typically involves fusing specific convolutional and pooling layers from within a neural network. That is, the starting layer is a convolutional or pooling layer, and multiple layers are fused sequentially, potentially including multiple convolutional and pooling layers, depending on the hardware capabilities. However, with the development of deep learning and neural networks, layer ordering has become more complex. For example, if an activation layer precedes a convolutional layer, this activation layer should also be considered for fusion with subsequent convolutional layers. Therefore, in addition to fusion solely based on convolutional and pooling layers, this disclosure offers diverse fusion methods that do not necessarily revolve around them. Instead, it employs specific strategies to flexibly select layers within the neural network for fusion. Even user-defined layers can be fused as long as they conform to the fusion strategy, thus optimizing overall performance.

[0063] Another embodiment of this disclosure is a novel fusion method, which utilizes the aforementioned Figure 1 , Figure 2 , Figure 3 and Figure 4This fusion is implemented using a hardware architecture, and such fusion is called a template fuse unit (TFU). The template fuse unit mainly uses a certain fusion strategy to flexibly fuse multiple layers into one layer to reduce the network's input / output overhead. This includes the aforementioned pyramid fusion and other fusion methods. The collection of these fused layers is the template fuse unit, which can be regarded as a new layer or a custom layer.

[0064] In this embodiment, the feature maps and weights required by the template fusion unit are loaded from DRAM 204 to the on-chip SRAM 308 in one go. The feature maps loaded into SRAM 308 are called on-chip cell maps. The on-chip cell maps are divided into sub-maps. Each time, a sub-map is loaded from SRAM 308 into the NRAM 431 of the processor core 306 assigned to calculate the sub-map, and the weights required to calculate the sub-map are also loaded from SRAM 308 into WRAM 432. After each sub-map is calculated, the corresponding intermediate result is obtained and stored back into SRAM 308. After all sub-maps have been calculated, the calculation results are stored back into DRAM 204 all at once. In other words, the corresponding results obtained by the on-chip cell maps and weights participating in the operator operations in the neural network model are transferred between DRAM 204 and SRAM 308, and the outputs (intermediate results) corresponding to the sub-maps are transferred between SRAM 308 and NRAM 431. From the perspective of the computing device 201, the data loading of the template fusion unit is done on a unit basis (on-chip unit graph), while the computation is done on a subgraph basis.

[0065] More specifically, SRAM 308 is one of the key reference indicators for the fusion strategy, and its size determines whether the template fusion cell is in large-image mode or small-image mode. Small-image mode and large-image mode refer to whether a feature map stored in DRAM 204 can be moved to SRAM 308 for processing at once. The processing device 203 compares the storage space required for the feature map with the available space in SRAM 308. If SRAM 308 has insufficient space and the feature map cannot be accommodated, it is in large-image mode; if SRAM 308 is large enough to accommodate the entire feature map, it is in small-image mode. It is important to note that in large-image mode, the on-chip cell map is only a part of the feature map; in small-image mode, if the available space in SRAM 308 is large enough, or the feature map is small enough, SRAM 308 may be able to accommodate multiple feature maps at once, meaning the on-chip cell map can include multiple feature maps.

[0066] If it is a large image mode, the feature map must be split before it can be loaded into the computing device 201. The processing device 203 will split the feature map on DRAM 204 until a sufficiently small on-chip cell map is generated to meet the space requirements of SRAM 308, so that the on-chip cell map can be moved to SRAM 308 for processing at once. During the splitting of the feature map, input-dependent operations and output-dependent operations may occur.

[0067] Input-dependent computations refer to situations where the split on-chip unit graphs at least partially overlap, and each subset requires some additional copies of the input for a complete computation. This leads to data redundancy in the splitting operation, where the same data is reused within the system. Input-dependent computations occur when template fusion units include layers such as convolution, pooling, or matrix multiplication.

[0068] Output-dependent operations refer to the process where, after each subgraph produces intermediate results, a reduction process is required to obtain the final computational result. Reduction involves breaking down the on-chip unit graph into subgraphs based on an understanding of its content, and then calculating each subgraph separately to reduce the computational scale. This minimizes the amount of data while preserving the original on-chip unit graph as much as possible, and then uses the subgraphs as a basis to restore or integrate the computational results. The computational results during reduction are interdependent. Output-dependent operations occur when the template fusion unit includes layers such as inner product, convolution, matrix multiplication, sorting, and counting.

[0069] This embodiment can process feature map data formats with dimensions N, H, W, and C, where N represents batch, H represents height, W represents width, and C represents channels. Taking image data as an example, N indicates the total number of images in the batch, H indicates the number of pixels in the vertical direction, W indicates the number of pixels in the horizontal direction, and C indicates the number of channels (e.g., a black and white image has 1 channel, while an RGB color image has 3 channels).

[0070] The ordering of these dimensions determines how the data is composed; common composition methods include NHWC and NCHW. Figure 9This diagram illustrates the difference between NCHW and NHWC formats, using an RGB color image as an example. In the diagram, R represents red pixels, G represents green pixels, and B represents blue pixels. Sequence 91 is in NCHW format, with N arranged on the outermost layer. Pixels within each channel are placed side-by-side and arranged in RGB order. The element with coordinates (n, c, h, w) has an offset in storage of ((n × C + c) × H + h) × W + w. Sequence 92 is in NHWC format, with C arranged on the innermost layer. RGB pixels corresponding to multiple channels are placed side-by-side. The diagram also shows the positions of input pixels 901, 902, and 903 under different arrangements. These three input pixels (901, 902, and 903) together represent the color of a point in the image. The coordinate offset of the element with coordinates (n, c, h, w) is calculated as ((n × H + h) × W + w) × C + c. NHWC is closer to the BMP image data storage format than NCHW. BMP files store data pixel by pixel, with each pixel containing the color values ​​of all channels. This eliminates the need for additional dimensionality transformations when reading input images. Therefore, NHWC offers better memory locality, producing one output pixel for every three input pixels. NCHW, on the other hand, requires all input channels to be ready before obtaining the final output, consuming more cache space.

[0071] This embodiment can use each layer of the data fusion neural network as a template fusion unit. Figure 10 The corresponding flowchart is shown.

[0072] In step 1001, the processing device 203 determines whether the storage space required for the feature map is greater than the available space of the SRAM 308. If so, it means that the feature map cannot be loaded into the SRAM 308 at once, so step 1002 is executed to split the feature map. In this embodiment, the processing device 203 preferentially chooses to split in the N-dimensional direction because there will be no input or output dependent operations. If splitting in the N-dimensional direction cannot meet the requirements, splitting in the H or W-dimensional direction will be considered, which may generate input or output dependent operations. This embodiment also supports splitting in the C-dimensional direction, especially splitting along the Cout direction. This splits a convolution into multiple convolutions through data optimization, so that the WRAM 432 can accommodate the weights, for example, splitting the weights onto four processor cores 306. Therefore, as long as splitting in a certain dimension is within the scope of this disclosure, it is within the scope of the computing device 201.

[0073] Furthermore, the processing device 203 can sequentially perform splitting at a specific granularity across the N, H, and W dimensions. This specific granularity can be a fixed or variable ratio, or represented by a function. In one application scenario, the processing device 203 splits the feature map or weights from largest to smallest. Taking a feature map as an example, firstly, in the N dimension, the feature map of dimension NHWC is split into feature maps of dimension N1HWC and N2HWC, where the specific granularity is a fixed ratio, with N1 and N2 each being half of N. If this is still not small enough, the processing device 203 continues to split the feature map of dimension N1HWC into feature maps of dimension N1H1WC and N1H2WC in the H dimension, where H1 and H2 each are half of H. If the feature map is still not small enough, the processing device 203 continues to split the N1H1WC feature map into N1H1W1C and N1H1W2C feature maps along the W dimension, where W1 and W2 are each half of W. The processing device 203 can continue to perform smaller-granularity splits along the N, W, and H dimensions, such as dividing it into quarters, eighths, or sixteenths, until the feature map is small enough to become an on-chip cell map that can be loaded into the SRAM 308 in one go.

[0074] Understandably, the processing device 203 can continue to split along one dimension until it can no longer split, and then it will choose another dimension to continue splitting. For example, it can continue to split along the H dimension, and if splitting to the smallest unit still cannot be loaded into the SRAM 308, it will then switch to splitting along the W dimension until splitting to the smallest unit.

[0075] It is important to note that because this splitting method proceeds from large to small, the required storage space for each split feature map, when it meets the conditions, is usually almost the same as the available space in SRAM 308. In other words, in large-map mode, DRAM 204 can only transfer one split feature map to SRAM 308 at a time, but in small-map mode, SRAM 308 may be able to load multiple feature maps from DRAM 204 at once.

[0076] In another application scenario, the processing unit 203 divides the feature map from small to large. The specific granularity can also be a fixed or variable ratio, or represented by a function. For example, it first divides the feature map in the N-dimensional space using a specific granularity as the smallest unit, i.e., 1×H×W×C. If the SRAM 308 can load the feature map, the processing unit 203 continues to enlarge the feature map division, for example, to 2×H×W×C. If it can still be loaded, it continues to enlarge until n×H×W×C can no longer be loaded. The size of the on-chip cell map is then (n-1)×H×W×C.

[0077] If the required storage space for 1×H×W×C exceeds the available space of SRAM 308, the processing device 203 will continue to split it from another dimension, for example, starting from the H dimension. The processing device 203 will then determine 1×1×W×C. If it is small enough, it will increase along the H dimension upwards until it finds a storage space required for 1×(h-1)×W×C that is close to but not greater than the available space of SRAM 308. If it still exceeds the available space of SRAM 308, the processing device 203 will continue to split it from another dimension, for example, from the W dimension. This process continues until the optimal input data that can be loaded into SRAM 308 at once is found. Here, "optimal" refers to the storage space required by the on-chip cell diagram that is closest to but not greater than the available space of SRAM 308.

[0078] After the processing device 203 splits the feature map, it returns to step 1001. The processing device 203 determines whether the storage space required for the split feature map is still greater than the available space of SRAM 308. If so, it executes step 1002 again to continue splitting.

[0079] If the processing device 203 determines that the storage space required for the split feature map is not greater than the available space of the SRAM 308, it means that the SRAM 308 can load the split feature map at once, and then executes step 1003, whereby the processing device 203 sets the split feature map as an on-chip cell map.

[0080] Finally, in step 1004, the processing unit 203 determines the template fusion unit based on the dimensions of the on-chip unit diagram. This step will be described in detail later.

[0081] In other application scenarios, after the processing device 203 repeatedly executes steps 1001 and 1002 multiple times, it indicates that the storage space required by the split feature map is getting closer and closer to the available space of the SRAM 308. For example, assuming the storage space required by the feature map is 100k and the available space of the SRAM 308 is 40k, in step 1001, the processing device 203 determines that the storage space required by the feature map is greater than the available space of the SRAM 308, so it executes step 1002, splitting it in half along the N-dimensional axis. At this time, the split feature map is 50k. Then it returns to step 1001, and the storage space required by the split feature map is still greater than the available space of the SRAM 308. It continues to execute step 1002, splitting it in half again along the N-dimensional axis. At this time, the split feature map is 25k. Then it returns to step 1001, and the storage space required by the split feature map is less than the available space of the SRAM 308, so it executes step 1003, and the processing device 203 sets the split feature map (size 25k) as an on-chip cell map.

[0082] The SRAM 308 has 40k of available space, while the on-chip cell diagram requires 25k of storage space, leaving 15k of space unused. This is because step 1002 splits the data in units of half, resulting in a granularity that is too large in the final split. This embodiment can gradually reduce the specific granularity of the split as the number of splits increases, making the required storage space for the split on-chip cell diagram as close as possible to the available space of the SRAM 308. For example, the specific granularity can initially be set to half, then three-quarters, and finally four-fifths. Taking a feature map requiring 100k of storage space and SRAM 308 having 40k of available space as an example, in step 1001, the processing device 203 determines that the storage space required by the feature map is greater than the available space of SRAM 308, so it executes step 1002, setting the specific granularity to half, resulting in a split feature map of 50k. Returning to step 1001, the storage space required by the split feature map is still greater than the available space of SRAM 308, so step 1002 is executed again, this time with the specific granularity adjusted to three-quarters, resulting in a split feature map of 37.5k. Returning to step 1001 again, the storage space required by the split feature map is less than the available space of SRAM 308, so step 1003 is executed, and the processing device 203 sets the split feature map (37.5k in size) as an on-chip cell map. 37.5k is closer to 40k than 25k, and the latter method makes better use of the available space of SRAM 308, resulting in higher efficiency. This embodiment does not limit the size of a specific granularity and can be set according to the application scenario.

[0083] After determining the size of the on-chip unit graph, step 1004 is executed. This step dynamically fuses the neural network according to the fusion strategy. Figure 11 This embodiment illustrates a method for dynamically fusing neural networks according to a fusion strategy.

[0084] In step 1101, the starting layer of the template fusion unit is selected according to the starting rules of the fusion strategy. The processing device 203 selects the starting layer of the template fusion unit according to the starting rules of the fusion strategy, that is, selects the layer to start fusion from the layers that have not yet been fused in the neural network.

[0085] In one application scenario, the starting rule could be that the starting layer is the first unfused layer in the neural network, and the processing device 203 will search for the first unfused layer. Figure 6 Taking the AlexNet neural network model as an example, it has 23 layers. Assuming that layers 1 to 5 have been fused, when the starting rule is that the starting layer is the first unfused layer in the neural network, the processing device 203 will select the ReLU activation layer of layer 6 as the starting layer and fuse backward (i.e., towards layer 7). It should be noted that under this starting rule, the starting layer is not necessarily a convolutional layer or a pooling layer.

[0086] In another application scenario, considering that convolutional and pooling layers consume the most input / output resources, the starting rule is that the initial layer is the first unfused convolutional or pooling layer. The processing device 203 will first identify all unfused convolutional and pooling layers in the neural network model and start fusing them from the first unfused convolutional or pooling layer. Similarly, Figure 6 Taking the AlexNet neural network model as an example, assuming that layers 1 to 9 have been fused, the processing device 203 will find all the unfused convolutional and pooling layers in the neural network model, namely layers 11, 13, and 15, and then start fusion from the first unfused convolutional or pooling layer, that is, the starting layer is layer 11.

[0087] In step 1102, fusion is performed based on the initial layer, and all rules of the fusion strategy are checked one by one to establish a template fusion unit. The processing device 203 performs fusion based on the initial layer, checking all rules of the fusion strategy one by one to establish a template fusion unit. Provided all rules are satisfied, the hardware resources of the computing device 201 are sufficient to support loading the data required to compute the template fusion unit at once, and then performing neural network computation based on the template fusion unit. In addition to the aforementioned initial rules, the fusion strategy may, exemplarily, include the following rules:

[0088] Rule 1: Backward fusion

[0089] Backward fusion refers to fusion from the initial layer towards the inference direction of the neural network model. Figure 6 For example, the fusion proceeds in the direction of first layer → second layer → third layer. If there are unfused layers before the starting layer, these unfused layers will not be considered in the template fusion unit under this rule.

[0090] Rule 2: Prioritize forward merging

[0091] Forward fusion refers to fusion in the reverse direction from the initial layer towards the inference layer of the neural network. Figure 6 For example, the fusion proceeds in the direction of the third layer → the second layer → the first layer. This rule is usually paired with the aforementioned starting layer rule, where the first unfused convolutional or pooling layer is the initial layer, because there may be unfused layers before the convolutional or pooling layer. After selecting the starting layer, the processing device 203 prioritizes forward fusion, attempting to incorporate the unfused layers before the starting layer into the template fusion unit. Similarly, Figure 6 Taking the AlexNet neural network model as an example, assuming that the first and second layers have been fused, the processing device 203 finds that the first unfused convolutional or pooling layer is the fifth layer, so the starting layer is the fifth layer. It prioritizes fusing the fourth and third layers forward. If it can continue to fuse, it then continues to fuse the sixth and seventh layers, etc.

[0092] Rule 3: Prioritize block structure.

[0093] When the neural network model has a block structure, this rule requires the processing device 203 to prioritize adding or removing template fusion units on a block-by-block basis rather than on a layer-by-layer basis. Only if the fusion of the computational logic of an entire block fails will fusion be considered from the layers of each branch. Figure 7 Taking a neural network model as an example, the processing device 203 will prioritize the fusion of subnetworks 701 or 702 as units.

[0094] When the neural network has a long chain structure, since there is no block structure, template fusion units are added or removed directly on a layer-by-layer basis. This rule does not apply to neural network models with long chain structures.

[0095] Rule 4: Single-branch output

[0096] The fusion strategy in this embodiment does not support template fusion units as multi-output networks. The reason is that the shape derivation implemented inside the template fusion unit mainly adopts the form of back-to-forward derivation. Multi-output networks mean that derivation needs to be performed from different outputs, and the derivation results will not necessarily be reduced to the same feature map, so convergence is impossible.

[0097] In other words, the output of the template fusion unit must be a single-branch output, meaning that the last layer of the template fusion unit can only have one output. Figure 7 Two fusion methods for subnetwork 701 are indicated. The first method merges layers 1 to 5 into a single template fusion unit 703, and the second method merges layers 1 to 6 into a single template fusion unit 704. Since the outputs of layers 3 and 5 are both outputs of template fusion unit 703, template fusion unit 703 is a multi-output network, i.e., multi-branch output. The output of layer 6 is the output of template fusion unit 704, producing only one output data; therefore, template fusion unit 704 is a single-output network, i.e., single-branch output. Processing unit 203 determines whether the output of the template fusion unit is a single-branch output. If this rule is not satisfied, processing unit 203 adds or removes layers within the template fusion unit until the rule is satisfied.

[0098] Rule 5: Includes at least 2 main layers

[0099] When the layer logic is too simple, the performance of the template fusion unit is worse than that of the unfused layers. Therefore, when using layer logic as the fusion strategy, the processing device 203 evaluates whether the operations of each fused layer are complex enough to generate benefits from the fusion. To generate benefits, the main layers should be included in the template fusion unit as much as possible. The main layers refer to layers that consume a lot of input / output resources, such as matrix multiplication, pooling, or convolution. Pooling here includes various types of pooling, such as max pooling or average pooling, and convolution also includes various types of convolution, such as ordinary convolution, convolution with average, and depthwise convolution. This rule requires that the template fusion unit includes at least two main layers. When the processing unit 203 determines that this rule is not met, the processing device 203 will adjust the template fusion unit until the rule is met.

[0100] Rule 6: Includes a continuous structure with the main layer, the main layer, and non-main layers adjacent in sequence.

[0101] This rule requires that template fusion units must include a continuous structure of main layers, main layers, and non-main layers; that is, a continuous structure where main layers, main layers, and non-main layers are sequentially adjacent. Such computation is complex enough to make fusion efficient. See also... Figure 6 The template fusion unit consists of layers 4, 5, and 6, where layer 4 is a max pooling layer, layer 5 is a convolutional layer, and layer 6 is a ReLU activation layer. This conforms to a continuous structure where the main layers and non-main layers are sequentially adjacent. Therefore, the template fusion unit including layers 4, 5, and 6 satisfies this rule. When the processing unit 203 determines that this rule is not satisfied, the processing device 203 adjusts the template fusion unit until the rule is satisfied.

[0102] Rule 7: Includes adjacent, continuous structures of scalar computation layers and vector computation layers.

[0103] This rule refers to a template fusion unit comprising a continuous structure of scalar computation layers and vector computation layers, i.e., a continuous structure where scalar computation layers and vector computation layers are sequentially adjacent. The scalar computation layer refers to an addition layer, subtraction layer, or multiplication layer, and the vector computation layer refers to an activation layer, batch normalization layer, or scaling layer. When processing unit 203 determines that this rule is not satisfied, processing device 203 will adjust the template fusion unit until the rule is satisfied.

[0104] Rule 8: The weights of a convolutional layer are not equal to the output of any other layer.

[0105] This rule states that the weights of the convolutional layers in the template fusion unit are not the outputs of any layer in the neural network, regardless of whether that layer is included in the template fusion unit. When the processing unit 203 determines that this rule is not met, the processing device 203 will remove this convolutional layer from the template fusion unit.

[0106] Rule 9: The weights of a convolutional layer should not be shared with any other layer in the neural network.

[0107] Because the weights of operators in the neural network model involved in the template fusion unit have a special arrangement, when the convolutional operator being fused shares weights with other operators, a conflict will occur in the weight arrangement logic. The rule is that the weights of the convolutional operators in the template fusion unit should not be shared with any layer of the neural network. When the processing unit 203 determines that this rule is not satisfied, the processing device 203 will remove this convolutional operator from the template fusion unit.

[0108] Rule 10: The weight is no greater than the available space of WRAM.

[0109] The large-image mode imposes fewer restrictions on WRAM 432 because the on-chip cell map loaded into SRAM 308 is only a part of the feature map. When calculating the template fusion unit, WRAM 432 only needs to store all the weights of that feature map. However, since the small-image mode may load multiple feature maps into SRAM 308, the required weights will increase, requiring careful evaluation of whether the available space of WRAM 432 is sufficient. The rule is that the storage space required for the weights in the on-chip cell map should not exceed the available space of WRAM 432. When the processing device 203 determines that this rule is not met, it will reduce the size of the on-chip cell map.

[0110] If the weights are split based on the output channel parameter Cout of dimension C, since the weights will be evenly distributed across multiple processor cores 306, this rule is adjusted as follows:

[0111]

[0112] Among them, W j denoted as , where is the storage space required for the weights involved in the on-chip cell graph j, n is the number of processor cores in the cluster, and W is the available space of WRAM 432.

[0113] Rule 11: Redundancy Percentage

[0114] The redundancy percentage is the ratio of the total redundancy generated by input-dependent operations and output-dependent operations to the normal input / output of the template fusion unit. Here, normal input / output refers to the amount of data in the on-chip cell graph that has no redundancy before it is split. The processing device 203 calculates the memory access size from DRAM 204 to SRAM 308 in the on-chip cell graph after the template fusion unit incorporates the current layer. TFU Size of normal input / output (excluding redundancy) ori The percentage of memory accesses, where size TFU This refers to the theoretical memory access size. oriAdd the redundancy sum. The formula is as follows:

[0115]

[0116] The processing device 203 incorporates the splitting information and shape derivation of the template fusion unit and sets a percentage threshold of 50%, 75%, 100%, 125%, or 150%, preferably 100%. For example, a percentage threshold of 100% means that fusion will cease when the total redundancy exceeds twice the normal input / output of the template fusion unit. This rule ensures that the total redundancy generated by splitting the on-chip unit diagram does not exceed a specific proportion related to the percentage threshold. If it does, it indicates excessive redundancy, and a large amount of resources will be consumed in calculating this redundancy, leading to decreased efficiency. Therefore, when the processing device 203 determines that this rule is not met, it will stop fusion.

[0117] It should be noted that in small-map mode, no redundancy occurs because at least one complete feature map is loaded at a time during the process from DRAM 204 to SRAM 308. This rule does not apply to small-map mode.

[0118] Rule 12: Input / output dimensions of on-chip unit diagram

[0119] Assuming the SRAM 308 has a space size of S, the on-chip cell diagram requires storage space IN, and the calculation result of the on-chip cell diagram requires storage space OUT, then the SRAM 308's space size must meet the following conditions:

[0120] If IN and OUT cannot reuse storage space, then IN+OUT

[0121] If IN and OUT can reuse storage space, then MAX(IN,OUT)

[0122] That is, if IN and OUT cannot reuse storage space, the sum of the storage space of the on-chip cell diagram and the storage space of the calculation result is less than the available space of SRAM 308; if IN and OUT can reuse storage space, the larger of the storage space of the on-chip cell diagram and the storage space of the calculation result is less than the available space of SRAM 308.

[0123] Rule Thirteen: W i +IN1+IN2≤S

[0124] In small-image mode, this rule requires the SRAM 308's space size to meet the following conditions:

[0125] W i +IN1+IN2≤S

[0126] ​​That is, the storage space W required for the weights of subgraph i. i The total storage space IN1 and cache space IN2 required by the on-chip cell diagram shall not exceed the available space of SRAM 308. When the processing device 203 determines that this rule is not met, the processing device 203 reduces the number of on-chip cell diagrams until the rule is met.

[0127] Rule Fourteen: SubINi+W i +IN2≤S

[0128] In small-image mode, this rule requires the SRAM 308's space size to meet the following conditions:

[0129] SubINi+W i +IN2≤S

[0130] That is, the required storage space SubINi for subgraph i and the required storage space W for the weights of subgraph i. i The sum of the cache space IN2 is not greater than the available space of SRAM 308. When the processing device 203 determines that this rule is not met, the processing device 203 reduces the number of on-chip cell diagrams until the rule is met.

[0131] Rule 15: SubOUTi+W i+1 +IN2≤S

[0132] In small-image mode, this rule requires the SRAM 308's space size to meet the following conditions:

[0133] SubOUTi+W i+1 +IN2≤S

[0134] That is, the storage space required for the intermediate result of subgraph i (SubOUTi) and the storage space required for the weights of the next subgraph (W). i+1 The sum of the cache space IN2 is not greater than the available space of SRAM 308. When the processing device 203 determines that this rule is not met, the processing device 203 reduces the number of on-chip cell diagrams until the rule is met.

[0135] Rule Sixteen: W i +W i+1 ≤W

[0136] In template fusion units, the weights involved in convolution operations are independently moved and reside in WRAM 432. In small-graph mode, if a subgraph contains multiple feature maps, considering the pipeline between subgraphs, WRAM 432 can store the weights of at most two adjacent subgraphs simultaneously. Assume the required storage space for each subgraph i is W. i And the total space of WRAM 432 is W. This rule requires that the space size of WRAM 432 must meet the following conditions:

[0137] W i +W i+1 ≤W

[0138] That is, the storage space W required for the weights of subgraph i. i The storage space W required for the weights of the next subgraph i+1 The total number of cells must not exceed the available space of WRAM 432. When the processing device 203 determines that this rule is not met, the processing device 203 reduces the number of on-chip cell diagrams until the rule is met.

[0139] Rule 17: The storage space required for a subgraph shall not exceed the available space of NRAM.

[0140] This rule states that the storage space required for a subgraph should not exceed the available space of NRAM 431. When the on-chip cell graph on SRAM 308 needs to be split into subgraphs and moved to NRAM 431, processing device 203 can perform fine-grained splitting along the N, H, and W dimensions. If the space in NRAM 431 is insufficient, processing device 203 will split the on-chip cell graph into finer subgraphs until this rule is met. Generally, NRAM 431 will have a reasonable amount of available space, allowing the on-chip cell graph to be split to a reasonable extent and loaded at once. From the perspective of the fusion strategy, the template fusion unit is not affected by the batch size. However, the smaller the on-chip cell graph is split (i.e., the more subgraphs), the slower the processing speed will be. Therefore, processing device 203 needs to evaluate the space of NRAM 431.

[0141] In some embodiments, the space of SRAM 308 corresponds to the number of NRAM 431 units in the processor cores 306 within cluster 305. For example, if cluster 305 includes four processor cores 306, then the space of SRAM 308 is four times the space of NRAM 431. In other words, on-chip units in large-image mode. Figure 1 Typically, data can be allocated to four processor cores (306). This architecture design already considers that data loaded into SRAM (308) can be allocated to all NRAM (431) cores at once. Therefore, this rule does not need to be considered in large-graph mode.

[0142] Rule 18: The number of feature maps should not exceed the feature map threshold.

[0143] In small-map mode, an on-chip unit graph may include multiple feature maps. The more feature maps there are, the more sub-graph transfers occur between SRAM 308 and NRAM 431, leading to decreased efficiency. Therefore, it is not always better for an on-chip unit graph to include more feature maps. The processing device 203 calculates an appropriate number of fusion layers based on the number of feature maps in the on-chip unit graph to maximize its efficiency. This rule is that the number of feature maps in the on-chip unit graph should not exceed a feature map threshold. When the processing device 203 determines that this rule is not met, it reduces the number of feature maps in the on-chip data until the rule is met.

[0144] Rule 19: Step Redundancy

[0145] Stride redundancy refers to the situation where, when a template fusion unit (SFU) fuses too many layers, and the dimensions of the convolutional and pooling kernels exceed the stride, the input data required for each output point overlaps. This overlap constitutes the aforementioned input-dependent computation, and this overlapping portion is stride redundancy. Stride redundancy forces each processor core 306 to read more data, but this reused data consumes on-chip and off-chip access resources. The more layers a SFU includes, the more severe the stride redundancy becomes. The rule is that the sum of the differences between the side length and stride of the convolutional or pooling kernel should not exceed the redundancy threshold.

[0146] In this embodiment, the redundancy threshold is defined as follows. Assume the length and width of the kernels for the convolutional and pooling layers are k. x and k y The step sizes in the length and width directions are s respectively. x and s y The stride redundancy in the longitudinal direction is k of all convolutional and pooling layers within the template fusion unit. x -s x The sum of all convolutional and pooling layers within the template fusion unit; similarly, the stride redundancy in the width direction is k. y -s y The sum of the values. The redundancy threshold in this embodiment can be 3, 4, 5, or 6, preferably 4. This rule is not satisfied as long as the step size redundancy in either the longitudinal or lateral direction is greater than the redundancy threshold. The processing device 203 adjusts the template fusion unit, typically by reducing the number of layers to be fused, until this rule is satisfied.

[0147] The fusion strategy sets an exception rule for step redundancy. If the layer to be fused contains multiple branches and the template fusion unit can fuse all multiple branches, the template fusion unit will perform better. In this case, the processing device 203 will ignore the step redundancy rule; that is, step redundancy will not restrict the template fusion unit from fusing multiple branches. In this embodiment, fusing multiple branches takes precedence over the restriction of step redundancy. In other words, step redundancy is only considered in the case of a single branch.

[0148] The above rules are merely examples. This disclosure does not restrict the order in which the rules are executed, nor does it restrict that these rules must be considered simultaneously. Those skilled in the art can add or delete rules according to the actual situation in different application scenarios to achieve a fusion strategy that conforms to the current application scenario.

[0149] Back Figure 11 In step 1103, neural network computation is performed based on the established template fusion unit. The computing device 201 is based on a three-level operation hierarchy of system-on-chip-cluster-processor core, and is equipped with a three-layer memory design of DRAM-SRAM-NRAM / WRAM. The template fusion unit is regarded as a custom layer in the neural network. The data required for computing the template fusion unit is loaded from DRAM 204 to SRAM 308 at one time, so that the data can be cached and computed at the appropriate level, forming a sufficient pipeline. After the computation is completed, the computation result is transferred from SRAM 308 to DRAM 204, which greatly reduces the input / output overhead in neural network computation.

[0150] When input data from fields such as computer vision, speech, natural language processing, and data mining is to be processed by various deep learning and machine learning algorithms, this disclosure, based on template fusion units, can reduce the input / output overhead in neural network computation. Another embodiment of this disclosure is a method for performing neural network computation using template fusion units. Figure 12 The process is shown.

[0151] In step 1201, a template fusion unit is determined according to the fusion strategy. The processing device 203 selects the starting layer of the template fusion unit based on the starting rule of the fusion strategy; and performs fusion based on the starting layer, checking all rules of the fusion strategy one by one to establish the template fusion unit. The previous embodiment has already illustrated the various rules of the fusion strategy in detail, and will not be repeated here.

[0152] In this step, the template fusion unit is presented as source code. The compiler then needs to convert this source code into machine language object code, also known as machine code. The following steps describe the process by which the compiler converts the source code of the template fusion unit into machine language object code.

[0153] In step 1202, the shape of the template fusion unit is derived. For the data that the template fusion unit needs to process, this embodiment uses a backward deduction method; the compiler works backward from the output to deduce the required input size. Figure 8For example, the process involves reverse deriving from feature map 803 to feature map 802, and then reverse deriving to feature map 801. In this step, the compiler not only derives the required input data based on the template fusion unit, but also further derives redundancy.

[0154] Next, step 1203 is executed to deduce the addresses. Based on the shape of the template fusion unit, the compiler deduces the on-chip memory addresses for the entire control flow graph and implements general-purpose address access to achieve the goal of simplifying computational resources and shortening computation time. A control flow graph is an abstract data structure used in compilers, representing all possible execution paths of a program, reflecting the possible flow of all nodes within the process in the form of a flowchart. A control flow graph is composed of nodes and the relationships between them. A node, also called a basic block (BB), is a sequence of statements executed as sequentially as possible in a program. Each basic block has only one entry and one exit point; execution begins at its entry point and exits at its exit point. A key characteristic of a basic block is that once the first instruction is executed, all instructions within that block will be executed sequentially.

[0155] Each basic block contains at least one instruction, and the instructions within a basic block may use pointers to point to specific on-chip memory locations. A pointer is a variable used to store an address in a specific address space. Through pointers, the processor core 306 can load data into or retrieve data from the specific address space pointed to by the pointer.

[0156] The compiler initially divides the basic blocks according to the partitioning of the template fusion unit, and then confirms the basic blocks and their interrelationships after iterative calculations, thus completing the target code for implementing the template fusion unit.

[0157] Furthermore, the compiler analyzes the reused data between consecutive template fusion units in the neural network, determines how much data from the previous template fusion unit can remain on-chip for use by the next template fusion unit, and plans the storage address of each data based on the determination result.

[0158] In this step, the compiler completes the derivation of addresses in the control flow graph.

[0159] In step 1204, on-chip memory space is allocated. Processing device 203 allocates physical space for SRAM 308, NRAM 431, and WRAM 432 based on the derivation of the template fusion cell address. In this step, the compiler completes the pointer assignment in the control flow graph.

[0160] Finally, step 1205 is executed to generate executable instructions. In this step, the linker links the object code generated by the compiler with external libraries, making it an executable file. More specifically, the object code is a program module that includes machine code and information available to the linker. The linker's job is to resolve undefined symbol references, replace placeholders in the object code with the addresses of symbols, and thus generate executable instructions. The executable instructions can be directly executed by the computing device 201 to complete the computation of the neural network.

[0161] Another embodiment of this disclosure is a computer-readable storage medium storing computer program code that dynamically fuses neural networks according to a fusion strategy. When the computer program code is run by a processor, it executes... Figure 10 , Figure 11 , Figure 12 The method described.

[0162] This disclosure dynamically determines the template fusion unit by setting a fusion strategy, fusions multiple layers in the neural network to form a new custom layer, and loads the data required to compute the template fusion unit at once to reduce input / output overhead.

[0163] Depending on the application scenario, the electronic devices or apparatus disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablets, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, dashcams, navigators, sensors, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, autonomous driving terminals, vehicles, home appliances, and / or medical devices. The vehicles include airplanes, ships, and / or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs. The electronic devices or apparatus disclosed herein can also be applied in fields such as the Internet, IoT, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and healthcare. Furthermore, the electronic devices or apparatus disclosed herein can also be used in application scenarios related to artificial intelligence, big data, and / or cloud computing, such as cloud computing, edge computing, and terminal applications. In one or more embodiments, the high-computing-power electronic devices or apparatuses according to the present disclosure can be applied to cloud devices (e.g., cloud servers), while the low-power electronic devices or apparatuses can be applied to terminal devices and / or edge devices (e.g., smartphones or cameras). In one or more embodiments, the hardware information of the cloud devices and the hardware information of the terminal devices and / or edge devices are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud devices to simulate the hardware resources of the terminal devices and / or edge devices based on the hardware information of the terminal devices and / or edge devices, so as to complete the unified management, scheduling and collaborative work of end-to-cloud or cloud-edge-end integration.

[0164] It should be noted that, for the sake of brevity, this disclosure describes some methods and their embodiments as a series of actions and combinations thereof. However, those skilled in the art will understand that the solutions disclosed herein are not limited by the order of the described actions. Therefore, based on the disclosure or teachings of this document, those skilled in the art will understand that some steps can be performed in a different order or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in this disclosure can be considered optional embodiments, that is, the actions or modules involved are not necessarily essential for the implementation of one or more solutions disclosed herein. In addition, depending on the solution, the description of some embodiments in this disclosure may have different emphases. In view of this, those skilled in the art will understand that parts not described in detail in a certain embodiment of this disclosure can also be referred to the relevant descriptions of other embodiments.

[0165] In terms of specific implementation, based on the disclosure and teachings of this document, those skilled in the art will understand that several embodiments disclosed herein can also be implemented in other ways not disclosed herein. For example, regarding the various units in the electronic device or apparatus embodiments described above, this document has divided them based on logical functions, but in actual implementation, there may be other ways of division. As another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. Regarding the connection relationships between different units or components, the connections discussed above in conjunction with the accompanying drawings can be direct or indirect couplings between units or components. In some scenarios, the aforementioned direct or indirect couplings involve communication connections utilizing interfaces, where the communication interface can support electrical, optical, acoustic, magnetic, or other forms of signal transmission.

[0166] In this disclosure, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network units. Furthermore, depending on actual needs, some or all of the units can be selected to achieve the purpose of the solution described in the embodiments of this disclosure. Additionally, in some scenarios, multiple units in the embodiments of this disclosure may be integrated into one unit or each unit may exist physically independently.

[0167] In some implementation scenarios, the integrated unit described above can be implemented as a software program module. If implemented as a software program module and sold or used as an independent product, the integrated unit can be stored in a computer-readable storage device (CMSDD). Therefore, when the disclosed solution is embodied in a software product (e.g., a computer-readable storage medium), the software product can be stored in a memory, which may include several instructions to cause a computer device (e.g., a personal computer, server, or network device) to execute some or all of the steps of the method described in the embodiments of this disclosure. The aforementioned memory may include, but is not limited to, various media capable of storing program code, such as USB flash drives, flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0168] In other implementation scenarios, the integrated units described above can also be implemented in hardware, i.e., as specific hardware circuits, which may include digital circuits and / or analog circuits. The physical implementation of the circuit's hardware structure may include, but is not limited to, physical devices, which may include, but are not limited to, transistors or memristors. Therefore, the various devices described herein (e.g., computing devices or other processing devices) can be implemented using appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage units or storage devices can be any suitable storage medium (including magnetic storage media or magneto-optical storage media), such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), ROM, and RAM.

[0169] The embodiments of this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this disclosure. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

Claims

1. An integrated circuit device for dynamically fusing neural networks according to a fusion strategy, comprising: The processing device is configured to: select the starting layer of the template fusion unit according to the starting rules of the fusion strategy; as well as The initial layer is used as a reference for fusion, and the rules within the fusion strategy are checked to establish the template fusion unit; A computing device for performing neural network calculations based on the template fusion unit; The computing device includes multiple clusters, each cluster including a shared storage unit. The processing device determines whether the storage space required by the feature map is greater than the available space of the shared storage unit. If so, the processing device splits the feature map into on-chip cell maps, and the storage space required by the on-chip cell maps is not greater than the available space of the shared storage unit. The fusion strategy is that the total redundancy generated by splitting the on-chip cell graph does not exceed a percentage threshold. When the processing device determines that the fusion strategy is not satisfied, the processing device stops fusion. The total redundancy is the difference between the memory access volume and the theoretical memory access volume.

2. The integrated circuit device according to claim 1, wherein the starting rule is that the starting layer is the first unfused layer in the neural network.

3. The integrated circuit device according to claim 1, wherein the starting rule is that the starting layer is the first unfused convolutional or pooling layer.

4. The integrated circuit device of claim 3, wherein the fusion strategy is fusion from the convolutional or pooling layer to a previous unfused layer.

5. The integrated circuit device of claim 3, wherein the fusion strategy is backward fusion from the convolutional or pooling layer.

6. The integrated circuit device according to claim 1, wherein the fusion strategy is to add or delete the template fusion unit on a block-structure basis when the neural network is a block structure.

7. The integrated circuit device according to claim 1, wherein the fusion strategy is to add or delete the template fusion unit on a layer-by-layer basis when the neural network has a long chain structure.

8. The integrated circuit device according to claim 1, wherein the fusion strategy is that the output of the template fusion unit is a single-branch output, and when the processing device determines that the fusion strategy is not satisfied, the processing device adds or deletes the template fusion unit until the fusion strategy is satisfied.

9. The integrated circuit device according to claim 1, wherein the neural network includes a plurality of main layers, the main layers being one of matrix multiplication, pooling, and convolution, and the rule of the fusion strategy is that the template fusion unit includes at least two main layers, and when the processing device determines that the fusion strategy is not satisfied, the processing device adjusts the template fusion unit until the fusion strategy is satisfied.

10. The integrated circuit device according to claim 1, wherein the neural network includes a plurality of main layers, the main layer being one of matrix multiplication, pooling, and convolution, the fusion strategy being that the template fusion unit includes a continuous structure of main layers, main layers, and non-main layers sequentially adjacent to each other, and when the processing device determines that the fusion strategy is not satisfied, the processing device adjusts the template fusion unit until the fusion strategy is satisfied.

11. The integrated circuit device of claim 10, wherein the structure is a single branch.

12. The integrated circuit device according to claim 1, wherein the fusion strategy is a continuous structure in which the template fusion unit comprises a scalar computation layer and a vector computation layer that are sequentially adjacent, and when the processing device determines that the fusion strategy is not satisfied, the processing device adjusts the template fusion unit until the fusion strategy is satisfied; in, The scalar computation layer includes one of an addition layer, a subtraction layer, and a multiplication layer, and the vector computation layer includes one of an activation layer, a batch normalization layer, and a scaling layer.

13. The integrated circuit device according to claim 1, wherein the fusion strategy is that the weights of the convolutional layers in the template fusion unit are not the outputs of any layer of the neural network, and when the processing device determines that the fusion strategy is not satisfied, the processing device removes the convolutional layers from the template fusion unit.

14. The integrated circuit device according to claim 1, wherein the fusion strategy is that the weights of the convolutional layer in the template fusion unit are not shared with any layer of the neural network, and when the processing device determines that the fusion strategy is not satisfied, the processing device removes the convolutional layer from the template fusion unit.

15. The integrated circuit device of claim 1, wherein the feature map includes N, H, W, and C dimensions, and the processing device performs a specific granularity split in one of the N, H, W, and C dimensions.

16. The integrated circuit device of claim 15, wherein the C dimension is an output channel parameter.

17. The integrated circuit device of claim 16, wherein each cluster further comprises a plurality of processor cores, each processor core comprising a weight storage unit, the fusion strategy being that the storage space required for the weights involved in the on-chip cell diagram divided by the number of processor cores is not greater than the available space of the weight storage unit, and when the processing device determines that the fusion strategy is not satisfied, the processing device reduces the size of the on-chip cell diagram.

18. The integrated circuit device of claim 1, wherein the rule within the fusion strategy is: in, size TFU To access the existing inventory, size ori This refers to the amount of data in the on-chip unit diagram.

19. The integrated circuit device of claim 1, wherein when the processing device determines that the storage space required for the feature map is not greater than the available space of the shared memory cell, the processing device further analyzes how many feature maps the available space of the shared memory cell can accommodate, and the set of all feature maps that can be accommodated is the on-chip cell map.

20. The integrated circuit device of claim 19, wherein the fusion strategy is that if the storage space of the on-chip cell diagram and the storage space of the calculation result of the on-chip cell diagram cannot be reused, the sum of the storage space of the on-chip cell diagram and the storage space of the calculation result is less than the available space of the shared storage unit, and when the processing device determines that the fusion strategy is not satisfied, the processing device reduces the number of feature maps in the on-chip cell diagram until the fusion strategy is satisfied.

21. The integrated circuit device of claim 19, wherein the fusion strategy is that if the storage space of the on-chip cell diagram and the storage space of the calculation result of the on-chip cell diagram can be reused, the larger of the storage space of the on-chip cell diagram and the storage space of the calculation result is less than the available space of the shared storage unit, and when the processing device determines that the fusion strategy is not satisfied, the processing device reduces the number of feature maps in the on-chip cell diagram until the fusion strategy is satisfied.

22. The integrated circuit device of claim 19, wherein the cluster further comprises a processor core and a storage core, the storage core splitting the on-chip cell graph into subgraphs, one of the processor cores computing the subgraphs, and the shared storage unit including a cache space.

23. The integrated circuit device of claim 22, wherein the fusion strategy is that the sum of the storage space required for the weights of the subgraph, the storage space required for the on-chip cell graph, and the cache space is not greater than the available space of the shared storage unit, and when the processing device determines that the fusion strategy is not satisfied, the processing device reduces the number of feature graphs in the on-chip cell graph until the fusion strategy is satisfied.

24. The integrated circuit device of claim 22, wherein the fusion strategy is that the sum of the storage space required by the subgraph, the storage space required by the weights of the subgraph, and the cache space is not greater than the available space of the shared storage unit, and when the processing device determines that the fusion strategy is not satisfied, the processing device reduces the number of feature maps in the on-chip unit graph until the fusion strategy is satisfied.

25. The integrated circuit device of claim 22, wherein the processor core includes a computation module for calculating the subgraph and generating intermediate results, the fusion strategy being that the sum of the storage space required for the intermediate results, the storage space required for the weight of the next subgraph, and the cache space is not greater than the available space of the shared storage unit, and when the processing device determines that the fusion strategy is not satisfied, the processing device reduces the number of feature maps in the on-chip unit graph until the fusion strategy is satisfied.

26. The integrated circuit device of claim 22, wherein each cluster further comprises a plurality of processor cores, each processor core comprising a weight storage unit, the fusion strategy being that the sum of the storage space required for the weights of the subgraph and the storage space required for the weights of the next subgraph is not greater than the available space of the weight storage unit, and when the processing device determines that the fusion strategy is not satisfied, the processing device reduces the number of feature maps in the on-chip unit graph until the fusion strategy is satisfied.

27. The integrated circuit device of claim 22, wherein each cluster further includes a storage core and a plurality of processor cores, each processor core including a neuron storage unit, the feature map including N, H, and W dimensions, the fusion strategy being that the storage space required for the sub-graph is not greater than the available space of the neuron storage unit, and when the storage core determines that the fusion strategy is not satisfied, the storage core performs a specific granularity split in one of the N, H, and W dimensions until the fusion strategy is satisfied.

28. The integrated circuit device of claim 22, wherein the rule of the fusion strategy is that the number of feature maps included in the on-chip cell diagram is not greater than a feature map threshold, and when the processing device determines that the rule is not satisfied, the processing device reduces the number of feature maps.

29. The integrated circuit device of claim 22, wherein the template fusion unit includes a convolutional or pooling layer, the fusion strategy is that the sum of the differences between the side length and the stride of the kernel of the convolutional or pooling layer is not greater than a redundancy threshold, and when the processing device determines that the fusion strategy is not satisfied, the processing device adjusts the template fusion unit until the fusion strategy is satisfied.

30. The integrated circuit device of claim 29, wherein the template fusion unit is a single branch.

31. A board comprising an integrated circuit device according to any one of claims 1 to 30.

32. A method for dynamically fusing neural networks according to a fusion strategy, comprising: According to the starting rules of the fusion strategy, the starting layer of the template fusion unit is selected; The template fusion unit is established by fusing the starting layer as a reference and checking the rules of the fusion strategy. as well as Perform neural network calculations based on the established template fusion unit; The computing device includes multiple clusters, each cluster including a shared storage unit. The processing device determines whether the storage space required by the feature map is greater than the available space of the shared storage unit. If so, the processing device splits the feature map into on-chip cell maps, and the storage space required by the on-chip cell maps is not greater than the available space of the shared storage unit. The fusion strategy is that the total redundancy generated by splitting the on-chip cell graph does not exceed a percentage threshold. When the processing device determines that the fusion strategy is not satisfied, the processing device stops fusion. The total redundancy is the difference between the memory access volume and the theoretical memory access volume.

33. A computer-readable storage medium having stored thereon computer program code for dynamically fusing neural networks according to a fusion strategy, wherein when the computer program code is run by a processing device, it performs the method of claim 32.

Images