Apparatus, board card, method and readable storage medium with fusion branch structure

Abstract

The application relates to an apparatus, a board card, a method and a readable storage medium for dynamically fusing branch structures of a neural network according to a fusion strategy, wherein the computing apparatus of the application is included in an integrated circuit apparatus, the integrated circuit apparatus includes a general-purpose interconnection interface and other processing apparatuses. The computing apparatus and the other processing apparatuses interact with each other to jointly complete a user-specified computing operation. The integrated circuit apparatus can further include a storage apparatus connected with the computing apparatus and the other processing apparatuses respectively, for data storage of the computing apparatus and the other processing apparatuses.

Description

Technical Field

[0001] This invention generally relates to the field of neural networks. More specifically, this invention relates to apparatus, boards, methods, and readable storage media for dynamically fusing the branch structures of a neural network according to a fusion strategy. Background Technology

[0002] A neural network is a system of multiple neurons connected according to certain rules. It is generally composed of the following four types of layer structures: input layer, convolution layer, pooling layer, and fully connected layer.

[0003] The input layer extracts a portion of the input data and transforms it into a feature matrix, containing the features corresponding to that portion. The convolutional layer is configured to receive the feature matrix from the input layer and extract features from the input data through convolution operations. Multiple convolutional layers can be built in practice. The pooling layer is configured to replace a region of data with a single value, typically the maximum or average value among all values ​​in that region. Pooling reduces model size and improves computational speed without losing too much information. The fully connected layer acts as a classifier in the entire convolutional neural network, essentially transforming the feature space, extracting and integrating all the useful information from the previous layers, and comparing information based on different classifications to determine whether the input data is similar to the comparison target.

[0004] With the development of technology, neural networks are becoming increasingly complex, with more and more layers. Many neural network models with branching structures have been developed, such as the ResNet model. Models with branching structures consume significant resources during computation and delay processing time.

[0005] Therefore, a mechanism to reduce input / output access in the branch structure of neural network models is urgently needed in the field of artificial intelligence. Summary of the Invention

[0006] To at least partially solve the technical problems mentioned in the background art, the present invention provides an apparatus, board, method, and readable storage medium for dynamically fusing the branch structure of a neural network according to a fusion strategy.

[0007] In one aspect, the present invention discloses an integrated circuit device for dynamically fusing the branch structure of a neural network according to a fusion strategy, comprising a processing unit and a computing unit. The processing unit is configured to establish a topology sequence based on the branch structure, perform fusion based on the starting layer of the topology sequence, and check rules within the fusion strategy to establish template fusion units. The computing unit is configured to perform neural network computation based on the template fusion units.

[0008] In another aspect, the present invention discloses a board including an integrated circuit device according to the foregoing.

[0009] In another aspect, the present invention discloses a method for dynamically fusing the branch structure of a neural network according to a fusion strategy, comprising: establishing a topology sequence according to the branch structure; fusing based on the starting layer of the topology sequence, checking the rules within the fusion strategy to establish a template fusion unit; and performing neural network computation according to the template fusion unit.

[0010] In another aspect, the present invention discloses a computer-readable storage medium storing computer program code that dynamically fuses the branch structure of a neural network according to a fusion strategy, wherein the aforementioned method is executed when the computer program code is run by a processing device.

[0011] This invention fuses branch structures to generate template fusion units. The input of the first layer and the output of the last layer in the template fusion unit serve as the interaction data between the template fusion unit and off-chip memory. During this process, the calculation of each layer does not require access to off-chip memory, which greatly reduces the frequency of on-chip and off-chip input / output access. Attached Figure Description

[0012] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. In the drawings, several embodiments of the invention are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts wherein:

[0013] Figure 1 This is a structural diagram of the board card according to an embodiment of the present invention;

[0014] Figure 2 This is a structural diagram illustrating an integrated circuit device according to an embodiment of the present invention;

[0015] Figure 3 This is a schematic diagram illustrating the internal structure of a computing device according to an embodiment of the present invention;

[0016] Figure 4 This is a schematic diagram illustrating the internal structure of the processor core according to an embodiment of the present invention;

[0017] Figure 5This is a diagram illustrating when one processor core wants to write data to another processor core in a different cluster;

[0018] Figure 6 This is a schematic diagram illustrating the AlexNet model;

[0019] Figure 7 This is a schematic diagram illustrating an exemplary neural network model;

[0020] Figure 8 This is a schematic diagram illustrating the fusion of two convolutional layers according to an embodiment of the present invention;

[0021] Figure 9 This is a schematic diagram showing the formats of NCHW and NHWC;

[0022] Figure 10 This is a flowchart illustrating an embodiment of the present invention that utilizes a template fusion unit to perform neural network computation;

[0023] Figure 11 This is a flowchart illustrating the dynamic fusion of neural networks according to a fusion strategy in an embodiment of the present invention;

[0024] Figure 12 This is a flowchart illustrating an embodiment of the present invention that utilizes a template fusion unit to perform neural network computation;

[0025] Figure 13 This shows an exemplary neural network model fragment;

[0026] Figure 14 This is a schematic diagram illustrating the topological sequence of the branch structure in an embodiment of the present invention;

[0027] Figure 15 This is a schematic diagram illustrating the transformation of a long chain structure into a branched structure according to an embodiment of the present invention;

[0028] Figure 16 This shows another exemplary neural network model fragment;

[0029] Figure 17 This is a schematic diagram illustrating the topological sequence of a branch structure according to another embodiment of the present invention; and

[0030] Figure 18 This is a flowchart illustrating a branch structure of a fused neural network according to another embodiment of the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] It should be understood that the terms "first," "second," "third," and "fourth," etc., in the claims, specification, and drawings of this invention are used to distinguish different objects, rather than to describe a specific order. The terms "comprising" and "including" used in the specification and claims of this invention indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0033] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this specification and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0034] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection."

[0035] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0036] A neural network consists of an input layer, convolutional layers, activation functions, pooling layers, and fully connected layers. It can have anywhere from a few layers to hundreds of layers. Each layer executes an operator; for example, a convolutional layer executes a convolution operator. The number of operators executed corresponds to the number of layers. In this invention, when a specific layer is mentioned, it refers to the operator corresponding to that layer.

[0037] In neural network computation, the input information and the output results of each layer of the model are different in each inference computation. They are considered variable data, which are generally represented by feature maps (matrices). In this invention, the input information of the entire neural network model and the input maps of each layer of the model are collectively referred to as feature maps. Once the feature maps are loaded onto the on-chip memory component, they are called on-chip cell maps in this invention. The parameters of the trained network model are usually not frequently changed after training is stable, or they can be compiled and generated after the network topology and hardware parameters are determined, and they do not change during the computation process. Therefore, they can be regarded as constant data. Constant data includes, but is not limited to, weights, biases, device hardware instructions, the mean and variance of batch normalization, etc. In this invention, weights are used to represent all constant data. When "data" is mentioned in this invention, it generally refers to the graph structure that allows the operation of corresponding operators in the neural network model to be fused together according to the fusion strategy. The variable data and constant data involved in this graph structure are the feature maps plus the corresponding weights.

[0038] Figure 1 A schematic diagram of the structure of a board 10 according to an embodiment of the present invention is shown. Figure 1 As shown, board 10 includes chip 101, which is a system-on-chip (SoC) integrating one or more combined processing units. These combined processing units are artificial intelligence computing units used to support various deep learning and machine learning algorithms, meeting the intelligent processing needs of complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in cloud intelligence. A significant characteristic of cloud intelligence applications is the large volume of input data, placing high demands on the platform's storage and computing capabilities. Board 10 in this embodiment is suitable for cloud intelligence applications, possessing massive off-chip storage, on-chip storage, and substantial computing power.

[0039] Chip 101 is connected to external device 103 via external interface device 102. External device 103 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 103 to chip 101 via external interface device 102. The calculation results from chip 101 can be transmitted back to external device 103 via external interface device 102. Depending on the application scenario, external interface device 102 may have different interface forms, such as a PCIe interface.

[0040] The board 10 also includes a storage device 104 for storing data, which includes one or more memory cells 105. The storage device 104 is connected to and transmits data with the controller 106 and the chip 101 via a bus. The controller 106 in the board 10 is configured to regulate the state of the chip 101. Therefore, in one application scenario, the controller 106 may include a microcontroller (MCU).

[0041] Figure 2 This is a structural diagram illustrating the combined processing device in chip 101 of this embodiment. (As shown) Figure 2 As shown, the combined processing device 20 includes a computing device 201, an interface device 202, a processing device 203, and a DRAM 204.

[0042] The computing device 201 is configured to perform user-specified operations. It is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0043] Interface device 202 is used to transmit data and control commands between computing device 201 and processing device 203. For example, computing device 201 can obtain input data from processing device 203 via interface device 202 and write it to on-chip storage device of computing device 201. Further, computing device 201 can obtain control commands from processing device 203 via interface device 202 and write them to on-chip control cache of computing device 201. Alternatively or optionally, interface device 202 can also read data from storage device of computing device 201 and transmit it to processing device 203.

[0044] The processing device 203, as a general-purpose processing device, performs basic controls including but not limited to data transfer and starting / stopping the computing device 201. Depending on the implementation, the processing device 203 may be one or more types of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), or other general-purpose and / or special-purpose processors. These processors include, but are not limited to, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, the computing device 201 of this invention can be considered as having a single-core structure or a homogeneous multi-core structure. However, when the computing device 201 and the processing device 203 are considered together, they are considered to form a heterogeneous multi-core structure.

[0045] DRAM 204 is used to store data to be processed. It is DDR memory, typically 16G or larger in size, and is used to store data in computing device 201 and / or processing device 203.

[0046] Figure 3 A schematic diagram of the internal structure of computing device 201 is shown. Computing device 201 is used to process input data such as computer vision, speech, natural language, and data mining. The computing device 201 in the figure adopts a multi-core hierarchical structure design. As a system-on-a-chip, computing device 201 includes multiple clusters, and each cluster includes multiple processor cores. In other words, computing device 201 is constructed in a hierarchical structure of system-on-a-chip, cluster, and processor core.

[0047] From the perspective of system-on-a-chip hierarchy, such as Figure 3 As shown, the computing device 201 includes an external storage controller 301, a peripheral communication module 302, an on-chip interconnect module 303, a synchronization module 304, and multiple clusters 305.

[0048] There can be multiple external storage controllers 301; two are shown as an example in the figure. These controllers are used to respond to access requests from the processor core to access external storage devices, such as… Figure 2The DRAM 204 in the chip allows data to be read from or written to external devices. The peripheral communication module 302 receives control signals from the processing device 203 via the interface device 202, initiating the computing device 201 to execute tasks. The on-chip interconnect module 303 connects the external storage controller 301, the peripheral communication module 302, and multiple clusters 305 to transmit data and control signals between modules. The synchronization module 304 is a global barrier controller (GBC) used to coordinate the working progress of each cluster and ensure information synchronization. The multiple clusters 305 are the computing core of the computing device 201; four are shown exemplary in the figure. With hardware development, the computing device 201 of this invention can also include eight, sixteen, sixty-four, or even more clusters 305. The clusters 305 are used to efficiently execute deep learning algorithms.

[0049] From the perspective of cluster hierarchy, such as Figure 3 As shown, each cluster 305 includes multiple processor cores (IPU cores) 306 and one memory core (MEM core) 307.

[0050] Four processor cores 306 are shown exemplarily in the figure, but the present invention does not limit the number of processor cores 306. Its internal architecture is as follows: Figure 4 As shown. Each processor core 306 includes three main modules: a control module 41, an arithmetic module 42, and a storage module 43.

[0051] The control module 41 coordinates and controls the operation of the computation module 42 and the storage module 43 to complete the deep learning task. It includes an instruction fetch unit (IFU) 411 and an instruction decode unit (IDU) 412. The instruction fetch unit 411 fetches instructions from the processing device 203, and the instruction decode unit 412 decodes the fetched instructions and sends the decoding result as control information to the computation module 42 and the storage module 43.

[0052] The computation module 42 includes a vector operation unit 421 and a matrix operation unit 422. The vector operation unit 421 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformations; the matrix operation unit 422 is responsible for the core computations of deep learning algorithms, namely matrix multiplication and convolution.

[0053] Storage module 43 is used to store or move related data, including neuron RAM (NRAM) 431, weight RAM (WRAM) 432, input / output direct memory access (IODMA) 433, and move direct memory access (MVDMA) 434. NRAM 431 is used to store feature maps for computation by processor core 306 and intermediate results after computation; WRAM 432 is used to store the weights of the deep learning network; IODMA 433 controls the memory access of NRAM 431 / WRAM 432 and DRAM 204 through broadcast bus 309; MVDMA 434 controls the memory access of NRAM 431 / WRAM 432 and SRAM 308.

[0054] Back Figure 3 The storage core 307 is primarily used for storage and communication, namely storing shared data or intermediate results among processor cores 306, and performing communication between cluster 305 and DRAM 204, communication between clusters 305, and communication between processor cores 306. In other embodiments, the storage core 307 has scalar operation capabilities and is used to perform scalar operations.

[0055] Storage core 307 includes a shared memory unit (SRAM) 308, a broadcast bus 309, a cluster direct memory access (CDMA) module 310, and a global direct memory access (GDMA) module 311. SRAM 308 acts as a high-performance data relay station. Data multiplexed between different processor cores 306 within the same cluster 305 does not need to be obtained from DRAM 204 by each processor core 306 individually. Instead, it is relayed between processor cores 306 via SRAM 308. Storage core 307 only needs to quickly distribute the multiplexed data from SRAM 308 to multiple processor cores 306, thereby improving inter-core communication efficiency and significantly reducing on-chip and off-chip I / O access.

[0056] Broadcast bus 309, CDMA 310, and GDMA 311 are used to perform communication between processor cores 306, communication between clusters 305, and data transfer between cluster 305 and DRAM 204, respectively. These will be explained below.

[0057] The broadcast bus 309 is used to complete high-speed communication between the processor cores 306 within the cluster 305. In this embodiment, the broadcast bus 309 supports inter-core communication methods including unicast, multicast, and broadcast. Unicast refers to point-to-point (i.e., data transmission from one processor core to another) data transmission. Multicast is a communication method that transmits a piece of data from SRAM 308 to several specific processor cores 306. Broadcast is a communication method that transmits a piece of data from SRAM 308 to all processor cores 306, and is a special case of multicast.

[0058] CDMA 310 is used to control SRAM 308 access between different clusters 305 within the same computing device 201. Figure 5 This diagram illustrates the operation of CDMA310 when one processor core attempts to write data to another processor core in a different cluster. In this application scenario, the same computing device comprises multiple clusters. For simplicity, only clusters 0 and 1 are shown in the diagram. Both clusters 0 and 1 contain multiple processor cores; similarly, for ease of explanation, only processor core 0 is shown in cluster 0, and only processor core 1 is shown in cluster 1. Processor core 0 intends to write data to processor core 1.

[0059] First, processor core 0 sends a unicast write request to write data into its local SRAM 0. CDMA 0 acts as the master and CDMA 1 acts as the slave. The master pushes the write request to the slave, that is, the master sends the write address AW and the write data W to transmit the data to SRAM 1 of cluster 1. Then, the slave sends a write response B as a response. Finally, processor core 1 of cluster 1 sends a unicast read request to read the data from SRAM 1.

[0060] Back Figure 3The GDMA 311, in conjunction with the external memory controller 301, controls memory access from the SRAM 308 to the DRAM 204 of the cluster 305, or reads data from the DRAM 204 into the SRAM 308. As previously described, communication between the DRAM 204 and the NRAM 431 or WRAM 432 can be achieved through two channels. The first channel is a direct connection between the DRAM 204 and the NRAM 431 or WRAM 432 via the IODAM 433; the second channel involves first transmitting data between the DRAM 204 and SRAM 308 via the GDMA 311, and then transmitting data between the SRAM 308 and the NRAM 431 or WRAM 432 via the MVDMA 434. Although the second channel appears to require more components and has a longer data flow, in some embodiments, the bandwidth of the second channel is actually much greater than that of the first channel. Therefore, communication between the DRAM 204 and the NRAM 431 or WRAM 432 may be more efficient via the second channel. The embodiments of the present invention can select the data transmission channel according to their own hardware conditions.

[0061] In other embodiments, the functions of GDMA 311 and IODMA 433 can be integrated into the same component. For ease of description, GDMA 311 and IODMA 433 are considered different components. For those skilled in the art, any component whose functions and technical effects are similar to those of this invention falls within the scope of protection of this invention. Furthermore, the functions of GDMA 311, IODMA 433, CDMA 310, and MVDMA 434 can also be implemented by the same component. Similarly, any component whose functions and technical effects are similar to those of this invention falls within the scope of protection of this invention.

[0062] Neural network models are mainly classified into two types: long-chain structures and branched structures. Long-chain structures refer to neural network models composed of layers connected in a single chain, with each layer having only one input and one output; the entire structure is single-branched, such as the VGG16 model. Figure 6 The AlexNet model shown is an example. A branching structure refers to a neural network where a subnetwork has only one input and one output, but contains multiple branches; that is, some layers of the subnetwork have multiple inputs or outputs. Examples include the ResBlock structure of ResNet50 and the Block structure of Inception_v3. Branching structures are shown below. Figure 7The diagram illustrates an exemplary neural network model, comprising subnetwork 701 and subnetwork 702. Subnetwork 701 has only one input and one output, comprising layers 1 through 6. Layer 1 has two outputs, and layer 6 has two inputs. Therefore, subnetwork 701 comprises two branches: one branch is layer 1 → layer 2 → layer 3 → layer 6, and the other branch is layer 1 → layer 4 → layer 5 → layer 6. Subnetwork 701 constitutes a branched structure. Similarly, subnetwork 702 also constitutes a branched structure.

[0063] Performing computations at each layer of deep learning requires numerous off-chip and on-chip accesses, particularly reading input data from DRAM 204 into computing device 201 and storing the computation results from computing device 201 back into DRAM 204. These frequent accesses consume significant hardware resources. To address this issue, this invention significantly reduces off-chip and on-chip data transfer by fusing adjacent layers of the neural network.

[0064] Figure 8 This diagram illustrates the fusion of two convolutional layers. The input to the first convolutional layer 810 is a 7×7 feature map 801. This layer convolves feature map 801 with a 3×3 kernel (not shown) to obtain feature map 802 of the first convolutional layer 810. The value of the 5×5 feature submap 804 affects the 3×3 feature submap 805. Assuming a stride of 1, after calculating the 5×5 feature submap 804, the first convolutional layer 810 will then calculate the 5×5 feature submap 806, and the value of the 5×5 feature submap 806 will affect the 3×3 feature submap 807.

[0065] During the computation of the second convolutional layer 811, feature map 802 becomes the input to the second convolutional layer 811, and is also convolved with a 3×3 kernel to obtain feature map 803 of the second convolutional layer 811. The value of the 3×3 feature sub-map 805 affects the 1×1 feature sub-map 808 within feature map 803. After calculating the 3×3 feature sub-map 805, the second convolutional layer 811 then calculates the 3×3 feature sub-map 807, and the value of the 3×3 feature sub-map 807 affects the 1×1 feature sub-map 809 within feature map 803.

[0066] If not merged, during the first convolutional layer 810, the computing device 201 reads a 5×5 feature sub-map 804 from DRAM 204, calculates it, and then stores a 3×3 feature sub-map 805 back into DRAM 204. Next, it reads a 5×5 feature sub-map 806 from DRAM 204, calculates it, and then stores a 3×3 feature sub-map 807 into DRAM 204. During the second convolutional layer 811, it similarly reads a 3×3 feature sub-map 805 from DRAM 204, calculates it, and then stores a 1×1 feature sub-map 808 into DRAM 204. Then, it reads a 3×3 feature sub-map 807 from DRAM 204, calculates it, and then stores a 1×1 feature sub-map 809 into DRAM 204. As can be seen from the above description, feature map 802 is repeatedly read and stored off-chip as intermediate data, which consumes considerable system resources.

[0067] If the first convolutional layer 810 and the second convolutional layer 811 are fused, that is, the feature map 802 is stored in NRAM 431 (the weights of the first convolutional layer 810 and the second convolutional layer 811 can also be stored in WRAM 432), the number of accesses between the computing device 201 and DRAM 204 can be reduced, thereby improving the overall execution efficiency of the neural network.

[0068] Neural network fusion typically involves fusing layers from specific convolutional and pooling layers within the neural network. That is, the starting layer is a convolutional or pooling layer, and multiple layers are fused sequentially, potentially including multiple convolutional and pooling layers, depending on the hardware capabilities. However, with the development of deep learning and neural networks, layer ordering has become more complex. For example, if an activation layer precedes a convolutional layer, this activation layer must also be considered for fusion with subsequent convolutional layers. Therefore, in addition to simply using convolutional and pooling layers as the core for fusion, this invention provides diverse fusion methods that do not necessarily rely on convolutional and pooling layers. Instead, it employs specific strategies to flexibly select layers within the neural network for fusion. Even user-defined layers can be fused as long as they conform to the fusion strategy, thereby optimizing overall performance.

[0069] Another embodiment of the present invention is a novel fusion method, which utilizes the aforementioned Figure 1 , Figure 2 , Figure 3 and Figure 4 This fusion is implemented using a hardware structure, and such fusion is called a template fuse unit (TFU). The template fuse unit mainly uses a certain fusion strategy to flexibly fuse multiple layers into one layer to reduce the input / output overhead of the network. It includes the aforementioned neural network fusion and other fusion methods. The collection of these fused layers is the template fuse unit, which can be regarded as a new layer or a custom layer.

[0070] In this embodiment, the feature map and weights required by the template fusion unit are loaded from DRAM 204 into the on-chip SRAM 308 in one go. The feature map loaded into SRAM 308 is called the on-chip cell map. The on-chip cell map is divided into sub-maps. Each time, a sub-map is loaded from SRAM 308 into NRAM 431 of the processor core 306 assigned to calculate the sub-map. The weights required to calculate the sub-map are also loaded from SRAM 308 into WRAM 432. After each sub-map is calculated, the corresponding intermediate result is obtained and stored back into SRAM 308. After all sub-maps have been calculated, the calculation results are stored back into DRAM 204 in one go.

[0071] More specifically, SRAM 308 is one of the key reference indicators for the fusion strategy, and its size determines whether the template fusion cell is in large-image mode or small-image mode. Small-image mode and large-image mode refer to whether a feature map stored in DRAM 204 can be moved to SRAM 308 for processing at once. The processing device 203 compares the storage space required for the feature map with the available space in SRAM 308. If SRAM 308 has insufficient space and the feature map cannot be accommodated, it is in large-image mode; if SRAM 308 is large enough to accommodate the entire feature map, it is in small-image mode. It is important to note that in large-image mode, the on-chip cell map is only a part of the feature map; in small-image mode, if the available space in SRAM 308 is large enough, or the feature map is small enough, SRAM 308 may be able to accommodate multiple feature maps at once, meaning the on-chip cell map can include multiple feature maps.

[0072] If it is a large image mode, the feature map must be split before it can be loaded into the computing device 201. The processing device 203 will split the feature map on DRAM 204 until a sufficiently small on-chip cell map is generated to meet the space requirements of SRAM 308, so that the on-chip cell map can be moved to SRAM 308 for processing at once. During the splitting of the feature map, input-dependent operations and output-dependent operations may occur.

[0073] Input-dependent computations refer to situations where the split on-chip unit graphs at least partially overlap, and each subset requires some additional copies of the input for a complete computation. This leads to data redundancy in the splitting operation, where the same data is reused within the system. Input-dependent computations occur when template fusion units include layers such as convolution, pooling, or matrix multiplication.

[0074] Output-dependent operations refer to the process where, after each subgraph produces intermediate results, a reduction process is required to obtain the final computational result. Reduction involves breaking down the on-chip unit graph into subgraphs based on an understanding of its content, and then calculating each subgraph separately to reduce the computational scale. This minimizes the amount of data while preserving the original on-chip unit graph as much as possible, and then uses the subgraphs as a basis to restore or integrate the computational results. The computational results during reduction are interdependent. Output-dependent operations occur when the template fusion unit includes layers such as inner product, convolution, matrix multiplication, sorting, and counting.

[0075] This embodiment can process feature map data formats with dimensions N, H, W, and C, where N represents batch, H represents height, W represents width, and C represents channels. Taking image data as an example, N indicates the total number of images in the batch, H indicates the number of pixels in the vertical direction, W indicates the number of pixels in the horizontal direction, and C indicates the number of channels (e.g., a black and white image has 1 channel, while an RGB color image has 3 channels).

[0076] The ordering of these dimensions determines how the data is composed; common composition methods include NHWC and NCHW. Figure 9 This diagram illustrates the difference between NCHW and NHWC formats, using an RGB color image as an example. In the diagram, R represents red pixels, G represents green pixels, and B represents blue pixels. Sequence 91 is in NCHW format, with N arranged on the outermost layer. Pixels within each channel are placed side-by-side and arranged in RGB order. The element with coordinates (n, c, h, w) has an offset in storage of ((n × C + c) × H + h) × W + w. Sequence 92 is in NHWC format, with C arranged on the innermost layer. RGB pixels corresponding to multiple channels are placed side-by-side. The diagram also shows the positions of input pixels 901, 902, and 903 under different arrangements. These three input pixels (901, 902, and 903) together represent the color of a point in the image. The coordinate offset of the element with coordinates (n, c, h, w) is calculated as ((n × H + h) × W + w) × C + c. NHWC is closer to the BMP image data storage format than NCHW. BMP files store data pixel by pixel, with each pixel containing the color values ​​of all channels. This eliminates the need for additional dimensionality transformations when reading input images. Therefore, NHWC offers better memory locality, producing one output pixel for every three input pixels. NCHW, on the other hand, requires all input channels to be ready before obtaining the final output, consuming more cache space.

[0077] This embodiment can use each layer of the data fusion neural network as a template fusion unit. Figure 10 The corresponding flowchart is shown.

[0078] In step 1001, the processing device 203 determines whether the storage space required for the feature map is greater than the available space of the SRAM 308. If so, it means that the feature map cannot be loaded into the SRAM 308 at once, so step 1002 is executed to split the feature map. In this embodiment, the processing device 203 chooses to split in any dimension. After splitting the feature map, the processing device 203 returns to step 1001 and determines whether the storage space required for the split feature map is still greater than the available space of the SRAM 308. If so, step 1002 is executed again to continue splitting.

[0079] If the processing device 203 determines that the storage space required for the split feature map is not greater than the available space of the SRAM 308, it means that the SRAM 308 can load the split feature map at once, and then executes step 1003, whereby the processing device 203 sets the split feature map as an on-chip cell map.

[0080] Finally, in step 1004, the processing device 203 determines the template fusion unit based on the size of the on-chip unit map. This step dynamically fuses the neural network according to the fusion strategy. Figure 11 This embodiment illustrates a method for dynamically fusing neural networks according to a fusion strategy.

[0081] In step 1101, the starting layer of the template fusion unit is selected according to the starting rules of the fusion strategy. The processing device 203 selects the starting layer of the template fusion unit according to the starting rules of the fusion strategy, that is, selects the layer to start fusion from the layers that have not yet been fused in the neural network.

[0082] In one application scenario, the starting rule could be that the starting layer is the first unfused layer in the neural network, and the processing device 203 will search for the first unfused layer. Figure 6 Taking the AlexNet neural network model as an example, it has 23 layers. Assuming that layers 1 to 5 have been fused, when the starting rule is that the starting layer is the first unfused layer in the neural network, the processing device 203 will select the ReLU activation layer of layer 6 as the starting layer and fuse backward (i.e., towards layer 7). It should be noted that under this starting rule, the starting layer is not necessarily a convolutional layer or a pooling layer.

[0083] In another application scenario, considering that convolutional and pooling layers consume the most input / output resources, the starting rule is that the initial layer is the first unfused convolutional or pooling layer. The processing device 203 will first identify all unfused convolutional and pooling layers in the neural network model and start fusing them from the first unfused convolutional or pooling layer. Similarly, Figure 6Taking the AlexNet neural network model as an example, assuming that layers 1 to 9 have been fused, the processing device 203 will find all the unfused convolutional and pooling layers in the neural network model, namely layers 11, 13, and 15, and then start fusion from the first unfused convolutional or pooling layer, that is, the starting layer is layer 11.

[0084] In step 1102, fusion is performed based on the initial layer, and all rules of the fusion strategy are checked one by one to establish a template fusion unit. The processing device 203 performs fusion based on the initial layer, checking all rules of the fusion strategy one by one to establish a template fusion unit. Provided all rules are satisfied, the hardware resources of the computing device 201 are sufficient to support loading the data required to compute the template fusion unit at once, and then performing neural network computation based on the template fusion unit. In addition to the aforementioned initial rules, the fusion strategy may, exemplarily, include the following rules:

[0085] Rule 1: Backward fusion

[0086] Backward fusion refers to fusion from the initial layer towards the inference direction of the neural network model. Figure 6 For example, the fusion proceeds in the direction of first layer → second layer → third layer. If there are unfused layers before the starting layer, these unfused layers will not be considered in the template fusion unit under this rule.

[0087] Rule 2: Prioritize forward merging

[0088] Forward fusion refers to fusion in the reverse direction from the initial layer towards the inference layer of the neural network. Figure 6 For example, the fusion proceeds in the direction of the third layer → the second layer → the first layer. This rule is usually paired with the aforementioned starting layer rule, where the first unfused convolutional or pooling layer is the initial layer, because there may be unfused layers before the convolutional or pooling layer. After selecting the starting layer, the processing device 203 prioritizes forward fusion, attempting to incorporate the unfused layers before the starting layer into the template fusion unit. Similarly, Figure 6 Taking the AlexNet neural network model as an example, assuming that the first and second layers have been fused, the processing device 203 finds that the first unfused convolutional or pooling layer is the fifth layer, so the starting layer is the fifth layer. It prioritizes fusing the fourth and third layers forward. If it can continue to fuse, it then continues to fuse the sixth and seventh layers, etc.

[0089] Rule 3: Prioritize branch structure as the unit.

[0090] When the neural network model has a branching structure, this rule requires the processing device 203 to prioritize adding or removing template fusion units based on the branching structure rather than on a layer-by-layer basis. Only if the fusion of an entire block of computational logic fails should fusion be considered from the layers of each branch. Figure 7 Taking a neural network model as an example, the processing device 203 will prioritize the fusion of subnetworks 701 or 702 as units.

[0091] When the neural network has a long chain structure, since there are no branches, template fusion units are added or removed directly at the layer level. This rule does not apply to neural network models with long chain structures.

[0092] Rule 4: Single-branch output

[0093] The fusion strategy in this embodiment does not support template fusion units as multi-output networks. The reason is that the shape derivation implemented inside the template fusion unit mainly adopts the form of back-to-forward derivation. Multi-output networks mean that derivation needs to be performed from different outputs, and the derivation results will not necessarily be reduced to the same feature map, so convergence is impossible.

[0094] In other words, the output of the template fusion unit must be a single-branch output, meaning that the last layer of the template fusion unit can only have one output. Figure 7 Two fusion methods for subnetwork 701 are indicated. The first method merges layers 1 to 5 into a single template fusion unit 703, and the second method merges layers 1 to 6 into a single template fusion unit 704. Since the outputs of layers 3 and 5 are both outputs of template fusion unit 703, template fusion unit 703 is a multi-output network, i.e., multi-branch output. The output of layer 6 is the output of template fusion unit 704, producing only one output data; therefore, template fusion unit 704 is a single-output network, i.e., single-branch output. Processing unit 203 determines whether the output of the template fusion unit is a single-branch output. If this rule is not satisfied, processing unit 203 adds or removes layers within the template fusion unit until the rule is satisfied.

[0095] Rule 5: Includes at least 2 main layers

[0096] When the layer logic is too simple, the performance of the template fusion unit is worse than that of the unfused layers. Therefore, when using layer logic as the fusion strategy, the processing device 203 evaluates whether the operations of each fused layer are complex enough to generate benefits from the fusion. To generate benefits, the main layers should be included in the template fusion unit as much as possible. The main layers refer to layers that consume a lot of input / output resources, such as matrix multiplication, pooling, or convolution. Pooling here includes various types of pooling, such as max pooling or average pooling, and convolution also includes various types of convolution, such as ordinary convolution, convolution with average, and depthwise convolution. This rule requires that the template fusion unit includes at least two main layers. When the processing unit 203 determines that this rule is not met, the processing device 203 will adjust the template fusion unit until the rule is met.

[0097] Rule 6: Includes a continuous structure with the main layer, the main layer, and non-main layers adjacent in sequence.

[0098] This rule requires that template fusion units must include a continuous structure of main layers, main layers, and non-main layers; that is, a continuous structure where main layers, main layers, and non-main layers are sequentially adjacent. Such computation is complex enough to make fusion efficient. See also... Figure 6 The template fusion unit consists of layers 4, 5, and 6, where layer 4 is a max pooling layer, layer 5 is a convolutional layer, and layer 6 is a ReLU activation layer. This conforms to a continuous structure where the main layers and non-main layers are sequentially adjacent. Therefore, the template fusion unit including layers 4, 5, and 6 satisfies this rule. When the processing unit 203 determines that this rule is not satisfied, the processing device 203 adjusts the template fusion unit until the rule is satisfied.

[0099] Rule 7: Includes adjacent, continuous structures of scalar computation layers and vector computation layers.

[0100] This rule refers to a template fusion unit comprising a continuous structure of scalar computation layers and vector computation layers, i.e., a continuous structure where scalar computation layers and vector computation layers are sequentially adjacent. The scalar computation layer refers to an addition layer, subtraction layer, or multiplication layer, and the vector computation layer refers to an activation layer, batch normalization layer, or scaling layer. When processing unit 203 determines that this rule is not satisfied, processing device 203 will adjust the template fusion unit until the rule is satisfied.

[0101] Rule 8: The weights of a convolutional layer are not equal to the output of any other layer.

[0102] This rule states that the weights of the convolutional layers in the template fusion unit are not the outputs of any layer in the neural network, regardless of whether that layer is included in the template fusion unit. When the processing unit 203 determines that this rule is not met, the processing device 203 will remove this convolutional layer from the template fusion unit.

[0103] Rule 9: The weights of a convolutional layer should not be shared with any other layer in the neural network.

[0104] Because the weights of operators in the neural network model involved in the template fusion unit have a special arrangement, when the convolutional operator being fused shares weights with other operators, a conflict will occur in the weight arrangement logic. The rule is that the weights of the convolutional operators in the template fusion unit should not be shared with any layer of the neural network. When the processing unit 203 determines that this rule is not satisfied, the processing device 203 will remove this convolutional operator from the template fusion unit.

[0105] Rule 10: The weight is no greater than the available space of WRAM.

[0106] The large-image mode imposes fewer restrictions on WRAM 432 because the on-chip cell map loaded into SRAM 308 is only a part of the feature map. When calculating the template fusion unit, WRAM 432 only needs to store all the weights of that feature map. However, since the small-image mode may load multiple feature maps into SRAM 308, the required weights will increase, requiring careful evaluation of whether the available space of WRAM 432 is sufficient. The rule is that the storage space required for the weights in the on-chip cell map should not exceed the available space of WRAM 432. When the processing device 203 determines that this rule is not met, it will reduce the size of the on-chip cell map.

[0107] If the weights are split based on the output channel parameter Cout of dimension C, since the weights will be evenly distributed across multiple processor cores 306, this rule is adjusted as follows:

[0108]

[0109] Among them, W j denoted as , where is the storage space required for the weights involved in the on-chip cell graph j, n is the number of processor cores in the cluster, and W is the available space of WRAM 432.

[0110] Rule 11: Redundancy Percentage

[0111] The redundancy percentage is the ratio of the total redundancy generated by input-dependent operations and output-dependent operations to the normal input / output of the template fusion unit. Here, normal input / output refers to the amount of data in the on-chip cell graph that has no redundancy before it is split. The processing device 203 calculates the memory access size from DRAM 204 to SRAM 308 in the on-chip cell graph after the template fusion unit incorporates the current layer. TFU Size of normal input / output (excluding redundancy) ori The percentage of memory accesses, where the size is [not specified]. TFU This refers to the theoretical memory access size. ori Add the redundancy sum. The formula is as follows:

[0112]

[0113] The processing device 203 incorporates the splitting information and shape derivation of the template fusion unit and sets a percentage threshold of 50%, 75%, 100%, 125%, or 160%, preferably 100%. For example, a percentage threshold of 100% means that fusion will cease when the total redundancy exceeds twice the normal input / output of the template fusion unit. This rule ensures that the total redundancy generated by splitting the on-chip unit diagram does not exceed a specific proportion related to the percentage threshold. If it does, it indicates excessive redundancy, and a large amount of resources will be consumed in calculating this redundancy, leading to decreased efficiency. Therefore, when the processing device 203 determines that this rule is not met, it will stop fusion.

[0114] It should be noted that in small-map mode, no redundancy occurs because at least one complete feature map is loaded at a time during the process from DRAM 204 to SRAM 308. This rule does not apply to small-map mode.

[0115] Rule 12: Input / output dimensions of on-chip unit diagram

[0116] Assuming the SRAM 308 has a space size of S, the on-chip cell diagram requires storage space IN, and the calculation result of the on-chip cell diagram requires storage space OUT, then the SRAM 308's space size must meet the following conditions:

[0117] If IN and OUT cannot reuse storage space, then IN+OUT

[0118] If IN and OUT can reuse storage space, then MAX(IN,OUT)

[0119] That is, if IN and OUT cannot reuse storage space, the sum of the storage space of the on-chip cell diagram and the storage space of the calculation result is less than the available space of SRAM 308; if IN and OUT can reuse storage space, the larger of the storage space of the on-chip cell diagram and the storage space of the calculation result is less than the available space of SRAM 308.

[0120] Rule Thirteen: W i +IN1+IN2≤S

[0121] In small-image mode, this rule requires the SRAM 308's space size to meet the following conditions:

[0122] W i +IN1+IN2≤S

[0123] That is, the storage space W required for the weights of subgraph i. i ​​The total storage space IN1 and cache space IN2 required by the on-chip cell diagram shall not exceed the available space of SRAM 308. When the processing device 203 determines that this rule is not met, the processing device 203 reduces the number of on-chip cell diagrams until the rule is met.

[0124] Rule Fourteen: SubINi+W i +IN2≤S

[0125] In small-image mode, this rule requires the SRAM 308's space size to meet the following conditions:

[0126] SubINi+W i +IN2≤S

[0127] That is, the required storage space SubINi for subgraph i and the required storage space W for the weights of subgraph i. i The sum of the cache space IN2 is not greater than the available space of SRAM 308. When the processing device 203 determines that this rule is not met, the processing device 203 reduces the number of on-chip cell diagrams until the rule is met.

[0128] Rule 15: SubOUTi+W i+1 +IN2≤S

[0129] In small-image mode, this rule requires the SRAM 308's space size to meet the following conditions:

[0130] SubOUTi+W i+1 +IN2≤S

[0131] That is, the storage space required for the intermediate result of subgraph i (SubOUTi) and the storage space required for the weights of the next subgraph (W). i+1 The sum of the cache space IN2 is not greater than the available space of SRAM 308. When the processing device 203 determines that this rule is not met, the processing device 203 reduces the number of on-chip cell diagrams until the rule is met.

[0132] Rule Sixteen: W i +W i+1 ≤W

[0133] In template fusion units, the weights involved in convolution operations are independently moved and reside in WRAM 432. In small-graph mode, if a subgraph contains multiple feature maps, considering the pipeline between subgraphs, WRAM 432 can store the weights of at most two adjacent subgraphs simultaneously. Assume the required storage space for each subgraph i is W. i And the total space of WRAM 432 is W. This rule requires that the space size of WRAM 432 must meet the following conditions:

[0134] Wi +W i+1 ≤W

[0135] That is, the storage space W required for the weights of subgraph i. i The storage space W required for the weights of the next subgraph i+1 The total number of cells must not exceed the available space of WRAM 432. When the processing device 203 determines that this rule is not met, the processing device 203 reduces the number of on-chip cell diagrams until the rule is met.

[0136] Rule 17: The storage space required for a subgraph shall not exceed the available space of NRAM.

[0137] This rule states that the storage space required for a subgraph should not exceed the available space of NRAM 431. When the on-chip cell graph on SRAM 308 needs to be split into subgraphs and moved to NRAM 431, processing device 203 can perform fine-grained splitting along the N, H, and W dimensions. If the space in NRAM 431 is insufficient, processing device 203 will split the on-chip cell graph into finer subgraphs until this rule is met. Generally, NRAM 431 will have a reasonable amount of available space, allowing the on-chip cell graph to be split to a reasonable extent and loaded at once. From the perspective of the fusion strategy, the template fusion unit is not affected by the batch size. However, the smaller the on-chip cell graph is split (i.e., the more subgraphs), the slower the processing speed will be. Therefore, processing device 203 needs to evaluate the space of NRAM 431.

[0138] In some embodiments, the space of SRAM 308 corresponds to the number of NRAM 431 units in the processor cores 306 within cluster 305. For example, if cluster 305 includes four processor cores 306, then the space of SRAM 308 is four times the space of NRAM 431. In other words, on-chip units in large-image mode. Figure 1 Typically, data can be allocated to four processor cores (306). This architecture design already considers that data loaded into SRAM (308) can be allocated to all NRAM (431) cores at once. Therefore, this rule does not need to be considered in large-graph mode.

[0139] Rule 18: The number of feature maps must not exceed the feature map threshold.

[0140] In small-map mode, an on-chip unit graph may include multiple feature maps. The more feature maps there are, the more sub-graph transfers occur between SRAM 308 and NRAM 431, leading to decreased efficiency. Therefore, it is not always better for an on-chip unit graph to include more feature maps. The processing device 203 calculates an appropriate number of fusion layers based on the number of feature maps in the on-chip unit graph to maximize its efficiency. This rule is that the number of feature maps in the on-chip unit graph should not exceed a feature map threshold. When the processing device 203 determines that this rule is not met, it reduces the number of feature maps in the on-chip data until the rule is met.

[0141] Rule 19: Step Redundancy

[0142] Stride redundancy refers to the situation where, when a template fusion unit (SFU) fuses too many layers, and the dimensions of the convolutional and pooling kernels exceed the stride, the input data required for each output point overlaps. This overlap constitutes the aforementioned input-dependent computation, and this overlapping portion is stride redundancy. Stride redundancy forces each processor core 306 to read more data, but this reused data consumes on-chip and off-chip access resources. The more layers a SFU includes, the more severe the stride redundancy becomes. The rule is that the sum of the differences between the side length and stride of the convolutional or pooling kernel should not exceed the redundancy threshold.

[0143] In this embodiment, the redundancy threshold is defined as follows. Assume the length and width of the kernels for the convolutional and pooling layers are k. x and k y The step sizes in the length and width directions are s respectively. x and s y The stride redundancy in the longitudinal direction is k of all convolutional and pooling layers within the template fusion unit. x -s x The sum of all convolutional and pooling layers within the template fusion unit; similarly, the stride redundancy in the width direction is k. y -s y The sum of the values. The redundancy threshold in this embodiment can be 3, 4, 5, or 6, preferably 4. This rule is not satisfied as long as the step size redundancy in either the longitudinal or lateral direction is greater than the redundancy threshold. The processing device 203 adjusts the template fusion unit, typically by reducing the number of layers to be fused, until this rule is satisfied.

[0144] The fusion strategy sets an exception rule for step redundancy. If the layer to be fused contains multiple branches and the template fusion unit can fuse all multiple branches, the template fusion unit will perform better. In this case, the processing device 203 will ignore the step redundancy rule; that is, step redundancy will not restrict the template fusion unit from fusing multiple branches. In this embodiment, fusing multiple branches takes precedence over the restriction of step redundancy. In other words, step redundancy is only considered in the case of a single branch.

[0145] The above rules are merely examples. This invention does not limit the order in which the rules are executed, nor does it limit the simultaneous consideration of these rules. Those skilled in the art can add or delete rules according to the actual situation in different application scenarios to achieve a fusion strategy that conforms to the current application scenario.

[0146] Back Figure 11 In step 1103, neural network computation is performed based on the established template fusion unit. The computing device 201 is based on a three-level operation hierarchy of system-on-chip-cluster-processor core, and is equipped with a three-layer memory design of DRAM-SRAM-NRAM / WRAM. The template fusion unit is regarded as a custom layer in the neural network. The data required for computing the template fusion unit is loaded from DRAM 204 to SRAM 308 at one time, so that the data can be cached and computed at the appropriate level, forming a sufficient pipeline. After the computation is completed, the computation result is transferred from SRAM 308 to DRAM 204, which greatly reduces the input / output overhead in neural network computation.

[0147] When input data from fields such as computer vision, speech, natural language processing, and data mining is to be processed by various deep learning and machine learning algorithms, this invention, based on a template fusion unit, can reduce the input / output overhead in neural network computation. Another embodiment of this invention is a method for performing neural network computation using a template fusion unit. Figure 12 The process is shown.

[0148] In step 1201, template fusion units are determined according to the fusion strategy. The processing device 203 selects the starting layer of the template fusion unit based on the starting rule of the fusion strategy; and performs fusion based on the starting layer, checking all rules of the fusion strategy one by one to establish the template fusion unit. The previous embodiment has already illustrated the various rules of the fusion strategy in detail, and will not be repeated here.

[0149] In this step, the template fusion unit is presented as source code. The compiler then needs to convert this source code into machine language object code, also known as machine code. The following steps describe the process by which the compiler converts the source code of the template fusion unit into machine language object code.

[0150] In step 1202, the shape of the template fusion unit is derived. For the data that the template fusion unit needs to process, this embodiment uses a backward deduction method; the compiler works backward from the output to deduce the required input size. Figure 8For example, the process involves reverse deriving from feature map 803 to feature map 802, and then reverse deriving to feature map 801. In this step, the compiler not only derives the required input data based on the template fusion unit, but also further derives redundancy.

[0151] Next, step 1203 is executed to deduce the addresses. Based on the shape of the template fusion unit, the compiler deduces the on-chip memory addresses for the entire control flow graph and implements general-purpose address access to achieve the goal of simplifying computational resources and shortening computation time. A control flow graph is an abstract data structure used in compilers, representing all possible execution paths of a program, reflecting the possible flow of all nodes within the process in the form of a flowchart. A control flow graph is composed of nodes and the relationships between them. A node, also called a basic block (BB), is a sequence of statements executed as sequentially as possible in a program. Each basic block has only one entry and one exit point; execution begins at its entry point and exits at its exit point. A key characteristic of a basic block is that once the first instruction is executed, all instructions within that block will be executed sequentially.

[0152] Each basic block contains at least one instruction, and the instructions within a basic block may use pointers to point to specific on-chip memory locations. A pointer is a variable used to store an address in a specific address space. Through pointers, the processor core 306 can load data into or retrieve data from the specific address space pointed to by the pointer.

[0153] The compiler initially divides the basic blocks according to the partitioning of the template fusion unit, and then confirms the basic blocks and their interrelationships after iterative calculations, thus completing the target code for implementing the template fusion unit.

[0154] Furthermore, the compiler analyzes the reused data between consecutive template fusion units in the neural network, determines how much data from the previous template fusion unit can remain on-chip for use by the next template fusion unit, and plans the storage address of each data based on the determination result.

[0155] In this step, the compiler completes the derivation of addresses in the control flow graph.

[0156] In step 1204, on-chip memory space is allocated. Processing device 203 allocates physical space for SRAM 308, NRAM 431, and WRAM 432 based on the derivation of the template fusion cell address. In this step, the compiler completes the pointer assignment in the control flow graph.

[0157] Finally, step 1205 is executed to generate executable instructions. In this step, the linker links the object code generated by the compiler with external libraries, making it an executable file. More specifically, the object code is a program module that includes machine code and information available to the linker. The linker's job is to resolve undefined symbol references, replace placeholders in the object code with the addresses of symbols, and thus generate executable instructions. The executable instructions can be directly executed by the computing device 201 to complete the computation of the neural network.

[0158] When determining the template fusion unit according to the aforementioned fusion strategy rules, it is not necessarily required to start the fusion process with a convolutional layer or a pooling layer. As mentioned in the previous embodiments, in one application scenario, the starting rule could be that the starting layer is the first unfused layer in the neural network, which can be a layer other than a convolutional or pooling layer. This starting rule makes the establishment of template fusion units more flexible, allowing for the appropriate selection of the starting layer for fusion based on the order of layers in different neural networks. It is not limited by the position or number of convolutional or pooling layers in the neural network model, thus adapting to various network models, making the fusion more comprehensive, and improving overall efficiency.

[0159] For example, with Figure 6 Taking a neural network model as an example, assuming that layers 1 to 5 have been fused, when building the next template fusion unit, if the starting rule is to use the first unfused convolutional or pooling layer as the starting layer, then the next convolutional or pooling layer will be the 8th layer. In other words, layers 6 and 7 may not be fused, affecting the overall efficiency.

[0160] Another embodiment of the present invention is a scheme for fusing neural networks, wherein the starting layer is a layer other than convolutional and pooling layers, i.e., a non-convolutional and non-pooling layer. This embodiment is also based on... Figures 1 to 4 This is implemented using the framework. This embodiment also executes as follows: Figure 11 The flowchart shown.

[0161] In step 1101, a starting layer is selected according to a fusion strategy. The processing device 203 selects a starting layer according to a fusion strategy, for example, the starting rule of the fusion strategy is that the starting layer is the first unfused layer in the neural network, which is a layer other than a convolutional layer or a pooling layer. The starting layer can be an element-wise layer, an addpadding layer, or a custom layer.

[0162] It should be noted that this step does not use the starting rule of selecting the first unfused convolutional or pooling layer. If the starting layer is selected according to this starting rule, it will restrict the starting layer to be a convolutional or pooling layer, and the advantage of this embodiment not being limited by the position and number of convolutional or pooling layers in the neural network model will not exist.

[0163] If the neural network includes a branching structure, according to Rule 3 mentioned above, fusion should be prioritized on a branching basis. However, sometimes the branching structure is too complex to be integrated into the template fusion unit, and based on the aforementioned rules, the fusion of the branching structure must be abandoned. Furthermore, Rule 4 requires the output of the template fusion unit to be a single-branch output, reflecting the necessity of fusion on a branching basis. In other words, the fusion strategies of Rules 3 and 4 are not favorable for neural network models with branching structures, resulting in poor fusion performance.

[0164] Another embodiment of the present invention is a device for dynamically fusing the branch structure of a neural network according to a fusion strategy, which also has Figures 1 to 4 The structure. This embodiment does not necessarily have to be merged with a complete branch structure. Figure 13 An exemplary neural network model fragment is shown, which includes a branch structure 1300, which starts at layer T1 and ends at layer T10. A first branch 1301 and a second branch 1302 are expanded from layer T1. The first branch 1301 includes layers T2 and T3, and the second branch 1302 includes layers T4 to T9.

[0165] When merging branch structure 1300, processing device 203 first establishes a topological sequence for branch structure 1300. A topological sequence refers to arranging all nodes in a directed acyclic graph into a linear sequence, satisfying the following two conditions: each node must appear exactly once; if there exists a path from node A to node B, then node A appears before node B in the sequence. Simply put, it's the process of obtaining a total order from a partial order on a set. Based on the aforementioned principles, when establishing the topological sequence, processing device 203 first identifies the start and end points of branch structure 1300, i.e., the start point is layer T1 and the end point is layer T10. Processing device 203 sets the start point of branch structure 1300 as the start point of the topological sequence, which is also set as the starting layer of the template fusion unit, and sets the end point of branch structure 1300 as the end point of the topological sequence. Then, it arranges the intermediate layers in branch structure 1300 according to the topology, with the following two arrangements.

[0166] The first arrangement method compares the number of layers in each branch and arranges the layers of the sub-branches from most to least. The second arrangement method compares the number of layers in each branch and arranges the layers of the sub-branches from least to most. This embodiment adopts the second arrangement method. The first branch 1301 has 2 layers, and the second branch 1302 has 6 layers. The first branch 1301 has fewer layers, therefore the layers in the first branch 1301 are placed before the layers in the second branch 1302. Based on this arrangement method, such as Figure 14As shown, a topological sequence is formed with layers T1 → T2 → T3 → T4 → T5 → T6 → T7 → T8 → T9 → T10. After transformation, the topological sequence of branch structure 1300 forms a long chain structure 1400.

[0167] This embodiment adds or removes template fusion units on a layer-by-layer basis in the topological sequence, rather than on an entire branch structure. The processing device 203 replaces the branch structure 1300 with a long chain structure 1400 and checks the rules within the fusion strategy to establish template fusion units. In other words, the processing device 203 treats the neural network model with the branch structure 1300 as a neural network model with a long chain structure 1400, and uses the starting layer (T1 layer) of the long chain structure 1400 as a reference for fusion. In this way, any rule within the aforementioned fusion strategy (except for rule three and rule four) can be selected to establish template fusion units.

[0168] In this embodiment, the template fusion unit does not necessarily need to include the entire branch structure 1300. For example, suppose the long chain structure 1400 can generate two template fusion units: the first template fusion unit 1401 includes layers T1 to T5, and the second template fusion unit 1402 includes layers T6 to T10. When the long chain structure 1400 is restored to the branch structure 1300, the shapes of the first template fusion unit 1401 and the second template fusion unit 1402 are as follows: Figure 15 As shown, the first template fusion unit 1401 has two branch outputs, which are respectively connected to the T10 layer and T6 layer of the second template fusion unit 1402. That is, the first template fusion unit 1401 has two output terminals, and the second template fusion unit 1402 has two input terminals.

[0169] To improve data transfer efficiency, when deriving the shape of the first template fusion unit 1401, the processing device 203 then determines whether the first template fusion unit 1401 includes the endpoint of the branch structure 1300. Since the first template fusion unit 1401 does not include the T10 layer, the processing device 203 further determines whether the available space in the NRAM 431 is large enough. If so, when deriving the address in step 1203, the processing device 203 causes the computing device 201 to store the two calculation results produced by the first template fusion unit 1401 (i.e., the intermediate results between the last layer T3 and T5) in the NRAM 431, because the second template fusion unit 1402 can directly retrieve values ​​from the NRAM 431 for calculation. If the available space in the NRAM 431 is insufficient, the processing device 203 further determines whether the available space in the SRAM 308 is large enough. If the available space in SRAM 308 is large enough, these two calculation results will be stored in SRAM 308, and the calculation of the second template fusion unit 1402 can be performed directly from SRAM 308.

[0170] Since these two calculation results are the on-chip cell diagram of the second template fusion unit 1402, the computing device 201 does not need to load the on-chip cell diagram from the DRAM 204 when calculating the second template fusion unit 1402. Instead, it can directly read the calculation from the NRAM 431 or SRAM 308, reducing on-chip and off-chip access.

[0171] If the available space in NRAM 431 and SRAM 308 is not large enough, the computing device 201 will store the two calculation results produced by the first template fusion unit 1401 back into DRAM 204. When the computing device 201 calculates the second template fusion unit 1402, it will load these two calculation results from DRAM 204 for calculation.

[0172] When deriving the shape of the second template fusion unit 1402, the processing device 203 determines whether the second template fusion unit 1402 includes the endpoint of the branch structure 1300. If the second template fusion unit 1402 does include the T10 layer, then when deriving the address in step 1203, the processing device 203 causes the computing device 201 to store the calculation result produced by the second template fusion unit 1402 back into the DRAM 204.

[0173] In summary, the processing device 203 of this embodiment converts the branch structure into a long chain structure. The long chain structure is simple and easy to generate template fusion units. Then, the long chain structure is restored to the branch structure for shape and address derivation, eliminating the need for fusion on a per-branch-structure basis. The computing device 201 performs neural network calculations based on the template fusion units.

[0174] Another embodiment of the present invention is also a device for fusing branch structures, which also has Figures 1 to 4 The structure. With Figure 15 The difference between this and other branching structures is that this embodiment can merge branching structures with sub-branches. Figure 16 An exemplary neural network model fragment is shown, comprising a branch structure 1600, which begins at layer T1 and ends at layer T11. From layer T1, there are a first branch 1601 and a second branch 1602. The first branch 1601 includes layers T2 to T7, and the second branch 1602 includes layers T8 to T10. The first branch 1601 includes a sub-branch structure, which begins at layer T3 and ends at layer T7. The first sub-branch 1603 includes layers T4 and T5, and the second sub-branch 1604 includes layer T6.

[0175] When merging branch structure 1600, processing device 203 first establishes a topology sequence for branch structure 1600. First, it identifies the start and end points of branch structure 1600, i.e., the start point is layer T1 and the end point is layer T11. Processing device 203 sets the start point of branch structure 1600 as the start point of the topology sequence, which is also set as the starting layer of the template fusion unit, and sets the end point of branch structure 1600 as the end point of the topology sequence. Processing device 203 further determines whether branch structure 1600 has sub-branch structures. If branch structure 1600 does have sub-branch structures, processing device 203 first identifies the start and end points of the sub-branch structures, i.e., layers T3 and T7, and then arranges the start point, end point, and intermediate layers in the sub-branch structures according to the topology. There are two possible arrangements.

[0176] The first arrangement method compares the number of layers in the sub-branches of the sub-branch structure, and arranges the layers of the sub-branches in order of the number of layers, from most to least. The first sub-branch 1603 has 2 layers, and the second sub-branch 1604 has 1 layer. Since the first sub-branch 1603 has more layers, its layers are placed before those of the second sub-branch 1604. Based on this arrangement, the topological order of the sub-branch structure is: T3 layer → T4 layer → T5 layer → T6 layer → T7 layer.

[0177] The second arrangement method compares the number of layers in the sub-branches within the sub-branch structure, arranging the layers of the sub-branches from fewest to most. The second sub-branch 1604 has fewer layers, therefore its layers are placed before those in the first sub-branch 1603. Based on this arrangement, the topological order of the sub-branch structure is: T3 layer → T6 layer → T4 layer → T5 layer → T7 layer.

[0178] After processing the topological sorting of the sub-branch structures, the processing device 203 continues to sort the branch structures 1600. In this embodiment, the sorting method of the branch structures 1600 is the same as that of the sub-branch structures. In other words, if the sub-branches adopt the first arrangement method (arranged from most to least number of layers), then the branch structures 1600 are also arranged from most to least number of layers. The first branch 1601 has more layers than the second branch 1602, therefore each layer of the first branch 1601 is placed before each layer of the second branch 1602, thus generating... Figure 17 The long chain structure 1701 shown; if the sub-branches adopt the second arrangement method (arranged in ascending order of the number of layers), then the layers of the second branch 1602 of the branch structure 1600 are placed before the layers of the first branch 1601, thus generating Figure 17 The long chain structure 1702 is shown.

[0179] Next, the processing device 203 replaces the branch structure 1600 with a long chain structure 1701 or 1702, adds or removes template fusion units on a layer-by-layer basis in the topology sequence, and checks the rules within the fusion strategy to establish template fusion units. Similarly, in this embodiment, the template fusion unit does not necessarily need to include the entire branch structure 1601 or 1602.

[0180] To improve data transfer efficiency, when deriving the shape of the template fusion unit of branch structure 1601 or 1602, the processing device 203 determines whether the template fusion unit includes the endpoint of a branch structure or sub-branch structure. If not, the processing device 203 further determines whether the available space of NRAM 431 is large enough. If so, when deriving the address in step 1203, the processing device 203 causes the computing device 201 to store the intermediate result of the final layer produced by the template fusion unit in NRAM 431. If the available space of NRAM 431 is not large enough, the processing device 203 further determines whether the available space of SRAM 308 is large enough. If the available space of SRAM 308 is large enough, the intermediate result of the final layer will be stored in SRAM 308, and the template fusion unit can be calculated directly from SRAM 308.

[0181] If the template fusion unit does not include the end point of the branch structure or sub-branch, it means that its output (the intermediate result of the last layer) is the on-chip cell diagram of the next template fusion unit. Therefore, when the computing device 201 calculates the next template fusion unit, it does not need to load the on-chip cell diagram from DRAM 204, but directly reads the calculation from NRAM 431 or SRAM 308, reducing on-chip and off-chip access.

[0182] However, if the available space of NRAM 431 and SRAM 308 is not large enough, the intermediate result of the last layer of the template fusion unit will be stored back into DRAM 204. When the computing device 201 calculates the next template fusion unit, it loads the result from DRAM 204 for calculation.

[0183] If the processing device 203 determines that the template fusion unit includes the branch structure 1600 or the end point of the sub-branch, then when the processing device 203 derives the address in step 1203, it causes the computing device 201 to store the intermediate result of the last layer produced by the template fusion unit back into the DRAM 204.

[0184] Although this embodiment is described with a branch structure including one sub-branch structure, those skilled in the art can easily extrapolate to the case of multiple sub-branches, so it will not be elaborated further. The processing device 203 in this embodiment converts the branch / sub-branch structure into a long chain structure. The long chain structure is simple and easy to generate template fusion units. Then, the long chain structure is restored to a branch structure for shape and address derivation. The computing device 201 performs neural network calculations based on the template fusion units.

[0185] Another embodiment of the present invention is a method for dynamically fusing the branch structure of a neural network according to a fusion strategy. This embodiment is composed of... Figures 1 to 4 The structure is used to merge branch structures with sub-branches. Figure 18 A flowchart illustrating this embodiment is shown.

[0186] In step 1801, a topological sequence is established for the branch structure. This step is further divided into the following steps.

[0187] In step 1802, the start and end points of the branch structure are identified. In step 1803, the start point of the branch structure is set as the start point of the topology sequence. In step 1804, this start point is set as the starting layer of the template fusion unit. In step 1805, the end point of the branch structure is set as the end point of the topology sequence. In step 1806, it is determined whether the branch structure has sub-branch structures. If so, step 1807 is executed to identify the start and end points of the sub-branch structures. In step 1808, the start point, end point, and intermediate layers in the sub-branch structure are arranged in a specific order. There are two ways to arrange them: comparing the number of layers on the sub-branches in the sub-branch structure and arranging the layers of the sub-branch from the most numerous to the least numerous; and comparing the number of layers on the sub-branch structures and arranging the layers of the sub-branch from the fewest numerous to the most numerous. After processing the topology sorting of the sub-branch structure, or if it is determined in step 1806 that the branch structure does not have sub-branch structures, step 1809 is executed to sort the layers of the branch structure in a specific order. In this embodiment, the branch structure is ordered in the same way as the sub-branch structure. Thus, this embodiment transforms the branch structure into a long chain structure.

[0188] Next, step 1810 is executed, replacing the branch structure with a long chain structure. Template fusion units are added or removed using layers in the topology sequence as units. Based on the starting layer set in step 1804, the rules within the fusion strategy are checked to establish template fusion units. This step is essentially replacing the branch structure with a long chain structure. Step 1102 is then executed; the technical details are not elaborated here.

[0189] When deriving the shape of the template fusion unit for a branch structure or sub-branch structure, this embodiment determines whether the template fusion unit includes the endpoint of the branch structure or sub-branch structure. If not, it further determines whether the available space of NRAM 431 is large enough. If so, during address derivation, the computing device 201 stores the intermediate result of the last layer produced by the template fusion unit in NRAM 431. If the available space of NRAM 431 is not large enough, it further determines whether the available space of SRAM 308 is large enough. If the available space of SRAM 308 is large enough, the intermediate result of the last layer is stored in SRAM 308, and the template fusion unit can be calculated directly from SRAM 308.

[0190] However, if the available space in NRAM 431 and SRAM 308 is not large enough, this embodiment will store the intermediate results of the last layer of the template fusion unit back into DRAM 204, and load them from DRAM 204 for calculation when calculating the next template fusion unit.

[0191] If the template fusion unit includes the endpoint of a branch structure or a sub-branch structure, this embodiment causes the computing device 201 to store the intermediate result of the last layer produced by the template fusion unit back into the DRAM 204 when deriving the address.

[0192] Finally, step 1811 is executed, where neural network computation is performed based on the template fusion unit.

[0193] Another embodiment of the present invention is a computer-readable storage medium storing computer program code that dynamically fuses the branch structure of a neural network according to a fusion strategy. When the computer program code is run by a processor, the method described in the foregoing embodiment is executed.

[0194] This invention dynamically determines the template fusion unit by setting a fusion strategy, fuses the branch structures in the neural network to form a new custom layer, and loads the data required to compute the template fusion unit at once to reduce input / output overhead.

[0195] Depending on the application scenario, the electronic devices or apparatus of the present invention may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablet computers, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, dashcams, navigators, sensors, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, autonomous driving terminals, vehicles, home appliances, and / or medical devices. The vehicles include airplanes, ships, and / or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs. The electronic devices or apparatus of the present invention can also be applied in fields such as the Internet, IoT, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and healthcare. Furthermore, the electronic devices or apparatus of the present invention can also be used in application scenarios related to artificial intelligence, big data, and / or cloud computing, such as cloud computing, edge computing, and terminal computing. In one or more embodiments, the high-computing-power electronic devices or apparatuses according to the present invention can be applied to cloud devices (e.g., cloud servers), while the low-power electronic devices or apparatuses can be applied to terminal devices and / or edge devices (e.g., smartphones or cameras). In one or more embodiments, the hardware information of the cloud devices and the hardware information of the terminal devices and / or edge devices are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud devices to simulate the hardware resources of the terminal devices and / or edge devices based on the hardware information of the terminal devices and / or edge devices, so as to complete the unified management, scheduling and collaborative work of end-to-cloud or cloud-edge-end integration.

[0196] It should be noted that, for the sake of brevity, this invention describes some methods and their embodiments as a series of actions and combinations thereof. However, those skilled in the art will understand that the solution of this invention is not limited to the order of the described actions. Therefore, based on the disclosure or teachings of this invention, those skilled in the art will understand that some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in this invention can be considered as optional embodiments, that is, the actions or modules involved are not necessarily essential for the implementation of one or more solutions of this invention. In addition, depending on the solution, the description of some embodiments of this invention also has different emphases. In view of this, those skilled in the art will understand that parts not described in detail in a certain embodiment of this invention can also refer to the relevant descriptions of other embodiments.

[0197] In terms of specific implementation, based on the disclosure and teachings of this invention, those skilled in the art will understand that the several embodiments disclosed herein can also be implemented in other ways not disclosed herein. For example, regarding the various units in the electronic device or device embodiments described above, this document has divided them based on logical functions, but in actual implementation, there may be other ways of division. As another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. Regarding the connection relationship between different units or components, the connection discussed above in conjunction with the accompanying drawings can be a direct or indirect coupling between units or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection utilizing an interface, wherein the communication interface can support electrical, optical, acoustic, magnetic, or other forms of signal transmission.

[0198] In this invention, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same position or distributed across multiple network units. Furthermore, depending on actual needs, some or all of the units can be selected to achieve the purpose of the solution described in the embodiments of this invention. Additionally, in some scenarios, multiple units in the embodiments of this invention may be integrated into one unit or each unit may exist physically independently.

[0199] In some implementation scenarios, the integrated unit described above can be implemented as a software program module. If implemented as a software program module and sold or used as an independent product, the integrated unit can be stored in a computer-readable storage device (CMSDD). Therefore, when the solution of this invention is embodied in a software product (e.g., a computer-readable storage medium), the software product can be stored in a memory, which may include several instructions to cause a computer device (e.g., a personal computer, server, or network device) to execute some or all of the steps of the method described in the embodiments of this invention. The aforementioned memory may include, but is not limited to, various media capable of storing program code, such as USB flash drives, flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0200] In other implementation scenarios, the integrated units described above can also be implemented in hardware, i.e., as specific hardware circuits, which may include digital circuits and / or analog circuits. The physical implementation of the circuit's hardware structure may include, but is not limited to, physical devices, which may include, but are not limited to, transistors or memristors. Therefore, the various devices described herein (e.g., computing devices or other processing devices) can be implemented using appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage units or storage devices can be any suitable storage medium (including magnetic storage media or magneto-optical storage media), such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), ROM, and RAM.

[0201] The embodiments of the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. An integrated circuit device for dynamically fusing branch structures of a neural network according to a fusion strategy, comprising: Processing device, for: Based on the branch structure, a topological sequence is established, wherein, in establishing the topological sequence, the starting point and ending point of the branch structure are identified, and the starting point is set as the starting layer; the starting point of the branch structure is set as the starting point of the topological sequence; the ending point of the branch structure is set as the ending point of the topological sequence; and the number of layers of each branch is compared, and the layers of the sub-branches are arranged in order of the number of layers from most to least, or in order of the number of layers from least to most, to sort the intermediate layers of the branch structure; and The fusion process is performed based on the starting layer of the topology sequence. Rules within the fusion strategy are examined to establish template fusion units. Specifically, the starting layer of the template fusion unit is selected according to the starting rule of the fusion strategy, and fusion is performed based on this starting layer. All rules of the fusion strategy are examined one by one to establish the template fusion unit. The template fusion unit flexibly merges multiple layers into one layer using a certain fusion strategy to reduce network input / output overhead. A computing device is used to perform neural network computation based on the template fusion unit, wherein the input of the first layer and the output of the last layer in the template fusion unit serve as the interaction data between the template fusion unit and off-chip memory, and the computation of each intermediate layer does not require access to off-chip memory.

2. The integrated circuit apparatus of claim 1, wherein the processing apparatus, when establishing the topology sequence, is further configured to: Determine whether the branch structure has a sub-branch structure; if so, then: Identify the start and end points of the sub-branch structure; The starting point, ending point, and layers in the sub-branch structure are arranged in a specific order.

3. The integrated circuit device according to claim 2, wherein the processing device, when arranging the start point, end point, and layer in the sub-branch structure in a specific order, is further configured to: Compare the number of levels on the sub-branches in the sub-branch structure; and The sub-branches are arranged in order of the number of layers, from most to least.

4. The integrated circuit device according to claim 2, wherein the processing device, when arranging the start point, end point, and layer in the sub-branch structure in a specific order, is further configured to: Compare the number of levels on the sub-branches in the sub-branch structure; and The sub-branches are arranged in order of increasing number of layers.

5. The integrated circuit device of claim 2, wherein the computing device comprises a plurality of clusters, each cluster comprising a shared storage unit, and the processing device is further configured to determine whether the template fusion unit includes the endpoint of the sub-branch structure; if not, the computing device stores the intermediate results of the last layer of each sub-branch in the sub-branch structure in the template fusion unit in the shared storage unit.

6. The integrated circuit device according to claim 2, wherein the computing device includes a plurality of clusters, each cluster includes a plurality of processor cores, each processor core includes a neuron storage unit, and the processing device is further configured to determine whether the template fusion unit includes the endpoint of the sub-branch structure; if not, the computing device stores the intermediate results of the last layer of each sub-branch in the sub-branch structure in the template fusion unit in the neuron storage unit.

7. The integrated circuit device of claim 1, wherein the computing device comprises a plurality of clusters, each cluster comprising a shared storage unit, and the processing device is further configured to determine whether the template fusion unit includes the endpoint of the branch structure; if not, the computing device stores the intermediate results of the last layer of each branch in the branch structure in the template fusion unit in the shared storage unit.

8. The integrated circuit device of claim 1, wherein the computing device comprises a plurality of clusters, each cluster comprising a plurality of processor cores, each processor core comprising a neuron storage unit, and the processing device is further configured to determine whether the template fusion unit includes the endpoint of the branch structure; if not, the computing device stores the intermediate results of the last layer of each branch in the branch structure in the template fusion unit in the neuron storage unit.

9. The integrated circuit device of claim 1, wherein the starting layer is a layer other than the convolutional layer and the pooling layer.

10. The integrated circuit device of claim 1, wherein the starting layer is the first unfused layer in the neural network.

11. The integrated circuit device of claim 1, wherein the fusion strategy is to add or delete the template fusion unit on a layer-by-layer basis in the topology sequence.

12. A board comprising an integrated circuit device according to any one of claims 1 to 11.

13. A method for dynamically fusing the branch structure of a neural network according to a fusion strategy, comprising: Based on the branch structure, a topological sequence is established, wherein the establishment steps include identifying the start and end points of the branch structure, setting the start point as the starting layer, setting the start point of the branch structure as the start point of the topological sequence, setting the end point of the branch structure as the end point of the topological sequence, and comparing the number of layers of each branch, arranging the layers of the sub-branches from most to least according to the number of layers or from least to most according to the number of layers to sort the intermediate layers of the branch structure; The fusion is performed based on the starting layer of the topology sequence. The rules within the fusion strategy are checked to establish a template fusion unit. The starting layer of the template fusion unit is selected according to the starting rule of the fusion strategy, and fusion is performed based on the starting layer. All rules of the fusion strategy are checked one by one to establish the template fusion unit. The template fusion unit flexibly merges multiple layers into one layer through a certain fusion strategy to reduce the input / output overhead of the network. as well as Neural network computation is performed according to the template fusion unit, wherein the input of the first layer and the output of the last layer in the template fusion unit serve as the interaction data between the template fusion unit and off-chip memory, and the computation of each intermediate layer does not require access to off-chip memory.

14. The method of claim 13, wherein the establishing step further comprises: Determine whether the branch structure has a sub-branch structure; if so, then: Identify the start and end points of the sub-branch structure; as well as The starting point, ending point, and layers in the sub-branch structure are arranged in a specific order.

15. The method of claim 14, wherein the arranging step comprises: Compare the number of layers on the sub-branches in the sub-branch structure; as well as The sub-branches are arranged in order of the number of layers, from most to least.

16. The method of claim 14, wherein the arranging step comprises: Compare the number of layers on the sub-branches in the sub-branch structure; as well as The sub-branches are arranged in order of increasing number of layers.

17. A computer-readable storage medium storing computer program code that dynamically fuses the branch structure of a neural network according to a fusion strategy, wherein when the computer program code is run by a processing device, it performs the method according to any one of claims 13-16.

Images