A method for mapping and placing convolutional neural networks based on time division multiplexing

Abstract

The application discloses a method for mapping and layout of convolutional neural network based on time division multiplexing, and relates to the technical field of network-on-chip and convolutional neural network research. Firstly, in the preprocessing stage, the target convolutional neural network model is flattened, and neuron calculation operations are described using similar mathematical representations. Secondly, in the model analysis stage, the operation amount of each layer of the target convolutional neural network is analyzed, and neurons are clustered into larger neurons. Thirdly, in the mapping stage, the operation process of the convolutional neural network is sliced, and the slicing is iteratively mapped onto the network-on-chip platform using a time division multiplexing mapping algorithm. Finally, in the layout stage, the constructed calculation unit is laid out in the network-on-chip. The method disclosed by the application is helpful to map and layout modern large convolutional neural networks onto a network-on-chip computing platform with limited resources, and the method is also helpful to cooperate with a multicast routing algorithm to reduce congestion and delay during network-on-chip communication.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of network on chip and convolutional neural network, and particularly relates to a method for mapping and layout of convolutional neural network based on time division multiplexing. BACKGROUND

[0002] Convolutional neural network (CNN) has been widely used in pattern recognition, prediction and other applications. In recent years, many large-scale CNN models have been proposed, such as VGG-16, AlexNet, etc., which have shown amazing high precision in many application fields. With the further development of CNN, more and more hardware accelerators have been proposed as an energy-saving solution for running CNN. Network on chip (NoC) as an efficient data transmission solution for multi-processor system-on-chip (MPSoC) has been applied to CNN computing.

[0003] However, in order to map the entire large-scale CNN model to the NoC platform, either the NoC scale is expanded or the computing power of the processing elements (PEs) is enhanced. However, considering the cost, power consumption and other factors, both of these solutions are actually not feasible when processing large-scale CNN models.

[0004] In addition, under the CNN computing application, the data flow pattern is significantly different from that in the traditional MPSoC: during CNN computing, more than 60% of the data packets will be transmitted through only 30% of the channels, and the data flow will be particularly concentrated on the transmission channels connecting two adjacent neuron layers, while the channels in the same neuron layer are mostly idle. Obviously, this communication situation will lead to a very unbalanced traffic pattern and cause congestion on the NoC.

[0005] In order to solve the above problems, the present application proposes a method for mapping and layout of convolutional neural network based on time division multiplexing, which helps to map and layout modern large convolutional neural networks to the network on chip computing platform with limited resources, and the method helps to cooperate with multicast routing algorithm to reduce congestion and delay during network on chip communication. SUMMARY

[0006] The present application aims to provide a method for mapping and layout of convolutional neural network based on time division multiplexing.

[0007] The purpose of the present application can be achieved by the following technical solutions:

[0008] A method for mapping and layout of convolutional neural network based on time division multiplexing, the method comprising:

[0009] In the preprocessing stage, the target CNN model is flattened, and the neuron calculation operation is described using similar mathematical representation;

[0010] In the model analysis stage, the operation amount of each layer of the target convolutional neural network is analyzed, the neurons are clustered into large neurons, and a clustered CNN is obtained;

[0011] In the mapping stage, the operation process of the clustered CNN is sliced, and the slicing is iteratively mapped onto the network-on-chip platform using a time division multiplexing mapping algorithm;

[0012] In the layout stage, the corresponding PE and NoC are generated, the generated PE is placed, and they are interconnected through NoC to build the corresponding CNN model.

[0013] As a further scheme of the present application: in the preprocessing stage, after the flattening process of the convolutional layer, the pooling layer and the fully connected layer of the CNN model, the different layers in the CNN model are converted into a fully connected or partially connected neural network, realizing the flattening of the CNN model.

[0014] As a further scheme of the present application: the two-dimensional convolution operation in the CNN model is constructed as formula (1):

[0015]

[0016] Where K represents a predefined convolution kernel; n represents the size of the target convolution; I and O represent the input and output of the convolution layer, respectively; for three-dimensional convolution, the kernel function in the above formula is kept unchanged, and the input is expanded to other dimensions;

[0017] The pooling operation in the CNN model uses the maximum pooling operation to sample the input of the pooling layer, and is constructed as formula (2):

[0018]

[0019] Where I and Op represent the input and output of the pooling layer; m represents the size of the pooling kernel; W is equal to 1; Defined as formula (3):

[0020]

[0021] The fully connected operation in the CNN model is constructed as formula (4):

[0022]

[0023] Where n is the number of neurons in a layer, and v is the output number after the neuron calculation in the layer.

[0024] As a further scheme of the present application: in the model analysis stage, the neurons in the flat CNN model are divided into several groups, and the neurons at different levels are divided into large neurons of different groups according to size constraints;

[0025] Each large neuron is respectively mapped to a PE, and the position of the PE on the NoC platform will depend on the mapping strategy.

[0026] As a further scheme of the present application: the PEs are divided into multiply-add PEs, max-pooling PEs, average-pooling PEs and addition PEs.

[0027] As a further scheme of the present application: in the mapping stage, in the slicing of the operation process of the clustered CNN, the largest layer of the clustered CNN needs to be completely executed on the NoC platform, and at least one layer is executed under a working time slot.

[0028] As a further scheme of the present application: the time-division multiplexing mapping algorithm process is: sharing the calculation resources on the CNN acceleration platform, mapping to the NoC platform at a time, and executing layer by layer and time by time based on the operation process of the clustered CNN, so that at least one layer is completely mapped to the NoC platform each time.

[0029] As a further scheme of the present application: in the layout stage, the PEs are connected using a 2D-mesh topology NoC of N*N, and N is defined as

[0030] After completing the mapping iteration calculation of each layer, the calculation result can be temporarily stored in the global storage for use in the next iteration calculation.

[0031] The present application has the beneficial effects that: through the stages of preprocessing, model analysis, time-division multiplexing mapping, layout, etc. of the target CNN model, the present application realizes the analysis of the operation amount of each layer of the CNN, the clustering of the neuron operation of the CNN, the slicing of the operation process of the CNN and the time-division multiplexing mapping to the NoC, and generates the corresponding PEs and NoC and constructs the CNN operation model; it is helpful to map and layout the modern large convolutional neural network to the on-chip network computing platform with limited resource scale, and the method is helpful to cooperate with the multicast routing algorithm to reduce the congestion and delay during on-chip network communication. BRIEF DESCRIPTION OF DRAWINGS

[0032] The present application will be further described below with reference to the accompanying drawings.

[0033] Figure 1 is a flowchart of the present application;

[0034] Figure 2 is a flat CNN model schematic diagram of the present application;

[0035] Figure 3 is the CNN model of the present application;

[0036] Figure 4 is the kind and number of PEs required after clustering of the CNN neurons of the present application;

[0037] Figure 5 is the flow chart of the CNN slicing and time-multiplexing mapping method of the present application;

[0038] Figure 6 is the schematic diagram of the CNN equalization mapping algorithm of the present application;

[0039] Figure 7 is the schematic diagram of the PE mapping position result of the CNN network of the present application;

[0040] Figure 8 is the schematic diagram of the time slot iteration calculation of the CNN network of the present application. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0042] Please refer to Figure 1 The present application is a method for mapping and layout of convolutional neural network based on time-multiplexing, which comprises the following steps:

[0043] S101: preprocessing stage: flattening the target CNN model, and using similar mathematical representation to describe the neuron calculation operation;

[0044] S102: model analysis stage: analyzing the operation amount of each layer of the target CNN model, and clustering the neurons into macro neurons, the model analysis stage is used to analyze the target CNN model and determine the neuron operation involved;

[0045] S103: mapping stage: slicing the operation process of the CNN, and using time-multiplexing mapping algorithm to iteratively map the slices to the network-on-chip platform;

[0046] S104: layout stage: generating corresponding PEs and NoC, placing the generated PEs, and interconnecting them through NoC to construct the corresponding CNN model.

[0047] In a specific embodiment: in the step S101, for a specific CNN model, it is usually composed of convolutional layers, pooling layers and fully connected layers: the convolutional layer extracts some features from the output by performing convolution with a predefined convolution kernel and the input matrix; the pooling layer accepts the output of the convolutional layer and is used to reduce the data dimension, wherein the max pooling and the average pooling are the most popular pooling methods in the current CNN model; the fully connected layer connects the neurons of a layer to all the neurons of the next layer, which is used to extract features from the convolutional layer and the pooling layer and perform classification;

[0048] In the mapping and layout method of the application, in order to evaluate the calculation amount and facilitate the future mapping to the same PE, a flattening process is first performed on the CNN for the three types of calculations described above; then the calculation types are unified, and the three types of operations are represented as similar multiply-add operations.

[0049] In an implementation, the flattening process schematic diagram of the convolutional layer, the pooling layer and the fully connected layer of the CNN model is as shown in Figure 2 After the flattening process, the different layers in the CNN model can be converted into a fully connected or partially connected neural network, which is referred to as a flat CNN in the application.

[0050] In the mapping and layout method of the application, for the two-dimensional convolution operation in the CNN, it is constructed as formula (1):

[0051]

[0052] Wherein K represents a predefined convolution kernel; n represents the size of the target convolution; I and O represent the input and output of the convolutional layer respectively; for the three-dimensional convolution case, the kernel function in the above formula is kept unchanged, and the input is expanded to other dimensions.

[0053] In the mapping and layout method of the application, for the pooling operation in the CNN, the maximum pooling operation is used to sample the input of the pooling layer, which is constructed as formula (2):

[0054]

[0055] Wherein I and Op represent the input and output of the pooling layer; m represents the pooling kernel size; W is equal to 1. It is defined as formula (3):

[0056]

[0057] In this way, the pooling operation can also be represented as a similar multiply-add operation.

[0058] In the mapping and layout method of the application, for the fully connected operation in the CNN, it is constructed as formula (4):

[0059]

[0060] where n is the number of neurons in a layer and v is the number of outputs of the neurons in the layer.

[0061] In one specific embodiment, the step S102 comprises analyzing the operation amount of each layer of the target CNN and clustering the neurons into larger neurons. The meaning of the clustering operation is that the CNN involves the calculation of millions or even tens of millions of neurons, and if a data packet is transmitted after each neuron finishes calculation, it will generate a huge traffic load. In order to reduce the network traffic load and improve the calculation efficiency, the calculation of multiple neurons can be allocated to one PE.

[0062] The mapping and layout method of the present application divides the neurons in the flat CNN model into several groups, each of which is called a large neuron. The maximum number of neurons in a group (i.e. the size of the large neuron) depends on the calculation capability of a PE. The CNN model after grouping is called a clustered CNN. In this clustering strategy, the neurons at different levels will be divided into different groups of large neurons according to the size constraint (the number of neurons in the clustered neuron group). Each large neuron is mapped to a PE, and the location of the PE on the NoC platform will depend on the mapping strategy. The calculation result of a PE can be transmitted to other PEs with large neuron connectivity. In this way, the output of a large neuron will become the input of several large neurons in the next layer, thereby simplifying the calculation process.

[0063] Since there is a feedforward calculation process in the conventional neural network operation, i.e. the input of each neuron layer usually reuses the output result of the previous neuron layer, the mapping and layout method of the present application adopts an inter-layer clustering strategy to support output reuse. In this way, the feedforward calculation flow of the CNN model can be preserved, and the traffic load on the NoC platform can be reduced.

[0064] In one implementation, a CNN example for illustration is defined as Figure 3 as shown, where the number of channels of the convolution layer 1 is 3, and the size of the convolution kernel is 5x5; the number of channels of the max pooling layer 1 is 3, and the size of the pooling kernel is 2x2; the number of channels of the convolution layer 2 is 3, and the size of the convolution kernel is 4x4; the number of channels of the max pooling layer 2 is 3, and the size of the pooling kernel is 2x2; the number of neurons of the fully connected layer 1 is 128, the number of neurons of the layer 2 is 32, and the number of neurons of the layer 3 is 10.

[0065] In an implementation, for the current mainstream CNN model, the PE can be divided into multiply-add PE, maximum pooling PE, average pooling PE, and addition PE. Among them, the multiply-add PE is used for convolution operation, and is usually used in convolution layer and full connection layer, and is used for multiplying and accumulating the input data and the corresponding weight data. Different convolution operations of different channels can be calculated in different multiply-add PEs. The maximum pooling PE is used to find the maximum value between the given kernel and the channel input data. The maximum pooling PE is used to find the maximum value of the partial region between the given channel input data. The average pooling PE is used to calculate the average value of the partial region between the given channel input data. The addition PE can be used for connection between two consecutive layers to realize cross-layer operation. The addition module adds two input data to generate an output result.

[0066] The specific implementation of the above-mentioned PE in the NoC can be homogeneous or heterogeneous. In the homogeneous case, a PE that is common to any calculation needs to be constructed, which should be able to perform different types of calculations in the convolution layer, the pooling layer, and the full connection layer. Therefore, the design of the PE should enable it to support different calculations and control its behavior at runtime.

[0067] In an implementation, for the current mainstream CNN model, the PE can be divided into multiply-add PE, maximum pooling PE, average pooling PE, and addition PE. Among them, the multiply-add PE is used for convolution operation, and is usually used in convolution layer and full connection layer, and is used for multiplying and accumulating the input data and the corresponding weight data. Different convolution operations of different channels can be calculated in different multiply-add PEs. The maximum pooling PE is used to find the maximum value between the given kernel and the channel input data. The maximum pooling PE is used to find the maximum value of the partial region between the given channel input data. The average pooling PE is used to calculate the average value of the partial region between the given channel input data. The addition PE can be used for connection between two consecutive layers to realize cross-layer operation. The addition module adds two input data to generate an output result. Figure 3 As shown in the CNN example, the types and numbers of PEs required after clustering of neurons of each layer are as shown in Figure 4 Each PE is set to process data from the same data dimension, that is, each multiply-add PE of the convolution layer and each maximum (average) pooling PE of the pooling layer calculates data of the same channel. Further, the maximum value of the calculation operation of each PE, that is, the number of neurons that can be calculated at each iteration, is set. According to the setting of the maximum value, the clustering of the PEs is completed, and then the number of PEs required by each type of PE can be finalized. From the CNN example in Figure 3 In the CNN example in

[0068] In a specific embodiment, in the S103 step, a CNN slicing method is provided, and the minimum requirement of the method is that the maximum layer (i.e., the layer containing the most neurons) of the CNN needs to be able to be completely executed on the NoC platform. In this way, it can be guaranteed that at least one layer is executed under a working time slot.

[0069] The mapping and layout method of the present application provides a time-multiplexing mapping method to share the computing resources on the CNN acceleration platform, which can be mapped to the NoC platform at a time. Figure 5 The implementation flow of the proposed CNN slicing and time-multiplexing mapping method is shown. The proposed slicing method is performed layer by layer and time by time to ensure that at least one layer is completely mapped to the NoC platform each time. The scheduling method of time-multiplexing is to maintain the CNN as a rotating time wheel in the order of execution, and to divide the time wheel according to the CNN slicing method. The execution time of each CNN is defined according to the WCRT. In order to be able to support all the computing operations in the CNN, the mapping and layout method of the present application considers the worst case to determine the number of each PE required. The method adopted is to select the maximum number of PEs required for each PE type in the same working time slot.

[0070] In an implementation, for Figure 3 For the CNN example shown, 3 MAC PEs are required to calculate the convolution layer 1; 3 pooling PEs are required to calculate the pooling layer 1; 3 MAC PEs are required to calculate the convolution layer 2; 3 pooling PEs are required to calculate the pooling layer 2; 4 MAC PEs are required to calculate the fully connected layer 1; 1 MAC PE is required to calculate the fully connected layer 2; and 1 MAC PE is required to calculate the fully connected layer 3. In this example, the convolution layer 2 and the max pooling layer 1 are divided as the boundary, and the two layers occupy different working time slots, which are respectively denoted as working time slot 1 and working time slot 2. Therefore, the two layers can time-multiplex the PEs for similar operations. In this example, the maximum number of PEs required in the same time slot is 9, i.e. 6 MAC PEs and 3 max pooling PEs are required in each of the working time slots 1 and 2, so using 9 PEs to calculate this CNN model is sufficient.

[0071] In an implementation, the simplest mapping method is to map each large neuron to the NoC platform in the order of dimensions. Although this dimension order mapping algorithm is easy to implement, serious traffic congestion may occur on the NoC platform because adjacent large neurons are mapped too densely. In order to solve this problem and obtain the minimum hop count and more balanced NoC traffic, a CNN balanced mapping algorithm is applied in the mapping and layout method of the present application, which can reduce the routing distance and data traffic between the NoCs, thereby reducing the delay and ensuring the traffic of the NoC communication to a certain extent, and the mapping algorithm is also suitable for applying a multicast routing algorithm to further optimize the network traffic.

[0072] The mapping and layout method of the present application provides a CNN balanced mapping algorithm. First, the following is defined:

[0073] Definition 1: A Convolutional Neural Network Communication Graph (CNNCG) is a directed graph represented by G(N, A), where each node n i represents a macro neuron in a clustered CNN, and the communication flow from one macro neuron n i to its connected macro neuron n j in the next layer is modeled as a directed edge a i,j.

[0074] Definition 2: An Architecture Feature Graph (ARCG) G'(U, L) is a directed graph representing the actual NoC, where each node u i represents a node in the NoC, and each edge l i represents a NoC communication link.

[0075] Definition 3: For an ARCG G(U, L), a deterministic routing function: R→P maps r i,j to a routing path p i,j, where p i,j ∈ P i,j.

[0076] Based on these definitions, CNN equalization mapping can be expressed as: given a CNNCG and an ARCG, a mapping function map() needs to be found that satisfies the following conditions, as expressed in equation (5):

[0077]

[0078] In one implementation, for a 2D-mesh network topology, it is assumed that the row and column numbers of the xth PE are rown,x and coln,x, respectively. Correspondingly, rown+1,y and coln+1,y are the row and column numbers of the yth connected PE in the adjacent layer. The CNN equalization mapping algorithm pseudo code is shown in Figure 6 The specific process of the algorithm is as follows:

[0079] The algorithm inputs the CNNCG (N, A) and the ARCG (U, L);

[0080] For each node in the CNNCG, it is mapped to the ARCG, and a node graph is iteratively formed;

[0081] For each directed edge in A, find the minimum value of ∑|rowi-rowj|+|col i-col j|, record this minimum value as Hmin, and record the node graph at this time as mapmin;

[0082] For all routing paths that satisfy the Hmin condition, find the sum of the communication traffic of each link under the largest working time slot, record this value as Packsum, and find a Packsum value for each node graph;

[0083] Output the node graph with the smallest Packsum value.

[0084] In one specific embodiment, the S104 step includes:

[0085] According to the mapping result in S103, a CNN is constructed and laid out.

[0086] In an implementation manner, for Figure 3 In the CNN example shown, the PEs are connected using a 2D-mesh topology NoC of NxN, where N can be defined as The NoC can be obtained as Figure 3 In an example, a 3x3 2D Mesh network connection can be used. The PEs are designed to be reusable according to the size of the required calculation, to meet the calculation requirements of each working time slot. Figure 3 The mapping position result of the PEs of the example CNN network is shown in Figure 7 .

[0087] In an implementation manner, after the mapping iteration calculation of each layer is completed, the calculation result can be temporarily stored in a global storage for use in the next iteration calculation. Figure 3 The PE iteration calculation schematic diagram of working time slot 1 and working time slot 2 of the example CNN network is shown in Figure 8 . The left graph is the mapping result of working time slot 1, the first row is the calculation of convolution layer 1; the third row is the calculation of max pooling layer 2; and the second row is the calculation of convolution layer 2. The right graph is the mapping result of working time slot 2, the first PE in the first row and the second row is the calculation of full connection layer 1; the second PE in the second row is the calculation of full connection layer 2; and the third PE in the third row is the calculation of full connection layer 3.

[0088] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes one or more available media sets. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

[0089] The above detailed description has shown, by way of example, an embodiment of the application. It is specifically contemplated that the application is not limited to the embodiments described herein, but rather, the application is intended to cover all modifications and equivalents thereof falling within the scope of the application.

Claims

1. A method for mapping and laying out convolutional neural networks based on time-division multiplexing, characterized in that, The method includes: In the preprocessing stage, the target CNN model is flattened, and the neuron computational operations are described using similar mathematical representations; In the model analysis phase, the computational complexity of each layer of the target convolutional neural network is analyzed, and neurons are clustered into large neurons to obtain a clustered CNN. In the mapping stage, the operation process of the clustering CNN is sliced, and the slices are iteratively mapped to the on-chip network platform using a time-division multiplexing mapping algorithm. During the layout phase, corresponding PEs and NoCs are generated, the generated PEs are placed, and they are interconnected through NoCs to build the corresponding CNN model; In the preprocessing stage, after the convolutional layers, pooling layers, and fully connected layers of the CNN model are flattened, the different layers in the CNN model are transformed into a fully connected or partially connected neural network, thus achieving the flattening of the CNN model. The two-dimensional convolution operation in the CNN model is constructed as shown in equation (1): Where K represents the predefined convolution kernel; n represents the size of the target convolution; I and O represent the input and output of the convolutional layer, respectively. For three-dimensional convolution, the kernel function in the above formula remains unchanged, and the input is extended to other dimensions. The pooling operation in the CNN model uses max pooling to sample the input of the pooling layer, as shown in equation (2): Where I and Op represent the input and output of the pooling layer; m represents the pooling kernel size; W equals 1; Defined as Equation (3): The fully connected operation in the CNN model is constructed as shown in equation (4): Where n is the number of neurons in a layer, and v is the number of outputs calculated by the neurons in that layer.

2. The method for mapping and laying out a convolutional neural network based on time-division multiplexing according to claim 1, characterized in that, During the model analysis phase, the neurons in the flat CNN model are divided into several groups, and the neurons at different levels are divided into different groups of large neurons according to size constraints. Each large neuron is mapped to a PE, and the location of the PE on the NoC platform will depend on the mapping strategy.

3. The method for mapping and laying out a convolutional neural network based on time-division multiplexing according to claim 2, characterized in that, The PE is divided into multiply-add PE, maximum pooling PE, average pooling PE, and summation PE.

4. The method for mapping and laying out a convolutional neural network based on time-division multiplexing according to claim 1, characterized in that, During the mapping phase, in the slice of the clustering CNN operation process, the largest layer of the clustering CNN must be able to be fully executed on the NoC platform, ensuring that at least one layer is executed in one working time slot.

5. A method for mapping and laying out a convolutional neural network based on time-division multiplexing according to claim 4, characterized in that, The time-division multiplexing mapping algorithm process is as follows: sharing the computing resources on the CNN acceleration platform, mapping to the NoC platform once, and executing the operation process of clustering CNN layer by layer and time by time, so that at least one layer is fully mapped to the NoC platform each time.

6. The method for mapping and laying out a convolutional neural network based on time-division multiplexing according to claim 1, characterized in that, During the layout phase, PEs are connected using an N×N 2D-mesh topology NoC, where N is defined as... After completing the mapping iteration calculation for each layer, the calculation results can be temporarily stored in global storage for use in the next iteration calculation.

Images