A method of modeling a convolutional neural network as a synchronous dataflow graph

Abstract

This invention relates to a method for modeling convolutional neural networks (CNNs) as synchronous data flow graphs, belonging to the technical fields of multi-core processor systems, on-chip networks, and CNN research. The method first accepts a CNN model saved as a JSON file; secondly, it parses the model and collects information on the computation and interconnections of each neuron layer; thirdly, it uses this information to calculate mathematical representations such as the number of tokens to be sent on each channel, the number and type of operations to be performed by each neuron layer, and the depth of each neuron layer; finally, it uses these mathematical representations to generate a synchronous data flow graph. The method proposed in this invention helps designers accelerate the analysis of the mapping and scheduling process from CNNs to multi-core processor systems through the generated synchronous data flow graph, and also facilitates the theoretical analysis of system throughput and other metrics.

Description

Technical Field

[0001] This invention belongs to the field of research technology of multi-core processor systems, on-chip networks and convolutional neural networks, and relates to a method for modeling convolutional neural networks as synchronous data flow graphs. Background Technology

[0002] In recent years, the development of Convolutional Neural Networks (CNNs) has attracted the attention of researchers in related fields, and the increasing number of new CNN models has also promoted the development of CNN computing platforms. Advanced CNNs invented in recent years have exhibited many irregular characteristics in the connections between neuron layers, because neuron layers are no longer simply connected to each other in a sequential manner, but may have multiple ingress and egress layers.

[0003] With the increasing application of Network-on-Chip (NoC) based heterogeneous multi-processor system-on-chip (MPSoC) computing to CNNs, different types of neuron layers can be scheduled and run on appropriate processors according to different computational needs. Therefore, performing CNN computations with irregular neuron layer connections on MPSoCs based on heterogeneous NoCs offers the following advantages: neuron layers that do not depend on each other's outputs can operate in parallel; computational results from neuron layers with multiple outgoing layers can be effectively reused; and neuron layers with specific computational needs can be mapped to dedicated processing cores to more effectively accelerate the computation of that layer. To effectively utilize these advantages, a mapping model is needed that can determine which processors each neuron layer should execute on; can arrange the computational order among these processors; and can schedule the input and output data streams moving between processors.

[0004] Synchronous Data Flow Graph (SDFG) is a formal model frequently considered in electronic design and software compilers to model communication between components with different rates. It is widely used in analog and digital signal processing and multimedia applications.

[0005] Currently, in most general-purpose computing platforms, all computations in one layer can only be executed after another layer has been completed. The complex connections between these CNN neuron layers create computational dependencies between different layers, making it difficult for current general-purpose computing platforms to perform CNN computations. Summary of the Invention

[0006] The purpose of this invention is to provide a method for modeling convolutional neural networks as synchronous data flow graphs. This invention uses SDFG as a theoretical mapping model to simulate the data scheduling and computation process that occurs during CNN execution. By studying the irregularities of advanced CNN structures, a method for modeling CNNs as MPSoC SDFGs is proposed. The method proposed in this invention helps designers accelerate the analysis of the mapping and scheduling process from CNN to MPSoC through the generated SDFG, and also helps to theoretically analyze system throughput and other indicators.

[0007] The objective of this invention can be achieved through the following technical solutions:

[0008] A method for modeling a convolutional neural network as a synchronous data flow graph, the method comprising the following steps:

[0009] (1) Accepts CNN models saved as JSON files;

[0010] (2) Analyze the model and collect computational and connectivity information of neuron layers;

[0011] (3) Calculate the number of tokens that need to be sent on each channel;

[0012] (4) Calculate the number and type of operations that need to be performed in each neuron layer;

[0013] (5) Calculate the depth of each neuron layer;

[0014] (6) Generate SDFG.

[0015] Furthermore, the specific method of step (3) is as follows:

[0016] For a specific CNN model, its modeling granularity is specified as the neuron layer, which is used as the modeling unit; its neurons are clustered into nodes, and the neuron layer is represented as a neuron cluster; after the modeling granularity is determined, the CNN modeling process is implemented.

[0017] Calculate the port rate at the end of each channel, which means calculating the number of input and output tokens;

[0018] Define the data transmitted between neurons or neuron layers as a three-dimensional matrix of size (w,h,m); the number of tokens transmitted between two roles on NoC is defined as the number of data transmitted, w*h*m.

[0019] For connected layers in a CNN, we define a connected layer as having input layers x1, x2, ..., x... n The input to this connection layer is n, and its output will be connected to layer y; the input to layer x is... iAfter i∈{1,2....,n} passes through this connection layer, the output is a value of size (w,h,m) i The data matrix of the next layer in the connection layer has a dimension of )

[0020] For the connection layers in CNN, the hidden connection layer only retains the connection between its input and output layers and does not use roles to represent it. That is, the connection layer does not perform any computation and no tokens flow through it.

[0021] For convolutional layers in a CNN, the convolution kernel is defined as a two-dimensional matrix of size (k, k). The output dimension of the convolutional layer depends on a parameter called `padding` defined by Tensorflow. If `padding="valid"`, the number of computations required to apply the convolution kernel to the input matrix is... At the same time, the dimension of the output matrix can be obtained as If padding="same", then the number of computations required to apply the convolution kernel to the input matrix is... At the same time, the dimension of the output matrix can be obtained as

[0022] For batch normalization and activation layers in CNNs, the output matrix dimension is the same as the input matrix dimension, both defined as (w,h,m);

[0023] For the padding layer in a CNN, we define the number of pixels p1 and p2 filled in the w and h dimensions, respectively, and we can obtain the dimension of its output matrix as (w+2p1,h+2p2,m).

[0024] For flat layers in a CNN, the dimension of the output matrix is ​​the same as the dimension of its input matrix, both defined as (w,h,m);

[0025] For pooling layers in a CNN, the output dimension depends on a parameter called `padding` defined by Tensorflow. If `padding="valid"`, the dimension of the output matrix will be [value missing]. If padding="same", the dimension of the output matrix will be...

[0026] For a fully connected layer in a CNN, its input is defined as a one-dimensional vector of size (1,1,m), and the number of neurons in the fully connected layer is u. Then, this neuron layer will produce u output data, and m*u multiplication and addition operations will be performed in the fully connected layer.

[0027] For the additive layer in a CNN, it is assumed that the dimension of all input data is the same, defined as (w,h,m), and the dimension of the output matrix is ​​also defined as (w,h,m).

[0028] Furthermore, the specific method of step (4) is as follows:

[0029] Estimate the number of operations required to compute each layer of a CNN on different processors;

[0030] These operations are categorized as follows:

[0031] Addition and subtraction operations: Perform addition or subtraction, where subtraction is performed by adding to a negative number;

[0032] Multiplication: Performs multiplication between two numbers, which is done by a multiplier unit;

[0033] Multiply-add: The operation of adding after multiplication, which can be implemented by cross switches, pulsating arrays or other hardware architectures specifically designed for efficient multiplication;

[0034] Division operation: Performs division between two numbers, which is done by a divider unit;

[0035] Comparison operation: Performs a comparison between two numbers and outputs true (1) or false (0) based on the comparison result. This operation includes comparisons of greater than or equal to or less than or equal to.

[0036] Initialization operations: Perform data alignment or shifting to fill or flatten data;

[0037] In the modeling method of this invention, for each type of CNN layer and its definition method, the computational quantity of each neuron layer of the CNN is defined as follows:

[0038] Two-dimensional convolution: nmk 2 w'h' times of multiplication and addition operations;

[0039] Batch standardization: 2whd addition and subtraction operations, whd multiplication and division operations;

[0040] Filling: 2p1 + 2p2 initialization operations;

[0041] Activation: whd comparison operations;

[0042] Flattening: whd initialization operation;

[0043] Average pooling: n(k) 2 -1)w'h'+1 multiplication operations and nw'h' division operations;

[0044] Max pooling: n(k) 2 -1)w'h' comparison operations;

[0045] Fully connected: mu multiplication and addition operations;

[0046] Global average pooling: (wh-1)d addition and subtraction operations, d division operations;

[0047] Addition: whd addition and subtraction operations;

[0048] On a specific processor, the computation time of these layers can be estimated using the following formula, which defines the computational cost of each operation on each specific processor:

[0049] Calculation cost = Number of addition / subtraction operations * time for alkali addition operation + Number of multiplication operations * time for multiplication operation + Number of multiplication / addition operations * time for multiplication / addition operation + Number of division operations * time for division operation + Number of comparison operations * time for comparison operation + Number of initialization operations * time for initialization operation.

[0050] Furthermore, the specific method of step (5) is as follows:

[0051] We propose a metric to evaluate the degree of data dependency in CNN models, which is defined as the depth of the neuron layer.

[0052] The depth of a neuron layer is defined as follows: First, the CNN model is viewed as a directed graph, where the nodes represent the neuron layers, and the edges represent the dependencies of one layer on the output of another. In a non-cyclic graph, the depth of node x is represented as d. x It is defined as the length of the longest path from the input node to node x. This definition method is based on breadth-first traversal, which measures the depth of each node and marks the depth of each neuron layer according to the order of traversal.

[0053] A connection is defined as a "long connection" when the depth difference between two nodes is greater than 1. In the mathematical expression of the neuron layer depth algorithm, the numerator is the sum of all "long connections" in the graph.

[0054] For a mathematical expression of the algorithmic measure of neuron layer depth, consider a graph G(V, E), and define the attributes of a node i (i∈V) as having an input neuron set K. i And a d i Depth, and neuron j (j∈K) i ) has a depth d j ;

[0055] Absolute depth can be defined as follows:

[0056]

[0057] Relative depth can be defined as follows:

[0058]

[0059] Furthermore, the specific method of step (6) is as follows:

[0060] Generate a representation of the SDFG, which includes the number of operations, the number of tokens, and connection information between layers;

[0061] The multiprocessor platform is defined as a set of tiles, placed in a static location determined during platform construction. This location depends on the NoC topology used by the researchers. Each tile consists of a processor, memory, communication auxiliary unit, and network interface.

[0062] Provide a user-modifiable configuration file that specifies the latency of each operation on each type of processor. This configuration file defines a tuple for each processor, including processor name, addition / subtraction, initialization, multiplication / addition, multiplication, division, and comparison. The processor name will be bound to the processor node specified in the multiprocessor platform to represent the performance differences when a certain layer is mapped to different types of processors.

[0063] Furthermore, the modeling method of this invention can output SDFG to an Extensible Markup Language (XML) file, which is suitable for the analysis of other indicators.

[0064] The beneficial effects of this invention are:

[0065] The method proposed in this invention helps designers accelerate the analysis of the mapping and scheduling process from convolutional neural networks to multi-core processor systems through the generated synchronous data flow graph, and also helps to theoretically analyze indicators such as system throughput. Attached Figure Description

[0066] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0067] Figure 1 This is an example diagram of SDFG in Embodiment 1 of the present invention;

[0068] Figure 2 This is a flowchart of a method for modeling a convolutional neural network as a synchronous data flow graph according to Embodiment 1 of the present invention;

[0069] Figure 3 This is a schematic diagram of a hidden CNN connection layer in Embodiment 1 of the present invention;

[0070] Figure 4 This is a schematic diagram illustrating how the depth of each neuron layer is obtained using a neuron layer depth algorithm in Embodiment 1 of the present invention.

[0071] Figure 5This is a schematic diagram illustrating an implementation of a multiprocessor platform in Embodiment 1 of the present invention. Detailed Implementation

[0072] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided.

[0073] Example 1

[0074] SDFG is commonly used to model modern digital signal processing (DSP) applications and to implement concurrent multimedia applications on MPSoCs. SDFG is primarily used to analyze key system performance characteristics such as throughput, execution time, communication bandwidth, and buffer size. In an SDFG, nodes are called roles, and edges are called channels. A role typically represents a logical function that accepts input data (in tokens) from its input ports and outputs output data (also in tokens) from its output ports. A channel is a directional connection between input and output ports; it can be on two different roles or on the same role. In the former case, it represents data dependencies between roles, while in the latter case, it is typically used to model the execution of roles in self-timed execution.

[0075] During a role's execution, the number of tokens consumed at the input port or generated at the output port is called the port rate. The port rate determines the exact number of tokens transmitted on its associated channel when a role executes. The port rate is usually noted at the role and the channel connected to it. Each role has a delay for completing its execution, which is represented as a number in the role.

[0076] like Figure 1 The diagram shows an example of an SDFG. Role 1 outputs tokens to Role 2, and Role 2's output is then used as input to Role 3. Role 1 completes each execution in one time unit. Similarly, Role 2 and Role 3 complete each execution in two time units, as indicated next to each role name. Role 1 generates two tokens per execution on channel d1, and Role 2 consumes three tokens per execution on channel d1. Therefore, the output port rate of a1 and the input port rate of a2 can be represented as follows.

[0077] Simultaneously, in SDFG, the buffer size of a channel can be modeled by adding a new channel in the opposite direction for each channel. The rate of the newly added port should be equal to the rate of the corresponding port on the original channel, and the initial number of tokens on the new reverse channel should be equal to the initial storage space constrained by the channel. Figure 1 In this context, channels d3 and d4 are not actual dependencies between roles, but rather auxiliary channels used to model the buffer capacities of d1 and d2, respectively. The initial number of tokens on d3 is 4, which indicates that the buffer size of d1 is 4 tokens.

[0078] The application graph of a convolutional neural network can be modeled as an SDFG (Simplified Chinese Graph Graph), where nodes represent the layers of the convolutional neural network model under consideration, connections between nodes represent the dependencies between the outputs of different layers, and tokens represent the inputs and outputs of a layer. These tokens are sent to other layers for further processing. When representing a neural network model with an SDFG, a pipeline is not required; the input and output port rates of the channels are equal. This means that the number of tokens received by the consuming layer is the same as the number of tokens sent by the production layer, and the production and consuming layers only need to execute relative to each other once to keep the number of tokens in the channel constant in each time unit.

[0079] A method for modeling convolutional neural networks as a synchronous data flow graph, the flowchart of which is as follows: Figure 2 As shown, the method includes the following steps:

[0080] S101: Use the Tensorflow deep learning framework to generate a CNN model saved as a JSON file;

[0081] S102: Analyze the model and collect information on the computational layers of each neuron and their interconnections;

[0082] S103: Using the information in S102, calculate the number of tokens that need to be sent on each channel;

[0083] S104: Using the information in S102, calculate the number and type of operations that need to be performed in each neuron layer;

[0084] S105: Using the information in S102, calculate the depth of each neuron layer;

[0085] S106: Generate a synchronous data flow graph using the mathematical representations calculated in S103-S105.

[0086] The specific method for step S103 is as follows:

[0087] For a specific CNN model, its modeling granularity is specified as the neuron layer, which is used as the modeling unit; its neurons are clustered into nodes, and the neuron layer is represented as a neuron cluster; after the modeling granularity is determined, the CNN modeling process is implemented.

[0088] Calculate the port rate at the end of each channel, which is to calculate the number of input and output tokens, the same as the number of input and output data in each layer. The relevant symbols are explained in Table 1 below.

[0089] Table 1

[0090]

[0091] Define the data transmitted between neurons or neuron layers as a three-dimensional matrix of size (w, h, m), where the matrix ∈ N. 3 The number of tokens transferred between two roles on NoC, regardless of whether they are mapped to the same processing unit, can be defined as the amount of data transferred, i.e., w*h*m;

[0092] In this embodiment, for the connection layers in a CNN, it is defined that when the input layer of a connection layer is x1, x2, ..., x... n The input to this connection layer is n, and its output will be connected to layer y; the input to layer x is... i After i∈{1,2....,n} passes through this connection layer, the output is a value of size (w,h,m) i The data matrix of the next layer in the connection layer has a dimension of )

[0093] In this embodiment, the connection layer is hidden, such as Figure 3 As shown, only the connection between its input and output layers is preserved, and no role is used to represent it. That is, no computation occurs in the connection layer, and no tokens flow through it.

[0094] In this embodiment, for the convolutional layer in a CNN, its convolutional kernel is defined as a two-dimensional matrix of size (k,k). During convolution, the operation of this convolutional kernel is applied to a partition of size (k,k) in the input matrix. After the operation, it moves a stride s along a certain dimension and is applied to the next (k,k) partition of the input matrix, and so on. The output dimension of the convolutional layer depends on a parameter called padding defined by Tensorflow. If padding="valid", the number of computations required to apply the convolutional kernel to the input matrix is ​​[value missing]. At the same time, the dimension of the output matrix can be obtained as If padding="same", then the number of computations required to apply the convolution kernel to the input matrix is... At the same time, the dimension of the output matrix can be obtained as

[0095] In this embodiment, for the batch normalization layer and activation layer in the CNN, since they both apply the same injection to all elements in their input matrix, the output matrix dimension of the batch normalization layer and activation layer is the same as the input matrix dimension, and is defined as (w,h,m).

[0096] In this embodiment, for the padding layer in the CNN, the number of pixels filled in the w and h dimensions are defined as p1 and p2, respectively, and the dimension of its output matrix can be obtained as (w+2p1,h+2p2,m).

[0097] In this embodiment, for the flat layer in CNN, since the flat layer only rearranges the input matrix, the dimension of the output matrix of the flat layer is the same as the dimension of its input matrix, both defined as (w,h,m).

[0098] In this embodiment, for the pooling layer in a CNN, the pooling operation is viewed as applying a function to each element of a certain region of the input matrix, and then applying the same function to a new region of the input matrix. The dimension of the output matrix is ​​defined similarly to that of a two-dimensional convolution. The output dimension depends on a parameter named `padding` defined by Tensorflow. If `padding="valid"`, the dimension of the output matrix will be [value missing]. If padding="same", the dimension of the output matrix will be...

[0099] In this embodiment, for the fully connected layer in CNN, since the input of the fully connected layer is always flat, its input is defined as a one-dimensional vector of size (1,1,m). The number of neurons in the fully connected layer is u. Then the neuron layer will produce u output data. Each output data is generated by the weighted sum of each input data. It can be seen that m*u multiplication and addition operations will be performed in the fully connected layer.

[0100] In this embodiment, for the additive layer in CNN, since the additive layer is similar to the connection layer and can accept multiple inputs, it is assumed that the dimension of all input data is the same, defined as (w,h,m). Since the dimension of the output matrix of the additive layer is the same as the dimension of its input matrix, the dimension of the output matrix is ​​also defined as (w,h,m).

[0101] The specific method for step S104 is as follows:

[0102] Estimate the execution time of each layer of the CNN on different processors, i.e., the number of computational operations performed;

[0103] For certain types of operations, different types of processing units will be optimized. In the modeling method of this embodiment, these operations are divided into the following categories:

[0104] Addition and subtraction operations: Perform addition or subtraction, where subtraction is performed by adding to a negative number;

[0105] Multiplication: Performs multiplication between two numbers, which is done by a multiplier unit;

[0106] Multiply-add: The operation of adding after multiplication, which can be implemented by cross switches, pulsating arrays or other hardware architectures specifically designed for efficient multiplication;

[0107] Division operation: Performs division between two numbers, which is done by a divider unit;

[0108] Comparison operation: Performs a comparison between two numbers and outputs true (1) or false (0) based on the comparison result. This operation includes comparisons of greater than or equal to or less than or equal to.

[0109] Initialization operations: Perform data alignment or shifting to fill or flatten data;

[0110] In the modeling method of this invention, for each type of CNN layer and its definition method, the computational quantity of various neuron layers in CNN is defined. The computational quantity of CNN neuron layers is shown in Table 2 below:

[0111] Table 2

[0112]

[0113] Two-dimensional convolution: nmk 2 w'h' times of multiplication and addition operations;

[0114] Batch standardization: 2whd addition and subtraction operations, whd multiplication and division operations;

[0115] Filling: 2p1 + 2p2 initialization operations;

[0116] Activation: whd comparison operations;

[0117] Flattening: whd initialization operation;

[0118] Average pooling: n(k) 2 -1)w'h'+1 multiplication operations and nw'h' division operations;

[0119] Max pooling: n(k) 2 -1)w'h' comparison operations;

[0120] Fully connected: mu multiplication and addition operations;

[0121] Global average pooling: (wh-1)d addition and subtraction operations, d division operations;

[0122] Addition: whd addition and subtraction operations;

[0123] On a specific processor, the computation time of these layers can be estimated using the following formula, which defines the computational cost of each operation on each specific processor:

[0124] Calculation cost = Number of addition / subtraction operations * time for alkali addition operation + Number of multiplication operations * time for multiplication operation + Number of multiplication / addition operations * time for multiplication / addition operation + Number of division operations * time for division operation + Number of comparison operations * time for comparison operation + Number of initialization operations * time for initialization operation.

[0125] The specific method for step S105 is as follows:

[0126] To address the irregularity of layer connections in CNNs, a metric is proposed to evaluate the degree of data dependency in CNN models. This metric is defined as the depth of the neuron layer.

[0127] The depth of a neuron layer is defined as follows: First, the CNN model is viewed as a directed graph, where the nodes represent the neuron layers, and the edges represent the dependencies of one layer on the output of another. In a non-cyclic graph, the depth of node x is represented as d. x , which is defined as the length of the longest path from the input node (a node that does not depend on the output of another node) to node x. This definition method is based on breadth-first traversal, so that the depth of each node is measured. The depth of each neuron layer is marked according to the order of traversal. The pseudocode of the neuron layer depth algorithm is shown in Table 3 below.

[0128] Table 3

[0129]

[0130] A schematic diagram illustrating the depth of each neuron layer obtained using a neuron layer depth algorithm is shown below. Figure 4 As shown, the mathematical expression for the measurement of neuron layer depth is standardized. Whether there is a serious data dependency between nodes depends on the edge length between two nodes and the number of similar connections in the CNN model. When the depth difference between two nodes is greater than 1, the connection is defined as a "long connection". In the mathematical expression for the measurement of neuron layer depth, the numerator is the sum of all "long connections" in the graph.

[0131] For a mathematical expression of the algorithmic metric of neuron layer depth, consider a graph G(V, E) (or a network with |V| layers, with |E| connections between layers), and define the attribute of a node i (i∈V) as having a set of input neurons K.i And a d i Depth, and neuron j (j∈K) i ) has a depth d j ;

[0132] Absolute depth can be defined as follows:

[0133]

[0134] If there are no "long connections", the denominator of this expression is the total depth difference of all connections, that is, the total number of connections in the CNN. The absolute depth represents the average depth difference of all connections in the CNN.

[0135] Relative depth can be defined as follows:

[0136]

[0137] In this embodiment, the mathematical representation of relative depth can provide a metric to describe the degree of long-term data dependence of this CNN compared to other CNNs; and normalize the metric in the range [0,1].

[0138] In CNNs without "long connections", there are no two connections with a depth difference greater than 1, which would result in a metric of zero. However, for layers with multiple inputs and / or multiple outputs, the maximum depth will be much smaller than the number of connections. This also means that there are layers with the same depth that do not depend on each other's outputs and can therefore be executed in parallel. Some of the irregularity representation parameters of mainstream CNNs are shown in Table 4 below.

[0139] Table 4

[0140]

[0141] When researchers explore the hardware design space of a particular application to execute these CNNs, the algorithmic metric of neuron layer depth can serve as a parameter that helps classify and describe the irregularity of the CNN.

[0142] The specific method for step S106 is as follows:

[0143] Generate a representation of the SDFG, which includes the number of operations, the number of tokens, and connection information between layers;

[0144] The multiprocessor platform described above is defined as a set of tiles placed in a static location determined during platform construction, depending on the NoC topology used by the researchers. Each tile consists of a processor (P), memory (M), communication auxiliary unit (CA), and network interface (NI).

[0145] An implementation of a multiprocessor platform, such as Figure 5 As shown, the sub-components are interconnected. There are three tiles in the figure that are interconnected through a NoC architecture, where the R component represents the router in the NoC.

[0146] When designers use the modeling method in this invention to map a CNN onto a heterogeneous MPSoC, the configuration of each tile, including the specifications of all the components listed above, can differ from one another. The configurable tile component parameters are as follows:

[0147] Processor: A type of processor optimized for different types of computing;

[0148] Memory: storage size;

[0149] Network interface: input bandwidth, output bandwidth, number of connections to other tiles, and each connection group should have a bandwidth value set.

[0150] To make the generated SDFG applicable to different multiprocessor platforms, this embodiment provides a user-modifiable configuration file that specifies the latency of each operation on each type of processor. This configuration file defines a tuple for each processor, including processor name, addition / subtraction, initialization, multiplication / addition, multiplication, division, and comparison. The processor name is bound to the processor node specified in the multiprocessor platform. This is to represent the many different types of processors contained in the heterogeneous MPSoC, and to show the performance differences when a certain layer is mapped to different types of processors.

[0151] In this embodiment, the modeling method can output SDFG to an Extensible Markup Language (XML) file, which is suitable for the analysis of other metrics, such as using other SDFG analysis frameworks to analyze system throughput.

[0152] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for modeling a convolutional neural network as a synchronous data flow graph, characterized in that, The method includes the following steps: (1) Accepts CNN models saved as JSON files; (2) Analyze the CNN model and collect computational and connection information of neuron layers; (3) Calculate the number of tokens that need to be sent on each channel; (4) Calculate the number and type of operations that need to be performed in each neuron layer; (5) Calculate the depth of each neuron layer; (6) Generate a synchronous data flow graph; The specific method for step (3) is as follows: For a specific CNN model, its modeling granularity is specified as the neuron layer, which is used as the modeling unit; its neurons are clustered into nodes, and the neuron layer is represented as a neuron cluster; after the modeling granularity is determined, the modeling process of the CNN model is implemented. Calculate the port rate at the end of each channel, which means calculating the number of input and output tokens; The data transmitted between neurons or neuron layers is defined as a three-dimensional matrix of size (w, h, m), where w represents the input data width, h represents the input data height, and m represents the input data depth, i.e., the number of channels. The number of tokens transmitted between two roles on NoC is defined as the number of data transmitted, w. h m; For the connection layers in a CNN model, we define a connection layer as having input layers x1, x2, ..., x... n The input to this connection layer is n, where n represents the number of cores, and its output will be connected to the next layer after the current connection layer; input layer x i After i∈{1,2....,n} passes through this connection layer, the output is a value of size (w,h,m) i The data matrix of the next layer in the connection layer has a dimension of ) m i Indicates the depth of the i-th input data; For the connection layers in a CNN model, the hidden connection layer only retains the connection between its input and output layers and does not use roles to represent it. That is, the connection layer does not perform any computation and no tokens flow through it. For convolutional layers in a CNN model, the convolution kernel is defined as a two-dimensional matrix of size (k, k), where k represents the kernel size. The output dimension of the convolutional layer depends on a parameter called `padding` defined by Tensorflow. If `padding="valid"`, the number of computations required to apply the convolution kernel to the input matrix is ​​[calculation details missing]. s represents the step size, and the dimension of the output matrix can be obtained as follows: If padding="same", then the number of times the convolution kernel is applied to the input matrix is... At the same time, the dimension of the output matrix can be obtained as follows: ; For the batch normalization layer and activation layer in a CNN model, the dimension of the output matrix is ​​the same as the dimension of its input matrix, and is defined as (w,h,m); For the padding layer in a CNN model, defining the number of pixels p1 and p2 p1 in the w and h dimensions respectively, we can obtain that the dimension of its output matrix is ​​(w+2). p1,h+2 p2,m); For flat layers in a CNN model, the dimension of the output matrix is ​​the same as the dimension of its input matrix, both defined as (w,h,m); For pooling layers in a CNN model, the output dimension depends on a parameter called `padding` defined by Tensorflow. If `padding="valid"`, the dimension of the output matrix will be [value missing]. If `padding="same"`, the dimension of the output matrix will be [value missing]. ; For a fully connected layer in a CNN model, its input is defined as a one-dimensional vector of size (1, 1, m), and the number of neurons in the fully connected layer is u. Then, this neuron layer will produce u output data points, resulting in m operations performed in the fully connected layer. u multiplication and addition operations; For the additive layer in a CNN model, it is assumed that the dimension of all input data is the same, defined as (w,h,m), and the dimension of the output matrix is ​​also defined as (w,h,m).

2. The method for modeling a convolutional neural network as a synchronous data flow graph according to claim 1, characterized in that, The specific method for step (4) is as follows: Estimate the number of operations performed on different processors for each layer of a CNN model. These operations are categorized as follows: Addition and subtraction operations: Perform addition or subtraction, where subtraction is performed by adding to a negative number; Multiplication: Performs multiplication between two numbers, which is done by a multiplier unit; Multiply-add: The operation of adding after multiplication, which is implemented by cross switches, pulse arrays or other hardware architectures specifically designed for efficient multiplication; Division operation: Performs division between two numbers, which is done by a divider unit; Comparison operation: Performs a comparison between two numbers and outputs true1 or false0 based on the comparison result. This operation includes comparisons of greater than or equal to, or less than or equal to. Initialization operations: Perform data alignment or shifting to fill or flatten data; For each type of layer in a CNN model and its definition method, define the computational count for each neuron layer in the CNN model: Two-dimensional convolution: n m k 2 w' h' multiply-add operations, w' represents the output data width, and h' represents the output data height; Batch standardization: 2 w h d addition and subtraction operations, w h d multiplication and division operations, where d represents the node depth; Fill: 2 p1+2 p2 initialization operations; Activate: w h d comparison operations; Flattening: w h d initialization operations; Average pooling: n (k 2 -1) w' h'+1 multiplication operations, n w' h' division operations; Max pooling: n (k 2 -1) w' h' comparison operations; Fully connected: m u multiplication and addition operations; Global average pooling: (w h-1) d addition and subtraction operations, d division operations; Addition: w h d addition and subtraction operations; On a specific processor, the computation time of these layers is estimated according to the following formula, defining the computational cost of each operation on each specific processor: Computational cost = Number of additions and subtractions Addition and subtraction time + multiplication operands Multiplication operation time + multiplication-addition operation numbers Multiplication and addition operation time + division operation number Divide operation time + comparison operands Comparison operation time + initialization operands Initialize computation time.

3. The method for modeling a convolutional neural network as a synchronous data flow graph according to claim 1, characterized in that, The specific method for step (5) is as follows: A metric is proposed to evaluate the degree of data dependency in a CNN model, which is defined as the depth of the neuron layer. The depth of a neuron layer is defined as follows: First, the CNN model is viewed as a directed graph, where the nodes represent the neuron layers, and the edges represent the dependencies of one layer on the output of another. In a non-cyclic graph, the depth of node x is represented as d. x It is defined as the length of the longest path from the input node to node x. This definition method is based on breadth-first traversal, which measures the depth of each node and marks the depth of each neuron layer according to the order of traversal. A connection is defined as a "long connection" when the depth difference between two nodes is greater than 1. In the mathematical expression of the neuron layer depth algorithm, the numerator is the sum of all "long connections" in the graph. For a mathematical expression of the algorithmic measure of neuron layer depth, consider a graph G(V, E), where V represents the set of nodes and E represents the set of edges. Define a node x (x∈V) with the attribute of having an input neuron set K. x And a d x Depth, and neuron j (j∈K) x ) has a depth d j ; Absolute depth is defined as follows: Relative depth is defined as follows: Where, k x Represents the input neuron set K x The x-th element in d max This represents the total number of elements, corresponding to the maximum depth of all neuron layers in the CNN model. Indicates from d max The total number of ways to combine any two layers from a neuron layer of different depths.

4. The method for modeling a convolutional neural network as a synchronous data flow graph according to claim 1, characterized in that, The specific method for step (6) is as follows: Generate a representation of the synchronous data flow graph, which includes the number of operations, the number of tokens, and connection information between layers; The multiprocessor platform is defined as a set of tiles, placed in a static location determined during platform construction. This location depends on the NoC topology used by the researchers. Each tile consists of a processor, memory, communication auxiliary unit, and network interface. Provide a user-modifiable configuration file that specifies the latency of each operation on each type of processor. This configuration file defines a tuple for each processor, including processor name, addition / subtraction, initialization, multiplication / addition, multiplication, division, and comparison. The processor name will be bound to the processor node specified in the multiprocessor platform to represent the performance differences when a certain layer is mapped to different types of processors.

5. The method for modeling a convolutional neural network as a synchronous data flow graph according to claim 1, characterized in that, Step (6) generates a synchronous data flow graph and outputs it to an Extensible Markup Language file.

Images