Block scheduling method for inference storage optimization of convolutional neural network

Abstract

The invention discloses a block scheduling method for inference storage optimization of a convolutional neural network, and the method comprises the steps: constructing a directed acyclic graph of a directed acyclic graph of the convolutional neural network, and defining block sub-graphs; generating all candidate block sub-graphs based on an algorithm of a message matrix; meanwhile, on the basis of the storage cost, defining the partitioning cost for each partitioned sub-graph, determining the partitioning mode of the partitioned sub-graphs, and selecting an optimal partitioned sub-graph combination by using a dynamic programming algorithm; and performing block scheduling on each block sub-graph according to the steps, optimizing storage occupation in the reasoning process, and merging the completed output blocks. According to the method, by designing the block scheduling strategy based on the block sub-graph, the storage occupation in the reasoning process is optimized to be within a limited range, and the extra overhead is minimized. Therefore, storage occupation is reduced, and extra cost caused by layer fusion is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of neural network inference, and particularly relates to a block scheduling method for optimizing convolution neural network inference storage. Background Art

[0002] The technical challenges faced when deploying a convolution neural network on a hardware platform mainly focus on data movement and storage requirements. Although the layer fusion strategy can eliminate off-chip feature map data transmission, there are still some technical deficiencies:

[0003] Computational efficiency issue: Although the layer fusion strategy reduces the cost of data movement, it does not consider storage size, resulting in reduced storage utilization.

[0004] Limitations of the cache mechanism: Although the use of a cache mechanism to store intermediate calculation results to reduce repeated calculations is mentioned in the literature, this strategy may have limited effectiveness for high-dimensional data or large networks.

[0005] Storage capacity and bandwidth limitations: As the image resolution changes and the storage requirements increase, the hardware platform may require higher storage capacity and bandwidth. However, this may be limited by the hardware, and increasing storage capacity and bandwidth may incur additional costs and power consumption.

[0006] Algorithm optimization and adjustment: For different applications and tasks, it may be necessary to optimize and adjust the convolution neural network. This involves adjusting aspects such as the network structure, parameters, and learning rate, which may require a relatively high level of technical expertise and experience.

[0007] In summary, although the layer fusion strategy provides an effective method for solving the data movement problem, there are still some technical challenges and deficiencies in practical applications. To further improve the performance and efficiency of convolution neural networks, it is necessary to further research and explore new optimization methods and technologies. Summary of the Invention

[0008] The object of the present invention is to overcome the deficiencies of the prior art. To achieve the above object, a block scheduling method for optimizing convolution neural network inference storage is adopted to solve the problems raised in the above background art.

[0009] A block scheduling method for optimizing convolution neural network inference storage includes the following steps:

[0010] Step S1, construct a directed acyclic graph of the convolution neural network and define a block subgraph;

[0011] Step S2, generate all candidate block subgraphs based on an algorithm of a message matrix;

[0012] Step S3: Define the block cost for each sub-block graph based on the storage cost, determine the block method of the sub-block graph, and use the dynamic programming algorithm to select the optimal combination of sub-block graphs;

[0013] Step S4: Perform block scheduling on each sub-block graph according to the above steps, optimize the storage occupancy during the inference process, and merge the completed output blocks.

[0014] As a further solution of the present invention: The specific steps in the step S1 include:

[0015] Construct a directed acyclic graph of the convolutional neural network. The convolutional neural network is a directed acyclic graph, and the sub-graph corresponding to the consecutive layers to be blocked is used as a sub-block graph;

[0016] Define a sub-block graph according to any sub-graph G'(V', E') of the directed acyclic graph G(V, E) of the convolutional neural network, then it satisfies:

[0017] Condition 1: |V'| > 1;

[0018] Condition 2: |{v' ∈ V': indegree(v') = 0}| = 1, |{v' ∈ V': outdegree(v') = 0}| = 1; that is, there is only one vertex with an in-degree of zero and only one vertex with an out-degree of zero, which are respectively denoted as the entry vertex v in and the exit vertex v out ;

[0019] Condition 3: All the adjacent edges of all vertices v' in V' in G, that is, the edges with v' as the starting point or the ending point, are included in E', and it does not include the edges with the entry vertex as the ending point and the edges with the exit vertex as the starting point.

[0020] As a further solution of the present invention: The specific steps in the step S2 include:

[0021] Search for all sub-block graphs based on the algorithm of the message matrix: Set |V| = n. For any given topological sequence of the graph G, number all vertices according to the topological order, denoted as v1, v2,..., v n ;

[0022] Traverse all vertices according to the topological order. The current vertex sets its own initial message to 1 and collects the messages of other vertices from all its predecessor vertices. If its predecessor vertex has multiple successor vertices, the message is equally divided according to the number of successor vertices, and the current vertex obtains one share;

[0023] When the current vertex collects the messages of the same vertex from different predecessor vertices, these messages are merged;

[0024] According to the above steps, the message matrix of the directed acyclic graph is generated, and then all candidate sub-block graphs are generated according to the message matrix;

[0025] Since the entry vertex blocks overlap with each other, the calculation of the overlapping part, as well as the block division of the entry vertices and the merging of the exit vertices, constitute the overhead of the sub-block graph to solve the storage problem.

[0026] As a further solution of the present invention: the specific steps of the optimal solution in step S3 include:

[0027] Set the vertices that do not meet the storage requirements to be recorded in topological order as The set of optional sub-block graphs is T, and the set of sub-block graphs of the solution Satisfy:

[0028]

[0029]

[0030] The optimization problem is:

[0031]

[0032] Traverse For each The original sequence is divided into two subsequences And At the same time, the set T is also divided into two disjoint sets T L , T R , such that

[0033]

[0034]

[0035] That is, the original set is divided into two subsets that only cover one of the subsequences, then The original problem is divided into two sub-problems The optimal solution combination of the sub-problems is the optimal solution of the original problem in this division case, and the sum of the minimum costs of the sub-problems is the minimum cost of the original problem in this division case;

[0036] According to the above steps, select a sub-block graph with the minimum cost from the sub-block graphs V' that cover all the current vertices that do not meet the storage requirements as the solution in the non-division case, and select the solution with the minimum cost from the set composed of this solution and the solutions in the division cases.

[0037] As a further solution of the present invention: the specific steps in step S4 include:

[0038] Chunk the input layer of the consecutive layers to be optimized, perform the inference of the entire consecutive layer for each chunk, optimize the memory occupation during the inference process, and merge the completed output chunks.

[0039] Compared with the prior art, the present invention has the following technical effects:

[0040] Adopting the above technical solution, by chunking the input layer of the consecutive layers and performing the inference of the entire consecutive layer for each chunk, and finally merging the completed output chunks. This solution can significantly reduce the memory occupation during the inference process, optimize it to the limited range, thus avoiding the occupation of storage space by a large number of intermediate results. At the same time, this solution can also minimize the additional overheads, such as computational and communication overheads. By means of chunk-by-chunk inference, the additional costs brought by layer fusion can be effectively reduced, thereby improving the efficiency and accuracy of the inference. Brief Description of the Drawings

[0041] The following will describe in detail the specific implementation manners of the present invention with reference to the accompanying drawings:

[0042] Figure 1 It is a schematic diagram of the steps of the chunk-by-chunk fusion scheduling method according to the disclosed embodiment of the present application;

[0043] Figure 2 It is a schematic diagram of the memory occupation of ResNet50 according to the disclosed embodiment of the present application;

[0044] Figure 3 It is a schematic diagram of the chunk processing of ResNet50 according to the disclosed embodiment of the present application;

[0045] Figure 4 It is a schematic diagram of the original memory occupation of the linear subgraph according to the disclosed embodiment of the present application;

[0046] Figure 5 It is a schematic diagram of the chunked memory occupation of the linear subgraph according to the disclosed embodiment of the present application;

[0047] Figure 6 It is a schematic diagram of the original memory occupation of the residual subgraph according to the disclosed embodiment of the present application;

[0048] Figure 7 It is a schematic diagram of the chunked memory occupation of two residual subgraphs according to the disclosed embodiment of the present application;

[0049] Figure 8 It is a schematic diagram of the chunked memory occupation of four residual subgraphs according to the disclosed embodiment of the present application;

[0050] Figure 9 It is a schematic diagram of the non-chunked subgraph according to the disclosed embodiment of the present application;

[0051] Figure 10Schematic diagram of special cases of sub-block subgraphs for disclosed embodiments of the present application;

[0052] Figure 11 Schematic diagram of message passing for disclosed embodiments of the present application;

[0053] Figure 12 Schematic diagram of message matrix for disclosed embodiments of the present application;

[0054] Figure 13 Schematic diagram of receptive field for disclosed embodiments of the present application;

[0055] Figure 14 Schematic diagram of topological order for disclosed embodiments of the present application. Detailed implementation manners

[0056] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0057] Please refer to Figure 1 , in the embodiments of the present invention, a block scheduling method for optimizing inference storage of a convolutional neural network includes the following steps:

[0058] Step S1, construct a directed acyclic graph of the convolutional neural network and define sub-block subgraphs;

[0059] Step S2, generate all candidate sub-block subgraphs based on the algorithm of the message matrix;

[0060] Step S3, at the same time, define block costs for each sub-block subgraph based on storage costs, determine the block methods of the sub-block subgraphs, and use the dynamic programming algorithm to select the optimal combination of sub-block subgraphs;

[0061] Step S4, perform block scheduling on each sub-block subgraph according to the above steps, optimize the storage occupancy during the inference process, and merge the completed output blocks.

[0062] In this embodiment, the specific steps of the sub-block subgraph are as follows:

[0063] The storage distributions of each layer of the convolutional neural network are not balanced. In some layers, it may lead to storage bottlenecks, while in other layers, the storage requirements may be relatively small. As Figure 2 shown, the figure shows the storage occupancy schematic diagram of ResNet50. The storage peak of the ResNet50 network occurs in the middle residual block part. If the available storage is between the peak storage and other storage, then only optimizing some layers can solve the storage bottleneck problem.

[0064] Different from the per-pixel inference of the layer fusion strategy, we adopt a block scheduling inference scheme. We divide the input layer of the consecutive layers to be optimized into blocks, perform the inference of the entire consecutive layer on each block, and then merge the completed output blocks. A convolutional neural network can be represented as a directed acyclic graph. The subgraph corresponding to the consecutive layers to be blocked is called a block subgraph. As Figure 3 shown, the figure shows the schematic diagram of the block processing of ResNet50, which is the storage occupancy obtained by dividing the three consecutive layers where the intermediate three peak layers of ResNet50 are located into four blocks.

[0065] In this embodiment, the specific steps for defining the block subgraph are as follows:

[0066] Assume that the size of the feature map of the consecutive layer remains unchanged (without upsampling or downsampling) and the receptive field is not considered during blocking. As shown in the following figure, for the input feature map A with a size of size A , when the layer B is executed, the overall peak storage occupancy is size A + size B = 2size A .

[0067] As Figure 4 shown, the figure shows the schematic diagram of the original storage occupancy of the linear subgraph;

[0068] If A is divided into two blocks A1 and A2 and executed separately, since the storage of A1 can be immediately released for subsequent use by B2 after A1 is executed, the overall peak storage occupancy is reduced to Reduced to

[0069] As Figure 5 shown, the figure shows the schematic diagram of the block storage occupancy of the linear subgraph;

[0070] For the common residual structure, since the input is used in the final addition layer, it needs to be resident in memory as an intermediate result, and the peak storage occupancy is 3siz input .

[0071] As Figure 6 shown, the figure shows the schematic diagram of the original storage occupancy of the residual subgraph;

[0072] After dividing the input into two blocks, the peak storage occupancy is reduced to 2siz input , which is of the original. When the number of branches increases, the peak storage occupancy will be reduced even more.

[0073] As Figure 7 shown, the figure shows the schematic diagram of the block storage occupancy of the two residual subgraphs;

[0074] Increasing the number of chunks can also reduce the peak storage. As shown in the figure, the storage occupancy when A is divided into four chunks is presented. It can be seen that the peak storage occupancy is reduced to Reduced to

[0075] such as Figure 8 shown. The figure shows the schematic diagram of the storage occupancy of four residual sub - graphs stored in chunks;

[0076] How does the number of chunks affect the storage occupancy? Assume that the input is divided into n chunks, and assume that the original peak storage occupancy is siz max . Then, for the execution process of the first chunk, its peak storage occupancy is However, at this time, the other chunks are stored in the memory. Therefore, the total peak storage occupancy is When the first chunk finishes execution, the result chunk replaces the input chunk and is stored in the memory and persists until the result merging process. For the last chunk, the total peak storage occupancy is Therefore, the peak storage occupancy of the entire execution process

[0077]

[0078] When n → ∞, size′ max → max{size Input , size output}. Assume that the storage upper limit is max. As long as max{size Input , size output} < max, it is possible to meet the storage requirements during the inference process through chunking.

[0079] From the principle derivation of chunking, it can be seen that if the output layer can release all intermediate results between the input and output layers, then the peak storage occupancy during the execution process can be reduced through chunking. Conversely, as Figure 9 shown. The figure shows the schematic diagram of non - chunked sub - graphs. Since the subsequent layers of the output layer depend on the results of the intermediate layers, even if chunked, they cannot be released. Therefore, the storage occupancy cannot be well optimized. For the third sub - graph, the intermediate layer depends on external inputs and cannot be directly chunked.

[0080] Therefore, for any sub - graph G′(V′, E′) of the directed acyclic graph G(V, E) of a convolutional neural network, a chunked sub - graph is defined if it satisfies:

[0081] Condition 1: |V′| > 1;

[0082] Condition 2: |{v′ ∈ V′: indegree(v′) = 0}| = 1 and |{v′ ∈ V′: outdegree(v′) = 0}| = 1, that is, there is only one vertex with in-degree zero and only one vertex with out-degree zero, denoted as the entry vertex v in and the exit vertex v out ;

[0083] Condition 3: All the adjacent edges of all vertices v′ in V′ in G (i.e., the edges with v′ as the starting point or the ending point) are included in E′, except for the edges with the entry vertex as the ending point and the edges with the exit vertex as the starting point.

[0084] Condition 3 excludes the above three possibilities.

[0085] In addition, as Figure 10 shown, the figure shows a schematic diagram of special cases of the sub-block graph. The above two sub-graphs can be regarded as variants of the above situation, and their sub-blocks are equivalent to the following two sub-graphs.

[0086] It can be seen that the sub-block graph corresponds one-to-one with the vertex pair composed of its entry vertex and exit vertex. That is, given a sub-block graph, its entry vertex and exit vertex are unique; given a pair of entry vertex and exit vertex, if they are the entry vertex and exit vertex of a certain sub-block graph, then such a sub-block graph is also unique.

[0087] In this embodiment, the specific steps for generating the sub-block graph are as follows:

[0088] Among them, the specific steps for generating the message matrix are as follows:

[0089] Use the algorithm based on the message matrix to search for all sub-block graphs. Assume |V| = n. For any given topological sequence of graph G, number all vertices according to the topological order, denoted as v1, v2,..., v n . Traverse all vertices according to the topological order. The current vertex sets its initial message to 1 and collects the messages of other vertices from all its predecessor vertices. If its predecessor vertex has multiple successor vertices, the message is equally divided according to the number of successor vertices, and the current vertex obtains one share of it. As Figure 11 shown, the figure shows a schematic diagram of message passing. Vertex A has two successor vertices, so both vertex B and vertex C obtain half of the message of their predecessor vertex A. When the current vertex collects the messages of the same vertex from different predecessor vertices, these messages are merged. For example, vertex D obtains half of the message of vertex A from its predecessor vertices B and C, and merges them into the complete message of vertex A.

[0090] By constructing the message matrix M n×n = {m ij} to record the message passing situation between vertices, m ijDenote vertex v i The vertex v it owns j The amount of messages, initialized to 0. For each traversed vertex v i , set the message m of this vertex itself ii to 1. Since the traversal is in topological order, all predecessor vertices of the current vertex have completed message collection. According to the indices of its predecessor vertices, find the vector of the corresponding row in the message matrix, which is the message vector of other vertices collected by its predecessor vertices. Accumulate the message vectors of all predecessor vertices to obtain the message vector of the current vertex and store it in the message matrix. This algorithm is a triple loop. The outermost loop is for all vertices, the middle loop is for all predecessor vertices of the current vertex, and the innermost loop is for vector accumulation. Therefore, the time complexity is O(n 3 ), where vector accumulation can be executed in parallel.

[0091] As Figure 12 shown, the figure shows a schematic diagram of the message matrix;

[0092] As shown in the following table, it is represented as the message matrix;

[0093]

[0094] In this embodiment, the steps to search for the sub-block graph are as follows:

[0095] All sub-block graphs can be easily found according to the message matrix. According to the definition of the sub-block graph, the exit vertex of the sub-block graph can always collect the messages of all vertices inside it, and the vertices corresponding to all non-zero messages of its entry vertex have the same value as the exit vertex. This is because there are no edges from outside the sub-block graph for the vertices inside the sub-block graph. Otherwise, the exit vertex would be able to collect messages that the entry vertex does not have, and their corresponding values would not be the same.

[0096] Design an algorithm based on this principle: Traverse all vertices in topological order. For each vertex v i , traverse the vertices v j behind it. The two may be able to form a sub-block graph with v i as the entry vertex and v j as the exit vertex. The judgment method is to subtract the row vector vector j with index j in the message matrix from the row vector vector i with index i to obtain a vector vector j-i of length n. If the values before index i of vector j-i are all 0, that is, the messages of the entry vertex and the exit vertex are the same, and the values after index i are all 1 (or 0), that is, the exit vertex has collected all the messages of the middle vertex (or the middle vertex is not on any path between the entry vertex and the exit vertex), then vi and v j can form a sub-block graph. For example, in Table 1-1, the index of vertex B is 1. Subtracting the row vector of vertex B from the row vector of vertex F, the values of the result vector before index 1 are all 0, and the values after index 1 are 1 or 0. Therefore, B and F can form a sub-block graph. However, subtracting the row vector of vertex D from the row vector of vertex B, the values of the result vector before index 1 are not all 0, so B and D cannot form a sub-block graph.

[0097] The algorithm is also a three-layer nested loop. The innermost layer is vector subtraction and value judgment, and the time complexity is O(n 3 ).

[0098] Note that the sub-block graph is only related to the structure of the directed acyclic graph of the network and has nothing to do with the selected topological order. Any topological order can be used to complete the search algorithm. The time complexity of generating the topological order is O(|V| + |E|), and the number of sub-block graphs does not exceed n 2 .

[0099] In this embodiment, the steps of selecting the sub-block graph are as follows:

[0100] Due to the existence of the receptive field, it is not possible to simply block the entry vertices of the sub-block graph to reduce the peak storage of the network because the regions corresponding to different regions of the exit vertices on the entry vertices may overlap. Therefore, in order to ensure that the blocks of the exit vertices can be assembled into a complete output, the exit vertices are first blocked, and then the block sizes of the entry vertices are calculated according to the receptive field relationship between the exit vertices and the entry vertices. These entry vertex blocks may overlap with each other. The calculation of the overlapping part, as well as the blocking of the entry vertices and the merging of the exit vertices, constitute the overhead of solving the storage problem by the sub-block graph. The overhead is related to the receptive field relationship between the entry and exit vertices of the sub-block graph, the number of blocks, and the selected sub-block graph. There may be multiple sub-block graphs that can be selected to solve the storage problem of the same layer, and one sub-block graph may be able to solve the storage problems of multiple layers. For a smaller sub-block graph, the receptive field of the block of the exit vertex at the entry vertex is relatively small, and the additional calculation overhead is reduced, but more sub-block graphs may be required to cover all the vertices that need to reduce storage, which increases the cost of blocking and merging; on the contrary, a larger sub-block graph results in a larger receptive field, increasing the calculation overhead, but only a few sub-block graphs may be needed to solve the problem, and there is not much overhead in blocking and merging.

[0101] In this embodiment, in deep learning, the receptive field is the size of the region in the input feature map of the convolutional neural network that produces the output feature. The idea of the receptive field is applicable to local operations (such as convolution, pooling, etc.). As Figure 13As shown in the figure, it is a schematic diagram of the receptive field; the receptive field of the output feature pixels in Layer3 in Layer2 is 3*3, and the receptive field in Layer1 is 5*5.

[0102] Among them, use r ij to represent the receptive field of the output features of the j-th layer of the convolutional neural network in the i-th layer. If the i-th layer is a convolutional layer, let k i and s i represent the kernel_size and stride of this layer respectively. Then the calculation method of the receptive field of the output feature pixels in Layer3 in Layer2 is:

[0103] r 23 =(r 33 -1)*s3 + k3;

[0104] The calculation method of the receptive field of the output feature pixels in Layer2 in Layer1 is:

[0105] r 12 =(r 22 -1)*s2 + k2;

[0106] Then the calculation method of the receptive field of the output feature pixels in Layer3 in Layer1 is:

[0107] r 13 =(r 23 -1)*s2 + k2 = ((r 33 -1)*s3 + k3 - 1)*s2 + k2.

[0108] The general calculation form of the receptive field is

[0109] r (i-1)j =(r ij -1)*s i + k i ;

[0110] Then for any two layers n, m, where m > n, the calculation method of the receptive field of the output feature pixels in the m-th layer in the n-th layer is:

[0111]

[0112] Traverse the convolutional neural network, and recursively calculate the receptive field relationship parameters between the previous layer and this layer for each layer, and the receptive field relationship matrix RF of the convolutional neural network can be obtained nn =(rf ij ), where the element rf of the matrix ij =(α ij , β ij) is the receptive field relationship parameter between the i-th and j-th layers. When there are different paths between two layers, the receptive fields calculated by different paths may be different. In this case, the larger receptive field is selected.

[0113] Through such a matrix, the size of the input block corresponding to the output block can be calculated. For example, if the size of the output block of the m-th layer is h m ×w m , then the size of the input block of the n-th layer is (h m *α nm +β nm )×(w m *α nm +β nm ). Note that it is assumed here that the kernel_size and stride in the horizontal and vertical directions of the convolution are equal. Otherwise, it is only necessary to calculate the receptive field relationship parameters for the two directions separately.

[0114] In this embodiment, the specific steps of the topological order are as follows:

[0115] The inference order of the network layers, that is, the topological order corresponding to the directed acyclic graph, will also affect the storage size required during the execution process, and thus will also affect the block division method and selection of the subgraphs. As Figure 14 shown, the figure is a schematic diagram of the topological order; assuming that the storage of each vertex is s, when choosing the execution order of A→B→C→D→E→F→G→H→I→J, the maximum storage occupancy is 4s, that is, when the intermediate results are vertices E, F, G, H; if choosing the execution order of A→B→C→D→F→G→H→I→E→J, the maximum storage occupancy is 5s, that is, when the intermediate results are vertices C, D, F, G, H.

[0116] The number of topological orders of a directed acyclic graph may be exponential. Therefore, it may be unrealistic to traverse all possible topological orders and find the optimal solution. Here, a heuristic method is adopted to select the topological order that can ensure the continuous execution of the vertices in each subgraph, that is, when selecting the next vertex of the topological order, the optional set is restricted to the set of all successor vertices of the current vertex.

[0117] Use this method to obtain the set of topological orders, traverse each topological order, find the corresponding optimal solution for each topological order, and finally select the optimal block division scheme and the corresponding topological order among all topological orders as the optimal solution to the problem.

[0118] In this embodiment, the steps for selecting the block division method are as follows:

[0119] Given a topological order, when each vertex finishes execution, it is determined which vertices are stored in the storage as intermediate results, and at this time the storage occupancy size is also determined. For all vertices, if the storage occupancy exceeds the available storage size when its execution ends, a corresponding chunking solution needs to be generated for this vertex. Suppose the set of such vertices is V overlimit , and the set of chunked subgraphs is T.

[0120] Traverse all the chunked subgraphs. For each chunked subgraph, the storage occupancy of its entry vertex and exit vertex when they finish execution should be less than the available storage size; otherwise, the storage occupancy before chunking or after the chunking inference ends will be greater than the available storage, and such chunked subgraphs are not selected. For other chunked subgraphs that meet the requirements, traverse V overlimit , when v overlimit ∈V overlimit is a non-entry vertex of the current chunked subgraph and has the largest storage occupancy, calculate the feasible number of chunks for this vertex. Suppose V c is the set of intermediate results when vertex v overlimit finishes execution, and the set of vertices that belong to the vertices of the current chunked subgraph is and the other vertices are V c ′. Denote the entry vertex of the chunked subgraph as v in , and the exit vertex as v out . Denote the storage occupancy of vertex v as s v , and the available storage as S. When the number of chunks is n, the maximum storage occupancy during the execution of each chunk is

[0121]

[0122] should satisfy s max <S, and we can solve for

[0123]

[0124] However, due to the existence of the receptive field, the value of n solved by this formula is only a theoretical lower bound. For the current chunked subgraph G′(V′,E′), assume that the size of the exit vertex block obtained by the chunking scheme with n chunks is

[0125]

[0126] Traverse all vertices v′∈V′. For each vertex v′, similarly, assume that the set of intermediate results when this vertex finishes execution is V c , and the set of vertices that belong to the vertices of the current chunked subgraph is and the other vertices are V c ′. Denote the entry vertex of the chunked subgraph as v in , and the exit vertex as v outAccording to the receptive field relationship matrix, the block size of the entrance vertex can be calculated as

[0127]

[0128] For The vertex calculated according to the receptive field relationship matrix The block size of is

[0129]

[0130] The complete storage occupancy of is Therefore, the storage occupancy of the current vertex v′ is

[0131]

[0132] For all vertices v′ ∈ V′, if there is always s v′ < S, then for the current sub-block graph, the number of sub-blocks n can make the sub-graph meet the storage requirements. Otherwise, increment n and repeat the above judgment process. An upper limit on the number of sub-blocks can be set, or in the worst case, the number of sub-blocks is the product of the height and width of the exit vertex, which is equivalent to the per-pixel scheme of the layer fusion strategy.

[0133] In this embodiment, the steps of the optimal solution are:

[0134] First, it is necessary to define the block cost for each sub-block graph. The additional computational cost caused by sub-blocking is related to the hardware operation method and computing power. The storage cost caused by moving data during the process of sub-blocking and merging in the memory is related to the hardware storage capacity. After the sub-blocking method of the sub-block graph is determined, these costs can be determined according to the specific hardware. For example, the additional computing cycles and the time spent on data transfer. It is appropriate to use these increased times as the cost of the sub-block graph because it can truly reflect the sub-blocking overhead. Here, specific hardware is not involved. Simply use the size of the overlapping part caused by sub-blocking as the additional computational cost, and use the sum of the total size of the entrance vertex and the size of the exit vertex after sub-blocking as the storage cost for data transfer. The weighted average of the two is used as the final block cost, which is a value that is determined accordingly after the sub-blocking method of the sub-block graph is determined.

[0135] After excluding some sub-block graphs that cannot reduce storage occupancy in the previous step and constructing corresponding block schemes for other sub-block graphs, if there are sub-block graphs providing solutions for all vertices that do not meet the storage requirements, then the combination with the lowest cost can be selected from these sub-block graphs as the optimal solution. According to the definition of sub-block graphs, there is either a nesting relationship or they are disjoint (except for the entry vertex and the exit vertex) between two sub-block graphs. Therefore, the final combination is a set of pairwise disjoint sub-block graphs that can completely cover all vertices that do not meet the storage requirements. This is similar to a weighted set covering problem, and its optimization problem is NP-hard. However, in this problem, the vertices are sorted in topological order and there are dependencies before and after, so the optimal solution can be found in polynomial time using the dynamic programming algorithm.

[0136] Suppose the vertices that do not meet the storage requirements are denoted in topological order as The set of optional sub-block graphs is T, and the set of sub-block graphs of the solution Satisfies:

[0137]

[0138]

[0139] The optimization problem is:

[0140]

[0141] Traverse For each Partition the original sequence into two subsequences And At the same time, partition the set T into two disjoint sets T L , T R , such that

[0142]

[0143]

[0144] That is, partition the original set into two subsets that only cover one of the subsequences. Then Partition the original problem into two sub-problems The combination of the optimal solutions of the sub-problems is the optimal solution of the original problem in this partitioning case, and the sum of the minimum costs of the sub-problems is the minimum cost of the original problem in this partitioning case. In addition, select a sub-block graph with the minimum cost in the sub-block graph V' that covers all the current vertices that do not meet the storage requirements as the solution in the non-partitioning case. The solution with the minimum cost is selected from the set composed of the solution in the partitioning case and this solution, which is the optimal solution and the minimum cost of the original problem. The sub-problems can be solved recursively. The smallest sub-problem is to select a sub-block graph with the minimum cost that only covers a single vertex that does not meet the storage requirements. The repeated sub-problems are optimized using dynamic programming.

[0145] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents and should be included within the protection scope of the present invention.

Claims

1. A block scheduling method for optimizing inference storage of a convolutional neural network, characterized in that, It includes the following steps: Step S1: Construct a directed acyclic graph of a convolutional neural network and define a subgraph block; Step S2: Generate all candidate subgraph blocks based on the algorithm of the message matrix; Step S3: Define the block cost for each subgraph block based on the storage cost at the same time, determine the block method of the subgraph block, and use the dynamic programming algorithm to select the optimal combination of subgraph blocks; Step S4: Perform block scheduling on each subgraph block according to the above steps, optimize the storage occupancy during the inference process, and merge the completed output blocks.

2. The block scheduling method for optimizing inference storage of a convolutional neural network according to claim 1, characterized in that, The specific steps in the above Step S1 include: Construct a directed acyclic graph of a convolutional neural network. The convolutional neural network is a directed acyclic graph, and the subgraph corresponding to the consecutive layers to be blocked is used as a subgraph block; Define a subgraph block according to any subgraph G′(V′, e′) of the directed acyclic graph G(V, E) of the convolutional neural network, and it satisfies: Condition 1: |V′| > 1; Condition 2: |{v′∈V′: indegree(v′) = 0}| = 1, |{v′∈V′: outdegree(v′) = 0}| = 1; that is, there is only one vertex with an in-degree of zero and only one vertex with an out-degree of zero, denoted as the entry vertex v in and the exit vertex v out ; Condition 3: All the adjacent edges of all vertices v′ in V′ in G, that is, the edges with v′ as the starting point or the ending point, are included in E′, and it does not include the edges with the entrance vertex as the ending point and the edges with the exit vertex as the starting point.

3. The block scheduling method for optimizing convolution neural network inference storage according to claim 1, characterized in that The specific steps in the above Step S2 include: The algorithm based on the message matrix searches all the subgraphs: Suppose |V| = n. For any given topological sequence of graph G, all vertices are numbered according to the topological order, denoted as v1, v2, …, v n ; Traverse all vertices according to the topological order. The current vertex initializes its own initial message to 1, and collects the messages of other vertices from all its predecessor vertices. If its predecessor vertex has multiple successor vertices, the message is equally divided according to the number of successor vertices, and the current vertex obtains one share; When the current vertex collects messages of the same vertex from different predecessor vertices, these messages are merged; Generate the message matrix of the directed acyclic graph according to the above steps, and then generate all candidate subgraph blocks according to the message matrix; Since there is an overlap between the entrance vertex blocks, the calculation of the overlapping part, as well as the blocking of the entrance vertices and the merging of the exit vertices, constitute the overhead of the subgraph block to solve the storage problem.

4. The block scheduling method for optimizing convolution neural network inference storage according to claim 1, characterized in that, The specific steps of the optimal solution in the above Step S3 include: Vertices that do not meet the storage requirements are recorded in topological order as The set of optional subgraphs for partitioning is T, and the set of subgraphs for partitioning in the solution Satisfy: The optimization problem is: Traverse For each that divides the original sequence into two subsequences and At the same time, the set T is also divided into two disjoint sets T L , T R such that If the original set is divided into two subsets that only cover one of the subsequences, then the original problem is divided into two subproblems the combination of the optimal solutions of the subproblems is the optimal solution of the original problem in this division case, and the sum of the minimum costs of the subproblems is the minimum cost of the original problem in this division case; Obtain the block subgraph V that covers all the vertices that currently do not meet the storage requirements according to the above steps ′ Select a block subgraph with the minimum cost as the solution for the non - partitioning case, where the minimum - cost solution is selected from the set composed of the solution for this non - partitioning case and the solution for the partitioning case.

5. The block scheduling method for optimizing convolution neural network inference storage according to claim 1, characterized in that, The specific steps in the above Step S4 include: Block the input layer of the consecutive layers to be optimized, perform the inference of the entire consecutive layer for each block, optimize the storage occupancy during the inference process, and merge the completed output blocks.

Images