Neural network on-chip resource scheduling method and device, electronic equipment and medium

Abstract

The invention discloses a neural network on-chip resource scheduling method and device, electronic equipment and a medium. The neural network on-chip resource scheduling method comprises the following steps: drawing a change trend chart based on an operator execution sequence and operator tensor memory usage of a current neural network, and segmenting the change trend chart to obtain network segments; calculating a target cache batch size of each network segment; and carrying out on-chip data splicing on the last cycle output data of the previous network segment in the current adjacent network segments and the residual input of the next network segment to obtain target input data of the operator corresponding to the next network segment, and carrying out data processing on the target input data based on the operator corresponding to the next network segment. According to the technical scheme provided by the embodiment of the invention, the on-chip cache space can be fully utilized, the repeated carrying of weight data and the fixed overhead of the system are reduced, and the optimized operation efficiency of on-chip cache is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of neural network on-chip scheduling, and in particular to a neural network on-chip resource scheduling method, device, electronic equipment and medium. Background Art

[0002] Although there are many types of neural networks, their data size scale or change cycle usually follows a certain change law, such as the data size scale changes from large to small, or from small to large, and the change cycle changes back and forth from large to small. Among them, neural networks with data scale in the shape of an "inverted pyramid" are particularly common, such as the typical resnet network (residual network), whose data size is the largest at the beginning of the network, and as the number of network layers deepens, the size gradually decreases, and the overall data scale is like an "inverted pyramid".

[0003] The logic of large-batch resource scheduling of traditional neural networks is as follows Figure 1 As shown in the figure, when optimizing the on-chip cache of this type of neural network, a single decision is usually made on the batch size based on the location with the largest network data size, that is, the peak memory size of the entire network. For example, a resnet network with a batch size of 32 can be cached with a batch size of 4 on a cache-determined AI (artificial intelligence) chip at the peak memory usage of the network. The entire network, from top to bottom, will use 4 as the batch size for cache optimization and cycle 8 times. Although this cache optimization method can reduce the interaction between the entire network and the external cache, there are still many problems: 1) Since the memory peak is used to determine the slice size, the entire network is sliced ​​too many times, and too many slices will cause a large amount of weight data to be repeatedly moved, and the system fixed overhead will increase exponentially, which will erode the benefits of the on-chip cache to a certain extent. 2) Using the slice size determined by the memory peak, in the initial stage of the network, the "inverted pyramid" memory trend makes the memory at the peak, so the cache utilization is also very high. However, as the network goes deeper layer by layer, the size of the tensor gradually decreases to half, one-quarter, or even one-eighth of the peak value, which causes the chip cache utilization in the back of the network to be reduced to half, one-quarter, or one-eighth of the peak value, causing serious cache waste. 3) In addition to cache waste, in the second half of the network, because the tensor is too small, the calculation becomes a bottleneck for weight transfer, and the idle state of calculation leads to idle and waiting of chip computing resources. Summary of the invention

[0004] The present invention provides a neural network on-chip resource scheduling method, device, electronic device and medium to solve the problems of low on-chip cache utilization, high number of repeated weight transfers, and waste of cache space and chip computing power in a batch scheduling method that uses memory peak to directly determine the entire network.

[0005] According to one aspect of the present invention, a method for scheduling resources within a neural network chip is provided, comprising:

[0006] Based on the operator execution order and operator tensor memory usage of the current neural network, draw a change trend graph, and segment the change trend graph to obtain each network fragment;

[0007] Calculate the target cache batch size for each network segment;

[0008] The last cycle output data of the previous network segment in the current adjacent network segment is concatenated with the remaining input of the next network segment to obtain the target input data of the operator corresponding to the next network segment, so as to process the target input data based on the operator corresponding to the next network segment.

[0009] According to another aspect of the present invention, a neural network on-chip resource scheduling device is provided, comprising:

[0010] The trend graph drawing and segmentation module is used to draw the change trend graph based on the operator execution order and operator tensor memory usage of the current neural network, and segment the change trend graph to obtain each network fragment;

[0011] A batch size calculation module, used to calculate the target cache batch size for each network segment;

[0012] The on-chip data splicing module is used to perform on-chip data splicing on the last cycle output data of the previous network segment in the current adjacent network segment with the remaining input of the next network segment to obtain the target input data of the operator corresponding to the next network segment, so as to perform data processing on the target input data based on the operator corresponding to the next network segment.

[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the neural network on-chip resource scheduling method described in any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the neural network on-chip resource scheduling method described in any embodiment of the present invention when executed.

[0018] The technical solution of the embodiment of the present invention draws a change trend graph based on the operator execution order and operator tensor memory usage of the current neural network, and segments the change trend graph to obtain each network segment, thereby calculating the target cache batch size of each network segment, and then performing on-chip data splicing of the last cycle output data of the previous network segment in the current adjacent network segment with the remaining input of the next network segment to obtain the target input data of the operator corresponding to the next network segment, so as to perform data processing on the target input data based on the operator corresponding to the next network segment. This solution calculates the cache batch size for each network segment, breaking the drawback of using memory peak to determine the batch size of the entire network. By dynamically changing the batch size in the cache, the number of loop slicing in the middle and tail segments of the network is greatly reduced to optimize the utilization of neural network memory resources. By optimizing the execution order of operators, the repeated movement of weight data and the fixed overhead of the system are reduced. This solves the problems of low on-chip cache utilization, frequent repeated weight movement, and waste of cache space and chip computing power in the batch scheduling method of directly determining the entire network using memory peak. It can make full use of the on-chip cache space, reduce repeated weight data movement and system fixed overhead, and greatly improve the optimized operation efficiency of the on-chip cache.

[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 A logical schematic diagram of a large batch resource scheduling of a traditional neural network provided by the present invention;

[0022] Figure 2 A flowchart of a neural network on-chip resource scheduling method provided in Embodiment 1 of the present invention;

[0023] Figure 3 A flowchart of a neural network on-chip resource scheduling method provided in Embodiment 2 of the present invention;

[0024] Figure 4 A schematic diagram of dividing a network into segments based on a change trend diagram provided in Embodiment 2 of the present invention;

[0025] Figure 5 A schematic diagram of a neural network on-chip resource scheduling logic provided in the second embodiment of the present invention;

[0026] Figure 6 A real-time operation diagram of an operator provided in Embodiment 2 of the present invention;

[0027] Figure 7 An operator operation diagram for traditional batch scheduling of an existing neural network provided in the second embodiment of the present invention;

[0028] Figure 8 A schematic diagram of the structure of a neural network on-chip resource scheduling device provided in Embodiment 3 of the present invention;

[0029] Fig. 9 A schematic diagram of the structure of an electronic device that can be used to implement an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "previous", "next", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] Embodiment 1

[0033] Figure 2This is a flowchart of a neural network chip resource scheduling method provided in the first embodiment of the present invention. This embodiment is applicable to the case of optimizing and scheduling neural network chip resources. The method can be executed by a neural network chip resource scheduling device. The neural network chip resource scheduling device can be implemented in the form of hardware and / or software. The neural network chip resource scheduling device can be configured in an electronic device. The electronic device can include but is not limited to a computer or a server. Figure 2 As shown, the method includes:

[0034] Step 110: based on the operator execution order and operator tensor memory usage of the current neural network, draw a change trend graph, and segment the change trend graph to obtain each network fragment.

[0035] The operator execution order may be the execution order of operators in the neural network. The operator tensor memory usage may be used to describe the operator tensor memory usage. The change trend graph may be used to describe the change trend of the tensor memory usage of the current neural network with the operator execution order. The network segment may be a segmentation result obtained by segmenting the current neural network based on the change trend graph.

[0036] In an embodiment of the present invention, the memory usage of the entire network can be analyzed during the compilation stage of the current neural network to obtain the operator execution order and operator tensor memory usage of the current neural network, and then a change trend graph can be drawn based on the operator execution order and operator tensor memory usage of the current neural network. Based on the change trend graph, several operators with relatively stable memory usage (the measurement standard for relatively stable memory usage can be set by oneself) are divided into one segment to achieve the division of the entire network of the current neural network and obtain each network fragment.

[0037] Step 120: Calculate the target cache batch size for each network segment.

[0038] The target cache batch size may be a batch size of a network segment cache.

[0039] In an embodiment of the present invention, the target cache batch size can be calculated for each network segment based on the corresponding memory peak value and the on-chip cache capacity of the chip where the current neural network is deployed, so as to dynamically change the size of the batch in the cache according to the tensor memory size in the network. Especially for neural networks such as "inverted pyramids", as the network deepens, the size of a single batch of tensors gradually becomes smaller, and the cache batch size gradually increases, which can make full use of the on-chip cache space and greatly reduce the number of loop slicing in the middle and tail segments of the network.

[0040] Step 130: perform intra-chip data splicing on the last cycle output data of the previous network segment in the current adjacent network segment and the remaining input of the next network segment to obtain the target input data of the operator corresponding to the next network segment, so as to perform data processing on the target input data based on the operator corresponding to the next network segment.

[0041] The current adjacent network segment may be composed of two current adjacent network segments, the network segment with the earlier operator execution order in the current adjacent network segments is the previous network segment, and the network segment with the later operator execution order is the next network segment. In-chip data splicing may be a process of combining the last cycle output data of the previous network segment with the remaining input of the next network segment. The target input data may be the data fully input to the next network segment in the current adjacent network segment.

[0042] In an embodiment of the present invention, the current adjacent network segment for which the operator execution order is currently being scheduled can be determined, and then the previous network segment and the next network segment in the current adjacent network segment can be determined, so that the last cycle output data of the previous network segment and the remaining input of the next network segment can be spliced ​​within the chip (there is no need to cache the last cycle output data of the previous network segment outside the chip), and the data splicing result is used as the target input data of the operator corresponding to the next network segment, and the target input data is further processed by the operator corresponding to the next network segment, thereby reducing the repeated transportation of weight data and the fixed overhead of the system, and can greatly improve the optimization operation efficiency of the on-chip cache.

[0043] The technical solution of the embodiment of the present invention draws a change trend graph based on the operator execution order and operator tensor memory usage of the current neural network, and segments the change trend graph to obtain each network segment, thereby calculating the target cache batch size of each network segment, and then performing on-chip data splicing of the last cycle output data of the previous network segment in the current adjacent network segment with the remaining input of the next network segment to obtain the target input data of the operator corresponding to the next network segment, so as to perform data processing on the target input data based on the operator corresponding to the next network segment. This solution calculates the cache batch size for each network segment, breaking the drawback of using memory peak to determine the batch size of the entire network. By dynamically changing the batch size in the cache, the number of loop slicing in the middle and tail segments of the network is greatly reduced to optimize the utilization of neural network memory resources. By optimizing the execution order of operators, the repeated movement of weight data and the fixed overhead of the system are reduced. This solves the problems of low on-chip cache utilization, frequent repeated weight movement, and waste of cache space and chip computing power in the batch scheduling method of directly determining the entire network using memory peak. It can make full use of the on-chip cache space, reduce repeated weight data movement and system fixed overhead, and greatly improve the optimized operation efficiency of the on-chip cache.

[0044] Embodiment 2

[0045] Figure 3 This is a flowchart of a neural network on-chip resource scheduling method provided by Embodiment 2 of the present invention. This embodiment is specific based on the above embodiment and provides a specific optional implementation method for calculating the target cache batch size of each network segment. Figure 3 As shown, the method includes:

[0046] Step 210: based on the operator execution order and operator tensor memory usage of the current neural network, draw a change trend graph, and segment the change trend graph to obtain each network fragment.

[0047] In an optional embodiment of the present invention, segmenting the change trend graph to obtain each network segment may include: determining the operator to be compared with the current tensor memory usage, and determining the operator tensor memory usage similarity of the operator to be compared with the current tensor memory usage; segmenting the change trend graph according to the operator tensor memory usage similarity and the lower limit threshold of the number of operators in a single network segment to obtain the current network segment.

[0048] The current tensor memory usage comparison operator may be an operator that currently needs to be compared in tensor memory usage. The operator tensor memory usage similarity may describe the similarity between tensor memory usages of different operators. The operator number lower limit threshold may be a preset minimum value of the number of operators in a single network segment.

[0049] In an embodiment of the present invention, the operator whose tensor memory usage currently needs to be compared can be determined first, that is, the operator whose current tensor memory usage is to be compared can be determined first, and then based on the pre-set operator tensor memory usage similarity evaluation rule, the operator tensor memory usage similarity of the operator whose current tensor memory usage is to be compared can be calculated, and then the operator tensor memory usage similarity and the lower limit threshold of the number of operators in a single network segment can be used to segment the change trend graph, so that the memory usage of the operators in the current network segment divided based on the change trend graph is relatively stable.

[0050] In an optional embodiment of the present invention, determining the operator tensor memory usage similarity of the operator to be compared with the current tensor memory usage may include: based on the operator tensor memory usage of the operator to be compared with the current tensor memory usage, determining the first memory usage similarity between the starting end operator and the ending end operator in the operator to be compared with the current tensor memory usage, or the second memory usage similarity between non-two-end point operators; and using the first memory usage similarity or the second memory usage similarity as the operator tensor memory usage similarity.

[0051] Among them, the starting end operator may be the operator with the earliest execution order among the operators to be compared for the current tensor memory usage. The ending end operator may be the operator with the latest execution order among the operators to be compared for the current tensor memory usage. The first memory usage similarity may be the similarity of the memory usage of the starting end operator and the ending end operator among the operators to be compared for the current tensor memory usage. The non-dual end point operator may be an operator that is not at the end point position at the same time among the operators to be compared for the current tensor memory usage. The second memory usage similarity may be the similarity of the memory usage of the non-dual end point operator.

[0052] In an embodiment of the present invention, based on the change trend graph, the operator tensor memory usage of the starting end operator and the ending end operator in the operators to be compared in the current tensor memory usage can be determined, and then the ratio of the operator tensor memory usage of the two endpoint operators can be used as the first memory usage similarity of the two endpoint operators. The ratio of the operator tensor memory usage of any two non-two-endpoint operators in the operators to be compared in the current tensor memory usage can also be calculated to obtain the second memory usage similarity between the non-two-endpoint operators, and the first memory usage similarity or the second memory usage similarity can be further used as the operator tensor memory usage similarity.

[0053] In an optional embodiment of the present invention, segmenting the change trend graph according to the similarity of operator tensor memory usage and the lower limit threshold of the number of operators in a single network segment to obtain the current network segment may include: when the number of operators to be compared in the current tensor memory usage is less than the lower limit threshold of the number of operators in a single network segment, or the similarity of the operator tensor memory usage meets the similarity judgment condition, updating the operator to be compared in the current tensor memory usage based on the operator in the next execution order of the terminal operator in the operator to be compared in the current tensor memory usage; and returning to execute the operation of determining the similarity of the operator tensor memory usage based on the operator tensor memory usage of the operator to be compared in the current tensor memory usage, until the number of operators to be compared in the current tensor memory usage is greater than the lower limit threshold of the number of operators in a single network segment, and the similarity of the operator tensor memory usage does not meet the similarity judgment condition, then using the operator to be compared in the current tensor memory usage to obtain the current network segment.

[0054] Among them, the similarity judgment condition can be a condition for judging the memory usage stability of the operator to be compared with the current tensor memory usage. Optionally, the similarity judgment condition can include that the first memory usage similarity is greater than 70% (which can be set by yourself), or the ratio of the maximum tensor memory usage of the operator to the minimum tensor memory usage of the operator in the non-two-endpoint operator is greater than 50% (which can be set by yourself).

[0055] In an embodiment of the present invention, if the number of operators to be compared in the current tensor memory usage is less than the lower limit threshold of the number of operators in a single network segment, or the similarity of the operator tensor memory usage meets the similarity judgment condition, then the operator characterizing the current tensor memory usage to be compared can be further expanded, thereby determining the operator of the next execution order of the terminal operator in the current tensor memory usage to be compared, and adding the newly determined operator of the next execution order to the current tensor memory usage to be compared operator, thereby returning to execute the operator tensor memory usage based on the current tensor memory usage to be compared operator, and determining the operation of the operator tensor memory usage similarity, until the number of operators to be compared in the current tensor memory usage is greater than the number of operators in a single network segment. The lower limit threshold of the number of operators to be compared is exceeded, and the similarity of the operator tensor memory usage does not meet the similarity judgment condition. At this time, it can be considered that the operator with relatively stable memory usage has been determined, which will cause the number of operators to be compared in the current tensor memory usage to be greater than the lower limit threshold of the number of operators in a single network segment, and the operator whose tensor memory usage similarity does not meet the similarity judgment condition (that is, the operator that was last added to the current tensor memory usage to be compared operator) is used as the demarcation point, and the network segment corresponding to the current tensor memory usage to be compared operator before the operator is added is used as the current network segment, and the operator is used as the initial operator of the next network segment, and the next network segment is divided according to the network segment division rule.

[0056] Step 220: Obtain the memory peak value within a single batch of each network segment.

[0057] The memory peak value within a single batch segment may be the memory peak value of the network segment when the batch size is 1.

[0058] In an embodiment of the present invention, if the acquired memory peak value of the network segment is not the memory peak value of a single batch, it is necessary to calculate the memory peak value of the network segment when the batch number is equal to 1, so as to obtain the memory peak value within a single batch segment of each network segment.

[0059] Step 230: Calculate the target cache batch size of each network segment based on the on-chip cache capacity of the target chip and the peak memory value of a single batch of each network segment.

[0060] Among them, the on-chip cache capacity can be used to describe the size of the on-chip cache of the chip.

[0061] In an embodiment of the present invention, the ratio of the on-chip cache capacity of the target chip to the memory peak value in a single batch segment of each network segment can be calculated, and the calculated ratio is used as the target cache batch size of the corresponding network segment.

[0062] In an optional embodiment of the present invention, after calculating the target cache batch size of each network fragment, it can also include: determining the network fragments to be fused that meet the network fragment fusion conditions according to the target cache batch size of each network fragment; fusing the network fragments to be fused to obtain fused processed network fragments, and updating the network fragments according to the fused processed network fragments.

[0063] The network segment fusion condition may be a condition for determining whether two adjacent network segments can be used as a network segment for cache batch adjustment. The network segments to be fused may be two adjacent network segments that meet the network segment fusion condition. The fused network segment may be a network segment composed of the network segments to be fused.

[0064] In an embodiment of the present invention, the target cache batch size of each network segment can be obtained first, and then the difference in the target cache batch sizes of two adjacent network segments can be calculated. If there are two adjacent network segments whose target cache batch size difference is not greater than a preset difference, the two adjacent network segments whose target cache batch size difference is not greater than the preset difference are used as network segments to be fused that meet the network segment fusion conditions, so that the network segments to be fused are fused to obtain a fused network segment, that is, the network segment to be fused is regarded as a complete network segment, and the determined network segment is updated based on the fused network segment.

[0065] Step 240: perform intra-chip data splicing on the last cycle output data of the previous network segment in the current adjacent network segment and the remaining input of the next network segment to obtain the target input data of the operator corresponding to the next network segment, so as to perform data processing on the target input data based on the operator corresponding to the next network segment.

[0066] In an optional embodiment of the present invention, after obtaining the target input data of the operator corresponding to the subsequent network segment, it may also include: after the operator corresponding to the subsequent network segment completes data processing of the target input data, updating the current adjacent network segment, and returning to execute the operation of performing in-chip data splicing of the loop output data of the previous network segment in the current adjacent network segment with the remaining input of the subsequent network segment to obtain the target input data of the operator corresponding to the subsequent network segment, until the target input data of the operator corresponding to the full amount of network segments is determined.

[0067] In an embodiment of the present invention, after data processing is performed on the target input data based on the operator corresponding to the subsequent network fragment, the subsequent network fragment and the next adjacent network fragment of the subsequent network fragment are used as the new current adjacent network fragment, so as to return to execute the operation of performing in-chip data splicing of the cyclic output data of the previous network fragment in the current adjacent network fragment with the remaining input of the subsequent network fragment to obtain the target input data of the operator corresponding to the subsequent network fragment, until the target input data of the operator corresponding to the full amount of network fragments is determined, so as to realize data processing of the corresponding target input data by all operators.

[0068] In a specific example, first analyze the tensor memory of the current neural network when the batch number is equal to 1, and draw a trend chart of the tensor memory usage of the current neural network as the operator execution order and the operator tensor memory usage. According to the trend chart, the network is segmented. Segmentation usually divides several operators with relatively stable memory usage into one segment. The network segmentation meets the following conditions: 1. Within the same network segment, the memory usage of each operator should be close, or the operators at the beginning and end of the network segment have relatively close memory usage to ensure that the memory cache has a high cache utilization rate within the entire network segment, because within the network segment, the size of the slice is determined by the memory peak operator within the network segment. 2. The network segment should include a certain number of operators (at least 3 or more) to ensure the overall benefit of the on-chip cache and balance the operator memory change range and the number of operators. If the memory usage fluctuates greatly and changes rapidly, dividing the network segment only according to the memory change trend will result in too few operators in each fusion segment, making it impossible to perform effective on-chip caching. The tensor memory change trend and the number of operators should be balanced to divide each operator into its own cache segment. A network segment is equivalent to a fusion segment. The computing operators in a fusion segment will be cached as a whole in the segment, and the continuous execution of computing operators in the fusion segment has data dependency.

[0069] Figure 4 A schematic diagram of dividing a network segment based on a change trend diagram is provided in the second embodiment of the present invention. Figure 4 As shown in the figure, the horizontal axis is the operator execution order (op) of the operators in the network, and the vertical axis is the operator tensor memory usage of each operator. The memory usage gradually decreases with the execution of the operator. Through this solution, the entire network can be divided into 4 segments where the memory is significantly reduced, and a separate cache analysis is performed inside each segment.

[0070] Figure 5 This is a schematic diagram of a neural network chip resource scheduling logic provided by the second embodiment of the present invention. Figure 5As shown in Figure 2, for a certain network segment (in Figure 5 In the abbreviation Stage, under the cache of fixed chip specifications, an optimal batch size for the cache, i.e., the target cache batch size, can be calculated with certainty. For different network segments, the optimal batch size for each network segment is also different, because their memory usage is usually different. Based on the optimization at the scheduling level, when two adjacent network segments are scheduled, the last cycle output data of the previous network segment is not moved out of the chip's on-chip cache, and the last cycle output data of the previous network segment is used as part of the input of the next network segment. The remaining input of the next network segment is continued to be moved into the chip, and the input splicing operation is completed directly on the on-chip cache.

[0071] The actual operation diagram of the operator of this scheme can be found in Figure 6 . Figure 6 The white dots in the figure represent batch tensors, which are used to reflect the propagation behavior of data in the network. The yellow dots represent concatenation behavior, which is used to merge the previous serial data into one tensor. The dotted arrows represent the order of operator execution, not data dependency. The solid arrows represent the order of operator execution and data dependency, which are unified. Figure 7 By comparing the operator operation diagram of the conventional batch scheduling of the existing neural network, we can intuitively see that the output data of the previous network segment is directly retained in the chip in this solution without external transfer, and is directly cached in the chip with the input of the next network segment, which can effectively reduce the redundant transfer of data. Figure 7 This describes how the inference stack is processed when the batch size is 16, that is, all batches form a tensor that passes through all the layers of the neural network layer by layer. Figure 7 The white dots in the middle represent the tensors of the batch, and the solid arrows represent the execution order of operators. Figure 7 Optimizing the execution order, disassembling batches, assembling them into appropriate sizes and executing them in sequence (there may be multiple evolutionary optimization versions), can improve the cache capacity of the L2 buf (L2 cache area) layer and make full use of the locality of L2 storage data.

[0072] This solution no longer focuses on the memory peak of the entire neural network, but on the memory usage change trend chart of the entire network. It is segmented according to the places where the memory usage jumps in the change trend chart. After segmentation, each segment is a network segment, so that the memory usage in the network segment is close, and the memory usage between network segments is quite different. This ensures that the memory usage of each network segment is similar, and the high peak value of individual operators in the network will not cause the low on-chip cache utilization rate of the remaining operators in the entire network. For each network segment after segmentation, the batch size that can be cached is calculated as the target cache batch size of the network segment. It does not rely on the memory peak of the entire network to perform a single division of the on-chip cache area, but divides the entire network into several network segments with similar memory usage according to the memory usage change trend, and each network segment is cached separately. The batch cache size within the network segment is only related to the memory peak within the current network segment, and is no longer linked to the memory peak of the entire network, ensuring the on-chip cache utilization rate within each network segment.

[0073] During the operation of the neural network, a new data dependency relationship is added by associating two consecutive network fragments. The output data of the previous network fragment is directly retained in the chip without being transferred to the outside, and is directly cached in the chip with the input of the next network fragment. In this way, the data interaction between adjacent fragments and outside the chip can be greatly reduced, and the redundant transfer step of the result data from inside the chip -> outside the chip -> inside the chip is eliminated, thereby improving performance. In addition, the scheduling method of this operator execution order does not depend on the network type and operator type. It only needs to pay attention to the memory usage of the operator, and has complete generalization performance.

[0074] The technical solution of the embodiment of the present invention is to draw a change trend graph based on the operator execution order and operator tensor memory usage of the current neural network, and segment the change trend graph to obtain each network segment, thereby obtaining the memory peak value within a single batch segment of each network segment, and then based on the on-chip cache capacity of the target chip and the memory peak value within a single batch segment of each network segment, calculate the target cache batch size of each network segment, and further perform on-chip data splicing of the last cycle output data of the previous network segment in the current adjacent network segment with the remaining input of the next network segment to obtain the target input data of the operator corresponding to the next network segment, so as to perform data processing on the target input data based on the operator corresponding to the next network segment. This solution calculates the cache batch size for each network segment, breaking the drawback of using memory peak to determine the batch size of the entire network. By dynamically changing the batch size in the cache, the number of loop slicing in the middle and tail segments of the network is greatly reduced to optimize the utilization of neural network memory resources. By optimizing the execution order of operators, the repeated movement of weight data and the fixed overhead of the system are reduced. This solves the problems of low on-chip cache utilization, frequent repeated weight movement, and waste of cache space and chip computing power in the batch scheduling method of directly determining the entire network using memory peak. It can make full use of the on-chip cache space, reduce repeated weight data movement and system fixed overhead, and greatly improve the optimized operation efficiency of the on-chip cache.

[0075] Embodiment 3

[0076] Figure 8 This is a schematic diagram of the structure of a neural network chip resource scheduling device provided by the third embodiment of the present invention. Figure 8 As shown, the device comprises:

[0077] The trend graph drawing and segmentation module 310 is used to draw a change trend graph based on the operator execution order and operator tensor memory usage of the current neural network, and segment the change trend graph to obtain each network fragment.

[0078] The batch size calculation module 320 is used to calculate the target cache batch size of each network segment.

[0079] The on-chip data splicing module 330 is used to perform on-chip data splicing on the last cycle output data of the previous network segment in the current adjacent network segment with the remaining input of the next network segment to obtain the target input data of the operator corresponding to the next network segment, so as to perform data processing on the target input data based on the operator corresponding to the next network segment.

[0080] The technical solution of the embodiment of the present invention draws a change trend graph based on the operator execution order and operator tensor memory usage of the current neural network, and segments the change trend graph to obtain each network segment, thereby calculating the target cache batch size of each network segment, and then performing on-chip data splicing of the last cycle output data of the previous network segment in the current adjacent network segment with the remaining input of the next network segment to obtain the target input data of the operator corresponding to the next network segment, so as to perform data processing on the target input data based on the operator corresponding to the next network segment. This solution calculates the cache batch size for each network segment, breaking the drawback of using memory peak to determine the batch size of the entire network. By dynamically changing the batch size in the cache, the number of loop slicing in the middle and tail segments of the network is greatly reduced to optimize the utilization of neural network memory resources. By optimizing the execution order of operators, the repeated movement of weight data and the fixed overhead of the system are reduced. This solves the problems of low on-chip cache utilization, frequent repeated weight movement, and waste of cache space and chip computing power in the batch scheduling method of directly determining the entire network using memory peak. It can make full use of the on-chip cache space, reduce repeated weight data movement and system fixed overhead, and greatly improve the optimized operation efficiency of the on-chip cache.

[0081] Optionally, the trend graph drawing and segmentation module 310 is specifically used to determine the operator to be compared with the current tensor memory usage, and determine the operator tensor memory usage similarity of the operator to be compared with the current tensor memory usage; according to the operator tensor memory usage similarity and the lower limit threshold of the number of operators in a single network segment, the change trend graph is segmented to obtain the current network segment.

[0082] Optionally, the trend graph drawing and segmentation module 310 is specifically used to determine the first memory usage similarity between the starting end operator and the ending end operator in the operator to be compared with the current tensor memory usage, or the second memory usage similarity between non-two-end point operators, based on the operator tensor memory usage of the operator to be compared with the current tensor memory usage; and use the first memory usage similarity or the second memory usage similarity as the operator tensor memory usage similarity.

[0083] Optionally, the trend graph drawing and segmentation module 310 is specifically used to update the current tensor memory usage to be compared operator based on the operator in the next execution order of the terminal end operator in the current tensor memory usage to be compared operator when the number of operators to be compared in the current tensor memory usage is less than the lower limit threshold of the number of operators in the single network segment, or the similarity of the operator tensor memory usage meets the similarity judgment condition; and return to execute the operation of determining the similarity of the operator tensor memory usage based on the operator tensor memory usage of the current tensor memory usage to be compared operator, until the number of operators to be compared in the current tensor memory usage is greater than the lower limit threshold of the number of operators in the single network segment, and the similarity of the operator tensor memory usage does not meet the similarity judgment condition, then use the current tensor memory usage to be compared operator to segment the change trend graph to obtain the current network segment.

[0084] Optionally, the batch size calculation module 320 is specifically used to obtain the memory peak value within a single batch segment of each of the network segments; based on the on-chip cache capacity of the target chip and the memory peak value within a single batch segment of each of the network segments, calculate the target cache batch size of each of the network segments.

[0085] Optionally, the neural network on-chip resource scheduling device also includes a network fragment update module, which is used to determine the network fragments to be fused that meet the network fragment fusion conditions based on the target cache batch size of each of the network fragments; fuse the network fragments to be fused to obtain fused processed network fragments, and update the network fragments based on the fused processed network fragments.

[0086] Optionally, the on-chip data splicing module 330 is specifically used to update the current adjacent network segment after the operator corresponding to the subsequent network segment completes data processing of the target input data, and return to execute the on-chip data splicing of the loop output data of the previous network segment in the current adjacent network segment with the remaining input of the subsequent network segment to obtain the target input data of the operator corresponding to the subsequent network segment, until the target input data of the operator corresponding to the full amount of network segments is determined.

[0087] The neural network on-chip resource scheduling device provided in the embodiment of the present invention can execute the neural network on-chip resource scheduling method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0088] Embodiment 4

[0089] Fig. 9A schematic diagram of an electronic device that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0090] like Fig. 9 As shown, the electronic device 10 includes at least one processor 11, and a memory such as ROM 12, RAM 13, etc., which is communicatively connected to the at least one processor 11, wherein the memory stores a computer program that can be executed by the at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the ROM 12 or the computer program loaded from the storage unit 18 to the RAM 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, ROM 12 and RAM 13 are connected to each other through a bus 14. The I / O interface 15 is also connected to the bus 14. Among them, ROM 12 is a read-only memory, RAM 13 is a random access memory, and the I / O interface 15 is an input / output interface.

[0091] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0092] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a neural network on-chip resource scheduling method.

[0093] In some embodiments, the neural network on-chip resource scheduling method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the neural network on-chip resource scheduling method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the neural network on-chip resource scheduling method in any other appropriate manner (e.g., by means of firmware).

[0094] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0095] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0096] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0097] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0098] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0099] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of traditional physical hosts and VPS servers, which are difficult to manage and have weak business scalability.

[0100] The embodiment of the present application also discloses a computer program product, which includes a computer program, and when the computer program is executed by a processor, the method for scheduling resources on a neural network chip provided by any embodiment of the present application is implemented. The program product and the method for scheduling resources on a neural network chip disclosed in each embodiment of the present application belong to the same inventive concept, so they are not described here in detail.

[0101] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0102] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A neural network on-chip resource scheduling method, characterized in that: include: Based on the operator execution order and operator tensor memory usage of the current neural network, a change trend graph is drawn, and the change trend graph is segmented to obtain each network segment; Calculating a target cache batch size for each of the network segments; The last cycle output data of the previous network segment in the current adjacent network segment is concatenated with the remaining input of the subsequent network segment to obtain the target input data of the operator corresponding to the subsequent network segment, so as to perform data processing on the target input data based on the operator corresponding to the subsequent network segment.

2. The method according to claim 1, characterized in that The change trend graph is segmented to obtain network segments, including: Determine an operator to be compared with the current tensor memory usage, and determine the similarity of the operator tensor memory usage of the operator to be compared with the current tensor memory usage; According to the similarity of the operator tensor memory usage and the lower limit threshold of the number of operators in a single network segment, the change trend graph is segmented to obtain the current network segment.

3. The method according to claim 2, characterized in that Determining the similarity of operator tensor memory usage of the operator to be compared with the current tensor memory usage includes: Based on the operator tensor memory usage of the operator to be compared with the current tensor memory usage, determine a first memory usage similarity between a starting end operator and an ending end operator in the operator to be compared with the current tensor memory usage, or a second memory usage similarity between non-two-end point operators; The first memory usage similarity or the second memory usage similarity is used as the operator tensor memory usage similarity.

4. The method according to claim 3, characterized in that According to the similarity of the operator tensor memory usage and the lower limit threshold of the number of operators in a single network segment, the change trend graph is segmented to obtain the current network segment, including: If the number of operators to be compared in the current tensor memory usage is less than the lower limit threshold of the number of operators in the single network segment, or the similarity of the operator tensor memory usage satisfies the similarity judgment condition, then based on the operator in the next execution order of the terminal end operator in the current tensor memory usage to be compared, update the current tensor memory usage to be compared operator; And return to execute the operation of determining the similarity of operator tensor memory usage based on the operator to be compared with the current tensor memory usage, until the number of operators to be compared with the current tensor memory usage is greater than the lower limit threshold of the number of operators in the single network segment, and the similarity of the operator tensor memory usage does not meet the similarity judgment condition, then use the current tensor memory usage to compare The operator is segmented to obtain the current network segment.

5. The method according to claim 1, characterized in that Calculating a target cache batch size for each of the network segments, including: Obtaining a single batch memory peak value of each of the network segments; Based on the on-chip cache capacity of the target chip and the memory peak value in a single batch segment of each of the network segments, the target cache batch size of each of the network segments is calculated.

6. The method according to claim 1, characterized in that After calculating the target cache batch size of each of the network segments, the method further includes: Determining the network segments to be fused that meet the network segment fusion conditions according to the target cache batch size of each of the network segments; The network segments to be merged are merged to obtain merged network segments, and the network segments are updated according to the merged network segments.

7. The method according to claim 1 or 6, characterized in that: After obtaining the target input data of the operator corresponding to the latter network segment, the method further includes: After the operator corresponding to the subsequent network segment completes data processing on the target input data, the current adjacent network segment is updated, and the operation of performing in-slice data splicing of the loop output data of the previous network segment in the current adjacent network segment with the remaining input of the subsequent network segment is returned to obtain the target input data of the operator corresponding to the subsequent network segment, until the target input data of the operator corresponding to the full amount of network segments is determined.

8. A neural network on-chip resource scheduling device, characterized in that: include: A trend graph drawing and segmentation module is used to draw a change trend graph based on the operator execution order and operator tensor memory usage of the current neural network, and segment the change trend graph to obtain each network fragment; A batch size calculation module, used to calculate a target cache batch size for each of the network segments; The on-chip data splicing module is used to perform on-chip data splicing on the last cycle output data of the previous network segment in the current adjacent network segment with the remaining input of the next network segment to obtain the target input data of the operator corresponding to the next network segment, so as to perform data processing on the target input data based on the operator corresponding to the next network segment.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the neural network on-chip resource scheduling method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the neural network on-chip resource scheduling method according to any one of claims 1 to 7 when executed.

Images