Neural network scheduling method and apparatus

Abstract

The application discloses a neural network scheduling method and device, relates to the field of neural networks, and can improve the utilization rate of on-chip storage capacity and the operation performance of hardware. The method comprises the following steps: determining a first batch size corresponding to each layer in a neural network (S701); based on the first batch size, the neural network is divided into a neural network comprising at least one first layer group (S702); based on the division result of the first layer group, the neural network is divided into a neural network comprising at least one second layer group (S703); and based on the division result of the second layer group, the neural network is scheduled (S704).

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a neural network scheduling method and apparatus. Background Technology

[0002] A neural network is a machine learning model that uses one or more layers to generate output data from received input data. In addition to input and output layers, a neural network also includes one or more hidden layers. The output data of each hidden layer serves as the input data for the next hidden or output layer. During computation, each hidden layer needs to store its input and output data in on-chip memory. Due to limitations in on-chip memory capacity, the input data for each hidden layer cannot be too large; otherwise, the cache requirements for some hidden layers may exceed the on-chip memory capacity. Therefore, the concept of batch processing is proposed, where input data is fed into the neural network in batches for computation, and the batch size limits the cache requirements of each hidden layer in the neural network.

[0003] However, determining the batch size only for the caching requirements of the input and output data of each hidden layer is limited by the hidden layer with the largest caching requirement, resulting in a small batch size of the neural network's input data and affecting the overall computational efficiency of the neural network. To address this, existing technologies have proposed a neural network scheduling method that leverages the correlation between the input and output data of each hidden layer and the hidden layers before and after it, as well as the characteristic that the output data size of some hidden layers is smaller than the input data size, thus reducing caching requirements. This method divides the neural network into multiple superlayers, each containing one or more hidden layers. By adjusting the scheduling order of the hidden layers in the neural network to construct superlayers, the caching requirements are reduced, and the batch size of the input data for the superlayers can be increased.

[0004] For example, suppose the on-chip memory capacity is 12 units. Figure 1 As shown in (a), a neural network contains 5 layers. The input batch size is 1, consisting of batches of 0 elements. Each layer has a different cache requirement. Limited by the 8-unit cache requirement of layer B, the maximum batch size of the input data for this neural network is 1. Based on this, as... Figure 1 As shown in (b) of the diagram, in the prior art, the neural network can be segmented, with layers A, B, and C forming a super L1, and layers D and E forming a super L2. The segmented neural network can process input data with a batch size of 2, consisting of batch elements 0 and 1, thus increasing the batch size.

[0005] In existing methods for scheduling neural networks using superlayers, the batch size of input data for each superlayer is the same, and this batch size is determined based on the cache requirement of the superlayer with the highest cache demand in the neural network. It can be noted that for the other superlayers in the neural network with smaller cache demands, processing input data of the same batch size cannot fully utilize the on-chip memory capacity, resulting in resource waste. Summary of the Invention

[0006] The embodiments of this application provide a neural network scheduling method and apparatus, which can improve the utilization rate of on-chip storage capacity and improve the operating performance of hardware.

[0007] In a first aspect, embodiments of this application provide a neural network scheduling method, the method comprising: determining the initial batch size corresponding to each layer in the neural network; dividing the neural network into neural networks comprising at least one first layer group based on the initial batch size; wherein each first layer group contains at least one layer in the neural network, the layers in each first layer group have the same initial batch size, and the cache requirement of each first layer group is less than or equal to the capacity of on-chip memory; dividing the neural network into neural networks comprising at least one second layer group based on the division result of the first layer group; wherein each second layer group contains at least one first layer group, and the cache requirement of each second layer group is less than or equal to the capacity of on-chip memory; at least one second layer group contains at least two first layer groups with different initial batch sizes; scheduling the neural network based on the division result of the second layer group.

[0008] It's important to note that because neural networks process data using operators within layers, the input data for each layer is the output data of the previous layer, and subsequent layers must perform operations based on the output data of the previous layer. In other words, data transmission within layers of a neural network is data-dependent. Therefore, layer grouping can only be done on adjacent layers, and the scheduling order of these subsequent layers must be consistent with their original scheduling order within the neural network. That is, neural network segmentation is a segmentation based on the sequence of the network's layers, and the layers in the resulting layer group must be consecutive. For example, if a neural network contains 5 layers, arranged sequentially from L1 to L5, then L1, L2, and L3 can be grouped together, but L1 and L3 cannot be grouped together.

[0009] Therefore, the batch size of each layer in the neural network is first determined based on the capacity of the on-chip memory. Then, layers with the same batch size are merged into a first layer group. Next, based on the cache requirements of the first layer groups and the capacity of the on-chip memory, multiple first layer groups are merged into a second layer group. The resulting second layer group contains first layer groups with different batch sizes. In other words, when the neural network is subsequently scheduled in units of second layer groups, input data is processed using different batch sizes. Thus, the cache requirements of each second layer group will not exceed the capacity of the on-chip memory, while improving the utilization of on-chip memory and enhancing hardware performance.

[0010] In one possible implementation, determining the initial size corresponding to each layer in the neural network includes: determining the initial size corresponding to each layer of the neural network based on the cache requirements of each layer and the capacity of the on-chip memory.

[0011] In a neural network, different layers may perform the same or different functions, and the operators and parameters of each layer may also be the same or different. Therefore, the batch size corresponding to each layer in the neural network may be the same or different. Thus, it is necessary to determine the batch size for each layer separately. In existing technologies, all layers in a neural network process data using the same batch size, which can lead to some layers not fully utilizing memory capacity, resulting in resource waste and reduced efficiency.

[0012] For example, assuming the on-chip memory capacity is 100, and the cache requirement for processing one image by L0 is 105, then the basic batch size for L0 is determined to be 1 / 2 images. The cache requirement for processing one image by L1 is 50, then the basic batch size for L1 is determined to be 2 images.

[0013] In one possible implementation, the initial size of each layer of the neural network is determined based on the cache requirements of each layer and the capacity of the on-chip memory. This includes determining the initial size of each layer of the neural network based on one or more input data and one or more output data of each layer, as well as the capacity of the on-chip memory. Herein, at least one input data or at least one output data of at least one layer of the neural network is stored in off-chip memory.

[0014] Each layer in a neural network can contain one or more input data and one or more output data, and each set of data can be stored in on-chip memory or off-chip memory.

[0015] In one possible implementation, for each layer of the neural network, considering one or more input data and one or more output data, and the capacity of the on-chip memory, determining the initial size corresponding to each layer includes: adjusting the storage location of one or more input data and / or one or more output data of at least one layer in the neural network based on the computational overhead of the neural network. The storage location may include on-chip memory or off-chip memory. During the adjustment of storage locations, the storage location of one or more input data and one or more output data of each layer of the neural network is obtained when the computational overhead is minimized. Based on the storage location of one or more input data and one or more output data of each layer of the neural network, and the capacity of the on-chip memory, the initial size corresponding to each layer of the neural network is determined.

[0016] The data processing in a neural network includes data input (reading in input data), computation, and data output (storing output data). First, a portion of the data is input, incurring overhead (head overhead). Then, the input, computation, and output processes run in parallel. Finally, the neural network outputs the final calculated data, storing it in memory; this incurring overhead (tail overhead).

[0017] Therefore, the process of scheduling neural networks for data processing incurs corresponding computational overhead, such as computation time and data transfer time. The computational overhead of a neural network can be used to measure its performance. Lower computational overhead generally indicates better performance.

[0018] Optionally, based on the characteristics of fast on-chip memory scheduling but small capacity, and slow off-chip memory scheduling but large capacity, a choice can be made between storing data in on-chip memory or off-chip memory. This allows for adjustments to the computational overhead of the neural network. For example, if the initial batch size for a certain layer is small, at least one input and / or output data for that layer can be moved to off-chip memory, thereby increasing the initial batch size of that layer. By adjusting the storage location and batch size, the storage location of each set of data and the batch size corresponding to each layer of the neural network are obtained when the computational overhead is minimized. Subsequently, the neural network is segmented based on this batch size.

[0019] In one possible implementation, based on the initial size, the neural network is divided into neural networks containing at least one first layer group, including: if the cache requirement when scheduling layers i to j as a whole in the neural network is greater than the capacity of the on-chip memory, and the cache requirement when scheduling layers i to (j-1) as a whole is less than or equal to the capacity of the on-chip memory, then based on the computational overhead of the neural network, layers i to (i+m) are determined as a first layer group; wherein, the initial size of layers i to j in the neural network is the same, i, j, m are positive integers, and (i+m)≤(j-1).

[0020] Optionally, starting from the first layer of the neural network, based on the initial size and the capacity of the on-chip memory, a segmentation method is determined to divide the neural network into groups containing the first layer. Alternatively, the initial, consecutive layers of the same size in the neural network are first identified, and these layers are used as segmentation units to perform the above segmentation steps. Then, according to the scheduling order of the layers in the neural network, the layer groups are segmented starting from the first layer in the segmentation unit.

[0021] In this way, the batch size of the layers in the first layer group of the partitioned neural network is the same. Therefore, when the neural network is scheduled using the first layer group as the scheduling unit, the utilization of on-chip memory capacity can be improved. This ensures that when the neural network is subsequently partitioned into second layer groups based on the partitioning results of the first layer group, the utilization of on-chip memory capacity can still be improved.

[0022] In one possible implementation, based on the computational overhead of the neural network, layers i to (i+m) are defined as a first layer group. This includes obtaining multiple computational overheads corresponding to scheduling layers i to t as a whole, where layer t is any one of layers (i+1) to (j-1); t is a positive integer, (i+1)≤t≤(j-1). The computational overhead of the neural network is minimized when scheduling layers i to (i+m) as a whole.

[0023] For example, suppose a neural network has layers L0, L1, L2, and L3, each with a 3x3 kernel size and a stride of 1 (less than the kernel's side length). Scheduling L0, L1, L2, and L3 as a single unit and using a padding algorithm to process the input data results in an overlap problem. Scheduling them as a single unit can also be understood as scheduling the layers within this unit as a layer group after the neural network is segmented. Scheduling L0-L2 as a single unit results in a cache requirement less than or equal to the on-chip memory capacity. However, if L3 is segmented into the same unit as L0-L2 (i.e., L0-L3 are also segmented as a single unit), the cache requirement exceeds the on-chip memory capacity due to the padding algorithm. Therefore, L3 should not be segmented into the L0-L2 unit to avoid exceeding the on-chip memory capacity during computation. Furthermore, the segmentation method within L0-L2 should be adjusted based on the computational overhead of the neural network. During the adjustment process, the computational overhead of the neural network is reduced. For example, for t=1 and t=2, the corresponding computational overhead is determined. If the computational overhead of the neural network when L0-L1 is determined as the first layer group is less than the computational overhead of the neural network when L0-L2 is determined as the first layer group, then L0-L1 is determined as a first layer group. That is, when m=1, the first and second layers of the neural network are determined as a first layer group.

[0024] In one possible implementation, based on the segmentation result of the first layer group, the neural network is segmented into a neural network containing at least one second layer group, including: if the cache requirement when scheduling the first layer group from a to b as a whole in the neural network is greater than the capacity of the on-chip memory, and the cache requirement when scheduling the first layer group from a to (b-1) as a whole is less than or equal to the capacity of the on-chip memory, then based on the computational overhead of the neural network, the first layer group from a to b is determined as a second layer group; or, the first layer group from a to (b-1) is determined as a second layer group; wherein, at least two of the first layer groups from a to b in the neural network have different first batch sizes, and a and b are positive integers.

[0025] Optionally, adjacent first-layer groups whose initial size is multiples of each other can be divided into a second-layer group. For example, if the initial size of the first first-layer group is 2 images, and the initial size of the second first-layer group is 8 images, and the second first-layer group is adjacent to the first first-layer group, and its initial size is 3 times that of the first first-layer group, then if the cache requirement of the first and second first-layer groups as a whole is less than or equal to the on-chip memory capacity, these two first-layer groups can be divided into a second-layer group.

[0026] Optionally, starting from the first first-layer group contained in the neural network, the first-layer groups in the neural network are traversed sequentially to divide the neural network into neural networks containing at least one second-layer group. For example, after dividing the first and second first-layer groups into a whole, it is determined whether the cache requirement of the current whole division exceeds the on-chip memory capacity. If the cache requirement of the whole division formed by the first and second first-layer groups exceeds the on-chip memory capacity, the first first-layer group is divided into a second-layer group, and the division continues from the second first-layer group to the next second-layer group. If the cache requirement of the current whole division formed by the first and second first-layer groups does not exceed the on-chip memory capacity, the first and second first-layer groups are divided into a second-layer group. Then, it is determined whether the cache requirement of the current whole division exceeds the on-chip memory capacity after dividing the third first-layer group into the whole formed by the first and second first-layer groups. Thus, by repeating the above steps and traversing all the first-layer groups contained in the neural network, the neural network is divided into neural networks containing at least one second-layer group.

[0027] In this way, when the segmented neural network is scheduled based on the second layer group, the utilization rate of on-chip memory capacity can be improved, thus improving hardware performance.

[0028] In one possible implementation, the method further includes: if the a-th first layer group to the b-th first layer group are determined as a second layer group, then the first batch size corresponding to the b-th first layer group or the (b-1)-th first layer group is reduced.

[0029] When scheduling the first layer group from the ath to the bth first layer group as a whole, the corresponding cache requirement exceeds the capacity of the on-chip memory. Therefore, by reducing the first batch size corresponding to the bth or (b-1)th first layer group, the cache requirement can be reduced, thereby ensuring that the cache requirement of the determined second layer group does not exceed the capacity of the on-chip memory.

[0030] For example, suppose the first batch size corresponding to the first layer group is 2 images, and the first batch size corresponding to the second layer group is 6 images. If the two layer groups are split into one, then the first layer group needs to be scheduled for 3 operations before the second layer group needs to be scheduled for 1 operation. Due to the gather problem, the layers in the first layer group generate additional on-chip memory cache requirements, causing the cache requirements of the second layer group to exceed the on-chip memory capacity. Therefore, the first batch size corresponding to the second layer group can be reduced, such as to 4 images. The additional on-chip memory cache requirements of the corresponding layers in the first layer group will be reduced accordingly. The two layer groups can then be split into one second layer group, and the cache requirements of the second layer group will be less than or equal to the on-chip memory capacity.

[0031] In one possible implementation, based on the computational overhead of the neural network, the first layer group a to the first layer group b are determined as a second layer group; or, the first layer group a to the (b-1)th first layer group are determined as a second layer group. This includes: when scheduling the first layer group a to the first layer group b as a whole, the computational overhead of the neural network is the first computational overhead; when scheduling the first layer group a to the (b-1)th first layer group as a whole, the computational overhead of the neural network is the second computational overhead. If the first computational overhead is less than the second computational overhead, then the first layer group a to the first layer group b are determined as a second layer group; if the second overhead is less than the first computational overhead, then the first layer group a to the (b-1)th first layer group are determined as a second layer group.

[0032] Thus, when scheduling based on the second layer group, the segmented neural network has the lowest computational overhead, improving the utilization of on-chip memory capacity and enhancing hardware performance.

[0033] In one possible implementation, during the scheduling of the neural network based on the segmentation results of the second layer group, the scheduling order of the layers in the second layer group is determined according to the scheduling order of each first layer group contained in the second layer group, and the scheduling order of the layers in the first layer group.

[0034] In one possible implementation, the scheduling order of the layers in the first layer group is the same as the scheduling order of the layers in the neural network before splitting, and the scheduling order of each first layer group contained in the second layer group is determined according to the first batch size and the scheduling order of the first and last layers in the first layer group.

[0035] For example, the neural network contains 6 convolutional layers L1-L6, and the scheduling order before splitting is L1→L2→L3→L4→L5→L6. L1-L3 form a first layer group, with an initial size of 2 images, and L4-L6 form another first layer group, with an initial size of 4 images. These two first layer groups are consecutive, and their initial sizes are doubled. After splitting these two first layer groups into a second layer group, the cache requirement of the second layer group is less than or equal to the on-chip memory capacity. Based on the splitting result of the second layer group, the input data of the neural network is processed. The input data is input into L1, with input data A0 and B0, each corresponding to an initial size of 2 images. The scheduling order of the first first layer group (L1-L3) is L1→L2→L3, and the scheduling order of the second first layer group (L4-L6) is L4→L5→L6. Based on the initial dimensions, it is determined that the first layer group needs to be scheduled twice before the second layer group needs to be scheduled once, and L3 is scheduled before L4 in the neural network before the split. Therefore, the scheduling order of the layers in the split neural network is L1→L2→L3→L1→L2→L3→L4→L5→L6.

[0036] In one possible implementation, at least one input or output data of a layer included in the second layer group is stored in on-chip memory, while the input data of the first layer and the output data of the last layer in the second layer group are stored in off-chip memory.

[0037] In other words, at least one piece of data in the split second-layer group can be quickly stored in on-chip memory, improving computational efficiency and on-chip memory utilization. Furthermore, the input and output data of the split second-layer group are stored in off-chip memory, thus ensuring that the cache requirements of the second layer do not exceed the capacity of the on-chip memory.

[0038] Secondly, embodiments of this application provide a neural network scheduling device, which may include: a determining unit, a segmenting unit, and a scheduling unit. The determining unit is used to determine the initial batch size corresponding to each layer in the neural network. The segmenting unit is used to segment the neural network into neural networks containing at least one first layer group based on the initial batch size; wherein each first layer group contains at least one layer of the neural network, the layers in each first layer group have the same initial batch size, and the cache requirement of each first layer group is less than or equal to the capacity of the on-chip memory. The segmenting unit is further used to segment the neural network into neural networks containing at least one second layer group based on the segmentation result of the first layer group; wherein each second layer group contains at least one first layer group, and the cache requirement of each second layer group is less than or equal to the capacity of the on-chip memory; at least one second layer group contains at least two first layer groups with different initial batch sizes. The scheduling unit is used to schedule the neural network based on the segmentation result of the second layer group.

[0039] In one possible implementation, a determining unit is used to determine the initial size of each layer of the neural network, specifically based on the cache requirements of each layer and the capacity of the on-chip memory.

[0040] In one possible implementation, a determining unit is specifically used to determine the initial size corresponding to each layer of the neural network for one or more input data and one or more output data of each layer in the neural network, as well as the capacity of the on-chip memory; wherein at least one input data or at least one output data of at least one layer in the neural network is stored in off-chip memory.

[0041] In one possible implementation, a determining unit is specifically used to adjust the storage location of one or more input data and / or one or more output data of at least one layer in the neural network based on the computational overhead of the neural network. The storage location includes on-chip memory or off-chip memory. During the adjustment of the storage location, the storage location of one or more input data and one or more output data of each layer of the neural network is obtained when the computational overhead of the neural network is minimized. Based on the storage location of one or more input data and one or more output data of each layer of the neural network, and the capacity of the on-chip memory, the initial size corresponding to each layer of the neural network is determined.

[0042] In one possible implementation, the segmentation unit is specifically used to determine layers i to j as a first layer group if the cache requirement when scheduling layers i to j in the neural network as a whole is greater than the capacity of the on-chip memory, and the cache requirement when scheduling layers i to (j-1) as a whole is less than or equal to the capacity of the on-chip memory. Based on the computational overhead of the neural network, layers i to (i+m) are defined as a first layer group. Here, the first batch size of layers i to j in the neural network is the same, i, j, and m are positive integers, and (i+m) ≤ (j-1).

[0043] In one possible implementation, the segmentation unit is specifically used to obtain multiple computational overheads corresponding to scheduling layers i to t as a whole, where layer t is any layer from layer (i+1) to layer (j-1); t is a positive integer, (i+1)≤t≤(j-1); when scheduling layers i to (i+m) as a whole, the computational overhead of the neural network is minimized.

[0044] In one possible implementation, the segmentation unit is specifically used to determine the first layer group from the a-th to the b-th first layer group in the neural network as a second layer group if the cache requirement when scheduling the first layer group from the a-th to the b-th first layer group as a whole is greater than the capacity of the on-chip memory, and the cache requirement when scheduling the first layer group from the a-th to the (b-1)-th first layer group as a whole is less than or equal to the capacity of the on-chip memory; or, the first layer group from the a-th to the (b-1)-th first layer group is determined as a second layer group; wherein, at least two of the first layer groups from the a-th to the b-th first layer group in the neural network have different first batch sizes, and a and b are positive integers.

[0045] In one possible implementation, the segmentation unit is further configured to reduce the first batch size corresponding to the b-th first layer group or the (b-1)-th first layer group if the a-th first layer group to the b-th first layer group are determined to be a second layer group.

[0046] In one possible implementation, the segmentation unit, specifically used to schedule the first layer group from the a-th to the b-th first layer group as a whole, has a first computational overhead in the neural network; when scheduling the first layer group from the a-th to the (b-1)-th first layer group as a whole, the computational overhead of the neural network is a second computational overhead; if the first computational overhead is less than the second computational overhead, then the first layer group from the a-th to the b-th first layer group is determined as a second layer group; if the second overhead is less than the first computational overhead, then the first layer group from the a-th to the (b-1)-th first layer group is determined as a second layer group.

[0047] In one possible implementation, during the scheduling of the neural network based on the segmentation results of the second layer group, the scheduling order of the layers in the second layer group is determined according to the scheduling order of each first layer group contained in the second layer group, and the scheduling order of the layers in the first layer group.

[0048] In one possible implementation, the scheduling order of the layers in the first layer group is the same as the scheduling order of the layers in the neural network before splitting, and the scheduling order of each first layer group contained in the second layer group is determined according to the first batch size and the scheduling order of the first and last layers in the first layer group.

[0049] In one possible implementation, the feature is that at least one input data or output data of a layer included in the second layer group is stored in on-chip memory, and the input data of the first layer and the output data of the last layer in the second layer group are stored in off-chip memory.

[0050] Optionally, the neural network scheduling device described in the second aspect may further include a receiving unit and a transmitting unit. The receiving unit is used to receive signals sent by other devices, such as receiving input data, and the transmitting unit is used to send signals to other devices, such as sending output data. Other devices may include, for example, other terminal devices or network devices.

[0051] It should be noted that the transmitting unit and the receiving unit can also be integrated together, such as a transceiver unit, implemented by a transceiver or transceiver-related circuit components. This application does not specifically limit the specific implementation of the receiving unit and the transmitting unit.

[0052] Optionally, the communication device described in the second aspect may further include a storage unit storing a program or instructions. When the determining unit, the segmenting unit, and the scheduling unit execute the program or instructions, the determining neural network scheduling device described in the second aspect can execute the neural network scheduling method described in the first aspect.

[0053] It should be noted that the neural network scheduling device in the second aspect can be a communication device or a chip (system), hardware circuit or other component or assembly that can be set in the communication device; this application does not limit this.

[0054] Furthermore, the technical effects of the neural network scheduling device in the second aspect can be referenced from the technical effects of the neural network scheduling method in the first aspect, and will not be elaborated here.

[0055] Thirdly, embodiments of this application provide a neural network scheduling apparatus that has the function of implementing the neural network scheduling method of any one of the first aspects described above. This function can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more units corresponding to the above-described function.

[0056] Fourthly, embodiments of this application provide a neural network scheduling device, including: a processor and a memory; the memory is used to store computer execution instructions, and when the neural network scheduling device is running, the processor executes the computer execution instructions stored in the memory to cause the neural network scheduling device to perform the neural network scheduling method as described in the first aspect above and any of its various optional implementations.

[0057] Fifthly, embodiments of this application provide a neural network scheduling apparatus, comprising: a processor; the processor is configured to be coupled to a memory, and after reading instructions from the memory, execute a neural network scheduling method as described in any of the preceding aspects according to the instructions.

[0058] Sixthly, embodiments of this application provide a neural network scheduling apparatus, which includes a processor, a memory, and a communication interface. The memory stores one or more programs. These programs include computer-executable instructions. When the apparatus is running, the processor executes the computer-executable instructions stored in the memory to enable the neural network scheduling method described in any of the first aspect and its various optional implementations.

[0059] In a seventh aspect, embodiments of this application provide a neural network scheduling device. This device can be a chip system, including a processor and potentially a memory, for implementing the neural network scheduling method described in any of the first aspect and its various optional implementations. The chip system can be composed of chips or may include chips and other discrete devices.

[0060] Eighthly, embodiments of this application provide a neural network scheduling apparatus, which can be a circuit system including a processing circuit configured to execute a neural network scheduling method as described in any of the above aspects.

[0061] Ninthly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed by a computer, perform the neural network scheduling method described in the first aspect and any of its various optional implementations.

[0062] In a tenth aspect, embodiments of this application provide a computer program product containing instructions that, when the computer program product is run on a computer, cause the computer to execute the neural network scheduling method described in any one of the first aspect and its various optional implementations. Attached Figure Description

[0063] Figure 1 A schematic diagram of a prior art neural network scheduling method provided in the embodiments of this application;

[0064] Figure 2A This is a schematic diagram of the neural network structure provided in an embodiment of this application;

[0065] Figure 2B A schematic diagram illustrating the neural network input / output method provided in an embodiment of this application;

[0066] Figure 3 A schematic diagram of the structure of an apparatus provided in an embodiment of this application;

[0067] Figure 4 A schematic diagram illustrating a neural network data processing process provided in an embodiment of this application;

[0068] Figure 5 This application provides an illustration of a neural network scheduling scenario. Figure 1 ;

[0069] Figure 6 A schematic diagram of the neural network computation overhead provided in the embodiments of this application;

[0070] Figure 7 A neural network scheduling method flow provided in the embodiments of this application Figure 1 ;

[0071] Figure 8 A schematic diagram illustrating the neural network data storage location provided in an embodiment of this application;

[0072] Figure 9 A flowchart of a neural network scheduling method provided in this application embodiment is shown in Figure 2.

[0073] Figure 10 A neural network scheduling method flow provided in the embodiments of this application Figure 3 ;

[0074] Figure 11 A second schematic diagram illustrating a neural network scheduling scenario provided in an embodiment of this application;

[0075] Figure 12 This application provides an illustration of a neural network scheduling scenario. Figure 3 ;

[0076] Figure 13 This application provides an illustration of a neural network scheduling scenario. Figure 4 ;

[0077] Figure 14 A schematic diagram of the structure of a neural network scheduling device provided in this application embodiment. Figure 1 ;

[0078] Figure 15A schematic diagram of a neural network scheduling device provided in this application embodiment is shown below;

[0079] Figure 16 This is a schematic diagram of the chip system provided in an embodiment of this application. Detailed Implementation

[0080] The following describes in detail, with reference to the accompanying drawings, a neural network scheduling method and apparatus provided in the embodiments of this application.

[0081] Generally, a neural network consists of an input layer, hidden layers, and an output layer. For example, ... Figure 2A The diagram shown is a schematic of a neural network structure.

[0082] The input layer of a neural network can process multidimensional data. Taking image processing as an example, the input layer can receive the pixel values ​​(a three-dimensional array) of an image, that is, two-dimensional pixels on a plane and the values ​​of the RGB channels.

[0083] The hidden layers of a neural network consist of one or more convolutional layers, one or more pooling layers, and one or more fully-connected layers. Typically, one or more convolutional layers are followed by a pooling layer. In some examples, the hidden layers of a neural network may not contain pooling layers.

[0084] The output layer of a neural network has the same structure and working principle as the output layer of a traditional feedforward neural network. For example, in a neural network for image classification, the output layer uses a logistic function or a normalized exponential function (softmax function) to output classification labels, such as people, scenery, and objects. In a neural network for object recognition, the output layer can be designed to output the center coordinates, size, and classification of the object.

[0085] The technical solution provided in this application, based on the specific structure of a neural network model, stores the feature data and weight data between every two layers of the neural network in a storage space during computation. For example, during forward computation, each layer first requests data from the previous layer (i.e., retrieves the data from the storage space), performs its own computation, and then stores the data back into the storage space as input data for the next layer. Similarly, during backward computation, the current layer calls the data output from the next layer before performing its computation. In other words, during neural network computation, each layer generates corresponding input and output data caching requirements and needs to interact with the storage space to retrieve or store data. Therefore, the size of the storage space and the power consumption for data retrieval both affect the performance of the neural network in processing data. The storage space includes on-chip memory and off-chip memory.

[0086] Each hidden layer in a neural network corresponds to one or more input data points and one or more output data points. For example, see... Figure 2B As shown, L0 contains two sets of output data, which are output to L1 and L3 respectively. Therefore, L3 contains two sets of input data, which are the output data of L0 and L2 respectively. Correspondingly, L3 will also generate two sets of output data. In the embodiments of this application, the storage location of each input data or output data of each layer includes storage in on-chip memory or off-chip memory.

[0087] Figure 3 This is a schematic diagram of the structure of a device provided in an embodiment of this application. The device can be an electronic device or server running the above-mentioned neural network, or a component (such as a chip system, circuit system, etc.) in an electronic device or server running the above-mentioned neural network, thereby realizing a specific function.

[0088] The specific functions mentioned above can be applications in computer vision such as image classification, object recognition, action recognition, pose estimation, and neural style transfer, or applications in natural language processing (NLP).

[0089] The device includes a neural network processing unit (NPU) 310, a central processing unit (CPU) (host CPU) 320, and off-chip memory 330.

[0090] The neural network processor NPU 310 is attached to the main CPU 320 as a coprocessor to allocate tasks. The core of the NPU 310 is the arithmetic circuit 331, and the controller 332 controls the arithmetic circuit 331 to extract data from the on-chip memory 333 and perform calculations.

[0091] In some implementations, the arithmetic circuit 331 internally includes multiple process engines (PEs). In other implementations, the arithmetic circuit 331 is a two-dimensional pulsating array. The arithmetic circuit 331 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In still other implementations, the arithmetic circuit 331 is a general-purpose matrix processor.

[0092] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit 331 retrieves the corresponding data for matrix B from the on-chip memory 333 and caches it on each PE in the arithmetic circuit 331. The arithmetic circuit 331 retrieves the data for matrix A from the on-chip memory 333 and performs matrix operations with matrix B. The partial or final result of the obtained matrix is ​​stored in the on-chip memory 333.

[0093] The bus interface unit 334 (BIU) is used to enable interaction between the main CPU 320, the off-chip memory 330, the arithmetic circuit 331 and the on-chip memory 333 via the bus.

[0094] Generally, the on-chip memory 333 can also be called a cache memory, which may include one or more independent cache memories or processing units with data caching capabilities, such as unified memory, input memory, weighted memory, and instruction fetch memory. It can access the off-chip memory 330 under the control of the controller 332. The on-chip memory 333 can be a fast, small-capacity static access memory, such as static random access memory (SRAM).

[0095] In some embodiments, the input data and / or output data of the arithmetic circuit 331 are stored in the on-chip memory 333. The fast interaction speed of the on-chip memory can improve the running speed of the neural network. However, due to its small capacity, it may be necessary to increase the number of interactions, resulting in increased power consumption.

[0096] The off-chip memory 330 is memory external to the NPU and may include one or more independent memories or processing units with data storage capabilities. The off-chip memory 330 is a slow-speed, high-capacity dynamic access memory. For example, it can be double data rate synchronous dynamic random access memory (DDR SDRAM), high bandwidth memory (HBM), or other readable and writable memory.

[0097] In some embodiments, the input data and / or output data of the arithmetic circuit 331 are stored in the off-chip memory 330. Taking advantage of its large capacity, it can process large amounts of data. However, due to its slow speed, the neural network processing efficiency is relatively slow.

[0098] To facilitate understanding of the technical solutions provided in the embodiments of this application, some terms used in the embodiments of this application will be explained first.

[0099] 1) Layer group

[0100] As mentioned above, neural networks contain multiple hidden layers. A neural network can be described as consisting of multiple layers arranged in a directed graph, with each layer having a corresponding set of parameters. Each layer group can be a partition of a directed graph comprising one or more layers.

[0101] Among them, a layer group can also be described as a super layer, a graph, etc., representing a single layer or multiple consecutive layers in a neural network.

[0102] In some examples, the neural network is scheduled to process input data in layers, with the scheduling order of the layers in the layer group being the same as the scheduling order of the layers in the neural network.

[0103] 2) Batch size

[0104] Due to the limited capacity of on-chip memory, the maximum amount of data that each layer in a neural network can process is the corresponding batch size. For example, if the on-chip memory capacity is 100, and the buffer requirement for L1 (layer 1) processing one image is 60, then L1 can process at most one image per scheduling, and the corresponding batch size for L1 is one image. If the buffer requirement for L2 processing one image is 30, then L2 can process at most three images per scheduling, and the corresponding batch size for L2 is three images.

[0105] It is understandable that different layers in a neural network can perform the same or different functions, and the operators and parameters of each layer can also be the same or different. Therefore, the batch size corresponding to the layers in the neural network can be the same or different.

[0106] 3) Overlap problem

[0107] In some scenarios where neural networks process images, due to limitations in on-chip memory capacity, it may be necessary to divide the entire image data into two or more parts as a batch of input data, where each part can be called non-integer image data. Convolutional layers can use padding algorithms to handle non-integer image input data. That is, before computation by the convolution kernel, the size of the input data is artificially increased by padding algorithms to offset the impact of size shrinkage during computation. Padding algorithms can be, for example, zero-padding, repeated boundary value padding, or other methods. In other words, if the input data is non-integer image data, padding algorithms are needed to process the input data; if the input data is integer image data, padding algorithms are not needed.

[0108] Taking the padding algorithm as an example, if a convolutional layer uses a padding algorithm, the input data needs to be padded before being flattened when interpreting the convolutional layer. If the stride of the convolutional kernel is less than the side length of the kernel (usually a square), the kernel and the original input matrix will overlap in their respective regions. If the stride is the same as the side length of the kernel, there will be no overlap. Specifically, if the input data size is (w*w), the padded data size will be (w+ks)*(w+ks). Here, k represents the side length of the convolutional kernel, s represents the stride, and the padded data is (ks).

[0109] For example, see Figure 4 As shown, suppose a neural network has layers L0, L1, L2, and L3, each with a 3x3 kernel size and a stride of 1. The stride is smaller than the kernel's side length. When L0, L1, L2, and L3 are treated as a single unit and a padding algorithm is used to process the input data, an overlap problem arises. Treating them as a single unit can also be understood as dividing the neural network into segments and then scheduling the layers within that single unit as a layer group.

[0110] For example, if the entire image is 56*56 pixels, and it's divided into four parts for processing, and L0-L2 are scheduled as a layer group, it's necessary to ensure that L2 outputs 14 rows of data, meaning the output data size of the layer group is 14*56, to ensure that L3 can process 1 / 4 of the image. Therefore, L2's input data needs to be padded with 2 rows of data, meaning the input data size is 16*56. Correspondingly, L1's input data size is 18*56, and L0's is 20*56. In other words, during the image segmentation process, ensuring the output data size increases the cache requirements of the layers within the layer group. Furthermore, the more layers in the layer group, the more data the previous layers need to padded. If the on-chip memory capacity is small, it limits the layer group size.

[0111] 4) Gathering and Scattering Problems

[0112] When multiple layers and / or groups of layers with different batch sizes in a neural network are scheduled as a single layer group, that is, when they are scheduled as a whole, some data will be retained in the on-chip memory cache space, thus generating additional on-chip memory cache requirements.

[0113] For example, such as Figure 5 As shown, assume that the neural network contains 6 layers, with the layer order being L0-L5. The batch size corresponding to L0, L1, L4 and L5 is 1, and the batch size corresponding to L2 and L3 is 2. The input data are batch 0 and batch 1, both with a batch size of 1. Figure 5 In the diagram, the input and output data corresponding to batch 0 of the neural network are represented by blank ellipses, while the input and output data corresponding to batch 1 of the neural network are represented by ellipses filled with diagonal lines. The neural network is then segmented, that is, one or more layers are divided into a layer group according to the scheduling order of the layers in the neural network. Subsequently, based on the segmentation results, the neural network is scheduled in units of layer groups. L0 and L1, with the same batch size, are segmented into layer group 0; L2 and L3, with the same batch size, are segmented into layer group 1; and L4 and L5, with the same batch size, are segmented into layer group 2.

[0114] It's important to note that because neural networks process data using operators within layers, the input data for each layer is the output data of the previous layer, and subsequent layers must perform operations based on the output data of the previous layer. In other words, data transmission within layers of a neural network is data-dependent. Therefore, layer grouping can only be done on adjacent layers, and the scheduling order of these subsequent layers must be consistent with their original scheduling order within the neural network. That is, neural network segmentation is a segmentation based on the sequence of the network's layers, and the layers in the resulting layer group must be consecutive. For example, if a neural network contains 5 layers, arranged sequentially from L1 to L5, then L1, L2, and L3 can be grouped together, but L1 and L3 cannot be grouped together.

[0115] The current neural network needs to process batches 0 and 1. If layer group 0 and layer group 1 are split into one layer group for scheduling, a gathering problem will occur. For example... Figure 5 As shown, L0 and L1 have a batch size of 1, so layer group 0 can process input data of size 1 at a time, meaning batch 0 and batch 1 are processed separately. After batch 0 is input into L0, it is processed by L0 and L1, and the output data of L1 is C0. L2 has a batch size of 2. At this time, C0 only corresponds to batch 0, which does not meet the processing requirements of L2, so C0 needs to be temporarily stored in the on-chip memory. Batch 1 is input into L0 for processing. After processing by L0 and L1, the output data of L1 is C1. At this time, L1 outputs two batches of data, which meets the processing requirements of L2. The on-chip memory contains two sets of data, C0 and C1. After aggregating C0 and C1, L2 can call the aggregated C0 and C1 for processing. Therefore, if layer group 0 and layer group 1 are divided into one layer group, during the process of scheduling L0 and L1 to process batch 1, C0 occupies the cache space of the on-chip memory, and the amount of data corresponding to C0 is the additional on-chip memory cache requirement of L0 and L1. In this process, the buffering requirement for the input data corresponding to L0 is the data volume corresponding to (C0+A1), and the buffering requirement for the output data is the data volume corresponding to (C0+B1); the buffering requirement for the input data corresponding to L1 is the data volume corresponding to (C0+B1), and the buffering requirement for the output data is the data volume corresponding to (C0+C1).

[0116] If layer group 1 and layer group 2 are split into a single layer group for scheduling, a scatter problem will occur. For example... Figure 5As shown, L3's input data consists of D0 for batch 0 and D1 for batch 1, and its output data consists of E0 for batch 0 and E1 for batch 1. L4's batch size is 1, so it cannot process E0 and E1 simultaneously. Therefore, L4 processes E0 first and temporarily stores E1 in on-chip memory. During the scheduling of L4 and L5 to process the data corresponding to batch 0, E1 occupies the on-chip memory cache space, and the amount of data corresponding to E1 represents the additional on-chip memory cache requirement for L4 and L5. In this process, L4's input data cache requirement is the amount of data corresponding to (E1 + E0), and its output data cache requirement is the amount of data corresponding to (E1 + F0); L5's input data cache requirement is the amount of data corresponding to (E1 + F0), and its output data cache requirement is the amount of data corresponding to (E1 + G0).

[0117] Therefore, when splitting layers and / or layer groups of different batch sizes, it is necessary to consider the additional on-chip memory cache requirements caused by gather or scatter problems, and determine whether the cache requirements of the split layer groups exceed the capacity of the on-chip memory.

[0118] 5) Neural network computational overhead

[0119] In the process of scheduling neural networks for data processing, corresponding computational overhead is incurred, such as computation time overhead and data transfer time overhead. The computational overhead of a neural network can be used to measure its performance. Lower computational overhead generally indicates better performance.

[0120] For example, such as Figure 6 The diagram illustrates a data processing procedure in a neural network, including a data input process (reading in input data), a computation process, and a data output process (storing output data). First, the neural network processes a batch of data, requiring the input of a portion of the data (the head overhead). Then, the input, computation, and output processes proceed in parallel. Finally, the neural network performs a data output process on the final calculated data, storing it in memory; this process incurs the tail overhead.

[0121] In this embodiment, layers process data in batches. During the processing of a batch of input data by a certain layer, the computation time = computational load of that layer / computing power of the chip carrying the neural network; the data transfer time = (input data volume + output data volume) / (on-chip memory bandwidth or off-chip memory bandwidth); and the total time overhead = head overhead + max(computation time, data transfer time) + tail overhead. It can be seen that if the batch size is too small, the time corresponding to the head overhead and tail overhead may be greater than or equal to the computation time, resulting in low neural network computation efficiency. The time overhead of a layer in the neural network can be obtained based on the storage location of the current layer's input data and / or output data, as well as the computing power of the chip carrying the neural network. The data storage location includes on-chip memory and off-chip memory.

[0122] Figure 7 This is a schematic diagram of a neural network scheduling method provided in an embodiment of this application. Figure 7 As shown, it includes: S701-S704.

[0123] S701. Determine the first batch size corresponding to each layer in the neural network.

[0124] In a neural network, the input layer receives and processes the input data, requiring scheduling of the network during processing. The input data is data from the dataset. Taking image processing as an example, the input data might consist of 32 images from the dataset.

[0125] Specifically, based on the cache requirements of each layer in the neural network and the capacity of the on-chip memory, the initial size corresponding to each layer of the neural network is determined.

[0126] In the process of data processing by the layers of a neural network, such as Figure 6 As shown, during certain time periods, data ingress and data egress processes occur simultaneously. This means that the caching requirements for each layer include caching requirements for one or more input data items and one or more output data items. Based on the caching requirements of a particular layer and the capacity of the on-chip memory, the corresponding data segmentation size, i.e., the corresponding batch size, can be determined.

[0127] Optionally, leveraging the high speed of on-chip memory scheduling, firstly, for each layer in the neural network, all input and output data of that layer are stored in on-chip memory to reduce data transfer time. Without considering layer splitting (i.e., without considering overlap and gather / scatter problems), the maximum batch size for each layer of the neural network is determined based on one or more input and output data points and the capacity of the on-chip memory. When a layer of the neural network processes its maximum batch size, the resulting cache requirement is less than or equal to the on-chip memory capacity. This maximum batch size is then used as the maximum amount of data that the current layer can process, serving as the base batch size for subsequent batch size adjustments.

[0128] For example, assuming the on-chip memory capacity is 100, and the cache requirement for processing one image by L0 is 105, then the basic batch size for L0 is determined to be 1 / 2 images. The cache requirement for processing one image by L1 is 50, then the basic batch size for L1 is determined to be 2 images.

[0129] After adjusting the data storage location based on the base batch size, the first batch size is obtained. For example, such as... Figure 8 As shown, each input and output data corresponding to each layer in a neural network can be selectively stored in on-chip memory or off-chip memory. Therefore, data corresponding to certain layers in a neural network with large cache requirements and small basic batch sizes can be stored in off-chip memory; that is, at least one input data or at least one output data of at least one layer in the neural network can be stored in off-chip memory. After adjusting the storage location, the basic batch size corresponding to this at least one layer can be increased to ensure that more data can be processed at a time, thereby reducing the proportion of head and tail overhead and reducing the number of interactions with storage space.

[0130] Based on the computational overhead of neural networks, this paper addresses the issue of minimizing computational overhead by adjusting the storage locations of one or more input data and / or one or more output data of at least one layer in a neural network, as well as the base batch size. This involves determining the storage locations of one or more input data and one or more output data of each layer in the neural network, and identifying the initial batch size corresponding to the current layer. This aims to solve the problem in existing technologies where limited on-chip memory capacity leads to small maximum batch sizes for layers with high cache requirements. A small maximum batch size results in a large proportion of head and tail overhead in a single data processing operation, while the actual computational overhead is relatively small. Furthermore, a small maximum batch size requires these layers to perform multiple computations to process the current dataset, increasing the number of interactions and thus increasing the overall actual overhead of the neural network.

[0131] For example, a certain layer in a neural network has a large caching requirement. Limited by the capacity of on-chip memory, the maximum batch size for this layer is 1 / 10 of an image, and this maximum batch size is used as the base batch size for this layer. In this case, the dataset contains 32 images. If the current base batch size is used to process this dataset, the number of interactions is large, resulting in high computational overhead for the neural network. Therefore, the storage location of at least one input data or at least one output data corresponding to this layer is adjusted to increase the batch size and reduce computational overhead. For example, based on computational overhead, after adjusting some data storage locations, the first batch size for this layer is determined to be 2 images.

[0132] Thus, based on the on-chip memory capacity and the computational overhead of the neural network, the storage locations of the input and output data for each layer in the neural network, as well as the corresponding initial size, are determined. By adjusting the storage of some neural network data to off-chip memory, instead of storing all input and / or output data to either on-chip or off-chip memory, the initial size is optimized from the perspective of the overall neural network performance. This reduces the number of data interactions during computation in layers with high cache requirements, while ensuring improved on-chip memory utilization, thereby improving the computational performance of the neural network.

[0133] S702. Based on the first batch size, the neural network is divided into neural networks containing at least one first layer group.

[0134] Each first layer group contains at least one layer of the neural network, the first batch size of each layer in each first layer group is the same, and the cache requirement of each first layer group is less than or equal to the capacity of the on-chip memory.

[0135] Optionally, starting from the first layer of the neural network, based on the initial batch size and the capacity of the on-chip memory, a partitioning method is determined to divide the neural network into groups containing the first layer. For example, if the batch size corresponding to L2 is determined to be the same as the initial batch size corresponding to L1, then it is further determined whether the cache requirement exceeds the capacity of the on-chip memory when L1 and L2 are partitioned into a single group. If the cache requirement does not exceed the capacity of the on-chip memory, then L1 and L2 are partitioned into a single group. Next, it is determined whether the batch size corresponding to L3 is the same as the initial batch size corresponding to L1 and L2. If the initial batch size corresponding to L3 is the same as the initial batch size corresponding to L1 and L2, then it is further determined whether the cache requirement exceeds the capacity of the on-chip memory when L1, L2, and L3 are partitioned into a single group. If the initial batch size corresponding to L3 is not the same as the initial batch size corresponding to L1 and L2, then L1 and L2 are partitioned into a single group, and the next group of first layers is partitioned starting from L3. If scheduling L1 and L2 as a single unit results in cache requirements exceeding the on-chip memory capacity, then L1 is divided into a first-layer group, and the process continues from L2 to the next first-layer group. This process is repeated until all layers in the neural network are traversed, resulting in a neural network containing at least one first-layer group.

[0136] Optionally, first, identify the first batch of contiguous layers with the same size in the neural network, and use these layers as segmentation units to perform the segmentation steps described above. Then, according to the scheduling order of the layers in the neural network, start segmenting the layer group from the first layer in the segmentation unit. For example, suppose the neural network contains 10 layers, L0-L3 correspond to the same first batch size, L4 corresponds to a separate first batch size, and L5-L9 correspond to the same first batch size. Then, based on the first batch size, the neural network contains 3 segmentation units: L0-L3, L4, and L5-L9. Next, according to the scheduling order of the layers in the neural network, select segmentation units L0-L3, and start segmenting the current segmentation unit from L0.

[0137] (L0-L3), the specific segmentation steps are described above and will not be repeated here. After segmenting the layers in L0-L3 into the first layer group, the next segmentation unit is continued according to the scheduling order of the layers in the neural network, that is, the segmentation unit corresponding to L4 is segmented. Since this segmentation unit contains only one layer, L4 is directly segmented into the first layer group containing one layer. Then, the segmentation unit consisting of L5-L9 is segmented. After sequentially traversing all segmentation units, the neural network is segmented into a neural network containing at least one first layer group. Of course, all segmentation units can also be processed in other orders to segment the neural network into a neural network containing at least one first layer group.

[0138] like Figure 9As shown, step S702 can be specifically implemented as steps S901-S906.

[0139] S901. Starting from the i-th layer of the neural network, traverse all layers sequentially.

[0140] S902. Determine whether to traverse all layers. If not, proceed to step S903. If yes, complete the process of dividing the entire neural network into neural networks containing at least one first layer group.

[0141] S903. Determine whether the cache requirements corresponding to layers i to j are greater than the on-chip memory capacity, and determine the first layer group for this partitioning. If the cache requirements corresponding to layers i to j are not greater than the on-chip memory capacity, proceed to step S904. If the cache requirements corresponding to layers i to j are greater than the on-chip memory capacity, proceed to step S905.

[0142] In this neural network, the first batch of layers from the i-th to the j-th layer has the same size, where i and j are positive integers.

[0143] Optionally, steps S901-S903 are specifically implemented as follows: starting from the first layer of the neural network, the neural network is segmented, that is, starting from i=1. All layers of the neural network are sequentially traversed, and consecutive layers with the same initial size are segmented into a single unit. During the segmentation process, it is determined whether the corresponding cache requirement exceeds the on-chip memory capacity. Consecutive layers are considered adjacent layers, and the operations performed on these layers can be the same or different. Since the neural network uses operators within layers to process data, the input data of each layer is the output data of the previous layer, and the subsequent layer must perform operations based on the output data of the previous layer. In other words, data transmission between layers in the neural network is data-dependent. Therefore, the segmentation of layer groups can only be performed on adjacent layers, and the scheduling order of these subsequent layers is consistent with the scheduling order of these layers in the neural network.

[0144] For example, such as Figure 4 As shown, assuming the first four layers of the neural network (L0-L3) each have an initial size of 1 / 4 of an image, the kernel size of L0-L3 is 3*3, and the stride of the kernel is 1. Due to the overlap problem, for each additional layer added to the current segment, the input data of the layers preceding that layer in that segment needs to be padded with two rows of data. Now, starting from the first layer L0 of the neural network, when L0 and L1 are segmented into a single unit, the input data of L0 needs to be padded with two rows of data. It is then determined whether the current cache requirement exceeds the capacity of the on-chip memory. That is, whether the cache requirement after adding padded data to L0 exceeds the capacity of the on-chip memory. If it does not exceed the capacity, step S904 is executed. If it does exceed the capacity, step S905 is executed.

[0145] S904. Divide the i-th layer to the j-th layer into a whole.

[0146] Optionally, after splitting layer j into the current split (i.e., including layer (j-1), if the cache requirement for scheduling layers i to j as a whole does not exceed the on-chip memory capacity, then steps S902-S904 are executed repeatedly to continue the current splitting process until the cache requirement exceeds the on-chip memory capacity. Then, step S905 is executed to determine the first layer group of this split. In other words, layer (j+1) is continued to be split into the current split, and it is continued to determine whether the cache requirement for scheduling layers i to (j+1) as a whole exceeds the on-chip memory capacity.

[0147] For example, such as Figure 4 As shown in the example of step S903, it is assumed that L0-L1 is divided into a whole. After that, after L2 is divided into the whole corresponding to the current division (i.e., the whole into which L0-L1 is divided), the cache requirement will not exceed the capacity of the on-chip memory. Therefore, L0-L2 is temporarily divided into a whole. That is, this division is to divide L0-L2 into a whole. Then, the process returns to step S902 to determine whether to traverse all layers. Figure 4 If the L3 of the neural network shown has not yet been traversed, then step S903 is executed to continue splitting L3 into the whole segment corresponding to the current segment, and to determine whether the cache requirement will exceed the capacity of the on-chip memory.

[0148] S905. Based on the computational overhead of the neural network, the i-th layer to the (i+m)-th layer is determined as a first layer group, (i+m)≤(j-1).

[0149] Optionally, if the cache requirement for scheduling layers i to j in the neural network as a whole is greater than the on-chip memory capacity, and the cache requirement for scheduling layers i to (j-1) as a whole is less than or equal to the on-chip memory capacity, then based on the computational overhead of the neural network, layers i to (i+m) are determined as a first layer group. Here, m is a positive integer, and (i+m) ≤ (j-1).

[0150] Specifically, we obtain the computational overhead when scheduling layers i through t as a whole, where layer t is any one of layers (i+1) through (j-1). t is a positive integer, (i+1) ≤ t ≤ (j-1). The computational overhead of the neural network is minimized when scheduling layers i through (i+m) as a whole.

[0151] For example, such as Figure 4As shown in the example of step S904, L0-L2 have been divided into a whole for scheduling, and the cache requirement is less than or equal to the on-chip memory capacity. Suppose that L3 is divided into the whole formed by L0-L2, i.e., L0-L3 is divided into a whole, the current cache requirement exceeds the on-chip memory capacity due to the use of a padding algorithm. Therefore, L3 cannot be divided into the whole formed by L0-L2 to avoid the amount of data during the operation exceeding the on-chip memory capacity. Furthermore, based on the computational overhead of neural networks,

[0152] Adjust the layer splitting method in L0-L2. During this adjustment, reduce the computational overhead of the neural network. For example, for t=1 and t=2, obtain the corresponding computational overhead. If the computational overhead of the neural network is less when L0-L1 is determined as the first layer group than when L0-L2 is determined as the first layer group, then L0-L1 is determined as a first layer group. That is, when m=1, the first and second layers of the neural network are determined as a first layer group.

[0153] S906. Determine the next first-level group starting from the (i+m+1)th level.

[0154] For example, such as Figure 4 As shown in the example of step S905, after determining L0-L1 as a first layer group, the process continues to divide the network starting from L2, determining the next first layer group, until all layers in the neural network have been traversed, at which point the division of the first layer groups stops. That is, after executing step S906, the process returns to step S902 to confirm whether all layers have been traversed. Traversing all layers includes iterative traversal of some layers. For example, in the example of step S904 above, L2 and L3 have already been traversed. In the example of step S905 above, Figure 4 The neural network shown in the figure, after determining L0-L1 as a first layer group, needs to continue to divide from L2 to determine the next first layer group, that is, to repeatedly traverse L2 and L3.

[0155] S703. Based on the segmentation results of the first layer group, the neural network is segmented into a neural network containing at least one second layer group.

[0156] Each second layer group contains one or more first layer groups in the neural network, and the cache requirement of each second layer group is less than or equal to the capacity of the on-chip memory. At least one second layer group contains at least two first layer groups with different initial sizes.

[0157] Optionally, adjacent first-layer groups whose initial size is multiples of each other can be divided into a second-layer group. For example, if the initial size of the first first-layer group is 2 images, and the initial size of the second first-layer group is 8 images, and the second first-layer group is adjacent to the first first-layer group, and its initial size is 3 times that of the first first-layer group, then if the cache requirement of the first and second first-layer groups as a whole is less than or equal to the on-chip memory capacity, these two first-layer groups can be divided into a second-layer group.

[0158] Optionally, starting from the first first-layer group contained in the neural network, the first-layer groups in the neural network are traversed sequentially to divide the neural network into neural networks containing at least one second-layer group. For example, after dividing the first and second first-layer groups into a whole, it is determined whether the cache requirement of the current whole division exceeds the on-chip memory capacity. If the cache requirement of the whole division formed by the first and second first-layer groups exceeds the on-chip memory capacity, the first first-layer group is divided into a second-layer group, and the division continues from the second first-layer group to the next second-layer group. If the cache requirement of the current whole division formed by the first and second first-layer groups does not exceed the on-chip memory capacity, the first and second first-layer groups are divided into a second-layer group. Then, it is determined whether the cache requirement of the current whole division exceeds the on-chip memory capacity after dividing the third first-layer group into the whole formed by the first and second first-layer groups. Thus, by repeating the above steps and traversing all the first-layer groups contained in the neural network, the neural network is divided into neural networks containing at least one second-layer group.

[0159] like Figure 10 As shown, step S703 can be specifically implemented as steps S1001-S1005.

[0160] S1001. Starting from the a-th first layer group of the neural network, traverse all first layer groups.

[0161] S1002. Determine whether to traverse all first-layer groups. If not, proceed to step S1003. If yes, complete the process of dividing the entire neural network into neural networks containing at least one second-layer group.

[0162] S1003. Determine whether the cache requirement when scheduling the a-th to b-th first-level groups as a whole is greater than the on-chip memory capacity, and determine the second-level group for this split. If the cache requirement when scheduling the a-th to b-th first-level groups as a whole is not greater than the on-chip memory capacity, proceed to step S1004. If the cache requirement when scheduling the a-th to b-th first-level groups as a whole is greater than the on-chip memory capacity, proceed to step S1005.

[0163] In this neural network, the first layer group from the a-th to the b-th layer group are consecutive first layer groups, and at least two of the first layer groups from the a-th to the b-th layer group have different first layer sizes, where a and b are positive integers. Because the data processed by the layers of the neural network has inter-layer data dependencies, the splitting of the second layer group can only be done on adjacent first layer groups.

[0164] Optionally, the specific implementation of steps S1001-S1003 above is as follows: starting from the first layer group of the neural network, the neural network is segmented, that is, starting from a=1. All first layer groups in the neural network are sequentially traversed to segment into second layer groups, and the cache requirement corresponding to the second layer group is less than or equal to the capacity of the on-chip memory. Here, based on the data dependency of data transmission between layers in the neural network, the first layer groups segmented into a second layer group are consecutive first layer groups.

[0165] For example, such as Figure 5 As shown, assume that L0 and L1 in the neural network form the first layer group, with an initial size of 2 images. L2 and L3 form the second layer group, with an initial size of 4 images. L4 and L5 form the third layer group, with an initial size of 2 images. These three layer groups are consecutive. The second layer group is divided starting from the first layer group. If the first and second layer groups are divided into a single unit, during the scheduling of neural network operations, the gather problem will cause L0 and L1 to generate additional on-chip memory cache requirements. If the second and third layer groups are divided into a single unit, during the scheduling of neural network operations, the scatter problem will cause L4 and L5 to generate additional on-chip memory cache requirements. It is necessary to determine whether the cache requirements of the second layer group exceed the on-chip memory capacity after adding additional on-chip memory cache requirements. If not, proceed to step S1004. If they exceed, proceed to step S1005.

[0166] S1004. Divide the first layer group from the ath layer group to the bth layer group into a whole.

[0167] Optionally, after splitting the b-th first-level group into the corresponding split whole (i.e., including the (b-1)-th first-level group as a second-level group), if the cache requirement when scheduling the a-th to b-th first-level groups as a whole does not exceed the on-chip memory capacity, then steps S1002-S1004 are executed repeatedly to continue the splitting of the second-level group until the cache requirement exceeds the on-chip memory capacity. Then, step S1005 is executed to determine the second-level group of the current split and to start determining the next second-level group. In other words, the (b+1)-th first-level group is continued to be split into the current split whole composed of the a-th to b-th first-level groups, and the cache requirement is continued to be determined to see if it exceeds the on-chip memory capacity.

[0168] For example, such as Figure 5 As shown, based on the example of step S1003, after dividing the first first layer group (L0 and L1) and the second first layer group (L2 and L3) into a whole, the cache requirement of the current whole division will not exceed the capacity of the on-chip memory. Then, after dividing L0-L3 into a whole, continue the division of the second layer group, that is, return to step S1002 to determine whether to traverse all first layer groups. Figure 5 In the neural network shown, the third first layer group (L4 and L5) has not yet been traversed. Continue to divide the third first layer group into the whole corresponding to the current division L0-L3, and determine whether the cache requirement of the current division will exceed the capacity of the on-chip memory.

[0169] S1005. Based on the computational overhead of the neural network, the first layer group from the a-th to the b-th first layer group is determined as a second layer group, and the next second layer group is determined starting from the (b+1)-th first layer group. Alternatively, the first layer group from the a-th to the (b-1)-th first layer group is determined as a second layer group, and the next second layer group is determined starting from the b-th first layer group.

[0170] Optionally, if the cache requirement for scheduling the first layer groups a to b in the neural network as a whole is greater than the on-chip memory capacity, and the cache requirement for scheduling the first layer groups a to (b-1) in the neural network as a whole is less than or equal to the on-chip memory capacity, then, based on the computational overhead of the neural network, the first layer groups a to b are determined as a second layer group. Alternatively, the first layer groups a to (b-1) are determined as a second layer group. If the first layer groups a to b are determined as a second layer group, then the initial batch size corresponding to the first layer group b or the first layer group (b-1) is reduced.

[0171] Specifically, when scheduling the first layer group from the a-th to the b-th first layer group as a whole, the computational cost of the neural network is the first computational cost. When scheduling the first layer group from the a-th to the (b-1)-th first layer group as a whole, the computational cost of the neural network is the second computational cost. If the first computational cost is less than the second computational cost, then the first layer group from the a-th to the b-th first layer group is determined as a second layer group. If the second cost is less than the first computational cost, then the first layer group from the a-th to the (b-1)-th first layer group is determined as a second layer group.

[0172] For example, such as Figure 5 As shown in the example of step S1004, the first first-layer group (L0 and L1) and the second first-layer group (L2 and L3) have been divided into a second-layer group, and the cache requirement is less than or equal to the on-chip memory capacity. Suppose that the third first-layer group (L4 and L5) is divided into the second-layer group formed by L0-L3. Due to the scatter problem, L4 and L5 generate additional on-chip memory cache requirements, resulting in the overall cache requirement of the current division exceeding the on-chip memory capacity. Therefore, after dividing the first and second first-layer groups into a second-layer group, the division of the second-layer group continues from the third first-layer group, traversing all first-layer groups, but not dividing the third first-layer group into the current second-layer group.

[0173] For example, suppose the first batch size corresponding to the first layer group is 2 images, and the first batch size corresponding to the second layer group is 6 images. If the two layer groups are split into one, then the first layer group needs to be scheduled for 3 operations before the second layer group needs to be scheduled for 1 operation. Due to the gather problem, the layers in the first layer group generate additional on-chip memory cache requirements, causing the cache requirements of the second layer group to exceed the on-chip memory capacity. Therefore, the first batch size corresponding to the second layer group can be reduced, such as to 4 images. The additional on-chip memory cache requirements of the corresponding layers in the first layer group will be reduced accordingly. The two layer groups can then be split into one second layer group, and the cache requirements of the second layer group will be less than or equal to the on-chip memory capacity.

[0174] Next, determine the next second-level group, return to step S1002, and confirm whether all first-level groups have been traversed.

[0175] Optionally, the loop traversal can be performed on all first-level groups, including some first-level groups. For example, in step S1005 above, after determining the first-level group from the a-th to the (b-1)-th first-level group as a second-level group, the next second-level group is determined starting from the b-th first-level group, that is, the b-th first-level group is repeatedly traversed.

[0176] Through steps S1001-S1005, the segmentation of the second layer groups in the neural network is stopped after traversing all the first layer groups. The neural network is segmented into a neural network composed of at least one second layer group. At least one input data or output data of the layer contained in the second layer group is stored in on-chip memory, and the input data of the first layer and the output data of the last layer in the second layer group are stored in off-chip memory.

[0177] S704. Based on the segmentation results of the second layer group, process the input data of the neural network.

[0178] Optionally, the input data of the neural network is the input data of the first second layer group, the output data of the second layer group is the input data of the next second layer group, and the output data of the last second layer group is the output data of the neural network. During the processing of the neural network's input data based on the segmentation results of the second layer groups, the scheduling order of the layers in the second layer group is determined according to the scheduling order of each first layer group contained within the second layer group, and the scheduling order of the layers in the first layer group.

[0179] The scheduling order of the layers in the first layer group is the same as the scheduling order of the layers in the neural network before splitting. The scheduling order of each first layer group in the second layer group is determined according to the first batch size and the scheduling order of the first and last layers in the first layer group.

[0180] For example, such as Figure 11 As shown, the neural network contains six convolutional layers L1-L6. The scheduling order before splitting is L1→L2→L3→L4→L5→L6. L1-L3 form a first-layer group, with an initial batch size of 2 images; L4-L6 form another first-layer group, with an initial batch size of 4 images. These two first-layer groups are consecutive, and their initial batch sizes are doubled. After splitting these two first-layer groups into a second-layer group, the cache requirement of the second-layer group is less than or equal to the on-chip memory capacity. Therefore, based on the splitting result of the second-layer group, the input data of the neural network is processed by inputting the input data into L1, such as... Figure 11 As shown, the input data are A0 and B0, corresponding to two images in the first batch. The scheduling order of the first layer group (L1-L3) is L1→L2→L3, and the scheduling order of the second layer group (L4-L6) is L4→L5→L6. Based on the first batch size, it is determined that the first layer group needs to be scheduled twice before the second layer group needs to be scheduled once, and L3 is scheduled before L4 in the neural network before segmentation. Therefore, as... Figure 11 As shown, the scheduling order of the layers in the split neural network is L1→L2→L3→L1→L2→L3→L4→L5→L6.

[0181] Therefore, the neural network scheduling method of this application first determines the batch size of each layer in the neural network based on the capacity of the on-chip memory, and then merges layers with the same batch size into a first layer group. Then, based on the cache requirements of the first layer group and the capacity of the on-chip memory, multiple first layer groups are merged into a second layer group. The resulting second layer group contains first layer groups with different batch sizes. In other words, when scheduling the neural network in units of second layer groups, input data is processed using different batch sizes. Thus, the cache requirements of each second layer group will not exceed the capacity of the on-chip memory, while improving the utilization rate of the on-chip memory and enhancing hardware performance.

[0182] The neural network scheduling method provided in this application embodiment will be described below with reference to specific examples.

[0183] Example 1: Input data is whole image data.

[0184] like Figure 12 As shown, based on the on-chip memory capacity and considering the overall performance of the neural network, the batch size for L0 and L1 is determined to be 1 image, the batch size for L2, L3, and L4 is 2 images, and the batch size for L5 and L6 is 4 images. Using the neural network scheduling method provided in this embodiment, L0 and L1 are divided into a first layer group, L2-L4 are divided into a second layer group, and L5 and L6 are divided into a third layer group. For the gather problem, based on the on-chip memory capacity and considering the overall performance of the neural network, the three first layer groups are divided into a second layer group, that is, L0-L6 are divided into a second layer group. The cache requirement of the second layer group is less than or equal to the on-chip memory capacity. The second layer group contains layers with different batch sizes. In the process of scheduling the layer groups in the neural network to process input data, the utilization rate of on-chip memory can be improved, and the hardware operating performance can be enhanced.

[0185] like Figure 12 As shown, assuming the dataset contains 8 images, L0 is the first layer of the second group, and the batch size is 1 image, the dataset is divided into 8 batches of input data. Figure 12 The batches shown (0-7) each consist of the complete image data corresponding to one image, and are input in batches (L0). Figure 12 As shown, during the processing of the input data of the current dataset, the first layer 1 group is scheduled twice, corresponding to the scheduling of the second layer 1 group once, i.e., the scheduling order is L0→L1→L0→L1→L2→L3→L4; the second layer 1 group is scheduled twice, corresponding to the scheduling of the third layer 1 group once, i.e., the scheduling order is L2→L3→L4→L2→L3→L4→L5→L6. Processing the input data of the current dataset requires scheduling the first layer 1 group 8 times, the second layer 1 group 4 times, and the third layer 1 group 2 times.

[0186] Example 2: The input data is not a complete image.

[0187] like Figure 13 As shown, based on the on-chip memory capacity and considering the overall performance of the neural network, the batch size corresponding to L0 and L1 is determined to be 1 / 4 of an image, and the batch size corresponding to L2, L3, and L4 is 1 / 2 of an image. Using the neural network scheduling method provided in this application embodiment, L0 and L1 are divided into the first layer group, and L2-L4 are divided into the second layer group. Regarding the overlap problem, as... Figure 13 As shown, the input data is not a complete image and requires a padding algorithm to process it, with the padding being the shaded areas. Considering the overall performance of the neural network and the on-chip memory capacity, the two first-layer groups are divided into a second-layer group, specifically L0-L4. The cache requirement of the second-layer group is less than or equal to the on-chip memory capacity. The second-layer group contains layers with different batch sizes. This improves the utilization of on-chip memory and enhances hardware performance during the scheduling of layer groups in the neural network to process input data.

[0188] like Figure 13 As shown, assuming the dataset contains 2 images, L0 is the first layer of the second group, and the batch size is 1 / 4 of an image, the dataset is divided into 8 batches of input data. Figure 13 The batches shown (0-7) each consist of non-full image data corresponding to 1 / 4 of an image, input in batches (L0). Figure 13 As shown, during the processing of the input data of the current dataset, the first layer 1 group is scheduled twice, corresponding to the scheduling of the second layer 1 group once, that is, the scheduling order is L0→L1→L0→L1→L2→L3→L4. Processing the input data of the current dataset requires scheduling the first layer 1 group eight times and the second layer 1 group four times.

[0189] The above combination Figure 7 , Figure 9 and Figure 10 This application provides a detailed description of the neural network scheduling method provided in its embodiments. The following is in conjunction with... Figure 14 This application provides a detailed description of the neural network scheduling device provided in its embodiments.

[0190] In one possible design, Figure 14 This is a schematic diagram of the structure of the neural network scheduling device provided in an embodiment of this application. Figure 14As shown, the neural network scheduling device 1400 includes: a determining unit 1401, a segmentation unit 1402, and a scheduling unit 1403. The neural network scheduling device 1400 can be used to implement the functions of the device involved in the above method embodiments. The neural network scheduling device 1400 can be the device itself, a functional unit or chip within the device, or a device used in conjunction with a communication device.

[0191] Optionally, the determining unit 1401 is used to determine the first batch size corresponding to each layer in the neural network.

[0192] Optionally, the segmentation unit 1402 is used to segment the neural network into neural networks containing at least one first layer group based on the first batch size.

[0193] Each first layer group contains at least one layer of the neural network, the first batch size of each layer in each first layer group is the same, and the cache requirement of each first layer group is less than or equal to the capacity of the on-chip memory.

[0194] Optionally, the segmentation unit 1402 is also used to segment the neural network into a neural network containing at least one second layer group based on the segmentation result of the first layer group.

[0195] Each second-layer group contains at least one first-layer group, and the cache requirement of each second-layer group is less than or equal to the capacity of the on-chip memory; at least one second-layer group contains at least two first-layer groups with different initial sizes.

[0196] Optionally, the scheduling unit 1403 is used to schedule the neural network based on the segmentation results of the second layer group.

[0197] In one possible implementation, the determining unit 1401 is specifically used to determine the initial size corresponding to each layer of the neural network based on the cache requirements of each layer and the capacity of the on-chip memory.

[0198] In one possible implementation, determining unit 1401 is specifically used to determine the initial size corresponding to each layer of the neural network for one or more input data and one or more output data for each layer of the neural network, as well as the capacity of the on-chip memory.

[0199] In this neural network, at least one input data or at least one output data of at least one layer is stored in off-chip memory.

[0200] In one possible implementation, the determining unit 1401 is specifically used to adjust the storage location of one or more input data and / or one or more output data of at least one layer in the neural network based on the computational overhead of the neural network.

[0201] The storage location includes on-chip memory or off-chip memory.

[0202] Furthermore, during the process of adjusting the storage location, the storage location of one or more input data and one or more output data of each layer of the neural network is obtained when the computational overhead of the neural network is minimized.

[0203] Then, based on the storage locations of one or more input data and one or more output data for each layer of the neural network, as well as the capacity of the on-chip memory, the initial size corresponding to each layer of the neural network is determined.

[0204] In one possible implementation, the segmentation unit 1402 is specifically used to determine the i-th to (i+m)-th layers as a first layer group if the cache requirement when scheduling the i-th to j-th layers in the neural network as a whole is greater than the capacity of the on-chip memory, and the cache requirement when scheduling the i-th to (j-1)-th layers as a whole is less than or equal to the capacity of the on-chip memory, based on the computational overhead of the neural network.

[0205] In this neural network, the first batch of layers from the i-th to the j-th layer has the same size, i, j, and m are positive integers, and (i+m)≤(j-1).

[0206] In one possible implementation, the segmentation unit 1402 is specifically used to obtain multiple computational overheads corresponding to scheduling layers i to t as a whole, where layer t is any one of layers (i+1) to (j-1). Here, t is a positive integer, (i+1) ≤ t ≤ (j-1). The computational overhead of the neural network is minimized when scheduling layers i to (i+m) as a whole.

[0207] In one possible implementation, the segmentation unit 1402 is specifically used to determine the first layer group from the a-th to the b-th first layer group in the neural network as a second layer group if the cache requirement when scheduling the first layer group from the a-th to the b-th first layer group as a whole is greater than the capacity of the on-chip memory, and the cache requirement when scheduling the first layer group from the a-th to the (b-1)-th first layer group as a whole is less than or equal to the capacity of the on-chip memory, based on the computational overhead of the neural network. Alternatively, the first layer group from the a-th to the (b-1)-th first layer group can be determined as a second layer group.

[0208] In this neural network, at least two of the first layer groups from the a-th to the b-th first layer group have different first-layer sizes, where a and b are positive integers.

[0209] In one possible implementation, the segmentation unit 1402 is further configured to reduce the first batch size corresponding to the b-th first layer group or the (b-1)-th first layer group if the a-th first layer group to the b-th first layer group are determined to be a second layer group.

[0210] In one possible implementation, the segmentation unit 1402 is specifically used to schedule the first layer group to the b layer group as a whole, where the computational overhead of the neural network is the first computational overhead. When the first layer group to the (b-1)th layer group is scheduled as a whole, the computational overhead of the neural network is the second computational overhead.

[0211] If the overhead of the first operation is less than the overhead of the second operation, then the first layer group from the ath to the bth first layer group is determined as a second layer group.

[0212] If the second overhead is less than the first computational overhead, then the first layer group from the a-th to the (b-1)-th first layer group is determined as a second layer group.

[0213] In one possible implementation, during the scheduling of the neural network based on the segmentation results of the second layer group, the scheduling order of the layers in the second layer group is determined according to the scheduling order of each first layer group contained in the second layer group, and the scheduling order of the layers in the first layer group.

[0214] In one possible implementation, the scheduling order of the layers in the first layer group is the same as the scheduling order of the layers in the neural network before splitting, and the scheduling order of each first layer group contained in the second layer group is determined according to the first batch size and the scheduling order of the first and last layers in the first layer group.

[0215] In one possible implementation, the feature is that at least one input data or output data of a layer included in the second layer group is stored in on-chip memory, and the input data of the first layer and the output data of the last layer in the second layer group are stored in off-chip memory.

[0216] Optional, Figure 14 The neural network scheduling device 1400 shown may further include a receiving unit and a transmitting unit. Figure 14 (Not shown in the image), the receiving unit is used to receive signals sent by other communication devices, and the sending unit is used to send signals to other communication devices.

[0217] Optionally, Figure 14 The neural network scheduling device 1400 shown may also include a storage unit ( Figure 14 (not shown in the image), this storage unit stores a program or instruction. When the determining unit 1401, the splitting unit 1402, and the scheduling unit 1403 execute the program or instruction, it causes... Figure 14The neural network scheduling device 1400 shown can perform Figure 7 , Figure 9 and Figure 10 The neural network scheduling method shown.

[0218] Figure 14 The technical effects of the neural network scheduling device 1400 shown can be referenced. Figure 7 , Figure 9 and Figure 10 The technical effects of the neural network scheduling method shown will not be elaborated here.

[0219] The receiving unit and the transmitting unit can be collectively referred to as the transceiver unit, which can be implemented by a transceiver or transceiver-related circuit components, and can be a transceiver or a transceiver module. The operation and / or function of each unit in this neural network scheduling device 1400 are respectively for realizing... Figure 7 , Figure 9 and Figure 10 The corresponding process of the neural network scheduling method shown is not described in detail here for the sake of brevity.

[0220] When using integrated units, Figure 15 A schematic diagram illustrating another possible composition of the neural network scheduling device involved in the above embodiments is shown. For example... Figure 15 As shown, the neural network scheduling device 1500 may include a processing module 1501.

[0221] Processing module 1501 is used to perform the above. Figure 14 The steps performed by the determining unit 1401, the splitting unit 1402, and the scheduling unit 1403 shown, and the operation and / or function of the processing module 1501, are for the purpose of implementing... Figure 7 , Figure 9 and Figure 10 The corresponding process of the neural network scheduling method shown is not described in detail here for the sake of brevity.

[0222] The neural network scheduling device 1500 may also include a storage module for storing the program code and data of the neural network scheduling device. The storage module may be a memory.

[0223] The processing module 1501 may be a processor or a controller. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0224] This application also provides a chip system, such as... Figure 16As shown, the chip system includes at least one processor 1601 and at least one interface circuit 1602. The processor 1601 and the interface circuit 1602 are interconnected via lines. For example, the interface circuit 1602 can be used to receive signals from other devices. As another example, the interface circuit 1602 can be used to send signals to other devices (e.g., the processor 1601). Exemplarily, the interface circuit 1602 can read instructions stored in memory and send those instructions to the processor 1601. When the instructions are executed by the processor 1601, the neural network scheduling device can perform the various steps of the neural network scheduling method in the above embodiments. Of course, the chip system may also include other discrete devices, and this application embodiment does not specifically limit this.

[0225] This application also provides a chip system, including: a processor coupled to a memory, the memory being used to store programs or instructions, wherein when the program or instructions are executed by the processor, the chip system implements the methods in any of the above method embodiments.

[0226] Optionally, the chip system may contain one or more processors. These processors can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor, implemented by reading software code stored in memory.

[0227] Optionally, the chip system may contain one or more memories. The memory may be integrated with the processor or disposed separately from it; this application does not limit this. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or disposed separately on different chips. This application does not specifically limit the type of memory or the arrangement of the memory and processor.

[0228] For example, the chip system may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a micro controller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0229] It should be understood that each step in the above method embodiments can be completed by integrated logic circuits in the processor hardware or by instructions in software form. The method steps disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.

[0230] Embodiments of this application also provide a storage medium for storing instructions used by the aforementioned communication device.

[0231] This application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are run on a server, the server executes the aforementioned method steps to implement the neural network scheduling method in the above embodiments.

[0232] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the neural network scheduling method described in the above embodiments.

[0233] Furthermore, embodiments of this application also provide an apparatus, which may specifically be a component or module. The apparatus may include one or more interconnected processors and a memory; wherein the memory is used to store computer programs, and the one or more computer programs include instructions. When the instructions are executed by one or more processors, the apparatus performs the neural network scheduling method in the above-described method embodiments.

[0234] In this application, the apparatus, computer-readable storage medium, computer program product or chip provided in the embodiments are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0235] The steps of the methods or algorithms described in this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an application-specific integrated circuit (ASIC).

[0236] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0237] In the several embodiments provided in this application, it should be understood that the disclosed methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of modules or units may be electrical, mechanical, or other forms.

[0238] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0239] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0240] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0241] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A neural network scheduling method, characterized in that, The method includes: Based on the caching requirements of each layer in the neural network and the capacity of the on-chip memory, the initial size corresponding to each layer of the neural network is determined; wherein, at least one input data or at least one output data of at least one layer in the neural network is stored in off-chip memory; Based on the first batch size, the neural network is divided into neural networks containing at least one first layer group; wherein each first layer group contains at least one layer of the neural network, the first batch size corresponding to the layers in each first layer group is the same, and the cache requirement of each first layer group is less than or equal to the capacity of the on-chip memory; Based on the segmentation results of the first layer group, the neural network is segmented into a neural network containing at least one second layer group; wherein, each second layer group contains at least one first layer group, and the cache requirement of each second layer group is less than or equal to the capacity of the on-chip memory; at least one second layer group contains at least two first layer groups with different initial sizes; Based on the segmentation results of the second layer group, the neural network is scheduled; Wherein, the step of dividing the neural network into a neural network containing at least one second layer group based on the segmentation result of the first layer group includes: If the cache requirement when scheduling the first layer group from the a-th to the b-th layer group in the neural network as a whole is greater than the capacity of the on-chip memory, and the cache requirement when scheduling the first layer group from the a-th to the (b-1)-th layer group as a whole is less than or equal to the capacity of the on-chip memory, then based on the computational overhead of the neural network, the first layer group from the a-th to the b-th layer group is determined as a second layer group; or, the first layer group from the a-th to the (b-1)-th layer group is determined as a second layer group; wherein, at least two of the first layer groups from the a-th to the b-th layer group in the neural network have different first batch sizes, and a and b are positive integers.

2. The neural network scheduling method according to claim 1, characterized in that, The determination of the initial size for each layer of the neural network, based on the cache requirements of each layer and the capacity of the on-chip memory, includes: For each layer of the neural network, the first batch size corresponding to each layer is determined based on one or more input data and one or more output data, as well as the capacity of the on-chip memory.

3. The neural network scheduling method according to claim 2, characterized in that, For each layer of the neural network, given one or more input data points and one or more output data points, and the capacity of the on-chip memory, determine the initial size corresponding to each layer of the neural network, including: Based on the computational overhead of the neural network, the storage location of one or more input data and / or one or more output data of at least one layer in the neural network is adjusted, wherein the storage location includes the on-chip memory or the off-chip memory; During the process of adjusting the storage location, the storage location of one or more input data and one or more output data of each layer of the neural network is obtained when the computational overhead of the neural network is minimized. Based on the storage locations of one or more input data and one or more output data of each layer of the neural network, and the capacity of the on-chip memory, the first batch size corresponding to each layer of the neural network is determined.

4. The neural network scheduling method according to any one of claims 1-3, characterized in that, The step of dividing the neural network into neural networks containing at least one first layer group based on the first batch size includes: If the cache requirement when scheduling layers i to j in the neural network as a whole is greater than the capacity of the on-chip memory, and the cache requirement when scheduling layers i to (j-1) as a whole is less than or equal to the capacity of the on-chip memory, then based on the computational overhead of the neural network, layers i to (i+m) are determined as a first layer group; wherein, the first batch size of layers i to j in the neural network is the same, i, j, m are positive integers, and (i+m) ≤ (j-1).

5. The neural network scheduling method according to claim 4, characterized in that, The computational overhead based on the neural network, which defines layers i to (i+m) as a first layer group, includes: Obtain multiple computational overheads when scheduling layers i to t as a whole, where layer t is any layer from layer (i+1) to layer (j-1); t is a positive integer, (i+1)≤t≤(j-1). The computational overhead of the neural network is minimized when the i-th to (i+m)-th layers are scheduled as a whole.

6. The neural network scheduling method according to claim 1, characterized in that, The method further includes: If the first layer group from the ath to the bth first layer group is determined as a second layer group, then the first batch size corresponding to the bth first layer group or the (b-1)th first layer group is reduced.

7. The neural network scheduling method according to claim 1 or 6, characterized in that, The computational overhead based on the neural network is used to determine the a-th first layer group to the b-th first layer group as a second layer group; or, to determine the a-th first layer group to the (b-1)-th first layer group as a second layer group; including: When the first layer group a to the first layer group b are scheduled as a whole, the computational overhead of the neural network is the first computational overhead; when the first layer group a to the (b-1)th first layer group are scheduled as a whole, the computational overhead of the neural network is the second computational overhead. If the first computational overhead is less than the second computational overhead, then the a-th first layer group to the b-th first layer group are determined as a second layer group; if the second computational overhead is less than the first computational overhead, then the a-th first layer group to the (b-1)-th first layer group are determined as a second layer group.

8. The neural network scheduling method according to any one of claims 1-3, characterized in that, In the process of scheduling the neural network based on the segmentation results of the second layer group, the scheduling order of the layers in the second layer group is determined according to the scheduling order of each of the first layer groups contained in the second layer group, and the scheduling order of the layers in the first layer group.

9. The neural network scheduling method according to claim 8, characterized in that, The scheduling order of the layers in the first layer group is the same as the scheduling order of the layers in the neural network before the split. The scheduling order of each of the first layer groups included in the second layer group is determined according to the first batch size and the scheduling order of the first and last layers in the first layer group.

10. The neural network scheduling method according to any one of claims 1-3, characterized in that, At least one input data or output data of a layer included in the second layer group is stored in the on-chip memory, and the input data of the first layer and the output data of the last layer in the second layer group are stored in the off-chip memory.

11. A neural network scheduling device, characterized in that, The device includes: a determination unit, a segmentation unit, and a scheduling unit; The determining unit is used to determine the first batch size corresponding to each layer of the neural network based on the cache requirements of each layer and the capacity of the on-chip memory; wherein at least one input data or at least one output data of at least one layer in the neural network is stored in off-chip memory; The segmentation unit is used to segment the neural network into neural networks containing at least one first layer group based on the first batch size; wherein each first layer group contains at least one layer of the neural network, the first batch size corresponding to the layers in each first layer group is the same, and the cache requirement of each first layer group is less than or equal to the capacity of the on-chip memory; The segmentation unit is further configured to segment the neural network into a neural network containing at least one second layer group based on the segmentation result of the first layer group; wherein each second layer group contains at least one first layer group, and the cache requirement of each second layer group is less than or equal to the capacity of the on-chip memory; at least one second layer group contains at least two first layer groups with different initial sizes; The scheduling unit is used to schedule the neural network based on the segmentation result of the second layer group; Specifically, the segmentation unit is used to determine the first layer group from the ath to the bth first layer group in the neural network as a second layer group if the cache requirement when scheduling the first layer group from the ath to the (b-1)th first layer group as a whole is greater than the capacity of the on-chip memory, and the cache requirement when scheduling the first layer group from the ath to the (b-1)th first layer group as a whole is less than or equal to the capacity of the on-chip memory; or, determine the first layer group from the ath to the (b-1)th first layer group as a second layer group; wherein, at least two of the first layer groups from the ath to the bth first layer group in the neural network have different first batch sizes, and a and b are positive integers.

12. The neural network scheduling device according to claim 11, characterized in that, The determining unit is specifically used to determine the first batch size corresponding to each layer of the neural network based on one or more input data and one or more output data of each layer in the neural network, as well as the capacity of the on-chip memory.

13. The neural network scheduling device according to claim 12, characterized in that, The determining unit is specifically used to adjust the storage location of one or more input data and / or one or more output data of at least one layer in the neural network based on the computational overhead of the neural network, wherein the storage location includes the on-chip memory or the off-chip memory; During the process of adjusting the storage location, the storage location of one or more input data and one or more output data of each layer of the neural network is obtained when the computational overhead of the neural network is minimized; based on the storage location of one or more input data and one or more output data of each layer of the neural network, and the capacity of the on-chip memory, the first batch size corresponding to each layer of the neural network is determined.

14. The neural network scheduling apparatus according to any one of claims 11-13, characterized in that, The segmentation unit is specifically used to determine layers i to j as a first layer group if the cache requirement when scheduling layers i to j in the neural network as a whole is greater than the capacity of the on-chip memory, and the cache requirement when scheduling layers i to (j-1) as a whole is less than or equal to the capacity of the on-chip memory, based on the computational overhead of the neural network. Wherein, the first batch size of layers i to j in the neural network is the same, i, j, and m are positive integers, and (i+m) ≤ (j-1).

15. The neural network scheduling device according to claim 14, characterized in that, The segmentation unit is specifically used to obtain multiple computational overheads corresponding to scheduling layers i to t as a whole, where layer t is any layer from layer (i+1) to layer (j-1); t is a positive integer, (i+1)≤t≤(j-1); when scheduling layers i to (i+m) as a whole, the computational overhead of the neural network is minimized.

16. The neural network scheduling device according to claim 11, characterized in that, The segmentation unit is further configured to reduce the first batch size corresponding to the b-th first layer group or the (b-1)-th first layer group if the a-th first layer group to the b-th first layer group are determined as a second layer group.

17. The neural network scheduling device according to claim 11 or 16, characterized in that, The segmentation unit is specifically used to schedule the first layer group to the bth layer group as a whole, where the computational overhead of the neural network is a first computational overhead; and to schedule the first layer group to the (b-1)th layer group as a whole, where the computational overhead of the neural network is a second computational overhead; if the first computational overhead is less than the second computational overhead, then the first layer group to the bth layer group is determined as a second layer group; if the second computational overhead is less than the first computational overhead, then the first layer group to the (b-1)th layer group is determined as a second layer group.

18. The neural network scheduling apparatus according to any one of claims 11-13, characterized in that, In the process of scheduling the neural network based on the segmentation results of the second layer group, the scheduling order of the layers in the second layer group is determined according to the scheduling order of each of the first layer groups contained in the second layer group, and the scheduling order of the layers in the first layer group.

19. The neural network scheduling device according to claim 18, characterized in that, The scheduling order of the layers in the first layer group is the same as the scheduling order of the layers in the neural network before the split. The scheduling order of each of the first layer groups included in the second layer group is determined according to the first batch size and the scheduling order of the first and last layers in the first layer group.

20. The neural network scheduling apparatus according to any one of claims 11-13, characterized in that, At least one input data or output data of a layer included in the second layer group is stored in the on-chip memory, and the input data of the first layer and the output data of the last layer in the second layer group are stored in the off-chip memory.

21. A computer-readable storage medium, characterized in that, Includes computer instructions that, when executed on the device, cause the device to perform the neural network scheduling method as described in any one of claims 1-10.

Images