Processing data for layers of a neural network

Abstract

The invention is titled "Processing data for layers of a neural network." The invention describes a method of processing input data for a given layer of a neural network using a data processing system that includes computing resources for performing convolution computations. The input data includes a given set of input feature maps (IFMs) and a given set of filters. The method includes generating a set of partial IFMs that includes a plurality of partial IFMs corresponding to respective IFMs in the given set of IFMs. The method also includes grouping the partial IFMs in the set of partial IFMs into a set of partial IFM selections. The method further includes computing a set of partial output feature maps by convolving the set of selections with the given set of filters by respective computing resources of the data processing system. The invention also describes a data processing system for processing input data for a given layer of a neural network.

Description

Background Technology Technical Field

[0002] This disclosure relates to processing data. Specifically, but not exclusively, this disclosure relates to processing input data for layers of a neural network using a data processing system that includes multiple computing resources.

[0003] Related technical specifications

[0004] Neural networks have become powerful tools for image processing, inference, machine learning, and related tasks. A neural network can include one or more convolutional layers. For example, a convolutional neural network (CNN) typically includes multiple convolutional layers. In a convolutional layer, an output data array (called an output feature map (OFM)) is computed through convolutions between an input data array (called an input feature map (IFM)) and one or more filters (also called kernels). Each output feature map contains multiple elements, where each element is computed via a series of convolutions between a subset of the elements of the data array and the corresponding weight values ​​of the kernel. Neural networks can also include other layer types, such as fully connected (FC) layers, deconvolutional layers, recurrent layers, etc.

[0005] Convolutional computations account for a large portion of the computational cost of performing inference or training a neural network, both in terms of processing time and the power required to switch bits within a register. Because these computations are repeated during inference or training, application-specific integrated circuits (ASICs) known as hardware accelerators have been developed.

[0006] A neural processing unit (NPU) is a hardware accelerator specifically designed for processing data based on neural networks, such as convolutional neural networks (CNNs). An NPU may also be referred to as a neural network accelerator (NNA), convolutional neural network accelerator (CNNA), machine learning accelerator, or artificial intelligence accelerator. An NPU comprises an array of dedicated convolutional engines (CEs), each containing, for example, multiply-accumulate (MAC) hardware to perform convolution operations. Summary of the Invention

[0007] According to a first aspect, a method is provided for processing input data for a given layer of a neural network using a data processing system, the data processing system including computational resources for performing convolutional computations, the input data including a given set of input feature maps (IFMs) and a given set of filters. The method includes performing computational performance improvement methods for processing the input data for the given layer, including: generating a set of partial IFMs, the set of partial IFMs including a first plurality of partial IFMs corresponding to a first IFM in the given set of IFMs and a second plurality of partial IFMs corresponding to a second IFM in the given set of IFMs. The method further includes grouping the partial IFMs in the set of partial IFMs into a set of partial IFM selections, including: selecting at least a first partial IFM from the first plurality of partial IFMs and a second partial IFM from the second plurality of partial IFMs to form a first partial IFM selection, and selecting at least a third partial IFM from the first plurality of partial IFMs and a fourth partial IFM from the second plurality of partial IFMs to form a second partial IFM selection. The method further includes convolving the set of selections with the given set of filters using the appropriate computational resources of the data processing system to compute a set of partial output feature maps—partial OFMs.

[0008] According to a second aspect, a data processing system is provided. The data processing system includes multiple computing resources and a storage device for storing data according to layers of a neural network. The data includes a set of input feature maps (IFMs) and a set of filters. The storage device is coupled to the data processing system. The data processing system is configured to generate a set of partial IFMs, the set of partial IFMs including a first plurality of partial IFMs corresponding to a first IFM in a given set of IFMs and a second plurality of partial IFMs corresponding to a second IFM in a given set of IFMs. The data processing system is further configured to group the partial IFMs in the set of partial IFMs into a set of partial IFM selections, including: selecting at least a first partial IFM from the first plurality of partial IFMs and a second partial IFM from the second plurality of partial IFMs to form a first partial IFM selection, and selecting at least a third partial IFM from the first plurality of partial IFMs and a fourth partial IFM from the second plurality of partial IFMs to form a second partial IFM selection. The data processing system is further configured to compute a set of partial output feature maps—partial OFMs—by convolving the set of selections with a given set of filters using a corresponding computing resource among the multiple computing resources.

[0009] According to a third aspect, a non-transitory computer-readable storage medium is provided, the non-transitory computer-readable storage medium including computer-executable instructions that, when executed by at least one processor, cause the at least one processor to: generate a set of partial IFMs, the set of partial IFMs including a first plurality of partial IFMs corresponding to a first IFM in a given set of IFMs and a second plurality of partial IFMs corresponding to a second IFM in a given set of IFMs; group the partial IFMs in the set of partial IFMs into a set of partial IFM selections, including: selecting at least a first partial IFM from the first plurality of partial IFMs and a second partial IFM from the second plurality of partial IFMs to form a first partial IFM selection, and selecting at least a third partial IFM from the first plurality of partial IFMs and a fourth partial IFM from the second plurality of partial IFMs to form a second partial IFM selection; and convolving the set of selections with a given set of filters using the corresponding computing resources of a data processing system to compute a set of partial output feature maps—partial OFMs. Attached Figure Description

[0010] Figure 1 The diagram schematically illustrates the data processing operations between the input data array and the kernel for generating the elements of the output data array;

[0011] Figure 2 A computing system configured to perform data processing operations is illustrated schematically according to an example;

[0012] Figure 3 schematically shown Figure 2 The neural processing unit (NPU) within the computing system;

[0013] Figure 4 This is a flowchart illustrating a method for processing data, based on an example.

[0014] Figure 5A It is the convolutional layer of the convolutional neural network in the example.

[0015] Figure 5B Yes Figure 5A An exemplary first operation of the convolutional layer of a convolutional neural network.

[0016] Figure 5C Yes Figure 5A An exemplary second operation of the convolutional layer of a convolutional neural network.

[0017] Figure 6 It is a computer-readable storage medium according to the example. Detailed Implementation

[0018] Referring to the accompanying drawings, the details of the systems and methods according to the examples will become apparent from the following description. In this specification, for purposes of explanation, numerous specific details of certain examples are set forth. References to "example" or similar language in this specification mean that a feature, structure, or characteristic described in connection with that example is included in at least one example, but not necessarily in other examples. It should also be noted that some examples are described schematically, in which certain features are omitted and / or must be simplified to facilitate explanation and understanding of the concepts on which the examples are based.

[0019] Some examples described herein relate to a method for processing input data for a given layer of a neural network using a data processing system, which includes computational resources for performing convolutional computations. The input data includes a given set of input feature maps (IFMs) and a given set of filters. The method includes computational performance improvement methods for processing the input data for a given layer. This method involves generating multiple partial IFMs from the set of IFMs, and convolving each of the partial IFMs by corresponding computational resources of the data processing system. By generating and convolving more partial IFMs than the existing corresponding IFMs, more computational resources of the data processing system can be utilized. This improved utilization can improve operational performance by leveraging the data processing system's ability to run computations in parallel on these multiple computational resources.

[0020] Figure 1An example of a convolution between an input data array 100 and a filter 102 for generating an output data array 104 is shown. In this example, the input data array 100 is a three-dimensional array with multiple channels, each channel comprising a planar array of numerical data elements referred to below as data values. The number of channels is referred to as the depth of the input data array 100. For example, a color image can be represented by a three-dimensional array where different locations in the plane represent different pixels and different planar slices represent different color channels. The filter 102 is another three-dimensional array with the same depth as the input data array 100. The filter 102 comprises stacked d two-dimensional arrays of size w×h. Each two-dimensional array in the d two-dimensional array will be referred to herein as a kernel. The elements of the filter 102 are referred to as weight values. The input data array 100 comprises stacked d two-dimensional arrays. Each two-dimensional array in the d two-dimensional array of the input data array will be referred to as an input feature map (IFM). According to this document, IFM can refer to the input data (e.g., an image) in the first layer of a CNN or the input data to subsequent layers of a CNN (e.g., the output from a previous layer of a CNN). In this example, d = 8, w = 3, h = 3. It should be understood that other sizes and shapes of the filter are possible without departing from the scope of the invention. For example, input data representing an RGB image may include three IFMs for each input data array: one IFM for each of the red, green, and blue data. Output data array 104 is a two-dimensional data array.

[0021] Convolution is defined such that for a given position of filter 102 relative to input data array 100, a multiply-accumulate (MAC) operation is performed, where each weight value of filter 102 is multiplied by the coincident data value of input data array 100, and the resulting products are summed to generate element 106 of output data array 104. For a given position of filter 102, a dot product is thus computed between filter 102 and the corresponding subarray of input data array 100. To generate the entire output data array 104, this dot product operation is performed for each of a predetermined set of positions of filter 102 relative to input data array 100, where these predetermined positions are separated by a fixed number of elements (called stride) in each direction within the plane of the array. In this example, the stride is 1 in each direction, such that adjacent positions of filter 102 relative to input data array 100 are adjacent to each other. In this example, zero padding is used, such that input data array 100 is surrounded by a boundary (not shown) with a width / height of two elements, where all elements are set to zero. This zero-padding ensures that the output data array 104 has the same in-plane size as the input data array 100. In other examples, other types of padding may be used. Furthermore, padding may be omitted entirely and / or a step size greater than 1 may be used in the in-plane size, in which case the output data array 104 will have a smaller in-plane size than the input data array 100.

[0022] Convolutional operations, such as those described above, are commonly used to generate output feature maps within the convolutional layers of a Convolutional Neural Network (CNN) during training or, once trained, during inference. In a given convolutional layer, each planar slice of a 3D Output Feature Map (OFM) is generated by performing convolutions between a 3D data array comprising multiple Input Feature Maps (IFMs) and corresponding filters, where different OFMs have different sets of filters. An OFM can indicate a corresponding set of features relevant to a given task, such as in computer vision or other image processing tasks. Different filters result in different sets of features, and while the weights of a given kernel can be predetermined, for most applications, the weights are likely learned during the training phase along with other parameters of the CNN, such as biases. Kernel sizes of varying sizes allow for the capture of different types of features from each IFM. Larger kernels may be able to capture more expressive features than smaller kernels, but for many applications, the features of interest are highly localized and therefore can be captured using smaller kernels, such as those of size 3×3. The generated OFM typically undergoes further processing, such as applying activation functions and optionally pooling, before being passed to subsequent layers of the CNN. Therefore, the processed OFM can become the IFM used in subsequent layers of the CNN. These subsequent layers can be, for example, another convolutional layer, a deconvolutional layer, or a fully connected layer.

[0023] Figure 2 An example of a computing system 200 arranged to perform layer processing according to a given neural network architecture is shown. The data processing system 200 includes a system bus 202 connected to a central processing unit (CPU) 204 and memory 206. Memory 206 includes volatile memory (including dynamic random access memory (DRAM), such as synchronous dynamic random access memory (SDRAM)), and non-volatile memory and storage devices such as flash memory, solid-state drives (SSDs), and non-volatile random access memory (NVRAM). The system bus 202 may also be connected to... Figure 2 Other components not shown include input / output devices, graphics processing units (GPUs), network interfaces, etc. The data processing system 200 also includes a neural processing unit (NPU) 208, which is a dedicated integrated circuit for processing data according to neural network architectures such as convolutional neural network (CNN) architectures. The NPU may also be referred to as a neural network accelerator (NNA), convolutional neural network accelerator (CNNA), machine learning accelerator, or artificial intelligence accelerator.

[0024] NPU 208 includes a control unit 210 that includes processing circuitry arranged to generate control data for multiple convolution engines or computing resources 212.1, 212.2, ..., 212.N (collectively referred to as convolution engines 212). In this example, NPU 208 includes N=16 convolution engines 212; however, it should be understood that different numbers of convolution engines 212 may be employed without departing from the scope of the invention. NPU 208 is arranged to retrieve data from memory 206 using direct memory access (DMA) 214. The data received from memory 206 may include, for example, image data or IFM data, as well as weight data associated with a given neural network layer, and any other parameters such as bias. NPU 208 is further arranged to write data such as OFM data to memory 206 using DMA 214.

[0025] Figure 3The NPU 208 is shown in more detail, including components of convolution engine 212.1. Other convolution engines 212.2, ..., 212.N include substantially the same components as convolution engine 212.1. Convolution engine 212.1 includes memory 302, which in this example includes static random access memory (SRAM). Convolution engine 212.1 also includes a MAC computation engine (MCE) 304, which is arranged to process data received from memory 302 and similar memory from the other convolution engines 212. MCE 304 includes components for performing MAC operations necessary for implementing layer processing for neural network architectures. Specifically, NPU 208 has components arranged for an optimized implementation of convolution (or deconvolution) between IFM or other data arrays and kernels to generate slices of OFM data, as described in more detail below.

[0026] The convolutional engine 212.1 also includes a post-computation resource 306 arranged to perform additional processing operations (such as pooling operations and applying activation functions) on slices of OFM data generated by the MCE 304. The post-computation resource 306 can be programmed to perform different operations on different layers within a given neural network, thereby allowing for the implementation of a wide range of neural network architectures. In this example, the post-computation resource 306 is arranged to output slices of processed OFM data to the memory 302 of the convolutional engine 212.1. Slices of processed OFM data can be stored in the memory 302 of the convolutional engine 212.1 if the memory 302 has sufficient capacity, can be transferred back to the memory 206 of the computing system 200, or can be broadcast to other convolutional engines 212 for further layer processing.

[0027] In one example, computer system 200 is used to perform inference on a set of IFMs based on a neural network architecture comprising multiple convolutional layers. Each convolutional layer of the neural network includes multiple filters, and therefore implementing each convolutional layer involves multiple convolution operations. A set of IFMs, along with weight data corresponding to the filters of the first convolutional layer of the neural network, is retrieved from memory 206. Both the set of IFMs and the filters are transferred to memory 302 of convolution engine 212.1. Convolution engine 212.1 performs convolutions between the set of IFMs and the filters to generate data representing OFMs, which is stored in memory 302. Each convolution engine in convolution engine 212 performs convolutions between the set of IFMs and the corresponding filters of the first layer in such a way that each convolution engine in convolution engine 212 stores data representing multiple OFMs. Once all the filters of the first layer have been applied, the entire set of processed OFMs is stored between convolution engines 212. This set of OFMs becomes a new set of IFMs for the second convolutional layer of the neural network. Convolutional engines 212 are arranged to broadcast a new set of stored IFMs to each other, making each convolutional engine 212 accessible to the entire new set of IFMs for the second layer. Weight data corresponding to the filters of the second layer is then transferred from memory 206 to the convolutional engines 212, and each convolutional engine in 212 performs a convolution between the new set of IFMs and the corresponding filters of the second layer. This process is repeated for subsequent convolutional layers of the neural network. When the final convolutional layer has been implemented, the resulting processed set of OFMs is transferred to memory 206 of computing system 200 or to on-board memory (including memory 302) of NPU 208. Additional layers of the neural network may reuse convolutional layers, such as deconvolutional layers, fully connected layers, or output layers. Although in this example, one layer of the neural network is processed at a time, in other examples, portions of multiple layers may be processed simultaneously, resulting in so-called fusion layer processing.

[0028] Figure 4 A method 400 is illustrated for processing input data for a given layer of a neural network using a data processing system, which includes computational resources for performing convolutional computations. The input data includes a given set of input feature maps (IFMs) and a given set of filters. Method 400 includes methods to improve the computational performance of processing the input data for the given layer. The data processing system may be... Figure 2 The computer system 200 is shown. In this example, the computing resource is the convolution engine 212.

[0029] Processing the input data for a given layer involves computing a set of OFMs by convolving a set of IFMs with a set of filters, as described above. An IFM is a data array comprising elements representing attribute values ​​of an image. The number of filters in this IFM is related to the number of OFMs produced. In standard convolution, the number of filters equals the number of OFMs produced. This is modified in method 400 to account for computational resources not utilized in standard convolution. For example, because each computational resource is configured to compute one OFM by convolving a set of IFMs with one filter from the set of filters, underutilization of computational resources may occur when the number of filters is less than the number of computational resources available for processing. For example, there might be 8 filters in a set of filters, and 16 computational resources in the data processing system. Computing the convolutional layer using standard convolution would result in 8 out of 16 computational resources being used to process the convolutional layer, thus achieving a 50% utilization rate of available computational resources.

[0030] At box 402, method 400 involves generating a set of partial IFMs, the set of partial IFMs including a first plurality of partial IFMs corresponding to a first IFM in a given set of IFMs and a second plurality of partial IFMs corresponding to a second IFM in a given set of IFMs. As described above, the data processing system includes computational resources. Computational resources may include multiple convolution engines, such as in Figure 2 In the computer system 200 shown, each convolution engine is configured to compute an OFM by convolving a set of IFMs with filters from a set of filters.

[0031] The generation of the first and second partial IFMs can be dynamically performed by the data processing system. To this end, the data processing system can be configured to determine the number of OFMs in a set of OFMs and the number of computational engines available for processing. In this case, the data processing system can determine that the number of OFMs in the set is less than the number of available computational engines. Then, the data processing system can generate the first set of partial IFMs based on the determined number of computational engines available for processing. For example, the data processing system may include 16 computational resources, and the layers of the convolutional neural network may have eight OFMs. In this case, the data processing system can generate two partial IFMs corresponding to the first IFM, such that each IFM in each filter convolutional partial IFM produces 16 corresponding partial OFMs. In this way, all available computational resources can be utilized to process the layers of the neural network.

[0032] In another example, the data processing system may include 32 computational resources, while the layers of the convolutional neural network may include 14 filters and 14 IFMs, thus generating 14 OFMs. In this case, the efficiency of the data processing system's computational resources in processing the convolutional neural network can be improved by generating 28 partial IFMs. Although not all of the data processing system's computational resources will be utilized, by generating partial IFMs from the IFMs, the efficiency of the data processing system can be increased from the maximum possible efficiency of 14 / 32 to 28 / 32.

[0033] Typically, when there are fewer filters than available computational resources for processing, multiple partial IFMs may be generated. Convolving these multiple partial IFMs with the set of filters can improve computational resource utilization. The number of partial IFMs in the first set does not need to be a power of 2. For example, the first set of partial IFMs may contain three, five, seven, or other partial IFMs. The number of generated partial IFMs depends on the number of filters in the set and the amount of computational resources available for processing.

[0034] How the partial IFMs in the first and second partial IFMs are generated can depend on certain properties of the convolutional layers. For example, if each kernel has a size of N×M, where N and M are both positive integers greater than 1, then the first partial IFM in the first multiple partial IFMs can include one or more elements that are also in the second partial IFM in the first multiple partial IFMs. That is, there may be some overlap between partial IFMs to ensure that, based on the first IFM convolved with a filter, subsequent convolutions produce the expected partial OFM. This is due to the nature of convolution when the kernel size is N×M, because the elements in the data array representing the OFM are the result of a MAC operation performed on multiple elements in the data array representing the IFM. A single element in the data array representing the IFM can participate in the computation of more than one element in the data array representing the OFM. Therefore, these elements can exist in more than one partial IFM in a given multiple partial IFMs.

[0035] However, if each kernel in the filters of this set of filters is 1×1, then the elements of each partial IFM in the multiple partial IFMs can be independent of each other; that is, each data array representing a partial IFM can be a disjoint subarray representing the array of the corresponding IFM. In any case, each data array representing a partial IFM can be considered a subarray representing the data array of the corresponding IFM. Furthermore, the composition and / or size of the generated partial IFMs can also depend on at least one of the stride and padding used by the convolutional neural network model.

[0036] The data array representing the first portion of the first plurality of IFMs may have the same size as the data array representing the second portion of the first plurality of IFMs. In one example, each data array representing each portion of the first plurality of IFMs may have the same size. For example, the first plurality of IFMs may include four portion IFMs corresponding to the first IFM, wherein each portion IFM has the same size and roughly corresponds to one-quarter of the data representing the first IFM.

[0037] In another example, the size of the data array representing a portion of the first plurality of partial IFMs can be different from the size of the data array representing a second portion of the first plurality of IFMs. For example, consider the following case: the first IFM is represented by a two-dimensional data array of size 10×10, the set of filters includes four filters, and the data processing system includes 12 available computational resources. In this case, the utilization of computational resources can be increased if three partial IFMs corresponding to the first IFM are generated. In this case, these three partial IFMs can be generated with sizes of 6×6, 6×6, and 6×10, because dividing the data array representing the first IFM into three partial IFMs of equal size may be difficult to implement.

[0038] Each of the partial IFMs in the second plurality of partial IFMs has the same size as the corresponding partial IFM in the first plurality of partial IFMs, such that the number of partial IFMs in the second plurality of partial IFMs equals the number of partial IFMs in the first plurality of partial IFMs. For example, if there is a partial IFM in the first plurality of partial IFMs corresponding to the top-left 3×3 element of the first IFM, then there will be a corresponding partial IFM in the second plurality of partial IFMs corresponding to the top-left 3×3 element of the second IFM. This process of generating partial IFMs for the corresponding IFMs in the input layer can be repeated for all IFMs.

[0039] A first plurality of partial IFMs and a second plurality of partial IFMs can be generated by manipulating the memory address of the first IFM. In this case, this may result in the first IFM being identified by the data processing system as a first plurality of partial IFMs. Therefore, a subsequent convolution operation will be performed on the identified first plurality of partial IFMs. Other examples are possible. In one example, the control unit 210 may generate multiple sets of partial IFMs based on the number of OFMs in the set of OFMs and the amount of computing resources available for processing. The number of OFMs and the amount of computing resources may be determined by the control unit 210. In any case, generating the first set of partial IFMs does not introduce additional data into the neural network model, and therefore does not increase memory usage.

[0040] At box 404, method 400 involves grouping the partial IFMs in the group of partial IFMs into a group of partial IFM selections. This grouping includes selecting at least a first partial IFM from a first plurality of partial IFMs and a second partial IFM from a second plurality of partial IFMs to form a first partial IFM selection, and selecting at least a third partial IFM from the first plurality of partial IFMs and a fourth partial IFM from the second plurality of partial IFMs to form a second partial IFM selection.

[0041] This grouping can be based on which part of the IFM is convolved to produce a given part OFM. For example, standard convolution can produce a given OFM by convolving a set of IFMs with a given filter. The top left quarter of the OFM can be the result of convolving the corresponding top left quarter of each IFM in that set with the given filter. Therefore, the IFMs corresponding to the top left quarter of each IFM in that set can be grouped such that convolving that set of IFMs produces the top left quarter of the corresponding OFM.

[0042] This grouping may involve identifying corresponding IFMs within a first plurality of partial IFMs and a second plurality of partial IFMs. For example, a first partial IFM corresponding to a subarray of a first 3×3 element of a first IFM and a second partial IFM corresponding to a subarray of a first 3×3 element of a second IFM may be grouped together.

[0043] Grouping can be achieved by manipulating the memory addresses of the first and second IFM portions. For example, the memory addresses can be manipulated such that the first and second IFM portions are recognized as sublayers of the IFM layer. Alternatively, the control unit 210 can be configured to group the first and second IFM portions such that they are received together by corresponding computational resources for convolution with corresponding filters.

[0044] Selecting the third and fourth IFM portions to form the second IFM selection can be done in the same way as forming the first IFM selection. The process of grouping the IFM portions from the first and second groups can be repeated until all corresponding IFM portions from both groups are grouped into IFM selections.

[0045] At block 406, method 400 involves computing a set of partial output feature maps (partial OFMs) by convolving the set of selections with a given set of filters using the appropriate computational resources of the data processing system. For example, convolving a first partial IFM selection with each filter in the set of filters produces a first set of partial OFMs, where each partial OFM in the first set of partial OFMs is the result of convolving the first partial IFM selection with the corresponding filter. It is determined at block 404 that the partial IFMs are grouped into the first partial IFM selections such that each resulting data array representing each partial OFM corresponds to a disjoint subset of the data array representing the corresponding OFM. In this way, when each partial IFM selection is convolved with the set of filters, the resulting partial OFMs are functionally equivalent to the OFMs produced by convolving the IFMs.

[0046] The result of this operation is that more computational resources of the data processing system can be utilized to compute the layers of the neural network. For example, if an IFM is convolved, instead of a partial IFM, the number of OFMs produced will be equal to the number of filters in that set of filters, because convolving that set of IFMs with a single filter results in a single OFM. However, according to method 400, the number of partial OFMs produced by the convolution process is equal to the number of partial IFMs for each IFM multiplied by the number of filters in that set of filters. This can be expressed mathematically as:

[0047] N O =N f *N PI Formula 1

[0048] Where N O N is the quantity of a portion of OFM. f N is the number of filters in this group of filters, and N PI It represents the number of partial IFMs for each IFM in a single convolutional layer.

[0049] Therefore, selecting an appropriate number of partial IFMs for each IFM determines the number of resulting partial OFMs, and thus determines how many computational resources are utilized in the computation of convolutional layers in the convolutional neural network. Equation 1 can be used to find the maximum possible N for the amount of available computational resources. OThis determines how many partial IFMs should be generated for each IFM. This can be done by the data processing system itself or by software on a device that includes the data processing system. For example, the data processing system might determine that convolving a given set of IFMs with a given set of filters will produce a set of OFMs including a given number of OFMs. If the given number of OFMs is less than or equal to half the given number of convolution engines, the data processing system determines that the set of partial OFMs may include partial OFMs with a number greater than the given number of OFMs and less than or equal to the given number of convolution engines.

[0050] At box 408, method 400 involves combining the group of partial OFMs to form a given set of OFMs and using this given set of OFMs as input data for subsequent layers of the neural network. As discussed above with respect to box 406, the data array representing each partial OFM is a disjoint subset of the data array representing the corresponding OFM. In some cases, for example, the output of a convolutional layer may be the input for subsequent convolutional or pooling layers. In this case, it may be expected that the output produced at box 406 will appear as if the convolutional layer did not process the data by first generating partial IFMs. An example of this is when subsequent layers of a convolutional network expect a certain number of OFMs from previous layers as input to subsequent layers. This can be useful in already trained convolutional neural networks where more complex processes in subsequent layers (such as channel rearrangement or further convolutions) expect a certain size of input data array.

[0051] The process of forming this set of OFMs can correspond to generating multiple partial IFMs. For example, the memory address of a partial OFM can be manipulated so that subsequent layers of the convolutional neural network will recognize the corresponding OFM as a partial OFM. Alternatively, the control unit 210 can be configured to form this set of OFMs.

[0052] In some cases, it may not be necessary to form a set of OFMs from a partial OFM. Method 400 can be repeated depending on the number of filters in a set of filters in subsequent layers, such as if the number of filters in subsequent layers is less than the amount of computational resources available for processing. This can be determined dynamically, as data is passed between layers of the convolutional neural network, or it can be determined before processing begins on the first convolutional layer.

[0053] In some examples, computational performance improvement method 400 may be selectively implemented. In this case, alternative methods, different from computational performance improvement method 400, may be implemented to process additional data for additional neural network layers using a data processing system, depending on the characteristics of the given network layer and the additional network layers. For example, method 400 may not be applied to every layer of the convolutional neural network, but may be selectively implemented based on the characteristics of the additional network layers. For example, these characteristics may include the number of OFMs generated by the additional network layers.

[0054] In one example, the additional layer includes another set of IFMs and another set of filters. The data processing system can determine that if the additional set of IFMs is convolved with the other set of filters, an additional set of OFMs will be computed, which includes an additional number of OFMs. If the additional number of OFMs is greater than half the given number of OFMs in the convolution engine, then the additional set of IFMs can be convolved with the first set of filters to compute the additional set of OFMs. Each OFM in the additional set of OFMs can be computed in different convolution engines within the convolution engine.

[0055] Method 400 provides an efficient way to process layers of a neural network by increasing the utilization of available computational resources. Furthermore, since a partial IFM is a subarray of the corresponding IFM, and the partial IFM is convolved with the same set of filters, Method 400 does not increase memory usage above the standard convolutional layers of the neural network. It should be noted that Method 400 is described with reference to the first IFM and the second IFM. However, Method 400 is applicable to layers comprising any number of IFMs greater than one.

[0056] Now refer to Figures 5A to 5C Discuss an example of method 400. Figure 5A This is an exemplary convolutional layer of a convolutional neural network 500, which has a set of IFMs 502 and 504, a set of filters 506-512, and a set of OFMs 514-520. Each filter 506-512 includes two kernels for convolutional of each IFM 502 and 504. Figure 5A The diagram illustrates the result of convolving a set of IFMs 502 and 504 with a set of filters 506-512 to produce a set of OFMs 514-520. In this case, OFM 514 is produced by convolving IFMs 502 and 504 with filter 506, OFM 516 is produced by convolving IFMs 502 and 504 with filter 508, OFM 518 is produced by convolving IFMs 502 and 504 with filter 510, and OFM 520 is produced by convolving IFMs 502 and 504 with filter 512.

[0057] exist Figures 5A to 5CIn the example shown, the data processing system (not shown) includes computational resources containing at least eight convolutional engines. Therefore, in Figure 5A In the example shown, only four of the eight compute engines will be used to process convolutional layer 500. Utilizing more compute engines to process convolutional layers would make the process more efficient.

[0058] Figure 5B An example of generating a set of partial IFM 502', 502", 504', 504" corresponding to IFM 522 and IFM 502 and 504 is shown. As mentioned above, the data processing system includes at least eight convolution engines. Therefore, in Figure 5B In the example shown, the two IFMs 502 and 504 are divided into four parts IFMs 502', 502", 504', and 504", such that eight parts OFMs are generated by convolving each part IFM with a set of filters, and thus all eight computational engines are utilized during the convolution process. The size of each part IFM 502', 502", 504', and 504" depends on the size of the kernel in filters 506-512. For example, if the kernel size in filters 506-512 is 1×1, then the parts IFMs 502', 502", 504', and 504" can represent disjoint subarrays of the corresponding IFMs 502 and 504. In one example, a portion IFM 502' may represent the upper half of an element in IFM 502, a portion IFM 502" may represent the lower half of an element in IFM 502, a portion IFM 504' may represent the upper half of an element in IFM 504, and a portion IFM 504" may represent the lower half of an element in IFM 504. Other divisions are possible. For example, portions IFM 502', 502", 504', and 504" may correspond to the vertical bisection of the respective IFM 502 and 504. In another example, portions IFM 502' and 504' have different dimensions than portions IFM 502" and 504".

[0059] If the kernel size in filters 506-512 is N×M, where N and M are positive integers greater than 1, then at least one element from part IFM 502' will be identical to an element from part IFM 502" in order to produce the correct partial OFM. In the case where IFM 502 is represented by a 16×16 data array, part IFM 502' may correspond to the upper 9×16 elements of IFM 502, while part IFM 502" may correspond to the lower 9×16 elements of IFM 502. In this case, the bottom row of part IFM 502' will be identical to the top row of part IFM 502" . Other partitioning is possible, as described above.

[0060] For the partial IFMs 502', 502", 504', 504", which are to be convolved to produce the expected partial OFM, the partial IFMs 502', 502", 504', 504" need to be properly grouped before convolution. Figure 5C This is an exemplary convolutional layer of a convolutional neural network 524, wherein portions IFM 502', 502”, 504', and 504” are grouped to form portions IFM selections 526 and 528. Portion IFM selection 526 includes portions IFM 502' and 504' corresponding to the first IFM 502 and the second IFM 504, respectively, while portion IFM selection 528 includes portions IFM 502” and 504” corresponding to the first IFM 502 and the second IFM 504, respectively. The data processing system can then process the first portion IFM selection 526 and the second portion IFM selection 528 as separate data input layers.

[0061] Then, the IFM selections 526 and 528 of the convolutional parts 506-512 are respectively passed through filters to generate two partial OFM selections 530 and 532. These two selections include partial OFMs 514', 516', 518', 520' and 514", 516", 518", 520", respectively. Partial OFMs 514', 514" correspond to OFM 514, partial OFMs 516', 516" correspond to OFM 516, partial OFMs 518', 518" correspond to OFM 518, and partial OFMs 520', 520" correspond to OFM 520. Specifically, partial OFMs 514', 516', 518', and 520' are generated by convolving partial IFM selection 526 with the corresponding filters 506-512, and partial OFMs 514', 516', 518', and 520' are generated by convolving partial IFM selection 528 with the corresponding filters 506-512. In this way, it can be understood that OFM selections 530 and 532 are functionally equivalent to... Figure 5A OFM 514-520.

[0062] It should be noted that method 400 can be provided by someone with Figure 2The data processing system of the computing device 200 shown, comprising multiple computing resources 212 and storage device 206, performs the operation. For example, storage device 206 stores data according to the layers of a neural network, including a set of input feature maps (IFMs) and a set of filters, wherein the storage element is coupled to the data processing system. The data processing system is configured to generate a set of partial IFMs, including a first plurality of partial IFMs corresponding to a first IFM in a given set of IFMs and a second plurality of partial IFMs corresponding to a second IFM in a given set of IFMs. The data processing system is further configured to group the partial IFMs in the set of partial IFMs into a set of partial IFM selections, including: selecting at least a first partial IFM from the first plurality of partial IFMs and a second partial IFM from the second plurality of partial IFMs to form a first partial IFM selection, and selecting at least a third partial IFM from the first plurality of partial IFMs and a fourth partial IFM from the second plurality of partial IFMs to form a second partial IFM selection. The data processing system is further configured to compute a set of partial output feature maps—partial OFMs—by convolving the set of selections with a given set of filters using the corresponding computing resources.

[0063] As described above, the data processing system can be a neural data processing system configured to process data representing layers of a convolutional neural network. Computational resources may include multiple convolution engines. The data processing system may be further configured to increase the number of convolution engines among the multiple convolution engines used to process input data for a given layer, relative to the number of convolution engines that would be used when convolving a given set of IFMs with a given set of filters. Additionally or alternatively, the data processing system may be configured to modify the memory address associated with a first IFM to generate a first plurality of partial IFMs. If this is the case, the data processing system may also be configured to modify the memory address associated with a first partial IFM from the first plurality of partial IFMs and a second partial IFM from a second plurality of partial IFMs to form a first partial IFM selection. The data processing system may be further configured to combine this set of partial OFMs to form a given set of OFMs and use this given set of OFMs as input data for subsequent layers of the neural network.

[0064] Figure 6A non-transitory computer-readable storage medium 600 including computer-executable instructions 604-610 is shown. When executed by processor 602, the computer-executable instructions 604-610 cause processor 602 to perform method 400. At a first block 604, method 400 includes generating a set of partial IFMs, the set of partial IFMs including a first plurality of partial IFMs corresponding to a first IFM in a given set of IFMs and a second plurality of partial IFMs corresponding to a second IFM in a given set of IFMs. At a second block 606, the method involves selecting at least a first partial IFM from the first plurality of partial IFMs and a second partial IFM from the second plurality of partial IFMs to form a first partial IFM selection, and selecting at least a third partial IFM from the first plurality of partial IFMs and a fourth partial IFM from the second plurality of partial IFMs to form a second partial IFM selection. At a block 608, the method involves convolving the set of selections with a given set of filters using appropriate computational resources of a data processing system to compute a set of partial output feature maps—partial OFMs. At box 610, the method involves combining the group of partial OFMs to form a given set of OFMs, and using the given set of OFMs as input data for subsequent layers of the neural network.

[0065] It should be understood that any feature structure described with respect to any embodiment may be used alone or in combination with other described feature structures, and may also be used in combination with one or more feature structures of any other embodiment, or in combination with any feature structure of any other embodiment. Furthermore, equivalents and modifications not described above may be employed without departing from the scope of the invention as defined in the appended claims.

Claims

1. A method for improving the use of computational resources in a neural processing unit, the neural processing unit being a hardware accelerator for processing data according to a neural network architecture and including a control unit and a plurality of convolutional engines, the control unit including processing circuitry arranged to generate control data for the plurality of convolutional engines, the method comprising processing input data for a given layer of a neural network, the input data including a given set of input feature maps (IFMs) and a given set of filters, the method comprising: The control unit generates a set of partial IFMs, the set of partial IFMs including a first plurality of partial IFMs corresponding to a first IFM in the given set of IFMs and a second plurality of partial IFMs corresponding to a second IFM in the given set of IFMs; The control unit divides the partial IFMs in the group of partial IFMs into a group of partial IFM selections, including: At least a first portion of IFM from the first plurality of IFM portions and a second portion of IFM from the second plurality of IFM portions are selected to form a first portion IFM selection; and At least a third IFM from the first plurality of IFM portions and a fourth IFM from the second plurality of IFM portions are selected to form a second IFM selection; and A set of partial output feature maps—partial OFMs—is computed by convolving the selected set with the given set of filters using the appropriate convolutional engine, where: If the given set of IFMs is convolved with the given set of filters, a set of OFMs including the given number of OFMs will be calculated. The given number of OFMs is less than or equal to half the given number of convolution engines among the plurality of convolution engines; and The set of partial OFMs includes a number greater than the given number of OFMs and less than or equal to the given number of partial OFMs of the convolution engine.

2. The method according to claim 1, wherein the method comprises: Combine the aforementioned set of partial OFMs to form a given set of OFMs; as well as The given set of OFMs is used as input data for subsequent layers of the neural network.

3. The method of claim 1, wherein the method: increases the number of convolution engines among the plurality of convolution engines used to process the input data for the given layer, relative to the number of convolution engines that would be used when convolving the given set of IFMs with the given set of filters.

4. The method of claim 1, wherein the method comprises computing each of the set of partial OFMs in different convolution engines in the convolution engine.

5. The method of claim 4, further comprising selectively performing the method and performing an additional method different from the method to process additional data for additional neural network layers using the neural processing unit, depending on the characteristics of the given network layer and the additional network layers, respectively.

6. The method according to claim 5, wherein: The additional layer includes another set of IFM and another set of filters; If the other set of IFMs is convolved with the other set of filters, another set of OFMs including an additional number of OFMs will be calculated. and The additional number of OFM is greater than half the given number of the convolutional engine, and The additional method mentioned above includes: The other set of OFM is calculated by convolving the other set of IFM with the other set of filters. as well as Each OFM in the other set of OFMs is computed in different convolution engines within the convolution engine.

7. The method of claim 1, wherein generating the set of partial IFMs includes modifying the memory addresses of the first IFM and the second IFM.

8. The method of claim 1, wherein at least one element of the data in the first part IFM is also in the third part IFM.

9. A neural processing unit, the neural processing unit being a hardware accelerator for processing data according to a neural network architecture and comprising: Multiple convolution engines; The control unit includes processing circuitry arranged to generate control data for the plurality of convolution engines; A storage device for storing data according to the layers of a neural network, the data including a given set of input feature maps (IFM) and a given set of filters, wherein the storage device is coupled to the plurality of convolutional engines. The control unit is configured as follows: Generate a set of partial IFMs, the set of partial IFMs including a first plurality of partial IFMs corresponding to a first IFM in the given set of IFMs and a second plurality of partial IFMs corresponding to a second IFM in the given set of IFMs; The partial IFMs in the aforementioned set of partial IFMs are grouped into a set of partial IFM selections, including: At least a first portion of IFM from the first plurality of IFM portions and a second portion of IFM from the second plurality of IFM portions are selected to form a first portion IFM selection; and At least a third IFM from the first plurality of IFM portions and a fourth IFM from the second plurality of IFM portions are selected to form a second IFM selection; and The plurality of convolutional engines are configured to convolve the set of selections with the given set of filters to compute a set of partial output feature maps—partial OFM, wherein: If the given set of IFMs is convolved with the given set of filters, a set of OFMs including the given number of OFMs will be calculated. The given number of OFMs is less than or equal to half the given number of convolution engines among the plurality of convolution engines; and The set of partial OFMs includes a number greater than the given number of OFMs and less than or equal to the given number of partial OFMs of the convolution engine.

10. A computer-readable medium comprising an instruction set that, when executed by a processor, causes the processor to perform the method according to any one of claims 1 to 8.

Images