Processing data stream modifications to reduce power effects during parallel processing

Abstract

Certain aspects of the present disclosure provide a method for performing parallel data processing, comprising: receiving data for parallel processing from a data processing requestor; generating a plurality of data sub-blocks; determining a plurality of data portions in each data sub-block of the plurality of data sub-blocks; changing an order of the plurality of data portions in at least one data sub-block of the plurality of data sub-blocks; providing the plurality of data sub-blocks to a plurality of processing units for parallel processing, the plurality of data sub-blocks including the at least one data sub-block including the plurality of data portions in the changed order; and receiving processed data associated with the plurality of data sub-blocks from the plurality of processing units.

Description

[0001] Priority claim according to 35 USC § 119

[0002] This application claims priority to U.S. Patent Application S / N. 16 / 834,986, filed March 30, 2020, entitled “PROCESSING DATA STREAMMODIFICATION TO REDUCE POWER EFFECTS DURING PARALLEL PROCESSING”, the entire application of which is incorporated herein by reference.

[0003] introduction

[0004] Various aspects of this disclosure relate to techniques for modifying processed data streams to reduce unintended and destructive power effects during parallel processing.

[0005] Parallel processing is increasingly being implemented to improve the performance of large data processing tasks. For example, the performance of machine learning and artificial intelligence systems can often be improved by implementing parallel processing in processing systems that include multiple processing units or across multiple processing systems.

[0006] While parallel processing can deliver significant performance advantages in many large data processing contexts, it also places significant stress on the processing system. This stress can lead to destructive power effects, such as current spikes and voltage drops, which negatively impact processing system performance. For example, if multiple processing units of a processing system (such as a System-on-Chip (SoC)) simultaneously initiate intensive data processing tasks, the entire system may become degraded or unstable during subsequent current spikes and / or voltage drops. In such scenarios, the benefits of parallel processing architectures are greatly reduced, and end-user performance is also compromised. These problems can be particularly severe in mobile devices where power considerations are amplified.

[0007] Therefore, there is a need for systems and methods to modify the processing data stream to reduce destructive power effects during parallel processing.

[0008] Brief Overview

[0009] Some aspects provide a method for performing parallel data processing, comprising: receiving data from a requester for parallel processing; generating a plurality of data sub-blocks; determining a plurality of data portions in each of the plurality of data sub-blocks; changing the order of the plurality of data portions in at least one of the plurality of data sub-blocks; providing the plurality of data sub-blocks to a plurality of processing units for parallel processing, the plurality of data sub-blocks including at least one data sub-block containing the reordered plurality of data portions; and receiving processed data associated with the plurality of data sub-blocks from the plurality of processing units.

[0010] A further aspect provides a method for performing parallel data processing, comprising: receiving data from a requester for parallel processing; determining a plurality of density ranges, wherein each of the plurality of density ranges is different from each of the other density ranges; determining a plurality of data portions of the data based on the density ranges; generating a plurality of data sub-blocks having reordered data portions; providing the plurality of data sub-blocks to a plurality of processing units for parallel processing; and receiving processed data associated with the plurality of data sub-blocks from the plurality of processing units.

[0011] A further aspect provides a non-transient computer-readable medium including instructions that, when executed by one or more processors of a processing system, perform the methods described above and those further described herein.

[0012] A further aspect provides a computer program product embodied on a computer-readable storage medium, the computer program product including code for performing the methods described above and further described herein.

[0013] A further aspect provides a processing system having one or more processors and / or other devices configured to perform the methods described above and those further described herein.

[0014] The following description and related figures illustrate certain illustrative features of one or more examples in detail. Brief description of the attached diagram

[0016] The accompanying drawings depict certain aspects of the one or more examples and are therefore not intended to limit the scope of this disclosure.

[0017] Figure 1 An example parallel data processing architecture is described.

[0018] Figure 2 An example is described where a processing coordinator generates multiple data sub-blocks based on received data blocks.

[0019] Figure 3 Another example of a parallel data processing architecture is described.

[0020] Figure 4 An example is depicted of dynamically reordering data for parallel processing to reduce power effects.

[0021] Figure 5 An example method of dynamically reordering data for parallel processing is depicted.

[0022] Figure 6 An example is depicted of dynamically reordering data for parallel processing to reduce power effects.

[0023] Figure 7 An example method of dynamically reordering data for parallel processing using density binning is depicted.

[0024] Figure 8 An example processing system configured to perform parallel processing using density-based reordering is depicted.

[0025] Figure 9 A chart depicting an example relationship between density in convolutional neural network data and power consumption of processing units is depicted.

[0026] Figure 10 An example of parallel processing of a multi-dimensional data block is depicted.

[0027] Figure 11A An example power effect associated with parallel processing is depicted.

[0028] Figure 12 An example of using density binning on machine learning data is depicted.

[0029] Figure 13 A comparison of regular processing of a machine learning data block with a kernel and density reordered processing of the machine learning data block is depicted.

[0030] Figure 14A An example power effect associated with parallel processing using density binning is depicted.

[0031] Figure 15 Useful properties of certain machine learning operations when using density reordered input data are depicted.

[0032] Figure 16 An example electronic device that can be configured to perform the methods described herein is depicted.

[0033] Figure 17 An example multi-processor processing system that can be implemented using the examples described herein is depicted.

[0034] For the sake of promoting an understanding of the principles of the application, the same reference numbers will be employed throughout the disclosure and when there is no possibility of misinterpretation, the same reference characters will be employed in the various drawings. Elements and features of one example can beneficially be incorporated into other examples without further recitation.

[0035] DETAILED DESCRIPTION

[0036] Aspects described herein relate to systems and methods for modifying processing data flow to reduce unintended and disruptive power effects during parallel processing. Such systems and methods improve the performance of parallel processing systems, such as those including multiple processing units and / or those coordinating multiple processing systems.

[0037] Generally, parallel processing capabilities can be provided in a single processor having multiple processing units (e.g., multiple processing cores of a processor), by multiple processors that can each have one or more processing units, by multiple processing systems each having its own processor, and by combinations thereof. Generally, the multiple processors for parallel processing can be designed to be homogeneous or heterogeneous, such as multiple central processing units (CPUs), or central processing units, graphics processing units (GPUs), neural signal processing units (NPUs), digital signal processors (DSPs), etc. working in concert. An NPU, such as described herein, is generally a specialized circuit configured to implement all the necessary control and arithmetic logic for executing machine learning algorithms, such as algorithms for processing artificial neural networks (ANN), deep neural networks (DNN), random forests (RF), etc.

[0038] As described in greater detail below, modifying processing data flow to reduce disruptive power effects during parallel processing can be achieved by analyzing data for parallel processing and reordering (or rearranging, reordering, or changing) the data to avoid concurrent power spikes from multiple processing units. Thus, the reordered processing data flow can reduce or eliminate undesirable power effects in the processing system, such as processing power spikes, voltage droops, overheating, additional leakage current, etc., which beneficially improves the performance and reliability of the processing system.

[0039] Further, the methods described herein can beneficially reduce peak current during parallel processing without resorting to conventional, deleterious performance degradation methods, such as frequency throttling and command stalling, to name a few examples.

[0040] Further, avoiding large processing power spikes can beneficially reduce thermal loading in the system (from high transient current). Reduced thermal loading can beneficially reduce cooling requirements, positively impacting packaging possibilities. Additionally, reduced thermal loading can reduce heat-induced leakage current, positively impacting power efficiency.

[0041] The above benefits can significantly improve the performance of mobile devices, particularly due to their inherent power design constraints.

[0042] Example parallel data processing architecture

[0043] Figure 1 An example parallel data processing architecture 100 is depicted. In the depicted example, an application 102 provides data for processing to a data processing system 110, which includes a data processing coordinator 104 for coordinating parallel processing. The application 102 can be a local application (e.g., on a processing system) or a remote application (e.g., in a client-server architecture) and can generally be referred to as a data processing requestor.

[0044] The processing coordinator 104 can generally be a function implemented within a processing system that is configured to coordinate parallel processing of data, such as in a multi-processor computing system (e.g., a computer, server, mobile device, system on a chip (SoC), Internet of Things (IoT), or edge device), or generally in any multi-processing unit system.

[0045] In this example, the data processing coordinator 104 receives data from the application 102 and coordinates parallel processing of the data by providing data sub-blocks A-D to processing cores 106A-D. The data sub-blocks A-D can be any type of data for parallel processing. In some examples, the data sub-blocks A-D can include machine learning model or artificial intelligence model input data, such as image data, video data, audio data, or other quantitative data.

[0046] A data sub-block can generally refer to a block, tile, tensor, or the like of data sent to multiple processing units (such as processing cores 106A-D) for parallel processing. In some examples, each data sub-block can represent a unique subset of a set of data for processing, while in other examples, each data sub-block can represent a repeated set of data to be processed differently on each processing unit (such as processing the same data with different models). Thus, generating data sub-blocks can include generating mutually exclusive subsets of data or repeated sets of data for distribution to multiple processing units.

[0047] Figure 2 An example of generating multiple data sub-blocks by a data processing coordinator (such as the processing coordinator 104 of Figure 1 FIG. 1.

[0048] In this example, the multi-dimensional data block 202 is provided to the processing coordinator 104 for processing by a plurality of processing devices, such as the plurality of processing cores 106A-D (e.g., for parallel processing). In some examples, the data block 202 can be a tensor. Generally, an n-order tensor in an m-dimensional space is a mathematical object with n indices and mn components and follows certain rules of transformation. Each index of a tensor is in the range of the dimension of the space. Thus, tensors can be a generalization of scalars (no indices), vectors (exactly one index), and matrices (exactly two indices) to any number of indices.

[0049] The data block 202 includes different portions 202A-D that have different densities (or, conversely, sparsities) (which are generally indicated by the different shading of those portions). In this example, portion 202B is the darkest shaded portion and, thus, is the densest portion of data 202, while portions 202A and 202C are one level lighter in shading and, thus, are less dense than portion 202B, but more dense than portion 202D. Data portion 202D is the lightest shaded and least dense (or most sparse) portion of data in the data block 202.

[0050] In one example, the density of a multi-dimensional data instance can be measured by the proportion of non-empty entries in each dimension of the data, such as an array, vector, matrix, or other multi-dimensional data format. Similarly, the sparsity of a multi-dimensional data instance can be measured by the proportion of empty entries in each dimension of the data. As a simple example, a data vector with entries [1, 0, 1] would have a density of 2 / 3 and a sparsity of 1 / 3. Generally, denser data requires more processing power, while sparser data requires less processing power.

[0051] It is worth noting that, for ease of explanation, Figure 2 The different densities and layouts of the data portions (202A-D) in FIG. 2 are intended to be only a simple example. Data can have significantly more complex density characteristics.

[0052] The processing order 203 of the data 202 indicates that, in processing the data 202, the processing cores 106A-D will encounter data of different densities at different times.

[0053] In this example, the processing coordinator 104 divides the data 202 into four sub-blocks 204A-D, each comprising a subset of portions 202A-D, as indicated by the matching shading bands. As depicted in Table 206, the processing of data sub-blocks 204A-D by processing cores 106A-D according to processing direction 203 results in a significant spike in total processing system power at 208, corresponding to time period 2 along the time axis. This is caused by each processing core (106A-D) simultaneously (or nearly simultaneously) arriving at the higher-density data portion 202B of data sub-blocks 204A-D.

[0054] The power spike at 208 is an example of an undesirable power effect that can cause various incidental problems in the processing system, such as voltage drops on the processing system power bus, which can lead to associated degradation such as processor frequency throttling, command pauses, and system instability. Furthermore, to mitigate the likelihood of these problems, higher operating voltages may be required, which would increase power dissipation.

[0055] As described in more detail below, the problem depicted in Figure 206 can be overcome by determining the data density in data sub-blocks (e.g., data sub-blocks 204A-D) and reordering the data to avoid concurrency of peak processing demands at each processing core (e.g., 106A-D in this example).

[0056] Parallel data processing architecture utilizing data reordering

[0057] Figure 3 Another example of a parallel data processing architecture 300 is depicted. In the depicted example, application 302 provides data for processing to data processing system 310, which includes a data processing coordinator 304, a data density analyzer 305, and a data sequencer 307. Again, application 302 can be referred to as a data processing requester and can be local to or remote from processing system 310.

[0058] As described above, the data processing coordinator 304 can generally be configured to coordinate the parallel processing of data in a multi-processor unit data processing system (such as processing system 310). In this example, the data processing coordinator 304 receives data from application 302 and generates data sub-blocks. As mentioned above, the received data can be a tensor.

[0059] In some examples, the processing coordinator 304 can be implemented as part of a compiler configured to determine how to split and allocate data sub-blocks onto processing cores during compile time. In other examples, the processing coordinator 304 can be implemented as part of a task scheduler or kernel in an operating system configured to determine how to split and allocate data sub-blocks onto processing cores at runtime. Notably, these are just two example implementations, and other example implementations are possible.

[0060] Unlike the examples in Figure 1 and 2 , here the data sub-blocks are subsequently analyzed by a density analyzer 305 prior to processing. The density analyzer 305 is configured to determine the density (or alternatively, sparsity) of the incoming data sub-blocks. In some examples, the density determination results in identifying a plurality of different data portions associated with different densities (or ranges of densities). In some examples, the data portions are arranged along a processing direction.

[0061] The density analyzer 305 provides the data sub-blocks and the determined densities to a sequencer 307, which then reorders the individual data sub-blocks based on the determined densities in order to immediately reduce or eliminate coincidences of peak density data portions being processed by multiple processing cores.

[0062] In one example, the sequencer 307 can generate reordered data sub-blocks by changing (e.g., by shuffling or shifting) the order (e.g., in processing order or time) of the data portions in each sub-block to reduce coincidences of peak densities in the processing order of data portions across multiple processing units, such as processing cores 306A-D.

[0063] In another example, the sequencer 307 can generate reordered data sub-blocks by retrieving data from different density bins in a particular order. For example, the sequencer 307 can reorder the data sub-blocks by extracting data from the density bins in a cyclic order of data densities, as described in more detail with respect to Figure 6 This likewise reduces coincidences of peak densities in the processing order across multiple processing units, such as processing cores 306A-D.

[0064] The sequencer 307 can be configured to not only reduce coincidences of peak density data portions, but also to reduce the average difference in data density (or sparsity) between consecutive data portions (e.g., along the processing order). In one example, this can be performed by reordering the individual data portions in an increasing or decreasing density order, such that the density of data portion x i is x i–1 < x i < x i+1 or x i–1x i x i+1 This density (or sparsity) based ordering beneficially reduces the magnitude of processing power variations (e.g., in current consumption) between data portions of different densities.

[0065] In the depicted embodiment, sequencer 307 provides reordered data sub-blocks to processing cores 306A-D, respectively, which are described in further detail with reference to Figure 4 As further defined with respect to Figure 6 In other examples, each processing core (e.g., 306A-D) can be assigned a default density based order and configured to read data portions based on the default density based order.

[0066] In some examples, after processing, sequencer 307 can receive the processed data and reorder it to match the original sequence order. In this way, density determination and ordering can be implemented without affecting existing parallel data processing coordination functionality. However, in other examples, if the order of data processing results is not important to the overall data processing results, processing cores 306A-D can provide processed data directly to processing coordinator 304.

[0067] Examples of density analyzer (305) and / or sequencer (307) can be implemented in the form of hardware logic and / or software logic. For example, in some examples, as described in further detail below with respect to Figure 8 Density analyzer (305) and sequencer (307) are implemented as part of DMA, including hardware logic and low-level firmware, in some examples.

[0068] Although shown as separate aspects in the present example, in alternative examples, the functionality of density analyzer 305 and sequencer 307 can be combined in a single element (such as in processing coordinator 304, or in another aspect of processing system 310). For clarity, these different functionalities are shown separately in Figure 3 .

[0069] Example of dynamic reordering of data for parallel processing

[0070] Figure 4 An example of dynamically reordering data for parallel processing to reduce power effects is depicted.

[0071] In the depicted example, multi-dimensional data block 302 is provided by an application (such as described above in Figure 3The data block 302 is described in more detail below. The data block 302 is shown as a two-dimensional block of data, but in other examples, the data block 302 can be a multi-dimensional block of data (e.g., three-dimensional, four-dimensional, etc.) for parallel data processing. Again, the data block 302 has portions (e.g., 302A-D) of varying data density (or sparsity). In this example, portion 302B is the darkest shaded portion, and thus the most dense data portion, while portions 302A and 202C are one shade lighter, and thus less dense than portion 302B, but more dense than portion 302D. Data portion 302D is the lightest shaded and least dense (or most sparse) data portion in the data block 202.

[0072] Notably, in this example, the density of each portion 302A-D is uniform, but in other examples, the density across one or more dimensions of the multi-dimensional data block 302 can not be uniform. For example, the data block 302 can represent image data, and each of the data portions 302A-D can represent a different channel (or layer) in the image with its own measurable density. As described above, the density can be measured by the number of null or zero entries in a portion of data, such as an image data slice having dimensions width (w) x height (h) x depth (d).

[0073] The data processing coordinator 304 receives the data block 302 and generates a plurality of data sub-blocks for parallel processing, which are then provided to the density analyzer 305. The density analyzer 305 analyzes the data sub-blocks and, for example, identifies portions of the data sub-blocks of different densities. In one example, these portions can correspond to a plurality of predetermined density ranges.

[0074] The density analyzer 305 then provides the data sub-blocks and density data to the sequencer 307, which dynamically reorders the data sub-blocks based on the determined density data.

[0075] In Figure 4In the example depicted in FIG. 4, the original relative order of the data portions (302A-D) is maintained in the reordered data sub-blocks 404A-D when considered in a circular fashion, but the processing time of each data portion is shifted by the reordering in each data sub-block. Specifically, in this example, the processing order of the data portions in data sub-block 404A is the same as in the original data block 302, while the processing order of the data portions in each subsequent data sub-block is incremented by one position, or in other words, incremented by one position for data sub-block 404B, incremented by two positions for data sub-block 404C, and incremented by three positions for data sub-block 404D. As depicted, when a data portion reaches the end of a sub-block, its next increment is circularly back to the start position. This reordering of data portions along the processing order of a data sub-block can be generally referred to as processing time reordering. The processing time reordering can have a defined stride, which defines how many positions a particular portion is shifted. In the depicted example, the stride is 1.

[0076] Figure 4 A beneficial result of the processing time reordering in FIG. 4 is that the reordered data sub-blocks 404A-D have different densities of data portions in different time periods of the processing order 406. The peak density data portions (and thus peak processing power) are distributed across different time periods for different data sub-blocks, which beneficially reduces the peak total processing system power compared to the graph 206 in FIG. 2. Figure 2 Specifically, the processing time reordering in this example results in the most dense data portions being processed in the second time period for data sub-block 404A, in the third time period for data sub-block 404B, in the fourth time period for data sub-block 404C, and in the first time period in data sub-block 404D (which has been rotated back to the first segment of the processing order). Moreover, the total system processing capacity across time periods 1-4 remains consistent in graph 408 compared to having a significant peak in graph 206 in FIG. 2. Figure 2

[0077] More generally, the processing time reordering generates multiple processing orders for different data portions in the data sub-blocks 404A-D. In this example, each data sub-block 404A-D has a different processing order for the data portions, but in other examples, the sub-blocks can have more data portions per sub-block, in which case certain processing orders can be repeated (such as the one depicted in FIG. 4). Figure 6

[0078] ​​Notably, while the number of data portions is the same as the number of data sub-blocks in this example, this need not be the case and is used in this example for ease of explanation. Further, while the number of data sub-blocks (404A-D) matches the number of processing cores (306A-D) in this example, this need not be the case. In some processing systems, the number of available cores can be dynamic, and the process can be performed according to dynamically available processing resources.

[0079] Although not depicted in Figure 4 , in other examples, the data portions of sub-blocks 404A-D can be further reordered according to a decreasing or increasing density order. In other words, the first data sub-block can be reordered such that the first data portion (at the beginning of the processing order) has the highest density, and each subsequent data portion has a lower density than the preceding data portion (i.e., in a decreasing density order). The subsequent data sub-blocks can then be reordered by time as described above. Alternatively, the first data sub-block can be reordered such that the first data portion has the lowest density, and each subsequent data portion has a higher density than the preceding data portion (i.e., in an increasing density order). This can be generally referred to as a density order reordering, and can beneficially reduce the variation in processing power between adjacent data portions. Once the data portions of the first data sub-block are ordered by density, the processing time ordering can be applied as described above.

[0080] Example aspects for dynamic reordering of data for parallel processing

[0081] Figure 5 An example method 500 of dynamically reordering data for parallel processing is depicted. Method 500 can be performed, for example, by a processing system such as processing system 310 of Figure 3 and 804 of Figure 8 . As described further herein, in some embodiments, aspects of method 500 can be performed by a DMA controller, as described with respect to Figure 8 .

[0082] Method 500 begins, at step 502, by receiving data for parallel processing from a data processing requestor, such as described with respect to Figure 3 , 4 , and 6. In various embodiments, the data processing requestor can be a local application, a remote application, a local processing subsystem (such as another processor in the same processing system), a local or remote process, etc.

[0083] Method 500 then proceeds to step 504, where a plurality of data sub-blocks are generated, such as described with respect to Figure 3 , 4As described in section 6. As mentioned above, in some respects, each data sub-block represents a subset of data to be processed in the same way, while in other respects, each sub-block is a set of copies of data to be processed in a different way.

[0084] Method 500 then proceeds to step 506, where multiple data portions are determined within each of the plurality of data sub-blocks. As described above, the data portions can be determined by referring to multiple densities or density ranges (such as by binning based on density).

[0085] In some aspects, determining multiple data portions within each of the plurality of data sub-blocks includes: determining a plurality of density ranges, wherein each of the plurality of density ranges is different from each of the other density ranges; and determining the plurality of data portions based on the plurality of density ranges.

[0086] Method 500 then proceeds to step 508, wherein the order of multiple data portions within at least one of the plurality of data sub-blocks is changed (or reordered). In this example, changing the order of data portions within a data sub-block results in a reordered data sub-block.

[0087] In some aspects, changing the order of multiple data portions of at least one of the multiple data sub-blocks includes: time-shifting each data portion of the multiple data portions in a cyclic order along the processing time direction, such as regarding Figure 4 and 6 As described.

[0088] In some aspects, changing the order of multiple data portions of at least one of the multiple data sub-blocks includes arranging the multiple data portions according to one of an ascending density order or a descending density order, such as regarding Figure 4 and 6 As described.

[0089] Method 500 then proceeds to step 510, wherein the plurality of data sub-blocks are provided to a plurality of processing units for parallel processing, the plurality of data sub-blocks including at least one data sub-block containing a plurality of data portions in a changed order.

[0090] Method 500 then proceeds to step 512, wherein processed data associated with the plurality of data sub-blocks is received from the plurality of processing units.

[0091] Method 500 then proceeds to step 514, in which the processed data is provided to the data processing requester.

[0092] In some aspects, the method 500 further includes changing an order of the processed data associated with the plurality of data sub-blocks based on an original order of the plurality of data portions in each data sub-block of the plurality of data sub-blocks prior to providing the processed data to the data processing requestor.

[0093] In some aspects, the method 500 further includes changing an order of the plurality of data portions in each data sub-block of the plurality of data sub-blocks.

[0094] In some aspects, determining the plurality of data portions in each data sub-block of the plurality of data sub-blocks is performed by a direct memory access (DMA) controller in data communication with the plurality of processing units, such as described with respect to Figure 8 .

[0095] In some aspects, changing the order of the plurality of data portions in at least one data sub-block of the plurality of data sub-blocks is performed by the DMA controller, such as described with respect to Figure 8 .

[0096] In some aspects, the data for parallel processing includes machine learning data associated with a machine learning model.

[0097] In some aspects, the plurality of processing units includes at least one neural signal processing unit or AI acceleration unit, such as described with respect to Figure 16 .

[0098] In some aspects, the method 500 can be used to perform neural network convolution processing across many neural processing units (NPUs) (or AI acceleration cores). Similarly, the method 500 can be used to perform neural network convolution processing across many arithmetic logic units (ALUs) within a GPU or many GPUs or across many CPUs. For example, the method 500 can be used to perform general image processing across many ALUs within a GPU or many GPUs.

[0099] Example of dynamic reordering of data for parallel processing based on density binning

[0100] Figure 6 Another example is depicted of dynamic reordering of data for parallel processing to reduce power effects.

[0101] In particular, Figure 6 A processing coordinator 604 is depicted that includes a density analyzer 606 and a sequencer 610. In this example, the processing coordinator 604 receives a multi-dimensional data block 602, which in some aspects can be a tensor. The processing coordinator 604 then uses the density analyzer 606 to examine the data block 602.

[0102] The density analyzer 606 identifies portions of the data block 602 having densities within predefined ranges associated with the density bins 608A-C. The density analyzer 606 then places the identified portions of the data block 602 (including 602A-D) into the bins 608A-C based on the determined densities. Generally, the multiple density bins will cover the entire range of possible densities, such that all data portions can be assigned to a unique bin.

[0103] As depicted in this example, data portion 602B is the highest density data portion and is stored in bin 608A based on its determined density falling into the density range associated with bin 608A (e.g., density > 66%); data portion 602A is the middle density data portion and is stored in bin 608B based on its determined density falling into the density range associated with bin 608B (e.g., 33% < density < 66%); and data portions 602C and 602D are low density data portions stored in bin 608C based on their determined densities falling into the density range associated with bin 608C (e.g., density < 33%).

[0104] In this example, the sequencer 610 then generates reordered data sub-blocks 612A-D by extracting data portions from the density bins (608A-C) in a defined order that is modified (e.g., incremented) for each data sub-block.

[0105] In the depicted example, the sequencer 610 uses the density bin sequence map 611 to map each processing core to a bin sequence (or order) in a circular fashion. For example, core 1 (616A) is mapped to sequence 1-2-3 (i.e., bins 608A-608B-608C), core 2 (616B) is mapped to sequence 3-1-2 (i.e., bins 608C-608A-608B), and so on. In this example, the bin sequence is incremented by one bin for each successive processing core in a circular order. Further, in this example, the fourth processing core repeats the sequence because there are only three bins. In other aspects, there can be more or fewer bins and more or fewer processing cores.

[0106] In other examples, the processing cores 616A-D can be configured to read data portions from the density bins 608A-C in a predetermined bin sequence order, such as shown in 611, without the action by the sequencer 610. In such examples, the predetermined density bin ordering assignment can be configured at installation, e.g., in a processing core firmware image.

[0107] After processing by the processing cores 616A-D, the processed data can be reordered and recombined to form a processed data output. In some examples, a bin sequence map can be used to reorder the processed data to match the original sequence of the data input. In this way, the data processing requestor need not be modified to receive the processed data.

[0108] Example method of dynamically reordering data for parallel processing using density binning

[0109] Figure 7 An example method 700 of dynamically reordering data for parallel processing using density binning is depicted.

[0110] The method 700 begins, at step 702, by receiving data for parallel processing from a data processing requestor.

[0111] The method 700 then proceeds to step 704, where a plurality of density ranges is determined, where each density range of the plurality of density ranges is different from each other density range of the plurality of density ranges.

[0112] The method 700 then proceeds to step 706, where a plurality of data portions of the data is determined based on the density ranges.

[0113] In some aspects, determining the plurality of data portions of the data based on the density ranges includes determining a density for each respective data portion of the plurality of data portions, and assigning the respective data portion to one of a plurality of density bins based on the density determined for the respective data portion of the plurality of data portions, where each density bin is associated with one of the plurality of density ranges, such as described above with respect to Figure 6

[0114] In some aspects, determining the plurality of data portions of the data based on the density ranges is performed by a direct memory access (DMA) controller in data communication with a plurality of processing units, such as described above with respect to Figure 8 The benefit of performing density (or sparsity) calculations in DMA is that it hides the associated computation latency, which can be done in parallel with moving and buffering the data of each data portion.

[0115] The method 700 then proceeds to step 708, where a plurality of data sub-blocks having reordered data portions is generated.

[0116] In some aspects, generating the plurality of data sub-blocks having reordered data portions includes adding data portions to the plurality of data sub-blocks according to a plurality of bin sequences, where each bin sequence of the plurality of bin sequences includes a unique sequence of the plurality of density bins, such as described above with respect to Figure 6 ​Described.

[0117] In some aspects, generating the plurality of data subchunks having reordered data portions is performed by a DMA controller, such as described with respect to Figure 8 Described.

[0118] Method 700 then proceeds to step 710, where the plurality of data subchunks is provided to a plurality of processing units for parallel processing.

[0119] In some aspects, providing the plurality of data subchunks to the plurality of processing units for parallel processing is performed by a DMA controller, such as described with respect to Figure 8 Described.

[0120] Method 700 then proceeds to step 712, where processed data associated with the plurality of data subchunks is received from the plurality of processing units.

[0121] In some aspects, receiving the processed data associated with the plurality of data subchunks from the plurality of processing units is performed by a DMA controller, such as described with respect to Figure 8 Described.

[0122] Method 700 then proceeds to step 714, where the processed data is provided to a data processing requestor. In various embodiments, the data processing requestor can be a local application (such as described with respect to Figure 3 and 6 a remote application, a local processing subsystem (such as another processor in the same processing system), a local or remote process, etc.

[0123] In some aspects, method 700 further includes assigning each bin sequence of the plurality of bin sequences to one or more processing units of the plurality of processing units, where each respective data subchunk of the plurality of data subchunks is provided to one processing unit of the plurality of processing units based on the assigned bin sequence of the respective data subchunk.

[0124] In some aspects, method 700 further includes determining the plurality of bin sequences according to one of an increasing density order or a decreasing density order and a processing order time shift.

[0125] In some aspects, the data for parallel processing is machine learning model input data for a convolutional neural network model.

[0126] In some aspects, each data portion of the plurality of data portions comprises a feature map. In some embodiments, a feature map can represent the output of one filter applied to a previous layer of a neural network, such as a convolutional neural network. In some cases, a feature map can be referred to as an intermediate feature map, which can represent output from an intermediate layer in a deep neural network, such as a convolutional neural network.

[0127] In some aspects, the method 700 further comprises reordering a convolution kernel (or filter) for each data sub-block of the plurality of data sub-blocks based on the sequence of data portions in the respective data sub-block; and applying the reordered convolution kernel to the respective data sub-block, such as described with respect to Figure 15 Described. In various embodiments, a convolution kernel or filter is a set of trainable values or parameters that are applied to data processed by layers of a convolutional neural network model to generate layer output data, such as feature maps. Such as described with respect to Figure 15 A convolution kernel as depicted in FIG. 8 can comprise multiple layers, which can be reordered to match reordered input data, such as reordered input feature maps.

[0128] In some aspects, the plurality of processing units comprises at least one neural signal processing unit or AI acceleration unit, such as described with respect to Figure 16 Described.

[0129] In some aspects, the method 700 can be used to perform neural network convolution processing across many neural signal processing units (NPUs) (or AI acceleration cores). Similarly, the method 700 can be used to perform neural network convolution processing across many arithmetic logic units (ALUs) within a GPU or many GPUs or across many CPUs. For example, the method 700 can be used to perform general image processing across many ALUs within a GPU or many GPUs.

[0130] Example system for parallel processing using density-based reordering

[0131] Figure 8 An example processing system 804 configured to perform parallel processing using density-based reordering is depicted, as discussed herein.

[0132] In this example, the processing system 804 receives data from the application 802 to be processed in parallel using a plurality of processing units, including neural signal processing units (NPUs) 812A-F, which can alternatively be referred to as AI acceleration cores or units, as described in further detail below. Notably, in other examples, other numbers and / or types of processing units can be used in addition to or instead of the NPUs 812A-F, such as described with respect to Figure 8Various types of processing units are described.

[0133] Data from the application 802 is received by a direct memory access (DMA) controller 806, which in this example includes a density analyzer 808 and a sequencer 810, such as those described above. Notably, in other examples, one or more of the density analyzer 808 and the sequencer 810 can be separate from the DMA controller 806. Further, as shown above, the density analyzer 808 and the sequencer 810 can be part of a processing coordinator (not shown). Alternatively, a separate processing coordinator can be provided and function as described herein. Figure 6

[0134] In other examples, a default density bin read order that can be followed for each sub-block can be assigned for each NPU, eliminating the need for the sequencer 810 to perform this function. For example, the density bin ordering assignment can be defined at installation time in, for example, an NPU firmware image. However, in some embodiments, the sequencer 810 can be used to overwrite the default ordering.

[0135] In this example, the DMA controller 806 performs the density analysis and reordering functions (as described above) and provides reordered data sub-blocks (not shown) to one or more of the NPUs 812A-F for parallel processing. The reordered data sub-blocks can be stored, for example, in memory associated with each NPU, such as tightly coupled memory (TCM) 814A-F. Generally, TCM provides low latency memory that a processor can use without the unpredictable characteristics of a cache. For example, TCM can be used to hold critical routines, such as interrupt handling routines or real-time tasks for which caching uncertainty is undesirable.

[0136] In one example, the processing system 804 can be configured to perform neural network convolution processing across the NPUs 812A-F.

[0137] Notably, Figure 8 This is merely one example of certain aspects of a processing system, and other aspects can exist and other configurations are possible.

[0138] Example parallel processing of neural network data using density-based reordering

[0139] ​Parallel processing of machine learning data, such as for neural networks, places high demands on processing systems. A number of factors can result in increased peak current consumption and increased current ramping over a short period of time, which is detrimental to processing system performance, as described above. For example, processing systems for machine learning data can generally operate at high frequencies in order to quickly process the various instructions and threads associated with machine learning operations. Moreover, machine learning data can generally be dense.

[0140] To illustrate the impact of dense machine learning data on processing capabilities, Figure 9 A graph 900 depicting an example relationship between density in convolutional neural network data and power consumption of processing units is depicted. As depicted, the amount of processing capability used generally increases as the density of the convolutional neural network data increases (i.e., as the sparsity decreases). Moreover, as the data size and computational complexity increase, the increased power consumption as the density increases further increases (e.g., floating point multiply accumulate consumes more power than integer multiply accumulate). In this particular example, 16-bit floating point convolution data processing consumes 40% more power than 8-bit integer convolution data processing at full density, such as in a saturated tensor.

[0141] Figure 10 An example of parallel processing of multi-dimensional data sub-blocks 1002A-D is depicted. In this example, each data sub-block 1002A-D includes a plurality of feature maps that are processed with convolution kernels 1004A-D (e.g., in a convolutional layer of a neural network). In this example, the density of the data sub-blocks 1002A-D varies along the various feature maps in each data sub-block, as depicted by the different shading of the feature maps along the convolution processing sequence 1010 of each sub-block, and thus each feature map can be considered a part of its respective data sub-block in this example.

[0142] A graph 1006 depicting power consumption of four processing units 1008A-D is depicted. Because the data sub-blocks 1002A-D each have denser portions (e.g., feature maps) at the same location along the convolution processing sequence 1010 of the data sub-blocks, the power consumption of each processing unit 1008A-D is simultaneously highest, resulting in a large increase in overall processing capability, as depicted at 1012. As described above, this large increase is detrimental to system performance because it can result in disruptive power effects, such as voltage droop (as described in more detail with respect to Figure 11A -C), which can subsequently impact other portions of the system (such as by causing throttling of processing operations).

[0143] Figure 11A An example graph depicting power consumption of a plurality of processing cores is depicted, similar to Figure 10 graph 1006 in

[0144] Figure 11B An example of total processing power consumption 1102 from cores 0-3 of Figure 11A is depicted. As depicted, total processing power generally varies over time, but reaches a maximum value of power spike 1105 at time increment 4 in Figure 11A . Moreover, the current variation 1104 (e.g., Δί) over time 1106 (e.g., Δί) results in a steep slope 1107 (e.g., Δί / Δί) of power consumption. This sudden change in power consumption causing a sudden change in current can result in destructive power effects, such as Figure 11C voltage droop depicted in

[0145] Figure 11C An example of system voltage 1008 over time is depicted, which corresponds in time to Figure 11A and 11B As depicted, a voltage droop 1112 occurs at time 4, which coincides with the power consumption spike 1105 in Figure 11B . Notably, the voltage droop 1112 takes the system voltage out of the stable system voltage band 1110, which can result in system instability and performance degrading actions by the system parts to maintain stability, such as frequency throttling.

[0146] Figure 12 An example of using density binning on machine learning data, such as feature maps depicted in Figure 10 is depicted. Specifically, bin 1 (1202A) includes an address pointer to feature map data 1204A, which has a density of 0-25%. Similarly, bin 2 (1202B) includes an address pointer to feature map data 1204B, which has a density of 26-50%; bin 3 (1202C) includes an address pointer to feature map data 1204C, which has a density of 51-75%; and bin 4 (1202D) includes an address pointer to feature map data 1204D, which has a density of 76-100%. The binning of address pointers to various feature map data of different densities can be performed by, for example, a density analyzer, such as described above (e.g., with respect to Figure 6 ).

[0147] Multiple processing units can then process the feature maps from the bins in a power efficient manner, such as by reducing the peak power slope (Δί / Δί). For example, a first processing unit can process one feature map from each bin (e.g., 1202A) before processing a feature map from the next bin (e.g., 1202B), such that the resulting processing order in this example is bin 4 (highest power), bin 3, bin 2, bin 1 (lowest power).

[0148] Further, as noted above, multiple processing units can stagger the bin processing start order to minimize the initial power ramp and aggregate current, which minimizes the total processing power peak and leakage power. Using the example of Figure 12 , the processing start order for four processing units can be: processing unit 1 : bin 4, bin 3, bin 2, bin 1 ; processing unit 2: bin 3, bin 2, bin 1, bin 4; processing unit 3: bin 2, bin 1, bin 4, bin 3; processing unit 4: bin 1, bin 4, bin 3, bin 2. Notably, these orders are just one example, and other examples are possible.

[0149] Once the parallel processing units start bin processing, the next binning operation can use, for example, a circular buffer to pipeline the current processing unit processing.

[0150] Figure 13 A comparison of the conventional processing of machine learning data sub-blocks 1002A-D with kernels 1004A-D versus the density reordering processing of machine learning data sub-blocks 1302A-D with kernels 1304A-D using processing units 1308A-D is depicted. Notably, this is an example where each data sub-block 1002A-D and 1302A-D has the same data operated on by different kernels.

[0151] As depicted, the data density distribution in sub-blocks 1002A-D is uniform, which results in the power spike of the most dense (darkest shade) data portion as described above. In contrast, the density distribution in data sub-blocks 1302A-D is varied due to its density-based reordering, which yields the benefits described above and with respect to Figure 14A -C described further. In particular, in this example, data sub-blocks 1302A-D have applied both temporal reordering and density order reordering.

[0152] Figure 14A An example graph depicting the power consumption of multiple processing cores processing data sub-blocks 1302A-D in Figure 13 is depicted. Notably, the peak power for each core is distributed as a result of the most dense portion of the distributed processed data, as depicted in data sub-blocks 1302A-D in Figure 13 . This is in contrast to the concentration of peak processing power in Figure 10 , which is based on the density distribution in data sub-blocks 1002A-D. Figure 11A

[0153] Figure 14B An example of the total processing power consumption 1402 from cores 0-3 of Figure 14A is depicted. As depicted, the total processing power again varies over time, but in Figure 11B ​The power spike depicted at 1105 in FIG. 11 is eliminated. In addition, the current change 1404 (e.g., Δί) as a function of time 1406 (e.g., Δί) results in a shallower power ramp 1407 (e.g., Δί / Δί) of power consumption. These improved power consumption characteristics avoid the above-described disruptive power effects, such as voltage droop. Figure 11C

[0154] Figure 14C An example of a system voltage as a function of time 1408 is depicted, which corresponds in time to Figure 14A and 14B As depicted, there is no significant voltage droop and the system voltage remains within a safe operating band 1410 as compared to the voltage droop 1112 depicted in FIG. 11. Figure 11C Thus, the density-based reordered order avoids system instability and performance degrading actions of system components when processing the same data.

[0155] Figure 15 Useful properties of certain machine learning operations when using density reordering of input data are depicted.

[0156] As depicted, applying the original sequence of feature maps 1502A to a kernel 1504A results in an output feature map 1506. Similarly, a reordered kernel 1504B can be applied to a reordered (e.g., time and / or density reordered) input feature map 1502B to obtain the same output feature map 1506. Thus, the mathematical results after reordering are the same as those before and no independent reordering of the output feature map is needed in this scenario. In addition, the same mathematical results are achieved with improved system performance, as described above.

[0157] Example electronic device for parallel data processing with data reordering

[0158] Figure 16 An example electronic device 1600 that can be configured to perform parallel data processing with data reordering as described herein, e.g., with respect to Figure 5 and 7 is depicted.

[0159] The electronic device 1600 includes a central processing unit (CPU) 1602, which in some examples can be a multi-core CPU. Instructions executed at the CPU 1602 can be loaded, e.g., from a program memory associated with the CPU.

[0160] ​The electronic device 1600 also includes additional processing components customized for particular functions, such as a graphics processing unit (GPU) 1604, a digital signal processor (DSP) 1606, a neural signal processing unit (NPU) 1608, a multimedia processing unit 1610, and a wireless connectivity component 1612.

[0161] An NPU, such as 1608, is generally a specialized circuit configured to implement all the necessary control and arithmetic logic for performing machine learning algorithms, such as algorithms for processing artificial neural networks (ANN), deep neural networks (DNN), random forests (RF), and the like. An NPU can sometimes be alternatively referred to as a neural signal processor (NSP), a tensor processing unit (TPU), a neural network processor (NNP), an intelligent processing unit (IPU), a vision processing unit (VPU), or a graphics processing unit.

[0162] An NPU, such as 1608, is configured to accelerate the performance of common machine learning tasks, such as image classification, machine translation, object detection, and various other predictive models. In some examples, multiple NPUs can be instantiated on a single chip, such as a system on a chip (SoC), while in other examples, multiple NPUs can be part of a specialized neural network accelerator.

[0163] An NPU can be optimized for training or inference, or in some cases can be configured to balance performance between training and inference. For NPUs capable of performing both training and inference, the two tasks can generally still be performed independently.

[0164] An NPU designed to accelerate training is generally configured to accelerate the optimization of a new model, which is a highly computation-intensive operation involving inputting an existing dataset (often labeled or tagged), iterating over the dataset, and then adjusting model parameters, such as weights and biases, in order to improve model performance. Generally, optimization based on error prediction involves passing back through the various layers of the model and determining gradients to reduce prediction error.

[0165] An NPU designed to accelerate inference is generally configured to operate on a complete model. Such an NPU can thus be configured to input a new piece of data and quickly process the data through a model that has already been trained to generate a model output (e.g., an inference).

[0166] In one implementation, the NPU 1608 is part of one or more of the CPU 1602, GPU 1604, and / or DSP 1606.

[0167] In some examples, the wireless connectivity components 1612 can include subcomponents, e.g., for third generation (3G) connectivity, fourth generation (4G) connectivity (e.g., 4G LTE), fifth generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The wireless connectivity processing components 1612 are further connected to one or more antennas 1614.

[0168] The electronic device 1600 can also include one or more sensor processing units 1616 associated with any manner of sensors, one or more image signal processors (ISPs) 1618 associated with any manner of image sensors, and / or a navigation processor 1620 that can include satellite-based positioning system components (e.g., GPS or GLONASS) and inertial positioning system components.

[0169] The electronic device 1600 can also include one or more input and / or output devices 1622, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, a speaker, a microphone, etc.

[0170] In some examples, the one or more processors of the electronic device 1600 can be based on an ARM or RISC-V instruction set.

[0171] The electronic device 1600 also includes memory 1624, which is representative of one or more static and / or dynamic memories, such as dynamic random access memory, static memory based on flash, etc. In this example, the memory 1624 includes computer-executable components that can be executed by one or more of the previously described processors of the electronic device 1600. Although not depicted in Figure 16 Although not depicted in FIG. 16, an electronic device can include a direct memory access (DMA) controller.

[0172] In particular, in this example, the memory 1624 includes a transmitting component 1624A, a receiving component 1624B, a processing component 1624C, a determining component 1624D, an outputting component 1624E, a sub-block component 1624F, a density estimation component 1624G, a sequence (or order) component 1624H, and a coordinating component 1624I. The depicted components, as well as other components not depicted, can be configured to perform aspects of the methods described herein.

[0173] In general, the electronic device 1600 and / or its components can be configured to perform the methods described herein.

[0174] Figure 17 An example multi-processor processing system 1700 is depicted that can be implemented using examples described herein. For example, the multi-processor system 1700 can be representative of the multi-processor system 1600 of FIG. 16. Figure 16processors of the electronic device 1600.

[0175] In this example, the system 1700 includes processors 1701, 1703, and 1705, although in other examples any number of individual processors can be used. Moreover, although depicted similarly, the processors 1701, 1703, and 1705 can represent various different kinds of processors in an electronic device, such as CPUs, GPUs, DSPs, NPUs, etc. as described herein.

[0176] Each of the processors 1701, 1703, and 1705 includes an instruction scheduler, various hardware subcomponents (e.g., hardware X, hardware Y, and hardware Z), and a local memory. In some examples, the local memory can be a tightly coupled memory (TCM). Note that although the components of each of the processors 1701, 1703, and 1705 are shown as being identical in this example, in other examples some or each of the processors 1701, 1703, and 1705 can have different hardware configurations, different hardware elements, etc.

[0177] Each of the processors 1701, 1703, and 1705 is also in data communication with a global memory, such as a DDR memory, or other type of volatile working memory. For example, the global memory 1707 can represent a memory that is shared by the processors 1701, 1703, and 1705. In some examples, the global memory 1707 can be a system memory of an electronic device, such as the memory 1624 of the electronic device 1600. Figure 16

[0178] In some implementations, in a multi-processor processing system, such as 1700, one of the processors can act as a master processor. For example, in this example, the processor 1701 can be the master processor. The master processor can include a compiler that, when executed, can determine how a model, such as a neural network, is to be processed by the various components of the processing system 1700. For example, hardware parallelism can be achieved by mapping processing portions of the model to various hardware (e.g., hardware X, hardware Y, and hardware Z) within a given processor (e.g., processor 1701), as well as mapping processing portions of the model to other processors (e.g., processors 1703 and 1705) and their associated hardware. For example, the various parallel blocks in the parallel block processing architecture described herein can be mapped to different portions of various hardware in the processors 1701, 1703, and 1705.

[0179] Example

[0180] ​Example 1 : A method for performing parallel data processing, comprising: receiving data for parallel processing from a data processing requestor; generating a plurality of data sub-blocks; determining a plurality of data portions in each data sub-block of the plurality of data sub-blocks; altering an order of the plurality of data portions in at least one data sub-block of the plurality of data sub-blocks; providing the plurality of data sub-blocks to a plurality of processing units for parallel processing, the plurality of data sub-blocks including the at least one data sub-block including the plurality of data portions in the altered order; and receiving processed data associated with the plurality of data sub-blocks from the plurality of processing units.

[0181] Example 2: The method of example 1, wherein determining the plurality of data portions in each data sub-block of the plurality of data sub-blocks comprises: determining a plurality of density ranges, wherein each density range of the plurality of density ranges is different from each other density range of the plurality of density ranges; and determining the plurality of data portions based on the plurality of density ranges.

[0182] Example 3: The method of any one of examples 1 or 2, wherein altering the order of the plurality of data portions in at least one data sub-block of the plurality of data sub-blocks comprises: time-shifting each data portion of the plurality of data portions in a processing time direction in a circular order.

[0183] Example 4: The method of any one of examples 1-3, wherein altering the order of the plurality of data portions in at least one data sub-block of the plurality of data sub-blocks comprises: arranging the plurality of data portions according to one of an increasing density order or a decreasing density order.

[0184] Example 5: The method of any one of examples 1-4, further comprising: altering an order of the processed data associated with the plurality of data sub-blocks based on an original order of the plurality of data portions in each data sub-block of the plurality of data sub-blocks; and providing the processed data to the data processing requestor.

[0185] Example 6: The method of any one of examples 1-5, further comprising: altering the order of the plurality of data portions in each data sub-block of the plurality of data sub-blocks.

[0186] Example 7: The method of any one of examples 1-6, wherein determining the plurality of data portions in each data sub-block of the plurality of data sub-blocks is performed by a direct memory access (DMA) controller in data communication with the plurality of processing units.

[0187] Example 8: The method of example 7, wherein altering the order of the plurality of data portions in at least one data sub-block of the plurality of data sub-blocks is performed by the DMA controller.

[0188] Example 9: The method of any of examples 1-8, wherein changing the order of the plurality of data portions in the at least one data sub-block of the plurality of data sub-blocks comprises reading the plurality of data portions by the plurality of processing units in a predefined order.

[0189] Example 10: The method of any of examples 1-9, wherein the plurality of processing units comprises at least one neural signal processing unit.

[0190] Example 11: A processing system for performing parallel data processing, comprising: a memory comprising computer-executable instructions; one or more processors configured to execute the computer-executable instructions and cause the processing system to perform the method of any of examples 1-10.

[0191] Example 12: A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of a processing system, cause the processing system to perform the method of any of examples 1-10.

[0192] Example 23: A computer program product embodied on a computer-readable storage medium comprising code for performing the method of any of examples 1-10.

[0193] Example 14: A method for performing parallel data processing, comprising: receiving data for parallel processing from a data processing requestor; determining a plurality of density ranges, wherein each density range of the plurality of density ranges is different from each other density range of the plurality of density ranges; determining a plurality of data portions of the data based on the density ranges; generating a plurality of data sub-blocks having reordered data portions; providing the plurality of data sub-blocks to a plurality of processing units for parallel processing; and receiving processed data associated with the plurality of data sub-blocks from the plurality of processing units.

[0194] Example 15: The method of example 14, wherein determining the plurality of data portions of the data based on the density ranges comprises: determining a density for each respective data portion of the plurality of data portions; and assigning the respective data portion to one of a plurality of density bins based on the determined density for the respective data portion of the plurality of data portions, wherein each density bin is associated with one of the plurality of density ranges.

[0195] Example 16: The method of example 15, wherein generating the plurality of data sub-blocks having reordered data portions comprises: adding data portions to the plurality of data sub-blocks according to a plurality of bin sequences, wherein each bin sequence of the plurality of bin sequences comprises a unique sequence of the plurality of density bins.

[0196] Example 17: The method of example 16, further comprising: assigning each bin sequence of the plurality of bin sequences to one or more processing units of the plurality of processing units, wherein each respective data sub-block of the plurality of data sub-blocks is provided to one processing unit of the plurality of processing units based on the assigned bin sequence of the respective data sub-block.

[0197] Example 18: The method of example 16, further comprising: determining the plurality of bin sequences according to one of an increasing density order or a decreasing density order and a processing order time shift.

[0198] Example 19: The method of any of examples 14-18, further comprising: providing processed data to a data processing requestor.

[0199] Example 20: The method of any of examples 14-19, wherein: the data for parallel processing is machine learning model input data for a convolutional neural network model, each data portion of the plurality of data portions comprises a feature map, and the method further comprises: reordering a convolutional kernel for each data sub-block of the plurality of data sub-blocks based on a sequence of data portions in the data sub-block; and applying the reordered convolutional kernel to the respective data sub-block.

[0200] Example 21 : The method of any of examples 14-20, wherein determining the plurality of data portions of the data based on the density range is performed by a direct memory access (DMA) controller in data communication with the plurality of processing units.

[0201] Example 22: The method of example 21, wherein generating the plurality of data sub-blocks with reordered data portions is performed by the DMA controller.

[0202] Example 23: The method of any of examples 14-20, wherein the plurality of processing units comprises at least one neural signal processing unit.

[0203] Example 24: A processing system for performing parallel data processing, comprising: a memory comprising computer-executable instructions; one or more processors configured to execute the computer-executable instructions and cause the processing system to perform the method of any of examples 14-23.

[0204] Example 25: A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of a processing system, cause the processing system to perform the method of any of examples 14-23.

[0205] Example 26: A computer program product embodied on a computer- readable storage medium comprising code for performing a method according to any of examples 14-23.

[0206] Additional Considerations

[0207] The preceding description is provided to enable any person skilled in the art to practice the various examples described herein. The examples discussed are not meant to be limiting of the scope, applicability, or examples set forth in the claims. Various modifications will come to mind to those skilled in the art from the teachings hereof, and the generic principles described herein can be applied to other examples. For example, the functionality and arrangement of the described elements can be altered to perform the same or similar functions, or additional functions. The various examples can omit, substitute, or add various procedures or components as appropriate. For instance, the methods described can be performed in an order different than that described, and other steps can be added, omitted, or combined. Also, features described with respect to some examples can be combined in other examples. For example, an apparatus or method can be implemented using any number of aspects described herein. In addition, the scope of the disclosure is intended to cover any patent, patent application, or print publication, including "translated" versions thereof, that falls within or relates to a claim of the disclosure. Any aspect of the disclosure can be implemented by one or more elements of a claim.

[0208] As used herein, the term "exemplary" means "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects.

[0209] As used herein, the phrase "at least one of" a list of items refers to any combination of those items, including single members. As an example, "at least one of a, b, or c" is intended to cover a, b, c, a-b, a-c, b-c, and a-b-c, as well as any combination with multiples of the same element (e.g., a-a, a-a-a, a-a-b, a-a-c, a-b-b, a-c-c, b-b, b-b-b, b-b-c, c-c, and c-c-c or any other ordering of a, b, and c).

[0210] As used herein, the term "determining" encompasses a wide variety of actions. For example, "determining" can include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, a database or another data structure), ascertaining and the like. Also, "determining" can include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory) and the like. Also, "determining" can include resolving, selecting, choosing, establishing and the like.

[0211] The methods disclosed herein comprise one or more steps or actions for achieving the methods. The method steps and / or actions can be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions can be modified without departing from the scope of the claims. Further, the various operations of methods described above can be performed by any suitable means capable of performing the corresponding functions. These means can include various hardware and / or software component(s) and / or module(s), including, but not limited to a circuit, an application specific integrated circuit (ASIC), or processor. Generally, where there are operations illustrated in figures, those operations can have corresponding counterpart means-plus-function components with similar numbering.

[0212] The following claims are not intended to be limited to the examples shown herein but are to be afforded all ranges consistent with the language of the claims. Within the claims, the reference to an element in the singular is not intended to mean "one and only one" unless specifically so stated, but rather "one or more." Unless specifically stated otherwise, the term "some" refers to one or more. No claim element is to be construed as a means plus function unless the element is expressly recited using the phrase "means for." The various aspects described throughout this disclosure can be all combinations of the elements recited, in any order, unless otherwise stated. All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are known or later come to be known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the claims. Moreover, nothing disclosed herein is intended to be dedicated to the public regardless of whether these disclosure are explicitly recited in the claims.

Claims

1. A method for performing parallel data processing, comprising: receiving data from a data processing requestor for parallel processing; generating a plurality of data sub-blocks based on the data; determining a plurality of data portions in each data sub-block of the plurality of data sub-blocks; determining a density of each data portion of the plurality of data portions in each data sub-block of the plurality of data sub-blocks; changing an order of the plurality of data portions in at least one data sub-block of the plurality of data sub-blocks based on the determined density of each data portion; providing the plurality of data sub-blocks to a plurality of processing units for parallel processing, the plurality of data sub-blocks including the at least one data sub-block including the plurality of data portions in the changed order; and receiving processed data associated with the plurality of data sub-blocks from the plurality of processing units.

2. The method of claim 1, wherein determining a plurality of data portions in each data sub-block of the plurality of data sub-blocks comprises: determining a plurality of density ranges, wherein each density range of the plurality of density ranges is different from each other density range of the plurality of density ranges; and determining the plurality of data portions based on the plurality of density ranges.

3. The method of claim 1, wherein changing the order of the plurality of data portions of the at least one of the plurality of data sub-blocks comprises: time shifting each data portion of the plurality of data portions in a loop order along a processing time direction.

4. The method of claim 1, wherein changing the order of the plurality of data portions in the at least one data sub-block of the plurality of data sub-blocks comprises: arranging the plurality of data portions according to one of an increasing density order or a decreasing density order.

5. The method of claim 1, further comprising: changing an order of the processed data associated with the plurality of data sub-blocks based on an original order of the plurality of data portions in each data sub-block of the plurality of data sub-blocks; and providing the processed data to the data processing requestor.

6. The method of claim 1, further comprising: changing the order of the plurality of data portions in each data sub-block of the plurality of data sub-blocks.

7. The method of claim 1, wherein determining the plurality of data portions in each data sub-block of the plurality of data sub-blocks is performed by a direct memory access (DMA) controller in data communication with the plurality of processing units.

8. The method of claim 7, wherein changing the order of the plurality of data portions in the at least one data sub-block of the plurality of data sub-blocks is performed by the DMA controller.

9. The method of claim 1, wherein changing the order of the plurality of data portions in the at least one data sub-block of the plurality of data sub-blocks comprises reading the plurality of data portions by the plurality of processing units in a predefined order.

10. The method of claim 1, wherein the plurality of processing units comprises at least one neural signal processing unit.

11. A processing system for performing parallel data processing, comprising: a memory comprising computer-executable instructions; one or more processors configured to execute the computer-executable instructions and cause the processing system to: receive data from a data processing requestor for parallel processing; generating a plurality of data sub-blocks based on the data; determining a plurality of data portions in each data sub-block of the plurality of data sub-blocks; determining a density of each data portion of the plurality of data portions in each data sub-block of the plurality of data sub-blocks; changing an order of the plurality of data portions in at least one data sub-block of the plurality of data sub-blocks based on the determined density of each data portion; providing the plurality of data sub-blocks to a plurality of processing units for parallel processing, the plurality of data sub-blocks including the at least one data sub-block including the plurality of data portions in the changed order; and receiving processed data associated with the plurality of data sub-blocks from the plurality of processing units.

12. The processing system of claim 11, wherein to determine a plurality of data portions in each data sub-block of the plurality of data sub-blocks, the one or more processors are further configured to: determine a plurality of density ranges, wherein each density range of the plurality of density ranges is different from each other density range of the plurality of density ranges; and determine the plurality of data portions based on the plurality of density ranges.

13. The processing system of claim 11, wherein to change an order of the plurality of data portions of the at least one data sub-block of the plurality of data sub-blocks, the one or more processors are further configured to time-shift each data portion of the plurality of data portions in a processing time direction in a circular order.

14. The processing system of claim 11, wherein to change an order of the plurality of data portions of the at least one data sub-block of the plurality of data sub-blocks, the one or more processors are further configured to: arrange the plurality of data portions according to one of an increasing density order or a decreasing density order.

15. The processing system of claim 11, wherein the one or more processors are further configured to: change an order of the processed data associated with the plurality of data sub-blocks based on an original order of the plurality of data portions in each data sub-block of the plurality of data sub-blocks; provide the processed data to the data processing requestor.

16. The processing system of claim 11, wherein the one or more processors are further configured to change an order of the plurality of data portions in each data sub-block of the plurality of data sub-blocks.

17. The processing system of claim 11, further comprising: a direct memory access (DMA) controller in data communication with the plurality of processing units, the DMA controller configured to determine the plurality of data portions in each data sub-block of the plurality of data sub-blocks are performed.

18. The processing system of claim 17, wherein the DMA controller is further configured to change an order of the plurality of data portions in the at least one data sub-block of the plurality of data sub-blocks.

19. The processing system of claim 11, wherein to change an order of the plurality of data portions in the at least one data sub-block of the plurality of data sub-blocks, the one or more processors are configured to read the plurality of data portions in a predefined order.

20. The processing system of claim 11, wherein the plurality of processing units comprises at least one neural signal processing unit.

21. A method for performing parallel data processing, comprising: receiving data for parallel processing from a data processing requestor; determining a plurality of density ranges, wherein each density range of the plurality of density ranges is different from each other density range of the plurality of density ranges; determining a plurality of data portions of the data based on the density ranges; generating a plurality of data sub-blocks having reordered data portions based on the density determined for each respective data portion of the plurality of data portions; providing the plurality of data sub-blocks to a plurality of processing units for parallel processing; and receiving processed data associated with the plurality of data sub-blocks from the plurality of processing units.

22. The method of claim 21, wherein determining the plurality of data portions of the data based on the density ranges comprises: determining a density for each respective data portion of the plurality of data portions; and assigning the respective data portion to one density bin of a plurality of density bins based on the density determined for each respective data portion of the plurality of data portions, wherein each density bin is associated with one density range of the plurality of density ranges.

23. The method of claim 22, wherein generating the plurality of data sub-blocks having reordered data portions comprises: adding data portions to the plurality of data sub-blocks according to a plurality of bin sequences, wherein each bin sequence of the plurality of bin sequences comprises a unique sequence of the plurality of density bins.

24. The method of claim 23, further comprising: assigning each bin sequence of the plurality of bin sequences to one or more processing units of the plurality of processing units, wherein each respective data sub-block of the plurality of data sub-blocks is provided to one processing unit of the plurality of processing units based on the assigned bin sequence of the respective data sub-block. determining the plurality of bin sequences according to: one of an ascending density order or a descending density order; and 25. The method of claim 23, further comprising: a processing order time shift. providing the processed data to the data processing requestor.

27. The method of claim 21, wherein: the data for parallel processing is machine learning model input data for a convolutional neural network model, 26. The method of claim 21, further comprising: each data portion of the plurality of data portions comprises a feature map, and the method further comprises: reordering a convolution kernel for each respective data sub-block of the plurality of data sub-blocks based on a sequence of data portions in the respective data sub-block; and applying the reordered convolution kernel to the respective data sub-block.

28. The method of claim 21, wherein determining the plurality of data portions of the data based on the density ranges is performed by a direct memory access (DMA) controller in data communication with the plurality of processing units.

29. The method of claim 28, wherein generating the plurality of data sub-blocks having reordered data portions is performed by the DMA controller. ​ ​ ​ 30. The method of claim 21, wherein the plurality of processing units includes at least one neural signal processing unit.

31. A processing system for performing parallel data processing, comprising: a memory comprising computer-executable instructions; one or more processors configured to execute the computer-executable instructions and cause the processing system to: receive, from a data processing requestor, data for parallel processing; determine a plurality of density ranges, wherein each density range of the plurality of density ranges is different from each other density range of the plurality of density ranges; determine a plurality of data portions of the data based on the density ranges; generate a plurality of data subchunks having reordered data portions based on the density determined for each respective data portion of the plurality of data portions; provide the plurality of data subchunks to a plurality of processing units for parallel processing; and receive, from the plurality of processing units, processed data associated with the plurality of data subchunks.

32. The processing system of claim 31, wherein to determine the plurality of data portions of the data based on the density ranges, the one or more processors are further configured to: determine a density for each respective data portion of the plurality of data portions; and assign the respective data portion to one density bin of a plurality of density bins based on the density determined for each respective data portion of the plurality of data portions, wherein each density bin is associated with one density range of the plurality of density ranges.

33. The processing system of claim 32, wherein to generate the plurality of data subchunks having reordered data portions, the one or more processors are further configured to: add data portions to the plurality of data subchunks according to a plurality of bin sequences, wherein each bin sequence of the plurality of bin sequences comprises a unique sequence of the plurality of density bins.

34. The processing system of claim 33, wherein the one or more processors are further configured to: assign each bin sequence of the plurality of bin sequences to one or more processing units of the plurality of processing units, wherein each respective data subchunk of the plurality of data subchunks is provided to one processing unit of the plurality of processing units based on the assigned bin sequence of the respective data subchunk.

35. The processing system of claim 33, wherein the one or more processors are further configured to determine the plurality of bin sequences according to: one of an ascending density order or a descending density order; and a processing order time shift.

36. The processing system of claim 31, wherein the one or more processors are further configured to provide the processed data to the data processing requestor.

37. The processing system of claim 31, wherein: the data for parallel processing is machine learning model input data for a convolutional neural network model, each data portion of the plurality of data portions comprises a feature map, and the one or more processors are further configured to: ​ reordering a convolution kernel for each respective data sub-block of the plurality of data sub-blocks based on a sequence of data portions in the respective data sub-block; and applying the reordered convolution kernel to the respective data sub-block.

38. The processing system of claim 31, further comprising a direct memory access (DMA) controller in data communication with the plurality of processing units, the DMA controller configured to determine the plurality of data portions of the data based on the density range.

39. The processing system of claim 38, wherein the DMA controller is further configured to generate the plurality of data sub-blocks with reordered data portions.

40. The processing system of claim 31, wherein the plurality of processing units comprises at least one neural signal processing unit.

Images