Mixed precision neural processing unit using spatial fusion with load balancing

Abstract

A mixed-precision neural processing unit using spatial fusion with load balancing is disclosed. A device can include a machine learning system. The machine learning system can include an accuracy determination circuit configured to determine an accuracy level of data and divide the data into data sub-portions. The machine learning system can utilize sparsity during computation of each sub-portion. The machine learning system can include a load balancing circuit configured to select a load balancing technique, where the load balancing technique includes loading a computation circuit alternately with at least a first data / weight sub-portion combination and a second data / weight sub-portion combination. The load balancing circuit can be configured to load the computation circuit with selected data sub-portions and selected weight sub-portions based at least in part on the load balancing technique. The machine learning system can include a computation circuit configured to compute partial computation results based at least in part on the selected data sub-portions and weight sub-portions.

Description

[0001] This application claims priority to Provisional Patent Application No. 63 / 003,883, filed April 1, 2020, entitled "Hybrid Precision NPU with Load Balancing Spatial Fusion". The subject matter of the earlier application is incorporated herein by reference. Technical Field

[0002] This specification relates to machine learning, and more specifically to the use of hybrid precision neural processing units (NPUs) with load-balanced spatial fusion. Background Technology

[0003] Artificial neural networks (ANNs), or connection systems, are computational systems that are often fuzzily inspired by the biological neural networks that make up the animal brain. Such systems “learn” a task by considering examples (i.e., progressively improving performance on the task) without typically requiring task-specific programming. For example, in image recognition, they can learn to recognize images containing cats by analyzing example images that have been manually labeled “cat” or “no cat” and using the results. They do this without any prior knowledge about cats (e.g., that they have fur, tails, whiskers, and cat-like faces). Instead, they deduce their own set of relevant features from the learning material they process.

[0004] Neural networks are typically based on a collection of connection units or nodes called artificial neurons. Each connection between artificial neurons (a simplified version of a synapse) sends a signal from one neuron to another. The receiving artificial neuron processes the signal and then sends a signal back to the artificial neurons connected to it. In common neural network implementations, the signals at the connections between artificial neurons are real numbers, and the output of each artificial neuron is calculated as a nonlinear function of the sum of its inputs. Artificial neurons and connections typically have weights that adjust as learning progresses. The weights increase or decrease the signal strength at the connection. Artificial neurons may have thresholds such that a signal is sent only if the aggregated signal exceeds that threshold. Typically, artificial neurons are organized into layers. Different layers can perform different kinds of transformations on their inputs. A signal may travel from the first (input) layer to the last (output) layer after multiple traversals of layers. Artificial intelligence accelerators or neural processing units (NPUs) are typically a class of dedicated hardware accelerators or computer systems designed to accelerate artificial intelligence applications, particularly artificial neural networks, machine vision, and machine learning. Summary of the Invention

[0005] According to one general aspect, an apparatus may include a machine learning system. The machine learning system may include: accuracy determination circuitry configured to determine an accuracy level for data and divide the data into sub-parts. The machine learning system may include: load balancing circuitry configured to select a load balancing technique, wherein the load balancing technique includes alternately loading computational circuitry with at least a first data / weight sub-part combination and a second data / weight sub-part combination. The load balancing circuitry may be configured to load computational circuitry with selected data sub-parts and selected weight sub-parts, at least in part, based on the load balancing technique. The machine learning system may include: computational circuitry configured to compute partial computation results, at least in part, based on the selected data sub-parts and weight sub-parts.

[0006] According to another general aspect, an apparatus may include a machine learning system. The machine learning system may include: a fusion circuit configured to combine a first portion of computation results with a second portion of computation results to form a combined computation result. The machine learning system may include: a first computing circuit configured to compute the first portion of computation results. The machine learning system may include: a second computing circuit configured to compute the second portion of computation results. The machine learning system may include: a load balancing circuit configured to select a load balancing technique, wherein the load balancing technique includes loading the first computing circuit with a slow data / weight sub-part combination or a faster data / weight sub-part combination in a preset mode, wherein the terms "fast" and "slow" refer to the computation associated with the data / weight sub-part combination.

[0007] According to another general aspect, an apparatus may be configured to provide a neural network. The apparatus may include: a fusion circuit configured to combine a first portion of computation results with a second portion of computation results to form a combined computation result. The apparatus may include: a first computing circuit configured to compute the first portion of computation results, wherein the first portion of computation results is associated with a first unit of data and is partially based on a first data sub-part of the first unit. The apparatus may include: a second computing circuit configured to compute a second portion of computation results associated with a second data sub-part of a second unit of data, at least partially concurrent with the computation of the first portion of computation results. The apparatus may include: a load balancing circuit configured to select a load balancing technique, wherein the load balancing technique includes rotating the data / weight sub-part combination between the first and second computing circuits to reduce the asynchrony between partial computation results of the first unit and partial computation results of the second unit.

[0008] Details of one or more embodiments are set forth in the accompanying drawings and the following description. Other features will be apparent from the specification, drawings, and claims.

[0009] As set forth more fully in the claims, and substantially as shown and / or described in conjunction with at least one figure, this disclosure relates to a system and / or method for machine learning, and more specifically to a hybrid precision neural processing unit (NPU) using spatial fusion with load balancing. Attached Figure Description

[0010] Figure 1 This is a block diagram of an example embodiment of a system based on the disclosed subject matter.

[0011] Figure 2 This is a block diagram of an example embodiment of a system based on the disclosed subject matter.

[0012] Figure 3 This is a block diagram of an example embodiment of a device based on the disclosed subject matter.

[0013] Figure 4 This is a block diagram of an example embodiment of a computational process based on the disclosed subject matter.

[0014] Figure 5 It may be a schematic block diagram of an information processing system that may include devices formed based on the principles of the disclosed subject matter.

[0015] The same reference numerals in the various figures denote the same elements. Detailed Implementation

[0016] Various exemplary embodiments will be described more fully below with reference to the accompanying drawings, in which some exemplary embodiments are illustrated. However, the subject matter of this disclosure may be presented in many different forms and should not be construed as limited to the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the subject matter to those skilled in the art. In the drawings, the dimensions and relative dimensions of layers and regions may be exaggerated for clarity.

[0017] It will be understood that when a component or layer is referred to as being “on,” “connected to,” or “coupled to” another component or layer, it may be directly on, connected to, or coupled to the other component or layer, or there may be intermediate components or layers. Conversely, when a component or layer is referred to as being “directly on,” “directly connected to,” or “directly coupled to” another component or layer, there are no intermediate components or layers. The same reference numerals always refer to the same component. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0018] It will be understood that although the terms first, second, third, etc., may be used herein to describe various elements, components, regions, layers, and / or parts, these elements, components, regions, layers, and / or parts should not be limited by these terms. These terms are used only to distinguish one element, component, region, layer, or part from another. Therefore, without departing from the teachings of this disclosure, the first element, component, region, layer, or part discussed below may be referred to as the second element, component, region, layer, or part.

[0019] For ease of description, this document uses spatial relative terms such as “below,” “under,” “lower,” “above,” and “over” to describe the relationship between one element or feature and another element (or features) or feature (or features) as shown in the figures. It will be understood that, in addition to the orientation depicted in the figures, the spatial relative terms are also intended to cover different orientations of the device during use or operation. For example, if the device in the figures is flipped, an element described as “below” or “under” other elements or features will be oriented as “above” other elements or features. Thus, the exemplary term “below” can cover both above and below orientations. The device may be in other orientations (rotated 90 degrees or in other orientations), and the spatial relative descriptors used herein are interpreted accordingly.

[0020] Furthermore, for ease of description, this document uses electrical terms such as “high,” “low,” “pull-up,” “pull-down,” “1,” “0,” etc., to describe voltage levels or currents relative to other voltage levels or relative to another element (or elements) or feature (or features) shown in the figures. It will be understood that, in addition to the voltages or currents depicted in the figures, the electrical relative terms are also intended to cover different reference voltages in the use or operation of the device. For example, if the device or signal in the figures is reversed or uses a different reference voltage, current, or charge, the element described as “high” or “pull-up” will be “low” or “pull-down” compared to the new reference voltage or current. Thus, the exemplary term “high” can cover both relatively low and high voltages or currents. Devices may also be based on different electrical reference frames, and the electrical relative descriptors used herein are interpreted accordingly.

[0021] The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to limit the subject matter of this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. It will be further understood that the term "comprising," as used in this specification, designates the presence of stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0022] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the subject matter of this disclosure pertains. It will be further understood that terms such as those defined in common dictionaries shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and shall not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0023] In the following text, exemplary embodiments will be explained in detail with reference to the accompanying drawings.

[0024] Figure 1 This is a block diagram of an example embodiment of system 100 according to the disclosed subject matter. In various embodiments, system 100 may include a neural processing unit (NPU), or another form of computing device, such as, for example, a laptop computer, desktop computer, workstation, personal digital assistant, smartphone, tablet computer, and other suitable computer or its virtual machine or virtual computing device. In various embodiments, system 100 may include a convolutional neural network (CNN) or another machine learning technique. It is understood that the above are merely a few illustrative examples of limitations to the disclosed subject matter.

[0025] In various embodiments, different quantization schemes may be used as part of deep learning techniques to quantize weights and / or activation functions. These quantization schemes may utilize or include different quantization precisions, such as, for example, 8 bits or 4 bits. In various embodiments, a mixed-precision NPU may support the execution of different deep learning techniques with different levels of precision (e.g., different numbers of bits per weight).

[0026] In such embodiments, low-precision multiplication units (e.g., computation circuitry 108) can be used to construct a mixed-precision system. Temporal or spatial fusion can be employed in the mixed-precision system to support higher-precision computation. In various embodiments involving spatial fusion, the multiplication unit (e.g., computation circuitry 108) can be divided into multiple sub-sections or circuit groups, referred to as “tiles”.

[0027] In some embodiments, high-precision or high-order data may be divided into low-precision or low-order components (e.g., an 8-bit byte divided into 4 nibbles), and assigned to each block according to the component position (e.g., most significant nibble (MSN), least significant nibble (LSN)). In this context, the most significant sub-part will be the higher or uppermost bit of the high-precision data, and the least significant or lower significant sub-part will be the least significant or lowermost bit of the high-precision data. In various embodiments, the division may be at a midpoint, but it is understood that the above is merely an illustrative example of a subject matter not being limited to this. Each block may use a low-precision multiplier or weights to perform partial computation of the low-precision sub-part. The final high-precision result can then be computed by fusing and aligning the partial computations of all sets or blocks.

[0028] In various embodiments, the NPU or system may leverage data sparsity to achieve performance improvements. In this context, "sparse" means that the NPU can skip invalid computations, such as multiplying by zero. In such an embodiment, a chunk receiving input data with high sparsity (i.e., many zeros) can skip many of its computations and will therefore complete faster than a chunk receiving input with fewer zeros. On the other hand, in this context, "dense" is the opposite of sparsity, meaning that the NPU does not skip invalid computations (e.g., multiplying by zero) and the chunk will perform multiplication even if the input is zero. In various embodiments, the percentage of data sparsity can vary across low / high precision sub-parts. For example, since MSN may have sparser data, the MSN sub-part may take less time to compute than the LSN sub-part. This is in addition to the sparsity caused by the data itself.

[0029] In some embodiments, this means that a set of computational blocks processing data sub-parts with different computation times (sparse ratios) may be paused to wait for other sets to complete their computations. In such embodiments, this pause allows results from all blocks (all blocks in a group) to be correctly fused into the high-precision data. In various embodiments, system 100 may employ different load balancing techniques to minimize or reduce pause times caused by differences in the sparsity distribution of data components.

[0030] In one embodiment, system 100 may include or have access to a dataset or feature map 102. In various embodiments, feature map 102 may include data to be processed by system 100. In some embodiments, feature map 102 may include raw data that has been filtered or processed by previous convolutions or other operations or layers (not shown). In various embodiments, the data or feature map 102 may be divided into units 103. In various embodiments, each unit may include pixels or groups of pixels. For example, in one embodiment, feature map 102 may include a series of video frames. In various embodiments, units 103 may be labeled or grouped (e.g., even and odd numbers) to aid processing.

[0031] In various embodiments, system 100 may include precision determination circuitry or unit 104. In the illustrated embodiment, precision determination circuitry or unit 104 may be configured to determine the precision level of data (e.g., unit 103). For example, the precision level may be one of a plurality of levels, such as high or low. In a particular embodiment, a high precision level may include 8 bits per data line or per unit, and a low precision level may include 4 bits per data line or per unit. It is understood that the above are merely a few illustrative examples that are not limited to the disclosed subject matter.

[0032] In some embodiments, the precision determination circuitry or unit 104 may be configured to divide or group data (e.g., unit 103) into or into smaller sub-parts 105. In various embodiments, unit data may be divided into smaller precision sub-parts. In such embodiments, each unit sub-part may have the same unit shape; for example, as described below, a unit comprising a vector of n numbers may be divided into sub-parts, where each sub-part comprises a vector of n numbers but with smaller precision. In one such embodiment, the precision determination circuitry 104 may divide 8-bit bytes of data into two 4-bit nibbles, MSN and LSN. Similarly, the precision determination circuitry 104 or similar precision circuitry (not shown) may acquire weighted data or values ​​and divide them into smaller precision weighted sub-parts 190. It is understood that the above is merely an illustrative example to which the disclosed subject matter is not limited.

[0033] In various embodiments, system 100 may include load balancing circuitry 106. In such embodiments, load balancing circuitry 106 may be configured to select a load balancing technique 116 to employ when distributing data to computing circuitry 108. In various embodiments, load balancing technique 116 may be configured to reduce pauses or periods of inactivity in computing circuitry 108 due to differences in the amount of time that may be spent processing each data sub-part or nibble 105. These load balancing techniques 116 are discussed in more detail below.

[0034] In the illustrated embodiment, load balancing circuit 106 may be configured to load or distribute (and another circuit may be loaded) specific data sub-parts 105 and weight sub-parts 190 to various computing circuits 108 or blocks. In such an embodiment, these data / weight combinations 192 may be created and distributed at least in part based on a selected load balancing technique 116.

[0035] In the illustrated embodiment, system 100 may include one or more computing circuits 108 or blocks. In such an embodiment, block 108 may be configured to compute portions 192 and 194 according to assigned data / weight combinations 192 and 194.

[0036] In the illustrated embodiment, system 100 may include fusion circuitry 110. As described above, fusion circuitry 110 may be configured to combine multiple portions and 194 to form a combined, fused, or complete unit and 196.

[0037] In various embodiments, the above processing may be repeated for the next unit 103 until dataset 102 has been processed. In such an embodiment, a series of complete sets and 196 may generate a new dataset (not shown). This new dataset may then be further processed by another layer of system 100, or it may be the final result of system 100.

[0038] Figure 2 This is a block diagram of an example embodiment of system 200 based on the disclosed subject matter. In the illustrated embodiment, as referred to above... Figure 1 As described, system 200 may employ computing circuitry 108 and fusion circuitry 110. It is understood that system 200 is merely an illustrative example, and the subject matter is not limited thereto.

[0039] In the illustrated embodiment, system 200 may include four blocks or computational circuits 108 grouped together. In various embodiments, the entire machine learning system may include more blocks 108 with the same or different configurations. In some embodiments, computational circuits 108 may include four 4-bit multiplier circuits configured to perform small-precision or low-precision computations.

[0040] In the illustrated embodiment, each block 108 may be assigned or loaded with a specific data / weight combination 192. In such an embodiment, the combination 192 may be assigned to the entire system 200, such that the entire unit processes data at a given time.

[0041] For example, in one embodiment, block 108-A may be loaded with a data / weight combination 192-A or a combination LL (lowest-lowest) including a data LSN and a weight LSN. Block 108-B may be loaded with a data / weight combination 192-B or a combination LM (lowest-highest) including a weight LSN and a data MSN. Block 108-C may be loaded with a data / weight combination 192-C or a combination ML (highest-lowest) including a weight MSN and a data LSN. Block 108-D may be loaded with a data / weight combination 192-D or a combination MM (highest-highest) including a data MSN and a weight MSN. In such an embodiment, the four data / weight combinations 192 may be referred to as LL, LM, ML, and MM.

[0042] As described above, block 108 can generate four parts and 194-A, 194-B, 194-C, and 194-D. System 200 may include a fusion circuit 110 configured to combine or fuse these parts and 194 into a complete unit and 196.

[0043] As described above, in one embodiment, the system 200 pipeline may require that, for a given unit, all data / weight combinations 192 be processed before the next unit can begin processing. In such an embodiment, this may involve pausing one or more blocks 108 while “slow” blocks 108 are completing computation.

[0044] In the illustrated embodiment, system 200 can be configured to adapt to the sparsity of the dataset and allow data / weight combinations from newer units (e.g., the next pixel of an image) to begin processing before all previous units have been computed.

[0045] Figure 3 This is a block diagram of an example embodiment of device 300 according to the disclosed subject matter. In various embodiments, device 300 may include, for example... Figure 1 Alternatively, the fusion circuit shown in Figure 2. In the illustrated embodiment, a fusion circuit 110 associated with four computing circuits is shown, but other levels of association, including dynamic rearrangements, are considered. It is understood that the above is merely an illustrative example to which the disclosed subject matter is not limited.

[0046] In one embodiment, as described above, device 300 may receive multiple (e.g., four) portions 194 from the computing unit. In some embodiments, portion calculation result 194-D may include MM calculation results (e.g., and), and may be represented by Equation 1 below. Portion calculation result 194-C may include ML calculation results and may be represented by Equation 2 below. Portion calculation result 194-B may include LM calculation results and may be represented by Equation 3 below. Portion calculation result 194-A may include LL calculation results and may be represented by Equation 4 below.

[0047] Equation 1: Partial Calculation Results MM =∑(W MSN ×A MSN )

[0048] Equation 2: Partial Calculation Results ML =∑(W MSN ×A LSN )

[0049] Equation 3: Partial Calculation Results LM =∑(W LSN ×A MSN )

[0050] Equation 4: Partial Calculation Results LL =∑(W LSN ×A LSN )

[0051] Where A is the data or activation function, and W is the weight.

[0052] In the illustrated embodiment, each pair of partial calculation results 194 can enter partial fusion circuit 302. In the illustrated embodiment, partial calculation results 194-D and 194-C can be assigned to partial fusion circuit 302-Y, and partial calculation results 194-B and 194-A can be assigned to partial fusion circuit 302-X. In various embodiments, in some pixels (e.g., even pixels), the input can enter a shifter, and in another case (e.g., odd pixels), the input can directly enter an adder. A switching block or multiplexer (not shown) can be used to route the input to the adder and shifter.

[0053] Each partial fusion circuit 302 may include a shifter 312 and a summer or adder 314. In the illustrated embodiment, a partial calculation result (e.g., partial calculation results 194-D and 194-B) may be shifted before being added to another partial calculation result (e.g., partial calculation results 194-C and 194-A). In various embodiments, the partial fusion circuit 302 and device 300 typically include registers or memory units to store calculation results 194, 392, and 196. Some of these registers are shown as blocks for calculating results 194, 392, and 196. In some embodiments, these registers may facilitate pipelined operation.

[0054] In one embodiment, a portion of the computation result 194 may be a 16-bit value. Shifter 312 may include a four-bit shifter that produces a 20-bit value. Adder 314 may output a 21-bit intermediate computation result 392. It is understood that the above is merely an illustrative example to which the disclosed subject matter is not limited.

[0055] In various embodiments, partial computation results 194 may be grouped and input into a partial fusion circuit 302 such that the number of shift operations (e.g., via shifter 312) required is less than or equal to the size or width of a nibble or data sub-part. For example, by not placing both the MM and LL partial computation results 194 into the same partial fusion circuit 302, only a shift equal to the size of a nibble (4 bits) is required instead of a shift equal to the size of a full byte (8 bits).

[0056] In the illustrated embodiment, these intermediate computation results 394 may be input into a third partial fusion circuit 302. The output computation result 196 or the full computation result 196 may be given by Equation 5 below.

[0057] Equation 5: Full result = [∑(W NSN ×A LSN ) + ∑(W LSN ×A MSN ) ＜＜N + [∑(W MSN ×A LSN ) + Σ(W MSN ×A MSN ) ＜＜N ＜＜N

[0058] Where A is the data or activation function, W is the weight, and "<<N" indicates a shift of n bits (e.g., a 4-bit shift).

[0059] In one embodiment, the partial computation results 194 may represent 4-bit × 4-bit values (4×4 values). The intermediate computation results 392 may represent 4×8 or 8×4 values. The full computation result 196 may represent 8×8 values. It is understood that the above is merely an illustrative example to which the disclosed subject matter is not limited.

[0060] Figure 4 is a block diagram of an example embodiment of computational flows 400, 401, and 402 according to the disclosed subject matter. In various embodiments, a machine learning system or device may be configured to select between one or more load balancing techniques. Figure 4 The computational flows 400, 401, and 402 of

[0061] show some possible examples. In the illustrated embodiment, the machine learning system includes four blocks or computational circuits, although other quantities are also contemplated. In such an embodiment, the machine learning system may compute partial computation results for a series of units in a pipelined manner.

[0062] ​Computation flow 400 illustrates a conventional computation flow in which each data / weight combination (e.g., LL, ML, LM, or MM) takes exactly the same amount of time to process. Computation flow 400 illustrates an example of a dense architecture that does not skip zeros. In computation flow 400, jobs or computational tasks in blocks 1, 2, 3, and 4 are shown as rows. Pipeline unit computations are shown as columns, such as unit 1 (C1), unit 2 (C2), unit 3 (C3), unit 4 (C4), unit 5 (C5), unit 6 (C6), etc., regardless of how many units are processed. In the illustrated embodiment, processing tasks are shown as tasks 4011-4046.

[0063] Computation flow 401 illustrates a computation flow utilizing sparsity (e.g., skipping zeros), where each data / weight combination (e.g., LL, ML, LM, or MM) takes a different amount of time to process. However, the system can still operate in a lockstep manner, allowing new data / weight combinations from new units to be left unprocessed until all data / weight combinations from the current unit have been processed. In the illustrated embodiment, the processing tasks are shown as tasks 4111-4144.

[0064] For example, although task 4121 (LM) of block 2 takes less time to process than task 4111 (LL) of block 1, block 2 may not move forward and begin processing task 4122 (LM) of the next sub-section or unit. Instead, block 2 must wait or pause 499 until the slowest task (e.g., task 4111) completes its calculation. In this way, the calculation results of all parts of a particular unit can be presented to the fusion circuit essentially simultaneously. For tasks that complete very quickly (e.g., task 4141 (MM) of block 4), the delay is even worse.

[0065] Computation flow 402 illustrates a specific embodiment of the system, in which the system leverages the sparsity of the dataset to accelerate computation and reduce pause times. In such an embodiment, cells may be divided into groups (e.g., even and odd numbers), and load balancing circuitry may alternate or rotate between groups. In the illustrated embodiment, one group of cells is colored white, and another group of cells is colored gray. It is understood that the above is merely an illustrative example to which the disclosed subject matter is not limited.

[0066] In such an embodiment, when the task of a (first) unit group (e.g., task 4221 of block 2) is completed, a task associated with another (second) unit group (e.g., task 4222 of block 2) can begin, even if another task associated with the first unit group is still being processed (e.g., task 4231 of block 3). In such an embodiment, computation time can be reduced due to the absence of pause 499. For example, in computation flow 402, many tasks associated with unit 3 (e.g., tasks 4223, 4233, and 4243) can be completed in similar tasks (e.g., tasks 4123, 4133, and 4143) even before computation flow 401 begins.

[0067] In various embodiments, the computing circuitry, fusion circuitry, or system as a whole may include one or more registers or queues to maintain and synchronize the results of partial computations completed at different times. In one embodiment, the registers or queues may be... Figure 3 A portion of the registers shown. It is understood that the above is merely an illustrative example of a subject matter not being limited to. In some embodiments, the system may include multiple fusion circuits, wherein each fusion circuit is associated with one or more unit groups (e.g., even and odd fusion circuits).

[0068] In various embodiments, pause 499 may be introduced to prevent partial calculation results from different units (e.g., unit 1 and unit 3) from being queued into the fusion circuit simultaneously. In another embodiment, pause 499 may not be introduced.

[0069] It is understood that the calculation processes 400, 401 and 402 shown are not to scale, and although the cell calculation times are shown as repeated for ease of illustration, each cell may involve its own calculation time and differences from the specification.

[0070] In some embodiments, the architecture speed or processing speed is limited by the slowest data / weight combination (e.g., computation flow 401). Typically, this could be the slowest data / weight combination in LL. In the illustrated embodiment, the machine learning system may attempt to distribute the slowest data / weight combination across different blocks or computation circuits over time to change which block is the slowest. In such embodiments, as shown in computation flow 402, the completion of unit tasks can occur relatively close together. This is in contrast to a pause-free version of computation flow 401 where block 1 is always the slowest and increasingly lags behind other blocks. Such latency would ultimately hinder the fusion circuit's ability to combine partial computation results of a given unit, as a partial computation result would be unavailable for a considerable period. In the illustrated embodiment, load balancing is employed to distribute latency or processing timeouts between blocks and thus more closely align (without resorting to pauses) the availability of all partial computation results for a given unit. In various embodiments, this may be referred to as synchronization of partial computation results of the unit.

[0071] In various embodiments, the load balancing circuit can be selected from one or more load balancing techniques. In the illustrated embodiment, calculation process 402 utilizes a static switching load balancing technique, in which each block is assigned one of two data / weight combinations, and the load balancing circuit alternates between the two. In the illustrated embodiment, the two data / weight combinations for each block are as follows:

[0072] Block 1: LL and MM

[0073] Block 2: LM and ML

[0074] Block 3: ML and LM

[0075] Block 4: MM and LL

[0076] In various embodiments, a round-robin or polling scheme may be employed. In one such embodiment, this allows each block or computational circuit to process each data / weight combination (e.g., LL, MM, etc.) and to more closely align partial computation results by amortizing the latency caused by long processing combinations. In one such embodiment, the data / weight combination for each block may be as follows:

[0077] Block 1: LL, then MM, then LM, then ML, then repeat.

[0078] Block 2: LM, then ML, then LL, then MM, then repeat.

[0079] Block 3: ML, then LM, then MM, then LL, then repeat.

[0080] Block 4: MM, then LL, then ML, then LM, then repeat.

[0081] In such an embodiment, it should be noted that, as described above, the fusion circuit will not have an LL and MM combination applied to the same part of the fusion circuit, and thus excessive bit shifting is avoided.

[0082] It is understood that the above are merely a few illustrative examples of publicly available topics, and are not limited to these.

[0083] In various embodiments, as described above, load balancing can be static or dynamic. In some embodiments, various forms of dynamic load balancing can be employed. In some embodiments, the form of load balancing can be configurable. In some embodiments, the dataset can be analyzed, and the form of load balancing can be automatically selected. For example, a first load balancing technique can be used for still images, a second load balancing technique can be used for still video, and a third load balancing technique can be used for other types of data (e.g., predictive data analysis). It is understood that the above are merely a few illustrative examples that are not limited to the disclosed subject matter.

[0084] In one embodiment, the spatial technology can be configurable. In one such embodiment, the machine learning system may be able to execute a intensive architecture (e.g., computation flow 400) and then switch to a sparse sensing technology (e.g., computation flow 402). Furthermore, load balancing techniques can be configured independently or in combination with specific aspects. In one such embodiment, the machine learning system may execute in a lockstep manner but employ load balancing to reduce pause times (e.g., computation flow 401). It is understood that the above are merely a few illustrative examples of the subject matter not being limited to these examples.

[0085] Figure 5 This is a schematic block diagram of an information processing system 500, which may include semiconductor devices formed according to the principles of the disclosed subject matter.

[0086] refer to Figure 5 The information processing system 500 may include one or more devices constructed according to the principles of the disclosed subject matter. In another embodiment, the information processing system 500 may employ or perform one or more techniques according to the principles of the disclosed subject matter.

[0087] In various embodiments, the information processing system 500 may include computing devices, such as, for example, laptop computers, desktop computers, workstations, servers, blade servers, personal digital assistants, smartphones, tablet computers, and other suitable computers or their virtual machines or virtual computing devices. In various embodiments, the information processing system 500 may be used by a user (not shown).

[0088] The information processing system 500 according to the disclosed subject matter may also include a central processing unit (CPU), logic circuitry, or processor 510. In some embodiments, processor 510 may include one or more functional unit blocks (FUBs) or combinational logic blocks (CLBs) 515. In such embodiments, combinational logic blocks may include various Boolean logic operations (e.g., NAND, NOR, NOT, XOR), stable logic devices (e.g., flip-flops, latches), other logic devices, or combinations thereof. These combinational logic operations may be configured in a simple or complex manner to process input signals to achieve a desired result. It is understood that although several illustrative examples of synchronous combinational logic operations have been described, the disclosed subject matter is not so limited and may include asynchronous operations or combinations thereof. In one embodiment, combinational logic operations may include a plurality of complementary metal-oxide-semiconductor (CMOS) transistors. In various embodiments, these CMOS transistors may be arranged as gates performing logic operations; however, it is understood that other techniques within the scope of the disclosed subject matter may be used.

[0089] The information processing system 500 according to the disclosed subject matter may also include volatile memory 520 (e.g., random access memory (RAM)). The information processing system 500 according to the disclosed subject matter may also include non-volatile memory 530 (e.g., hard disk drive, optical memory, NAND, or flash memory). In some embodiments, volatile memory 520, non-volatile memory 530, or a combination or portion thereof may be referred to as a "storage medium". In various embodiments, volatile memory 520 and / or non-volatile memory 530 may be configured to store data in a semi-permanent or substantially permanent form.

[0090] In various embodiments, the information processing system 500 may include one or more network interfaces 540, wherein the one or more network interfaces 540 are configured to allow the information processing system 500 to be part of a communication network and to communicate via the communication network. Examples of Wi-Fi protocols may include, but are not limited to, IEEE 802.11g and IEEE 802.11n. Examples of cellular protocols may include, but are not limited to, IEEE 802.16m (also known as Advanced Wireless-MAN (Metropolitan Area Network)), Advanced Long Term Evolution (LTE), Enhanced Data Rate GSM (Global System for Mobile Communications) Evolution (EDGE), and Evolved High Speed ​​Packet Access (HSPA+). Examples of wired protocols may include, but are not limited to, IEEE 802.3 (also known as Ethernet), Fibre Channel, and power line communication (e.g., HomePlug, IEEE 1901). It is understood that the above are merely a few illustrative examples, not limited to the disclosed subject matter.

[0091] The information processing system 500 according to the disclosed subject matter may also include a user interface unit 550 (e.g., a display adapter, a haptic interface, a human-machine interface device). In various embodiments, the user interface unit 550 may be configured to receive input from a user and / or provide output to a user. Other types of devices may also be used to provide interaction with the user; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or haptic feedback; and input from the user may be received in any form, including sound, speech, or haptic input.

[0092] In various embodiments, the information processing system 500 may include one or more other devices or hardware components 560 (e.g., a display or monitor, keyboard, mouse, camera, fingerprint reader, video processor). It is understood that the above are merely a few illustrative examples, and the subject matter is not limited thereto.

[0093] The information processing system 500 according to the disclosed subject matter may also include one or more system buses 505. In such embodiments, the system bus 505 may be configured to communicatively connect the processor 510, volatile memory 520, non-volatile memory 530, network interface 540, user interface unit 550, and one or more hardware components 560. Data processed by the processor 510 or data input from outside the non-volatile memory 530 may be stored in the non-volatile memory 530 or the volatile memory 520.

[0094] In various embodiments, the information processing system 500 may include or run one or more software components 570. In some embodiments, software component 570 may include an operating system (OS) and / or applications. In some embodiments, the OS may be configured to provide one or more services to applications and to manage or act as an intermediary between applications and various hardware components of the information processing system 500 (e.g., processor 510, network interface 540). In such embodiments, the information processing system 500 may include one or more native applications, wherein the native applications may be installed locally (e.g., within non-volatile memory 530) and configured to be executed directly by processor 510 and to interact directly with the OS. In such embodiments, the native applications may include pre-compiled machine-executable code. In some embodiments, the native applications may include a script interpreter (e.g., C shell (csh), AppleScript, AutoHotkey) or a virtual execution machine (VM) (e.g., Java Virtual Machine, Microsoft Common Language Runtime), wherein the script interpreter or VM is configured to translate source code or object code into executable code that is subsequently executed by processor 510.

[0095] Various packaging technologies can be used to package the aforementioned semiconductor devices. For example, any of the following technologies can be used to package semiconductor devices constructed according to the principles of the disclosed subject matter: package-on-package (POP) technology, ball grid array (BGA) technology, chip-scale package (CSP) technology, plastic leaded chip carrier (PLCC) technology, plastic dual in-line package (PDIP) technology, die technology in wafer packaging, die technology in wafer form, chip-on-board (COB) technology, ceramic dual in-line package (CERDIP) technology, plastic metered quad flat package (PMQFP) technology, plastic quad flat package (PQFP) technology, small outline package (SOIC) technology, reduced small outline package (SSOP) technology, thin small outline package (TSOP) technology, thin quad flat package (TQFP) technology, system-in-package (SIP) technology, multi-chip package (MCP) technology, wafer-level fabrication package (WFP) technology, wafer-level processing stacked package (WSP) technology, or other technologies as known to those skilled in the art.

[0096] The method steps may be executed by one or more programmable processors running a computer program to perform a function by manipulating input data and generating output. The method steps may also be executed by special-purpose logic circuitry (e.g., an FPGA (Field-Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit)), and the device may be implemented as special-purpose logic circuitry (e.g., an FPGA or an ASIC).

[0097] In various embodiments, the computer-readable medium may include instructions, wherein, when executed, the instructions cause the apparatus to perform at least a portion of the method steps. In some embodiments, the computer-readable medium may be contained in a magnetic medium, an optical medium, other media, or combinations thereof (e.g., a CD-ROM, a hard disk drive, a read-only memory, a flash drive). In such embodiments, the computer-readable medium may be an article of manufacture that is tangibly and non-transitory.

[0098] Although the principles of the disclosed subject matter have been described with reference to exemplary embodiments, it will be apparent to those skilled in the art that various changes and modifications may be made thereto without departing from the spirit and scope of these disclosed concepts. Therefore, it should be understood that the above embodiments are not restrictive but merely illustrative. Consequently, the scope of the disclosed concepts will be determined by the broadest permissible interpretation of the claims and their equivalents, and should not be constrained or limited by the foregoing description. Therefore, it will be understood that the appended claims are intended to cover all such modifications and changes falling within the scope of the embodiments.

Claims

1. An apparatus comprising: Machine learning systems, including: The accuracy determination circuit is configured as follows: Determine the precision level of the data. The data is divided into data sub-parts, each sub-part comprising the most significant nibble and the least significant nibble. Obtain the weight value. The weight value is divided into a weight sub-part, which includes the highest valid half-byte and the lowest valid half-byte. The load balancing circuit is configured as follows: Selecting a load balancing technique, wherein the load balancing technique includes alternately loading computing circuitry with at least a first data / weight sub-part combination and a second data / weight sub-part combination, wherein the first data / weight sub-part combination includes least significant nibble of data and least significant nibble of weight, and the second data / weight sub-part combination includes most significant nibble of data and most significant nibble of weight. At least in part, based on load balancing techniques, the computing circuit is loaded with selected data sub-parts and selected weight sub-parts; and The computing circuit is configured to calculate a partial computation result based at least in part on a selected data sub-part and a selected weight sub-part. The computing circuit includes: A first computing circuit is alternately loaded with a first data / weight sub-combination and a second data / weight sub-combination by a load balancing circuit; and The second computing circuit is alternately loaded with the second data / weight sub-combination and the first data / weight sub-combination by the load balancing circuit.

2. The device as claimed in claim 1, wherein, The machine learning system further includes a fusion circuit configured to combine a first portion of the computation result with at least a second portion of the computation result to form a combined computation result.

3. The device as described in claim 2, wherein, The load balancing circuit is configured to select a load balancing technique that performs the following operation: shifts a portion of the calculation result by a factor not greater than the width of the data sub-part.

4. The device as claimed in claim 1, wherein, The load balancing circuit employs load balancing techniques that include the following operations: Based on the combined arrangement of data sub-parts and weight sub-parts, at least a first data / weight sub-part combination and a second data / weight sub-part combination should be created; and The computation circuit is loaded using a round-robin approach, combining data / weight sub-parts.

5. The device as claimed in claim 1, wherein, The first data / weight sub-combination and the second data / weight sub-combination are selected from different units of the data.

6. The device as claimed in claim 1, wherein, Load balancing techniques include static load balancing techniques.

7. The device as claimed in claim 1, wherein, The load balancing circuit is configured to select a load balancing technique that reduces the amount of downtime in the computing circuit.

8. An apparatus comprising: Machine learning systems, including: The fusion circuit is configured to combine the first part of the calculation result with the second part of the calculation result to form a combined calculation result; The first computing circuit is configured to calculate a first part of the calculation result; The second computing circuit is configured to calculate the second part of the calculation result; and A load balancing circuit is configured to select a load balancing technique, wherein the load balancing technique includes: alternately loading a first computing circuit and a second computing circuit with a first data / weight sub-part combination and a second data / weight sub-part combination in a preset mode. The data is divided into data sub-parts, each containing the most significant nibble and the least significant nibble. Similarly, the weight values ​​are divided into weight sub-parts, each containing the most significant nibble and the least significant nibble. The first data / weight sub-part combination includes the least significant nibble of data and the least significant nibble of weight, while the second data / weight sub-part combination includes the most significant nibble of data and the most significant nibble of weight. In this circuit, the first computing circuit is alternately loaded with a first data / weight sub-combination and a second data / weight sub-combination by the load balancing circuit, and the second computing circuit is alternately loaded with a second data / weight sub-combination and a first data / weight sub-combination by the load balancing circuit.

9. The device as claimed in claim 8, wherein, The fusion circuit is configured to shift a portion of the calculation result by a width no greater than the width of the data sub-part.

10. The device as claimed in claim 8, wherein, The load balancing circuit employs a load balancing technique that includes the following operations: loading the first computing circuit in a round-robin manner with a first data / weight sub-combination, a second data / weight sub-combination, and at least a third data / weight sub-combination.

11. The device as claimed in claim 8, wherein, The first data / weight sub-combination and the second data / weight sub-combination are selected from different units of the data.

12. The device as claimed in claim 8, wherein, Load balancing techniques include static load balancing techniques.

13. The device as claimed in claim 8, wherein, The load balancing circuit is configured to select a load balancing technique that reduces the amount of downtime in the first computing circuit.

14. The device as claimed in claim 8, wherein, The load balancing circuit is also configured as follows: In response to the first computing circuit generating a first part of the calculation result and without considering that the second computing circuit did not generate a second part of the calculation result, the first computing circuit is loaded with a third data / weight sub-part combination.

15. The device as claimed in claim 14, wherein, The first computing circuit is configured to begin calculating the third part of the calculation result without waiting for the second computing circuit to generate the second part of the calculation result.

16. An apparatus configured to provide a neural network, the apparatus comprising: The fusion circuit is configured to combine the first part of the calculation result with the second part of the calculation result to form a combined calculation result; A first computing circuit is configured to compute a first portion of a computation result, wherein the first portion of the computation result is associated with a first unit of data and is partially based on a first data sub-part of the first unit, wherein the first data sub-part of the first unit of data includes the most significant half-byte of data, wherein the first computing circuit is alternately loaded with a first data / weight sub-part combination and a second data / weight sub-part combination by a load balancing circuit. A second computing circuit is configured to compute, at least partially simultaneously with the computation of the first partial computation result, a second partial computation result associated with a second data sub-part of the second unit of the data, wherein the second data sub-part of the second unit of the data includes the least significant nibble of data, and wherein the second computing circuit is alternately loaded with a second data / weight sub-part combination and a first data / weight sub-part combination by a load balancing circuit; and A load balancing circuit is configured to select a load balancing technique, wherein the load balancing technique includes rotating data / weight sub-combinations between a first computing circuit and a second computing circuit to reduce the asynchrony between partial calculation results of the first unit and partial calculation results of the second unit. The weight value is divided into a weight sub-part, which includes the highest-weighted valid nibble and the lowest-weighted valid nibble. The first data / weight sub-part combination includes the least significant half-byte of data and the least significant half-byte of weight, while the second data / weight sub-part combination includes the most significant half-byte of data and the most significant half-byte of weight.

17. The device as claimed in claim 16, wherein, The fusion circuit is configured to shift a portion of the calculation result by a width no greater than the width of the data sub-part.

Images