On-chip non-zero unpacking and dispersion

Abstract

This invention provides an integrated circuit and system that can reduce wasted resource usage due to zero weight values ​​in neural network computations. [Solution] On-chip non-zero value unpacking and distribution is implemented by a plurality of multiply-accumulate (MAC) units, a memory that communicates with the plurality of MAC units, and an unpacker that receives packed non-zero values ​​from the memory, correlates each non-zero value among the packed non-zero values ​​with the corresponding MAC unit among the plurality of MAC units, combines the non-zero value with the address of the corresponding MAC unit, and transmits the non-zero value and address to the corresponding load path connected to the corresponding MAC unit among the plurality of load paths.

Description

Background Art

[0001] Neural network inference chips utilize a plurality of multiply-and-accumulate (MAC) units arranged in a systolic array. The weight registers of the MAC units are connected to the chip's memory and other internal components through one of a plurality of execution paths, where one execution path is connected in series to a plurality of MAC units. Pages containing ordered weight values are sent through the execution path such that each subsequent weight register receives a subsequent weight value, which is sometimes referred to as a shift-based method.

Brief Description of the Drawings

[0002] Aspects of the present disclosure are best understood from the following detailed description when read in conjunction with the accompanying drawings. Note that various features are not drawn to scale in accordance with standard practice in the industry. In fact, the dimensions of various features may be arbitrarily increased or reduced for the clarity of the discussion.

[0003] [Figure 1] FIG. 1 is a schematic diagram of a system for on-chip non-zero value unpacking and dispersion according to at least some embodiments of the present disclosure.

[0004] [Figure 2] FIG. 2 is a schematic diagram of an integrated circuit according to at least some embodiments of the present disclosure.

[0005] [Figure 3] FIG. 3 is a schematic diagram of a systolic array 310 according to at least some embodiments of the present disclosure.

[0006] [Figure 4] FIG. 4 is a schematic diagram of a MAC unit 420 according to at least some embodiments of the present disclosure.

[0007] [Figure 5]This is an operational flow for on-chip non-zero value unpacking and distribution according to at least some embodiments of the present disclosure.

[0008] [Figure 6] This is an operational flow for loading weight values ​​according to at least some embodiments of the present disclosure.

[0009] [Figure 7] This is an operational flow for unpacking a value according to at least some embodiments of the present disclosure. [Modes for carrying out the invention]

[0010] The following disclosure provides many different embodiments or examples to implement different features of the subject matter provided. Specific examples of components, values, behaviors, materials, arrangements, or the like are described below for the sake of brevity of this disclosure. Naturally, these are merely examples and not intended to be limiting. Other components, values, behaviors, materials, arrangements, or the like are contemplated. In addition, this disclosure may repeat reference numbers and / or letters in various examples. This repetition is for the sake of brevity and clarity and does not in itself define relationships between the various embodiments and / or configurations discussed.

[0011] A page contains weight values ​​for each MAC unit connected to the execution path, even if those weight values ​​are zero. However, before the MAC units of the systolic array are loaded with weight values, the weight registers are reset or "cleared" so that each weight register stores data equivalent to a zero value. In other words, each weight register already stores a zero value. In many cases of neural network inference, the weight values ​​are mostly zero, a condition sometimes referred to as "sparse weights." The use of resources to send zero values ​​to the weight registers is sometimes considered unnecessary. Pages containing only a small proportion of zero values ​​rarely pose a problem. However, when sending a sparse weight page with only a small proportion of non-zero values, most resources are used for values ​​already stored in the weight registers.

[0012] In at least some embodiments described herein, the integrated circuit for neural network inference comprises a systolic array including a load path in addition to a normal execution path, and a hardware unpacker at the interface between the systolic array and other components of the integrated circuit. In at least some embodiments, the unpacker is configured to unpack packed non-zero values ​​received from memory and distribute the unpacked non-zero values ​​to each MAC unit through the load path. In at least some embodiments, each load path is connected to an address decoder and load register of the MAC unit connected to the load path, and the load register is connected to the weight register of the MAC unit. In at least some embodiments, the unpacker transmits each non-zero weight value in combination with an address that identifies the MAC unit. In at least some embodiments, the address decoder of the identified MAC unit stores the non-zero weight values ​​received through the load path in the load register in response to verifying the address. In at least some embodiments, an "update" command is transmitted through the execution path, thereby storing the weight values ​​stored in the load register in the weight register. In at least some embodiments, the unpacker is configured to apply one of several unpacking methods to suit the packing method of the packed weights. In at least some embodiments, the packing method includes element-level packing, mask-based coding, address-based coding, lookup table indexing, or any other method of data compression. In at least some embodiments, the packed weights are initially packed according to a packing method determined by the compiler, based on the minimum memory size.

[0013] In at least some embodiments, the packed weights do not need to include weight values ​​equal to zero, and therefore the data size of the packed weights is reduced, especially in the case of sparse weights. In at least some embodiments, the reduced data size of the packed weights reduces bandwidth and memory size requirements.

[0014] Figure 1 is a schematic diagram of a system for on-chip non-zero unpacking and distribution according to at least some embodiments of the present disclosure. The system for on-chip non-zero unpacking and distribution includes an integrated circuit 100 and a host computer 102.

[0015] The integrated circuit 100 is a component of a system for a hardware configuration for non-zero value variance. In at least some embodiments, the integrated circuit 100 is configured to house an unpacker 104, memory 106, controller 108, and systolic array 110 for neural network inference. In at least some embodiments, the integrated circuit 100 is configured to facilitate non-zero value variance through a load path. In at least some embodiments, the integrated circuit 100 is configured to interface with a host computer 102. In at least some embodiments, the integrated circuit 100 is configured to receive packed non-zero values ​​from the host computer 102 and store the packed non-zero values ​​in memory 106. In at least some embodiments, the integrated circuit 100 is one of the following: ASIC (Application-Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), SoC (System on Chip), etc.

[0016] The host computer 102 is a component of the system for on-chip non-zero value unpacking and distribution. In at least some embodiments, the host computer 102 is configured to determine a packing method for non-zero values. In at least some embodiments, the host computer 102 is configured to pack non-zero values. In at least some embodiments, the host computer 102 is configured to transmit packed non-zero values ​​to the integrated circuit 100. In at least some embodiments, the host computer 102 is configured to interface with the controller 108 to manage data flow. In at least some embodiments, the host computer 102 is configured to perform general-purpose computing tasks, run software applications, manage peripherals, etc. In at least some embodiments, the host computer 102 is one or more desktop computers, servers, workstations, cloud computing instances, etc. In at least some embodiments, the host computer 102 communicates with the integrated circuit. In at least some embodiments, the host computer is configured to determine a packing method for packed non-zero values, pack the non-zero values ​​according to the packing method to produce packed non-zero values, and send the packed non-zero values ​​to memory.

[0017] The unpacker 104 is a component of the integrated circuit 100. In at least some embodiments, the unpacker 104 is configured to receive packed non-zero values ​​from memory 106. In at least some embodiments, the unpacker 104 is configured to unpack non-zero values. In at least some embodiments, the unpacker 104 is configured to distribute non-zero values ​​to MAC units through a load path. In at least some embodiments, the unpacker 104 is configured to perform unpacking of packed data using any of a plurality of packing methods. In at least some embodiments, the unpacker 104 consists of gates and registers within the integrated circuit 100.

[0018] Memory 106 is a component of the integrated circuit 100. In at least some embodiments, memory 106 is configured to store packed non-zero values ​​and activation values. In at least some embodiments, memory 106 stores non-zero value packages. In at least some embodiments, memory 106 is configured to provide data to the unpacker 104 and the systolic array 110. In at least some embodiments, memory 106 is configured to receive instructions from the controller 108 that manage the data flow. In at least some embodiments, memory 106 is configured to serve a general data storage purpose for various components of the integrated circuit 100. In at least some embodiments, memory 106 is in the form of flash memory or other types of on-chip memory.

[0019] The controller 108 is a component of the integrated circuit 100. In at least some embodiments, the controller 108 is configured to manage the flow of non-zero and activation values. In at least some embodiments, the controller 108 is configured to interface with memory 106, an unpacker 104, and a systolic array 110. In at least some embodiments, the controller 108 is configured to communicate with a host computer 102. In at least some embodiments, the controller 108 is in the form of one or more microcontrollers, control units, etc. In at least some embodiments, the controller 108 is configured to send packed non-zero values ​​from memory 106 to the unpacker 104, send multiple activation values ​​from memory 106 to multiple MAC units, and store multiple output sum values ​​from multiple MAC units on memory 106. In at least some embodiments, the controller 108 is configured to send a non-zero value package from memory 106 to the unpacker 104, send a plurality of activation values ​​from memory 106 to a plurality of MAC units, and store a plurality of output sum values ​​from the plurality of MAC units on memory 106.

[0020] The systolic array 110 is a component of the integrated circuit 100. In at least some embodiments, the systolic array 110 is configured to perform parallel processing of values ​​for neural network inference. In at least some embodiments, the systolic array 110 is configured to interface with the umpacker 104, the memory 106, and the controller 108. In at least some embodiments, the systolic array 110 includes a plurality of MAC units for data processing. In at least some embodiments, the systolic array 110 is configured to interface with various data processing and storage units.

[0021] Figure 2 is a schematic diagram of an integrated circuit according to at least some embodiments of the present disclosure. The integrated circuit includes an unpacker 204, a memory 206, a systolic array 210, an execution path 212, a load path 214, and a result path 218. The description of the unpacker 104 in Figure 1 applies to the unpacker 204. The description of the memory 106 in Figure 1 applies to the memory 206. The description of the systolic array 110 in Figure 1 applies to the systolic array 210.

[0022] The unpacker 204 is a component of the integrated circuit. In at least some embodiments, the unpacker 204 in the integrated circuit is configured to unpack packed non-zero values ​​received from memory 206, such as packed non-zero values ​​211. In at least some embodiments, the unpacker 204 is connected to multiple MAC units by multiple load paths, such as load path 214. In at least some embodiments, the unpacker 204 is configured to distribute these unpacked values ​​to the systolic array 210 through load paths, such as load path 214. In at least some embodiments, the unpacker 204 is configured to receive packed non-zero values ​​from memory 206, correlate each non-zero value among the packed non-zero values ​​with the corresponding MAC unit among the multiple MAC units, combine the non-zero value with the address of the corresponding MAC unit, and send the non-zero value and address to the corresponding load path connected to the corresponding MAC unit among the multiple load paths. In at least some embodiments, the unpacker is further configured to determine the corresponding load path based on a row identifier, where the address is a column identifier. In at least some embodiments, the unpacker 204 is configured to read a non-zero value package from memory 206, associate each non-zero value among the multiple non-zero values ​​in the non-zero value package with an associated MAC unit among the multiple MAC units, combine the non-zero value with the address value of the associated MAC unit, and transmit the non-zero value and address value through a corresponding load path, such as a load path 214, connected to the associated MAC unit.

[0023] Memory 206 is a component of the integrated circuit. In at least some embodiments, memory 206 is configured to store packed non-zero values and activation values. In at least some embodiments, memory 206 is configured to supply the packed non-zero values to unpacker 204. In at least some embodiments, memory 206 is configured to provide activation values to systolic array 210. In at least some embodiments, memory 206 is configured to store the output sum value received from systolic array 210.

[0024] Systolic array 210 is a component of the integrated circuit. In at least some embodiments, systolic array 210 is configured to perform parallel processing of data using a plurality of MAC units. In at least some embodiments, systolic array 210 interacts with other components by receiving unpacked non-zero values and activation values from unpacker 204 and memory 206. In at least some embodiments, systolic array 210 is configured to transmit the output sum value to the result path. [[ID=**6]]

[0025] Execution path 212 is a component of the integrated circuit. In at least some embodiments, execution path 212 is configured for transmitting activation values from memory 206 to systolic array 210. In at least some embodiments, execution path 212 is configured to facilitate the flow of data during the calculation phase.

[0026] Load path 214 is a component of the integrated circuit. In at least some embodiments, load path 214 is configured for transmitting unpacked non-zero values and addresses to systolic array 210. In at least some embodiments, load path 214 connects unpacker 204 to systolic array 210 for data distribution.

[0027] The result path 218 is a component of the integrated circuit. In at least some embodiments, the result path 218 is configured for transmitting processed data from the systolic array 210 to the memory 206. In at least some embodiments, the result path 218 is configured for transmitting output sum values ​​from the systolic array 210 to the memory 206.

[0028] The packed value 211 is the format of the data processed by the integrated circuit. In at least some embodiments, the packed value 211 represents compressed data to be unpacked and processed. In at least some embodiments, the packed value 211 optimizes memory usage by storing only non-zero values. In at least some embodiments, the packed value 211 is stored in memory 206 and unpacked by unpacker 204.

[0029] Figure 3 is a schematic diagram of a systolic array 310 according to at least some embodiments of the present disclosure. The systolic array 310 includes a plurality of MAC units such as MAC unit 320, a plurality of execution paths such as execution path 312, a plurality of load paths such as load path 314, a plurality of input summation paths such as input summation path 315, a plurality of output summation paths such as output summation path 316, and a plurality of result paths such as result path 318. The description of execution path 212 in Figure 2 applies to execution path 312. The description of load path 214 in Figure 2 applies to load path 314. The description of result path 218 in Figure 2 applies to result path 318.

[0030] The MAC unit 320 is a component of the systolic array 310. In at least some embodiments, the MAC unit 320 is configured to perform multiply-accumulate operations. In at least some embodiments, the MAC unit 320 is configured to receive non-zero weight values ​​from the unpacker via the load path 314. In at least some embodiments, the MAC unit 320 is configured to receive activation values ​​from memory via the execution path 312. In at least some embodiments, the MAC unit 320 is configured to output a sum value to a downstream MAC unit via the output sum path 316. In at least some embodiments, the MAC unit 320 is configured to handle common arithmetic operations such as multiplication and addition. In at least some embodiments, each of the multiple MAC units includes a register configured to store a non-zero value received from the corresponding load path, and an address decoder configured to instruct the register to store a non-zero value in response to validating an address received from the corresponding load path. In at least some embodiments, each MAC unit among a plurality of MAC units includes a register configured to store non-zero values ​​received from a corresponding load path among a plurality of load paths, and an address decoder configured to store non-zero values ​​in the register in response to validating an address value coupled to a non-zero value. In at least some embodiments, the registers include a load register and an active register, and the controller is further configured to instruct each MAC unit among the plurality of MAC units to transfer a non-zero value from the load register to the active register. In at least some embodiments, each MAC unit is identified by a row identifier and a column identifier.

[0031] The input sum path 315 is a component of the systolic array 310. In at least some embodiments, the input sum path 315 is configured for sending input sum values ​​from an upstream MAC unit to MAC unit 320 for accumulation.

[0032] The output sum path 316 is a component of the systolic array 310. In at least some embodiments, the output sum path 316 is configured for transmitting output sum values ​​from MAC unit 320 to downstream MAC units.

[0033] Figure 4 is a schematic diagram of a MAC unit 420 according to at least some embodiments of the present disclosure. The MAC unit 420 includes an address decoder 422, a load register 424, an active register 425, a multiplier 427, and an adder 429. The descriptions of the execution path 212 in Figure 2 and the execution path 312 in Figure 3 apply to the execution path 412. The descriptions of the load path 214 in Figure 2 and the load path 314 in Figure 3 apply to the load path 414. The description of the input summation path 315 in Figure 3 applies to the input summation path 415. The description of the output summation path 316 in Figure 3 applies to the output summation path 416.

[0034] The address decoder 422 is a component of the MAC unit 420. In at least some embodiments, the address decoder 422 is configured to decode an address received via the load path 414 to determine whether the MAC unit 420 is a location where a non-zero value combined with the address should be stored. In at least some embodiments, the address decoder 422 instructs the load register 424 to store a non-zero value in response to validating the address.

[0035] The load register 424 is a component of the MAC unit 420. In at least some embodiments, the load register 424 is configured to temporarily store non-zero values ​​received via the load path 414 until the non-zero values ​​are transferred to the active register 425. In at least some embodiments, when the load register 424 receives an update command, it transfers the stored values ​​to the active register 425. In at least some embodiments, the load register 424 is configured for general data storage and transfer. In at least some embodiments, the load register 424 is of a type typically used for temporary data storage in CPUs, GPUs, and other digital circuits. In at least some embodiments, the controller is further configured to instruct each of the multiple MAC units to clear the registers.

[0036] The active register 425 is a component of the MAC unit 420. In at least some embodiments, the active register 425 is configured to store non-zero values ​​that are actively used in multiplication operations within the MAC unit 420. In at least some embodiments, the active register 425 receives values ​​from the load register 424. In at least some embodiments, the active register 425 provides values ​​to the multiplier 427 for calculation. In at least some embodiments, the active register 425 is configured for general data storage and transfer. In at least some embodiments, the active register 425 is of a type typically used for temporary data storage in CPUs, GPUs, and other digital circuits.

[0037] The multiplier 427 is a component of the MAC unit 420. In at least some embodiments, the multiplier 427 is configured to multiply a non-zero value from the active register 425 by an activation value from the execution path 412 to produce a product value. In at least some embodiments, the multiplier 427 is configured to receive a non-zero value from the active register 425. In at least some embodiments, the multiplier 427 is configured to receive an activation value from memory via the execution path 412. In at least some embodiments, the multiplier 427 is configured to send the product value to an adder. In at least some embodiments, the multiplier 427 is configured for general multiplication operations in digital systems. In at least some embodiments, the multiplier 427 is in a form suitable for FPGA modules, ASICs, CPUs, GPUs, DSPs, etc. In at least some embodiments, each of the multiple MAC units further includes a multiplier configured to multiply a non-zero value from a register and an activation value from the execution path to produce a product value.

[0038] The adder 429 is a component of the MAC unit 420. In at least some embodiments, the adder 429 is configured to add the product value from the multiplier and the input sum value to produce an output sum value. In at least some embodiments, the adder 429 is configured to receive the product value from the multiplier 427. In at least some embodiments, the adder 429 is configured to receive the input sum value from an upstream MAC unit via the input sum path 415. In at least some embodiments, the adder 429 is configured to send the output sum value to a downstream MAC unit via the output sum path 416. In at least some embodiments, the adder 429 is configured for general addition operations in digital systems. In at least some embodiments, the adder 429 is in a form suitable for FPGA modules, ASICs, CPUs, GPUs, DSPs, etc. In at least some embodiments, each of the multiple MAC units further includes an adder configured to add the product value from the multiplier and the input sum value to produce an output sum value.

[0039] In Figure 4, the MAC unit 420 includes two registers, a load register 424 and an active register 425. In at least some embodiments, each of the multiple MAC units includes a register configured to store a non-zero value received from a corresponding load path among a plurality of load paths, an address decoder configured to store a non-zero value in the register in response to validating an address value coupled to a non-zero value, and a multiplier configured to multiply the non-zero value from the register and the activation value from the execution path to produce a product.

[0040] Figure 5 shows an operational flow for on-chip non-zero unpacking and distribution according to at least some embodiments of the present disclosure. In at least some embodiments, the operational flow provides a method for on-chip non-zero unpacking and distribution. In at least some embodiments, the method is performed by a controller of an integrated circuit, such as controller 108 in Figure 1.

[0041] In S530, the controller or a section thereof clears the weight values. In at least some embodiments, the controller sends a command to each MAC unit to clear the contents of those load registers, such as the load register 424 in Figure 4. In at least some embodiments, the controller causes the load registers to effectively store zero values. In at least some embodiments, the controller causes the load registers to reset to a default state, which is equivalent to storing zero values. In at least some embodiments, clearing the weight values ​​ensures that non-zero weight values ​​are not mistakenly carried over and used in the next neural network inference process.

[0042] In S532, the controller or a section thereof loads weight values. In at least some embodiments, the controller causes the unpacker to receive packed non-zero weight values ​​from memory, unpack them, and distribute them to the appropriate MAC units via the load path. In at least some embodiments, the controller performs the operation flow shown in Figure 6 below.

[0043] In S534, the controller or a section thereof activates the weight values. In at least some embodiments, the controller sends an "update" command through the execution path to the MAC unit, causing the weight values ​​stored in the load registers to be transferred to their respective active registers. In at least some embodiments, the controller updates the weight registers in the MAC unit with the new weight values.

[0044] In S536, the controller or a section thereof inputs an activation value. In at least some embodiments, the controller sends the activation value from memory to the MAC unit via the execution path. In at least some embodiments, the controller causes the activation value to be sent to the multiplier in the MAC unit.

[0045] In S538, the controller or a section thereof performs MAC operations. In at least some embodiments, the controller causes a MAC unit to perform a sum-of-products operation using weight values ​​and activation values. In at least some embodiments, the controller causes a multiplier to multiply the weight values ​​and activation values ​​to produce a product. In at least some embodiments, the controller causes an adder to add the product to the input sum to produce an output sum. In at least some embodiments, the controller stores the output sum generated by the downstream MAC unit in memory.

[0046] In S539, the controller or a section thereof determines whether all activation values ​​have been entered. In response to determining that not all activation values ​​have been entered, the operation flow returns to the activation value input in S536. In response to determining that all activation values ​​have been entered, the operation flow terminates.

[0047] Figure 6 shows an operation flow for loading weight values ​​according to at least some embodiments of the present disclosure. In at least some embodiments, the operation flow provides a method for loading weight values. In at least some embodiments, the method is performed by an ampucker of an integrated circuit, such as ampucker 104 in Figure 1.

[0048] In S640, the unpacker receives the packed values. In at least some embodiments, the controller sends the packed non-zero weight values ​​from memory to the unpacker. In at least some embodiments, the unpacker receives the packed values ​​through multiple paths connecting the unpacker to memory.

[0049] In S643, the unpacker unpacks the values. In at least some embodiments, the unpacker decodes or decompresses the packed non-zero values. In at least some embodiments, the unpacker unpacks according to a packing method. In at least some embodiments, the packing method is one of addressing, masking, or indexing. In at least some embodiments, the unpacker converts the packed non-zero values ​​into a format usable by the MAC unit. In at least some embodiments, the unpacker performs the operation flow shown in Figure 7 below.

[0050] In S646, the ampucker sends a non-zero value to each row of the MAC unit. In at least some embodiments, the ampucker sends each unpacked non-zero value, along with its corresponding address, to the appropriate load path. In at least some embodiments, the ampucker routes the non-zero values ​​to reach the identified MAC unit. In at least some embodiments, the ampucker is further configured to determine the corresponding load path based on the row identifier.

[0051] In S649, the unpacker determines whether all values ​​have been unpacked. In response to determining that not all values ​​have been unpacked, the operation flow returns to unpacking values ​​in S643. In response to determining that all values ​​have been unpacked, the operation flow terminates. In at least some embodiments, the unpacker unpacks and transmits a number of values ​​equal to or less than the number of load paths during a clock cycle.

[0052] Figure 7 shows an operation flow for unpacking a value according to at least some embodiments of the present disclosure. In at least some embodiments, the operation flow provides a method for unpacking a value. In at least some embodiments, the method is performed by an unpacker of an integrated circuit, such as the unpacker 104 in Figure 1.

[0053] In S750, the unpacker determines whether the packing method is mask-based coding. In response to determining that the packing method is mask-based coding, the operation flow proceeds to mask reading in S751. In response to determining that the packing method is not mask-based coding, the operation flow proceeds to identifier reading in S752. In at least some embodiments, the unpacker makes the decision according to a signal from the controller. In at least some embodiments, the unpacker makes the decision according to the format of the packed value.

[0054] In S751, the unpacker reads the mask. In at least some embodiments, the unpacker reads the mask value associated with the packed non-zero value. In at least some embodiments, the unpacker uses the mask to determine which MAC unit corresponds to the non-zero value. In at least some embodiments, the unpacker identifies the non-zero value and its position in the kernel matrix. In at least some embodiments, the packed non-zero value includes the mask value associated with each group of non-zero values ​​among the packed non-zero values. In at least some embodiments, multiple non-zero values ​​are separated into groups of non-zero values, each group associated with a mask value, and the unpacker is further configured to determine, for each non-zero value among the multiple non-zero values, the corresponding load path and the address value of the associated MAC unit based on the mask value. In at least some embodiments, the packed non-zero value includes one or more mask values ​​which are not associated with a non-zero value due to a situation where all values ​​potentially associated with those mask values ​​were zero.

[0055] In S752, the unpacker reads an identifier. In at least some embodiments, the unpacker reads an identifier associated with a packed non-zero value. In at least some embodiments, the unpacker uses the identifier to determine which MAC unit corresponds to the non-zero value. In at least some embodiments, a packed non-zero value includes an identifier for the corresponding MAC unit associated with each non-zero value among the packed non-zero values. In at least some embodiments, a non-zero value package includes, for each non-zero value among a plurality of non-zero values, an identifier for the corresponding MAC unit associated with the non-zero value, and the unpacker is further configured to determine, for each non-zero value among a plurality of non-zero values, the corresponding load path and the address value of the associated MAC unit based on the identifier.

[0056] In S754, the unpacker determines whether a non-zero value is an index value. In at least some embodiments, the unpacker determines whether a packed non-zero value is an index value. In response to determining that a non-zero value is an index value, the operation flow proceeds to index value correlation in S755. In response to determining that a non-zero value is not an index value, the operation flow proceeds to row value and column value correlation in S757. In at least some embodiments, the unpacker makes a decision based on a signal received from the controller. In at least some embodiments, the packed non-zero value includes an index value. In at least some embodiments, the unpacker is further configured to replace each index value among a plurality of index values ​​with a non-zero value associated with that index value by the index.

[0057] In S755, the unpacker correlates an index value with a non-zero value. In at least some embodiments, the unpacker correlates each index value with a corresponding non-zero value by referencing an index. In at least some embodiments, the unpacker is further configured to correlate each index value with a non-zero value by referencing an index.

[0058] In S757, the unpacker correlates the value with the row and column of the MAC unit. In at least some embodiments, the unpacker correlates each non-zero value with the corresponding row and column identifier. In at least some embodiments, the row directly corresponds to a load path. In at least some embodiments, correlating a non-zero value with the corresponding MAC unit includes determining the corresponding load path and address based on the identifier. In at least some embodiments, correlating a non-zero value with the corresponding MAC unit includes determining the corresponding load path and address based on the mask value. In at least some embodiments, the unpacker is further configured to associate a non-zero value with an associated MAC unit among a plurality of MAC units by determining the row and column identifiers encoded in the non-zero value package.

[0059] In S759, the ampucker combines the value with the column address. In at least some embodiments, the controller combines each non-zero value with its corresponding column address. In at least some embodiments, the ampucker prepares the non-zero values ​​and addresses for transmission to the load path. In at least some embodiments, the ampucker combines the non-zero values ​​with addresses to ensure that each non-zero value is validated by the correct MAC unit. In at least some embodiments, the ampucker is further configured to combine the non-zero values ​​with address values ​​based on the column identifier and determine the corresponding load path based on the row identifier.

[0060] While embodiments of the present invention have been described, the technical scope of any claimed subject matter is not limited to the embodiments described above. Those skilled in the art will understand that various modifications and improvements are possible to the embodiments described above. Those skilled in the art will also understand that such modifications or improvements to the embodiments are included within the technical scope of the present invention, beyond the scope of the claims.

[0061] The operations, procedures, steps, and stages of each process performed by the apparatus, system, program, and method shown in the claims, embodiments, or figures may be performed in any order, unless their order is indicated by "prior to" or "before" or similar, and unless the output from a previous process is used in a later process. Even if a process flow is described in the claims, embodiments, or figures using words such as "first" or "next," such description does not necessarily mean that the processes must be performed in the order described.

[0062] On-chip non-zero unpacking and distribution is implemented by a plurality of multiply-accumulate (MAC) units; a memory communicating with the plurality of MAC units; and an unpacker configured to receive packed non-zero values ​​from the memory, correlate each non-zero value among the packed non-zero values ​​with the corresponding MAC unit among the plurality of MAC units, combine the non-zero value with the address of the corresponding MAC unit, and transmit the non-zero value and the address to the corresponding load path connected to the corresponding MAC unit among a plurality of load paths.

[0063] In at least some embodiments, each of the plurality of MAC units further includes a multiplier configured to multiply the non-zero value from the register and the activation value from the execution path to produce a product, and an adder configured to add the product from the multiplier and the input sum to produce an output sum. In at least some embodiments, on-chip non-zero unpacking and distribution are further implemented by a controller configured to send the packed non-zero value from the memory to the unpacker, send a plurality of activation values ​​from the memory to the plurality of MAC units, and store a plurality of output sums from the plurality of MAC units on the memory. In at least some embodiments, the controller is further configured to instruct each of the plurality of MAC units to clear the register. In at least some embodiments, the register includes a load register and an active register, and the controller is further configured to instruct each of the plurality of MAC units to transfer the non-zero value from the load register to the active register. In at least some embodiments, the packed nonzero value includes an identifier for the corresponding MAC unit associated with each nonzero value among the packed nonzero values, and correlating the nonzero value with the corresponding MAC unit includes determining the corresponding load path and the address based on the identifier. In at least some embodiments, the packed nonzero value includes a mask value associated with each group of nonzero values ​​among the packed nonzero values, and correlating the nonzero value with the corresponding MAC unit includes determining the corresponding load path and the address based on the mask value. In at least some embodiments, the packed nonzero value includes an index value, and the unpacker is further configured to correlate each index value with the nonzero value by referring to the index.In at least some embodiments, each MAC unit is identified by a row identifier and a column identifier, the unpacker is further configured to determine the corresponding load path based on the row identifier, and the address is the column identifier. In at least some embodiments, on-chip non-zero value unpacking and distribution is further implemented by a host computer communicating with the integrated circuit, the host computer is configured to determine a packing method for the packed non-zero values, pack the non-zero values ​​according to the packing method to produce the packed non-zero values, and transmit the packed non-zero values ​​to the memory. In at least some embodiments, the packing method is one of addressing and masking. In at least some embodiments, the packing method includes indexing.

[0064] On-chip non-zero unpacking and distribution is implemented by having a plurality of multiply-accumulate (MAC) units, each of the plurality of MAC units having a register configured to store non-zero values ​​received from a corresponding load path of a plurality of load paths, and an address decoder configured to store the non-zero values ​​in the register in response to validating an address value coupled to the non-zero values; a memory communicating with the plurality of MAC units, the memory storing non-zero value packages; and an umpacker connected to the plurality of MAC units by the plurality of load paths, the umpacker configured to read the non-zero value packages from the memory, associate the non-zero value with a related MAC unit of the plurality of MAC units for each non-zero value among the plurality of non-zero values ​​in the non-zero value package, combine the non-zero value with the address value of the related MAC unit, and transmit the non-zero value and the address value through the corresponding load path connected to the related MAC unit.

[0065] In at least some embodiments, each of the plurality of MAC units further includes a multiplier configured to multiply the non-zero value from the register and the activation value from the execution path to produce a product, and an adder configured to add the product from the multiplier and the input sum to produce an output sum. In at least some embodiments, on-chip non-zero unpacking and distribution further includes a controller configured to send the non-zero package from the memory to the unpacker, send a plurality of activation values ​​from the memory to the plurality of MAC units, and store a plurality of output sums from the plurality of MAC units on the memory. In at least some embodiments, the controller is further configured to instruct each of the plurality of MAC units to clear the register. In at least some embodiments, the register includes a load register and an active register, and the controller is further configured to instruct each of the plurality of MAC units to transfer the non-zero value from the load register to the active register. In at least some embodiments, the non-zero value package includes, for each non-zero value among the plurality of non-zero values, an identifier for the corresponding MAC unit associated with the non-zero value, and the umpacker is further configured to determine, for each non-zero value among the plurality of non-zero values, the corresponding load path and the address value of the associated MAC unit based on the identifier. In at least some embodiments, the plurality of non-zero values ​​are separated into groups of non-zero values, each group associated with a mask value, and the umpacker is further configured to determine, for each non-zero value among the plurality of non-zero values, the corresponding load path and the address value of the associated MAC unit based on the mask value. In at least some embodiments, the umpacker is further configured to replace each non-zero value among the plurality of non-zero values ​​with an index value associated with the non-zero value by index.In at least some embodiments, the unpacker is further configured to associate the non-zero values ​​with associated MAC units among the plurality of MAC units by determining row identifiers and column identifiers encoded in the non-zero value package, to combine the non-zero values ​​with the address values ​​based on the column identifiers, and to determine the corresponding load path based on the row identifiers.

[0066] The foregoing briefly describes some embodiments so that those skilled in the art may better understand aspects of this disclosure. Those skilled in the art should understand that this disclosure can be readily used as a basis for designing or modifying other processes and structures to perform the same purposes and / or achieve the same advantages as the embodiments introduced herein. Those skilled in the art should also recognize that such equivalent configurations will not depart from the spirit and scope of this disclosure, and that various changes, substitutions, and modifications within this specification are possible without departing from the spirit and scope of this disclosure.

Claims

1. It is an integrated circuit, Multiple multiply-accumulate (MAC) units; A memory that communicates with the aforementioned multiple MAC units; Receive a packed non-zero value from the memory, and for each non-zero value among the packed non-zero values, The non-zero value is correlated with the corresponding MAC unit among the plurality of MAC units. The non-zero value is combined with the address of the corresponding MAC unit, The non-zero value and the address are transmitted to the corresponding load path connected to the corresponding MAC unit among the multiple load paths. An ampaca configured in such a way; Equipped with, Each of the aforementioned MAC units is A register configured to store the non-zero value received from the corresponding load path, and An address decoder configured to instruct the register to store the non-zero value in response to validating the address received from the corresponding load path. An integrated circuit that utilizes a special function.

2. Each of the aforementioned MAC units is A multiplier configured to multiply the non-zero value from the register and the activation value from the execution path to generate a product value, and An adder configured to generate an output sum by adding the product value from the multiplier and the input sum value. The integrated circuit according to claim 1, further comprising the above.

3. The packed non-zero value is sent from the memory to the unpacker. From the memory, multiple activation values ​​are transmitted to the multiple MAC units. The memory stores the sum of multiple outputs from the multiple MAC units. The integrated circuit according to claim 1, further comprising a controller configured as such.

4. The integrated circuit according to claim 3, wherein the controller is further configured to instruct each of the plurality of MAC units to clear the register.

5. The aforementioned registers include a load register and an active register. The integrated circuit according to claim 3, wherein the controller is further configured to instruct each of the plurality of MAC units to transfer the non-zero value from the load register to the active register.

6. The packed non-zero value includes the identifier of the corresponding MAC unit associated with each non-zero value among the packed non-zero values, The integrated circuit according to claim 1, wherein correlating the non-zero value with the corresponding MAC unit includes determining the corresponding load path and the address based on the identifier.

7. The packed nonzero values ​​include mask values ​​associated with each group of nonzero values ​​among the packed nonzero values, The integrated circuit according to claim 1, wherein correlating the non-zero value with the corresponding MAC unit includes determining the corresponding load path and the address based on the mask value.

8. The aforementioned packed non-zero value includes the index value, The integrated circuit according to claim 1, wherein the unpacker is further configured to correlate each index value with the non-zero value by referring to the index.

9. Each MAC unit is identified by a row identifier and a column identifier. The ampucker is further configured to determine the corresponding load path based on the row identifier, The integrated circuit according to claim 1, wherein the address is the column identifier.

10. The integrated circuit according to any one of claims 1 to 9; and A host computer that communicates with the aforementioned integrated circuit, the host computer is The packing method for the aforementioned packed non-zero values ​​is determined, The non-zero values ​​are packed according to the packing method described above to generate the packed non-zero values. The packed non-zero value is sent to the memory. It is structured in such a way. A system that includes these features.

11. The system according to claim 10, wherein the packing method is one of addressing and masking.

12. It is an integrated circuit, Multiple sum-of-products (MAC) units, each of the multiple MAC units is, A register configured to store non-zero values ​​received from a corresponding load path among multiple load paths, and An address decoder configured to store the non-zero value in the register in response to validating the address value coupled to the non-zero value. Having; A memory that communicates with the plurality of MAC units, the memory stores non-zero value packages; and An ampucker connected to the plurality of MAC units by the plurality of load paths, the ampucker reads the non-zero value package from the memory, and for each non-zero value among the plurality of non-zero values ​​in the non-zero value package, The non-zero value is associated with the associated MAC unit among the plurality of MAC units, The non-zero value is combined with the address value of the associated MAC unit, The non-zero value and the address value are transmitted through the corresponding load path connected to the associated MAC unit. It is structured in such a way. An integrated circuit comprising:

13. Each of the aforementioned MAC units is A multiplier configured to multiply the non-zero value from the register and the activation value from the execution path to generate a product value, and An adder configured to generate an output sum by adding the product value from the multiplier and the input sum value. The integrated circuit according to claim 12, further comprising the above.

14. The non-zero value package is sent from the memory to the unpacker. From the memory, multiple activation values ​​are transmitted to the multiple MAC units. The memory stores the sum of multiple outputs from the multiple MAC units. The integrated circuit according to claim 12, further comprising a controller configured as such.

15. The integrated circuit according to claim 14, wherein the controller is further configured to instruct each of the plurality of MAC units to clear the register.

16. The aforementioned registers include a load register and an active register. The integrated circuit according to claim 14, wherein the controller is further configured to instruct each of the plurality of MAC units to transfer the non-zero value from the load register to the active register.

17. The non-zero value package includes, for each non-zero value among the plurality of non-zero values, an identifier for the corresponding MAC unit associated with the non-zero value, The integrated circuit according to any one of claims 12 to 16, wherein the unpacker is further configured to determine, for each non-zero value among the plurality of non-zero values, the corresponding load path and the address value of the associated MAC unit based on the identifier.

18. The aforementioned multiple non-zero values ​​are separated into groups of non-zero values, and each group is associated with a mask value. The integrated circuit according to any one of claims 12 to 16, wherein the unpacker is further configured to determine the address value of the corresponding load path and the associated MAC unit based on the mask value for each of the plurality of non-zero values.

19. The integrated circuit according to any one of claims 12 to 16, wherein the unpacker is further configured to replace each of the plurality of non-zero values ​​with an index value associated with the non-zero value by index.

20. The aforementioned ampaca is, By determining the row identifier and column identifier encoded in the non-zero value package, the non-zero values ​​are associated with the associated MAC units among the plurality of MAC units. Based on the column identifier, the non-zero value is combined with the address value, Based on the row identifier, the corresponding load path is determined. The integrated circuit according to any one of claims 12 to 16, further configured as follows.

Images